As of now, there's an example how to start phex without gui.
Advanced Questions - PhexWiki
java -Xmx512m -Djava.awt.headless=true -jar phex.jar
This is a Java feature and still needs an xserver, since some java classes depend on it, but it doesn't need to have a display and can be run serverside.
You can setup the sharing diretory directly in the config files, and you can also switch from headless to full gui by just starting Phex differently (with the normal starting command).
Everything else should be a matter of stripping code.
Besides: The Phex subversion repository contains a shell script which builds and runs phex headless:
build-and-run-headless
How to get the code:
Subversion - PhexWiki
Besides (2): We'd be glad, if you'd work in our public subversion repository (in a branch), so we can easily transfer those changes which would also benefit the mainline client.