I'm not sure if it still works... and I'm not sure why it runs mplayer, but try to set the following properties in phexCorePrefs.properties. I post them with there default value, you only need to add the modified values to the properties file.

SystemTools.OpenVideoCmdOtherOS = xterm -e mplayer %filepath%

SystemTools.OpenImageCmdOtherOS = ee %filepath%

SystemTools.OpenAudioCmdOtherOS = xterm -e mplayer %filepath%

SystemTools.OpenBrowserCmdOtherOS = mozilla -remote openURL(%filepath%,new-window)

The file phex.cfg is obsolete since 3.0

__________________