Is there a way to alter which program that limewire will use for file types its own player can't handle?

Yes .. setting the default program for opening for each type of file. But you first must make a choice between using the LW media player or default player for audio files.
Setting default audio program
So .. potentially, you could have one type of audio program open mp3's, another audio program open m4a files for example.