Oh. The first time I changed my port, I must have clicked the Advanced->Firewalls panel. It still says 6350 (I didn't know why I had to click the Force IP box, but it's still checked).

After that, I always used the Advanced->Port panel. It still says 6351.

So this is working as designed?