Are you running on a Linksys router? If so I have the same problem.
But my issue is the newer versions of the WRT54G models (Above v5) can't handle multiple source downloading from what I understand. The router chokes when you try and download from too many people. + My bittorrent download speed has drastically dropped from like 800kb/s to 200kb/s for single torrents. I really hope they fix this with a new firmware upgrade.

I'll be honest, I don't know what kind of router I am using but I do believe that might be the problem.