Does this only happen when you're downloading?
How many files are you downloading at once? Try reducing the number of simultaneous downloads to just a couple.
Try disabling ultrapeer capabilities (tools > options > speed). Also disable any iTunes sharing.
Reduce your upload & download bandwidth a bit.