Sounds very much like ISP p2p bandwidth shaping. The fact it takes a few seconds before the speed drops is a typical sign. Bad ISPs - AzureusWiki though there might be more recent ones out there than is listed. I think one of the Sydney radio ISP's also shapes p2p bandwidth.

Does this only happen at certain times of day, such as peak periods of day? Have you tried very late at night (after midnight or 2 am or during the morning? This is worth checking.)

Are the files in question torrent files or regular gnutella network shared files?