When anybody chooses to download a file, the first thing they do is download the file's Hash. That should only take a second or few. So what you are seeing are people downloading the hash & probably waiting in queue after that. The rest (other 5) are actually downloading the file.

To reduce drag that LW causes, there's a few things you can try.

Go to LW's prefs>Speed & tick disable Ultrapeer Cap. & press apply button.
Set both uplds & downlds to autoclear.
Close all LW windows you don't need open at that time.
If you're not in front of your comp, then hide LW in the dock.
Tip: Purge the Downloads Queue! (click on blue link)