it could be the upload from the users to you with it being slow..everyone has different connections.

you may be behind nat if you use random port mapping (dont know if this random port mapping applies to frostwire)

open your ports also.