Small chunk uploads?

[BaalDemon]

Alright first of all I am a Shareaza user and not a LimeWire user but please dont treat this as a flame just for that reason. I have recently made 'raza only work on the subnet for my schools WAN and have noticed a few things about LimeWire. First of all it is the majority client at my school by far, mainly due to the fact our school comes right out and says use LimeWire because it is simple to set up to work on just the WAN, which has become the only way for p2p to work here with the new packetShaper routers installed which limits p2p bandwidth to unimaginably slow speeds. But anyways with the majority of users being LW users I have come across a few observations and would like to get peoples opinions and possibly some technical reasons to why I see this. Over 90% of my uploads go to people using various versions of LimeWire, however when I connect to g1 UltraPeers I have not once seen a connection to a LW Ultrapeer. Is there something going on with LimeWire or Shareaza which prevents this interaction between the two clients? Also another thing I have noticed is an overall slowness in ups to LimeWire users. I know everyone has the same connection who I will be uploading to or downloading from because they are all on the same network, we have two t3 pipelines each with a packetShaper router. This slowness in uploading to LimeWire seems to be caused by the way LimeWire downloads in very small chunks. This is what I see to most LimeWire clients, uploading at around 95kB/sec for about a second, it finishes the chunk, asks for another chunk, and repeats. This doesnt seem like the most efficent way to download a file. I know by downloading in chunks it is easier to get all these chunks to various places and everything, however it will greatly slow down downloads. Wouldnt it be better if, while the user had a connection established that they continue downloading until reaching the end of a file, or until reaching another chunk of the file that has already been downloaded from another user? What is the great advantage to this small chunk file downloading which LimeWire implements?

[trap_jaw]

The smaller chunk size doesn't really matter because the TCP connection is kept alive the whole time. So what is really happening is that after 100KB have been requested HTTP headers are exchanged (a few hundred bytes) and then the next 100KB are uploaded. If the uploading client has an efficient HTTP1.1 implementation, the time spent for exchanging HTTP headers is neglegible and there is no noticeable loss in speed (maybe 1% at most).

Gnutella clients with slow HTTP1.1 implementations could cause a noticeable drop in the upload through-put. That is something LimeWire was willing to accept for a cleaner implementation of swarming.

[BaalDemon]

What makes this swarming effect so wanted.... what are the advantages of this over a non swarming effect which seems to be more efficient.

[trap_jaw]

After requesting a chunk, there is no way to abort the download without dropping the TCP connection. So if you keep the chunk sizes small, slow remote hosts don't reduce the overall download speed as much, because all the other chunks can easily be requested from faster nodes.

If you have a 4MB file and download in 1MB chunks, you can swarm from 4 hosts. If one of them is slow (say 1K/s) you would have to wait for this slow host to finish. With 100K chunks one slow remote host does not matter as much.

In addition alternate locations will be propagated much faster, because they are exchanged much more often.

[BaalDemon]

This swarming effect though if it always starts at the beginning and progresses itself through the file, which is what I have seen wouldnt it be better for a more random distribution of the chunks? Like instead of picking up chunks in what it looks like 100k then skip 100k then get 100k wouldnt it be more efficient for the swarm if each chunk of 100k came from a pseudo-random part of the file? Or does it actually do this?

Also can you answer why I never see a connection to a LimeWire Ultrapeer when they take up well over 90% of my ups. I know there has to be some out there, but why am I not connecting to them.

[trap_jaw]

LimeWire does not skip chunks when downloading a file. It requests one chunk from one host, the next chunk from another host and so on. Once most of the chunks are finished it iterates through the unfinished chunks, stops all stalled downloads and downloads those chunks from "good" hosts.

Randomly downloading chunks does not necessarily make sense, - even if LimeWire supported PFSP which it does not.

[BaalDemon]

Well I guess my next logical question would be what is the big deal with swarming if you cant share partials. I guess I just dont really understand what is so great about this swarmming system that you have in place. Would an even better improvement be to relate whether a file should be swarmed to the users bandwidth. Lets say for example a user with a 56k modem is downloading at 5kB/sec from one user..... wouldnt it be better for the user to continue downloading from that person the whole file, than swarming and getting different chunks from different people. I dont know I guess maybe swarmming in my head just doesnt seem to be very efficient, or maybe its just becasue I dont use a broadband connection and dont really pay attention all that much because when im at home I use a 56k so when I have a broadband I dont complain much. I would however like to know why I don't see LimeWire Ultrapeer connections, I heard that you only reserve two spots on an Ultrapeer for non LimeWire clients due to some trouble with Morpheous or something like that..... im just curious.

[Joakim Agren]

Hello!

Why Swarming?

Well the advantage of swarming is not present for modem users. However if you are on a broadband connection and have found a popular file that is available from multiple hosts then you will get the file downloaded at higher speeds and maybe even reduce the amount of traffic on individual hosts. My record so far was when downloading a mp3 file from 8 hosts at the same time and reached the download speed of 770Kbit/s( I am on a 8Mbit/s connection) and yeat the first initial host that I clicked to download it from only allowed a downloadspeed of 5Kbit/s so there is a huge advantage. Whenever I have started downloading a file and I see that it is kind of slow I always try to find alternate locations for the same file by performing a new search for the file in question and if LW finds another source it will attempt to connect to that host or those hosts too and several times I will get much higher speeds if LW finds an alteranate source to also download it from for instance it can be upped from 5Kbit/s to in average 200-300Kbit/s when downloading from 2-3 hosts a clear difference in speed dont you think?.

Now to this sensitive question about whty you do not see any LW UltraPeers in your connections?

Shareaza is considered a hostile client and the developers of Gnutella clients especially between Bearshare developers and Mike the developer of Shareaza. This has lead to some clients blocking Shareaza and I think that LW does it to in the most recent versions. Why is Shareaza considered hostile?. You can read all about it here but be warned there is alot of reading to do so make sure you have the time to do it. Here is the spot to check out:

http://www.mp3newswire.net/stories/2002/Shareaza.html

[BaalDemon]

I know why Shareaza is considered hostile, but in my opinion BS took the only actions that were truly hostile. Shareaza has released the g2 protocol specs and g2 doesnt harm the gnutella network in anyway because it is a seperate protocol the shareaza client can access either of the clients without bothering the other one. But I really dont want to turn this into a big war about that so I will just drop it here. Thank you for letting me know about swarming, I basically see it as a type of PFS, is there any advantage of swarming of pfs, because I guess I see them as doing about the same thing, or am I completely wrong?

[osu_uma]

OK, just to be clear on this:

LimeWire only connects to Shareaza Leafs and refuses connections with Ultrapeers?

That explains a lot. I occasionally run Shareaza on my PC and I upload mostly to G1 clients. Mostly BearShare, Gnucelus and GTK, almost no LimeWire at all.

I don't think LimeWire is doing its users a favor by blocking Shareaza Ultrapeers.

Tell me this is not true.