Thoughts about Segmented Downloading
 

First of all, let me say that uploading a file that is not yet complete, or maybe even in the process of downloading is GREAT, since it helps spread the data around faster.

But I have a small concern about segmented downloading. I mean, getting the same file from multiple hosts at the same time. We all know that this will tie up many upload slots, which would prevent other users from getting files. The idea is that since you are getting less of the file, the slots will be released faster. That is all fine and good in an ideal world, but I think it won't work like that in the real world without some safeguards.

Let me just make an example. Let's say that there are 2 hosts with this very rare file. They have infinite upload bandwidth. Download bandwidth is limited in such a way that it will take an hour to download this file.

In the current situation, we have a timeline that looks like this:

0.00hours - 2 files on the network
1.00h - 4 files on the network
2.00h - 8 files on the network

Now, with an ideal situation in which we use segmented downloading and people have infinite download bandwidth, and each person uses every host available with the file:

0.00h - 2
0.50h - 3
0.83h - 4
1.08h - 5
1.28h - 6
1.45h - 7
1.59h - 8

So we got to the same number of copies in less time, which is always good. But now, lets thing about someone with very limited download bandwidth trying to do the same. Lets say that even if they get a whole lof of segments going, they will never be able to get it faster than in one hour:

0.00h - 2
1.00h - 3
2.00h - 4
3.00h - 5

In this case, the file spreads much slower.

Of course, since we are always uploading little chunks, the "graphs" will look different, but we can't ignore the fact that slow bandwidth people will destroy any benefits that segmented downloading might offer. Segmented Downloading can work, but we need to study the bandwidth that each servent has available to it, and adjust the limits according to that. Sadly, the user cannot be trusted to tell us how big is their pipe.

Please let me know if I am completly wrong on this! :)

Low bandwidth users don't use swarmed downloads by default; only high speed ones do (unless the client is poorly written). Your point is well said, and low bandwidth users cannot benefit when it comes to download speed. However, it does help them when uploading. People with faster connections can download small chunks from modem users. This way, modem users can share without being penalized for it. This would encourage them to share and not be a complete freeloader.

:-)

totaly agree! :-)

When you think about the huge amount of modem users, the benefit from swarming would be big for the network. When thinking about an intelligent swarming the benfit could be even more. A intelligent swarming can look like this: a) use a minimum bandwith while the user is downloading itself, use a higher bandwith automatically when the user has finished downloads and stays online. b) optional run the Gnutella client allways in background and automatically share with a minimum bandwith when the user is online, e.g. while he surfs the web.

How are swarmed downloads currently implemented, like with Xolox and LimeWire? Does Gnucleus swarm too? Are they 'inteligent', or how do they determine when swarming should be used?

:-)

I think, correct me if I'm wrong, we currently have no Gnutella client with swarming... only with segmented downloads.

Xolox has two healthy feature (inspired of eDonkey): First it does allow to upload unfinished files to spread new and big files much faster (the other client has to know how to handle this, okay only Xolox itself can do this now). Second it has a tiny intelligent upload: You did choose a minimum upload bandwith + a optional higher bandwith when the user is being idle. Actually this option was only suitable for high bandwith user (which Xolox was designed for), not for modem users. An intelligent uploading or swarming mechanism as described above does only exist in fantasy yet. :-)

The implementation prerequisites for swarming are IMHO: a stable client, multisegmented downloads, upload of unfinished partials, automatic researches (for automatic resume) and file hashs (for efficient research/resume). So far the technical aspects... there is one more aspect to make more use of modem users: superpeers (ultrapeers) are needed, they give back modem users some of their bandwith, which they can then better use for swarming.

Trustar...

gnucleus should be getting segmented as soon as 1.6 is released and swabby had made the statement about including the supernode stuff after Limewire has had a chance to work the bugs out and check it's usefulness... most of the new info on gnuc's is on www.gnucleus.net ... there is a forum up now...

I disagree completely with your point of view:

- you said that modem users could not profit much from swarming. it is correct that they do not profit as much as highspeed users. but concerning the current download speed of the network, swarming could easily double make their download speed twice or three times as much, and i call that profit.

- you only tell half the truth: modem users do profit much less of it. but of course then they also harm the network much less! a highspeed user who is simultaneously downloading 5 files could well use up to 30 hosts for swarm sources. a modem user could hardly ever use more than 3 hosts until his bandwidth is maxed out.

- even with swarming modem users can download very slow. that is why most modem users are downloading far less then highspeed users. how many modem users do you know who are downloading complete divx movies ? and how many highspeed users do you know who don't do it ? and using swarming, highspeed users will even download more than know.

- you say that modem users should not be swarm downloaders, but swarm uploaders, if i got you right. are you crazy??? modem users have about 8 kbps of bandwith. when you download a huge file from 6 modem users, you decrease the bandwidth of 6 modem users who are theirselves not allowed to benefit from swarming.

modem users have so little bandwith! it's bad enough that you don't want them to max out the little bit they have, but it's even worse that you want to take something from them in order to max out yours! it must be said clearly: it is NOT modem users who shorten the available upload slots. one highspeed connection can easily consume ten times as many slots as a modem user. and they DO it. they are not only consuming more slots, they are also downloading more frequently. your thoughts are not at all a means to help the network. it is a means to exploit modem users. i have the strong impression that none of you is actually using a modem to share files. modem users like me have 8 kbps and i give half of it for uploading. Tru, do you really think you would encourage me to anything when you tell me that you want do take from me, but you are not willing to give to me? we ARE the majority of the network and i for instance am strongly against any such kind of elitist "upper-class" ideology. on a peer-to-peer Network, either everyone should be allowed to use swarming, or noone should.

-- comments ?

Hi milhouse

Thanks for the link!
:-)

Hi Abaris
Plenty...

I'm glad you have an opinion of your own. But please do not force it on others. We are all entitled to our own opinion.

To be in the right context, please read (or skim/review/whatever) this thread:
http://www.gnutellaforums.com/showth...&threadid=6349

Now, time for comments....

You seem to stress equality for all. However this is not practical, nor realistic. (this is NOT about civil rights, so don't even go there) If the Internet followed your philosophy, everyone would have the same speed connection. No one would be able to have a smaller or larger bandwidth than any other person. This is an slightly extreme, yet viable example, but the idea remains. When it comes to Internet bandwidth, what you pay for is what you get.

As the gnutella network stands, too much of a modem user's bandwidth is used up by Gnutella traffic (pings/pongs/etc). So much, in fact, that hardly any bandwidth is left for downloading or uploading. By restructuring the network similarly to LimeWire's UltraPeer network topology, much less bandwidth will be wasted. If a more efficient protocol is put in place, perhaps then it would be practical for modem users to use swarm downloads.
What you fail to take into account is that 30 hosts are busy for 1/30 of the time that a 5 hosts would be busy. This, in effect, would always leave free hosts. Of course, the number of upload slots would need to be increased on each node, but that is a simple matter to fix. In a peer to peer network, information is distributed more evenly and as not as centralized as some classic client/server models. Why not have the responsibility of uploading laid on the the network more evenly so that popular hosts are not always bogged down with uploading massive amounts of information on their own?
Correct!! If people download more, that's just a consequence of having a more efficient network. When Intel (or IBM for you Macsters out there ;-)) releases a faster processor, we develop applications that require more processing power. Any benefits we develop tend to be taken advantage of, and then they become the norm instead of the advantage. But this is how advancement works. This law of computing has been proven time and time again.
Yes, I am crazy - you better start calling the asylum now, or I just might go ballistic...
Most modem users (the majority of the network) are forced to be freeloaders because uploading anything would not allow them to download anything. If they had more bandwidth freed up, they would be more willing to share.
I never said that. Like I said earlier, right now Swarming will not help modem users. Perhaps it will when their bandwidth is freed up a bit by more efficient Gnutella protocol standards.
This might shock you, but I am a modem user too. And yes, I do share. About 6 GBs worth of stuff. All recent movie soundtracks. I am distressed to see that I cannot upload very many files. I get disconnected often. The files are only partially uploaded, and the downloader may have problems resuming their download. Swarming would allow me to upload the maximum amount possible, and I wouldn't have to worry about whether it finished or not. I WANT to upload, but I can't upload much of anything because I am restricted by my modem bandwidth and choked with Gnutella traffic.

{END TRANSMISSION}

{AND SARCASM}

Greetings,

> I'm glad you have an opinion of your own. But please do not
> force it on others. We are all entitled to our own opinion.

I didn't mean to. of course everyone should think for himself.

> this is NOT about civil rights, so don't even go there

I never said that. it's about fairness. see the difference ?

> What you fail to take into account is that 30 hosts are busy for
> 1/30 of the time that a 5 hosts would be busy. This, in effect,
> would always leave free hosts.

Which is true for modem users as well. when i download a file from 3 sources, they are used for 1/3 of the time. so i don't decrease the numer of upload slots either.

> Most modem users (the majority of the network) are forced to
> be freeloaders because uploading anything would not allow
> them to download anything. If they had more bandwidth freed
> up, they would be more willing to share.

Make them share. Don't allow them to decrease their upload bandwidth as they like. I for instance allow uploading with half my bandwidth and I don't like being taken for a freeloader just because I use a modem.

> right now Swarming will not help modem users. Perhaps it will
> when their bandwidth is freed up a bit by more efficient
> Gnutella protocol standards.

Look what i meant. I'm giving 4 kbps upload bandwith to the network. I am tired of having downloads with 1-2 kbps even at times when noone downloads from me. i don't know why you think that traffic eats up all my bandwith, for this is not what i observe. when i download multiple different files i get a much larger download speed than 1-2 kbps, but most times i don't want to download 2 different files. i want to download one file from 2 sources. where is the great difference concerning network performance ?

> This might shock you, but I am a modem user too. And yes, I do
> share. About 6 GBs worth of stuff.

It doesn't. So do I (although it's not 6 gigs i share).

i would like to swarm in order to use all my bandwidth more effective, just as you explained. i don't see why i can download two or three different files at once, but i cannot download one file from two or three different sources.

at last i would like to say that i live in a region where i cannot get a cable or dsl connection. also i do not really want more bandwith than i have. all i want is to max out what i have.

if my last post was too offensive to you, i beg your pardon. i didn't mean to start a war between highspeed and modem users. i just want fair swarming. i do see that we can profit of the feature as well as they do. i don't see why we should harm the network more than they do. as far as i understand you, it all depends on whether or not network traffic is the main reason for my low transfer rate. i think it isn't. when i am interested in one file only, i am definitely wasting bandwith.

A swarming prerequisites for modem user is the existance of supernotes (or named Ultrapeers for Limewire). A Supernote will protect a modem user from backbone traffic and so enable the possibility of using this small slice of bandwith for swarming (swarming upload).
For sure a modem user can download multisegmented... as long as he has enough bandwith! It makes no sense to use another download stream, if the first stream does allready use the whole available download bandwith. Same for cable/DSL users, e.g. XoloX makes sure it does not use more download streams as necessary to use the available download bandwith.

Greets, Moak