Swarming, Multisegmented Downloading, Partial uploads...
 

Hi there,
it seems those phrases often cause confusion, I'll try to give a short definition (as far as I understood):

Multisegmented Downloading vs. Swarming
'Multisegmented Downloading' is what modern download manager and Gnutella clients do, download a file from multiple sources at the same time (increasing download speed). 'Swarming' means the same + spreading small file partials over the network automatically (increasing availability and download speed, distributing often requested files, balance load).

Most time people mean 'Multisegmented Downloading' if they talk about or 'Swarmed Downloads' or 'Swarming'! If you mean the real swarming better talk about 'Real Swarming'. To show you more about the difference: When we talk about downloading multisegmented the partials (file chunks) are typically very big, up to the full file size. Partials/chunks from Real Swarming are much smaller, a few 100KBs or whatever will fit best to the needs of making modem users part of the network. Segmented downloading and Swarming does NOT exclude each other! Swarming is just a bonus and actually not available on Gnutella these days. Real Swarming is far far future on Gnutella.

Swarmed Downloads or Parallel Downloads
People mean usually 'Multisegmented Downloads', those are only further synonyms... see above.

Partial upload
When a client does upload a file before it is finished with downloading the whole file itsef. This way especially big files get distributed faster and more often.

Automatic resume
When a client automatically downloads the full file and if a peer drop (or all) it full automatically searches more peers and continues the download without any further user interaction is needed. Automatic resume does need automatic requeries or other kind of techniques to find more peers offering the original file.

Feel free tor add more phrases & definitions below.... :-)

PS: Morgwen forced me to write this post, hehe. Here some older posts with more details:
"Improve Download Host/Speed" - http://www.gnutellaforums.com/showth...light=swarming
"A better way to promote sharing" - http://www.gnutellaforums.com/showth...&threadid=5939
"Thoughts about Segmented Downloading" - http://www.gnutellaforums.com/showth...&threadid=6396

A note from me (summary from the links above): The bandwith of modem users is completely wasted right now and modem users are often "forced" to be freeloader. Actually this egoizm is contraproductive and means less files for all. While high bandwith users can share the traditional way, swarming might be a key to make low bandwith users share with the network, without any significant negative effect or traffic-overhead. For both, for high and low bandwith users, sharing should mean there is a guaranteed high percentage of bandwith available for their own use! Conclusion, there are many ideas to envolve Gnutella and all work together. 1. A superpeer concept for dynamic traffic routing = reducing backbone traffic + improves network toplogy + increases horizon (more available files). 2. Search-caches for reducing double/multiple routed traffic = reducing high amout of search backbone traffic. 3. Swarming technology (as last step) = make use of the high amout of wasted bandwith + will spread often requested files + balance load + less "busy" servants (more available files). Please discuss about swarming and other possible improvements in the threads above and use this post for Gnutella definitions.

Re: Swarming, Multisegmented Downloading, Partial uploads...
 

Thank you "VERY MUCH"! :)

Morgwen

Thanks, now my confusion vanished (at least for a while)

I get it now, Thanks Moak!

the reason people want this is even as a modem user you are limited by a modem's upload speed of 31K on a 56K modem. so a modem user can get faster downloads from two sources at once. most cable modems have upload limits too.
cable modem users can suck the bandwidth from several thousand modem users at a time and get everything they ever wanted in 2 seconds.
thus cable modem users will prevent modem users from the benefit of this feature because all uploads are taken.
ok, so a modem user gets the file faster. now does he turn his client off cause he's happy? no, he's not happy cause he wants it all and wants it now and will never be happy. he will even go buy another 160GB hard drive if his gets full.
so he goes on to the next file and the next file and modem users get screwed.
does he share? not much! it might slow down his downloading a bit!
and developers are acting like this is something good.

Mr. Unregistered!

You prooved already , that you have no background information:

http://www.gnutellaforums.com/showth...&threadid=7981

So now you will tell us that Fasttrack is slow? That modem users suck and go offline - sure I have DSL too I suck my stuff and go off than I have it...

But the point is, when you are sharing while you are downloading all can get their stuff! And swarming supports this!

Do you really think Gnutella was/is faster than Fasttrack?

Swarming is the "FUTURE" of Gnutella!

Morgwen

Real Swarming
 

Does anything have real swarming?
What about freenet?

BS already calls multi-segment dling 'swarming'
 

Bearshare already calls their multi-segment downloading 'swarming'.

It is pointless to try to give the 'proper' definition of swarming here flying in the face of what they have posted on their web site.

They have 1G x as many users as you have here.

You will have to come up with a better word for swarming.

I actaully agree with calling multi-segment downloading swarming.

I also think it is all that is needed .. swarming sounds like a lot of work for benefit that can't really be measured yet. How do you know it will improve things, not just use up bandwidth on people's clients without benefitting them?

I wonder what percentage of people never download a 'commonly downloaded' file?

I would be interested to see a protocol where this happens, but I think until it is a proven benefit for like 95% of users (at LEAST!!!) then it should not be in gnutella, which works.

I would like to see file hashes and multi-segment downloads (should I just give in and call them swarmed-downloads? It's shorter to type and sounds less technical for the users) implemented widely (>75% of users) before I resorted to implement 'swarming'. You might find that it is unnecessary (I think so :-)

Commonly downloaded files are just that, common. They are everywhere. With file-hashes, you will be able to find them in an instant. With multi-segment downloads you will download from many many sources at once. How does it improve things to make some people's clients download even more copies of these files (partially)? Like I said, the benefit is so far unmeasured.

I am not just arguing against it, I am curious to see it, but I think realistically once hashes and ms downloads are wide, then it will be put on a low priority and eventually won't happen (in most clients anyway).

So I think it is OK to call ms downloads 'swarming' and just give up on the other idea. :-)

Real Swarming already proven
 

'Real Swarming' does indeed exist, in Edonkey2000. Partial files are used immediately for uploading (once one chunk is complete), and upload priority is given to the rarest chunk that you have.

The result is that even HUGE archives can be uploaded to thousands of people in just a few days. Take for example the 7 CD archive of a certain giant software company's latest visual development tool. We're talking about over 2.5 GB of data (7 CDs were not all full), released as 7 image files.

Starting just 2 days after this download was announced on an Edonkey2000 forum, it took only 1-2 days to download everything. That is the advantage of real swarming - the distribution pyramid that forms to spread files happens as quickly as a single chunk can be spread. What users see is that hot new releases are easy to download right away, not weeks or months later.

So, in answer to your question:
"I wonder what percentage of people never download a 'commonly downloaded' file?"

The answer is that everyone who ever wanted a 'hot' release of some new large file, like a movie or game, were in fact asking for a 'commonly downloaded' file. That's most people, I bet.

Oggs

Re: Real Swarming already proven
 

No!

This feature is missing:

spreading small file partials over the network automatically (increasing availability and download speed, distributing often requested files, balance load).

This means you share also partials from files you don´t have!

AFAIK only mojo nation and freenet have REAL swarming, but I am not sure I didn´t test them yet!

Morgwen

P.S.:

Xolox (Gnutella) has also the ability to share partials but only these you REALLY share!