3 Queues for different file sizes
 

Hi,

Today I saw again someone trying to download a 27kB file stuck in my queue behind 4 >200MB downloads.

That's why I think, you should implement the proposed 3 different queue-sizes:

One reserved slot for files whoose size is below or exactly 1MB

One reserved slot for files, whoose size is between 1MB and 15MB and

One Slot for Files, whoose size is 15MB or larger.

That would mean: every LimeWire would grant at lest 3 upload slots. All slots above that should be unspecified (not size-limited).

I propose an alternative. I really don't want 3 queues (although my example looks a lot like 3 queues).

Add an option for "File Size Management" to the Option menu:
< > Check here to enable File Size Management
1. For files greater than "a" and less than "b" allow "x" uploads
2. For files greater than "c" and less than "d" allow "y" uploads
3. For files greater than "e" and less than "f" allow "z" uploads
(x+y+z cannot exceed max upload slots)

The uploads queue would still display in the same manner with user-selectable sorts.

The advantage is that advanced users are given the greatest control over managing their bandwidth to the shared file sizes in their library. All users will benefit from the individuals that manage these settings appropriately.

Here are two examples to clarify the above criteria:
-High-bandwidth Setting Recommendation
Max Uploads set to 34
<x> Enable File Size Management
1. For files greater than 50M and less than 999M allow 4 uploads
2. For files greater than 9M and less than 50M allow 10 uploads
3. For files greater than 0 and less than 9M allow 20 uploads

-Low-bandwidth Setting Recommendation
Max Uploads set to 6
<x> Enable File Size Management
1. For files greater than 9M and less than 999M allow 0 uploads (yep, it's a filter)
2. For files greater than 120k and less than 9M allow 1 uploads
3. For files greater than 0 and less than 120k allow 5 uploads

Note: Files greater than 9M should not be returned in search results (in the 2nd example).

The filter for low bandwith would eliminate the advantage from Partial File Sharing for big files.

What would happen in your example, if 20 Users wanted to download big files?

Would then the small slots be still free, so that small downloads can pass by unhindered?

What would happen if teh slots for small files and large files are filled and a downloader of small files would get into the queue?

Would he/she then have to wait, till the downloaders in the Queue before him/her have begun downloading, that means till a slot in Big-Files is free?

If not, you'd either have no Queue for small files (your client simply answers "busy"), or a seperate one (to be able to get the download at once, when a slot is free for small files, feels a lot like a second queue).

For a page containing teh summary of a a discussion (and the discussion itself) about Queues generally (so we don't repeat everything here again), you can look here:

http://draketo.de/inhalt/krude-ideen...sscussion.html

Actually Limewire uploads files by fragments of max 100KB (with a 10 bytes overlay). This allows faster propagation of the download mesh with partial file sharing is enabled in the downloader, that performs successive requests on its connection to upload new fragments and advertize its locations, and to give feedback to the uploader that it now shares some fragments of the uploaded file.

If there's something that may be impoved is to allow interleaving of download requests: for example, after uploading 1MB to the downloader (10 fragments), its connection would be rescheduled (in the active queue), to allow concurrent uploads to perform their swarmed requests, allowing serving more clients.

I think that LimeWire could maintain a higher number of active queuing connections, while also maintaining a low number of active uploads. For example the soft maximum would be set to 5 for low low bandwidth connections, but there should be no problem to maintain a list of 64 active queues.

Even on "symetric" broadband connections, the upload stream is most often limited face to the download stream (most often the effective upload bandwidth is about a quarter of the download bandwidth). If you're not convinced, look at websites that compare the speed of various ISPs (ZDNet has such a online tool for US, in France we've got grenouille.com): you can have a symetric 2Mbit/s cable connection that delivers upstream only about 512kbit/s. (Cable providers do not indicate measures of this assymetry, I think it's a lie to their customers, which is more obvious on ADSL where advertized upstreams are much more accurate and effective).

Smarter management of upload queues would allow non-infinite delay for transfers from hosts with limited upstream bandwidth, so that very large files will still continue to be uploaded when allowing also to deliver smaller files. My opinion is that filesize should not be taken too much into consideration, but rather the diversity (because now swarmed transfers work perfectly and with the same performance on slow and broadband connections).

For this reason, I think that bandwidth filters in searches should be removed. There's as much value for a swarmed transfer from modem users or from broadband users. Queueing should be based on smaller units than just full files.

Gnutella has never been so fast today. If we want even faster transfers we must maximize the effect of swarmed transfers by allowing more hosts downloading concurrently the same files, with PFSP enabled.

There's less risk to upload a single large file to many hosts (that will then collaborate to interchange their file fragments), than to let a single host download a very large file and then disconnect immediately once the transfer finishes. This would make the mesh much more resistant to single points of failures (a host that disconnects just after its transfer terminates). It would really increase the variety of available contents on the Gnet, and larger sets of alternate locations for large files.

Please don't do it like that!
Someone out there already does and you can imagine how I hated downloading a file by a 100kB Junk, the being put behind some Modem-User.
I think feeling is far too important for this.

This is about making it possible to download small files, while many big filles are in the queue, but a downloader of a big file would also hate not really being able to download continuously, at least I would. THe difference between theh Small and te big file is: Small File downloaders come, grab the file and are content, Big file downloaders grab your Upload slot for some time. but if small-file-downloaders could get in between, the big-file-downloaders would sit in queue most of the time, that's why I'd give those small-file-downloaders an extra upload slot. It doesn't stop other downloads (which would make reconnection necessary) and allows them to spread quickly and effectively.

Also this wouldn't gain anything, when siimply five big-files were in the upload-queue (or the queue would grow indefinitely). Since swarming I often get 10 to 20 uploads to other downloaders while I download a big file, so there would be o space for a small file in between (especially because I prefer to have an upload done, and not to have it linger endlessly in Queue, because the first assures me, that the file is completely avaible on the Gnet.

Filesize makes a difference for me as User. I'm not patient when waiting for a small file, but when I wait for big files, I have no problems with being in a queue, because I know I'll get the file at some time (and it will take 3 hours anyway, so why bother with 15 minutes in queue?). For the first, quick download is more important, while for the second it's reliability.

Complete agreement, as long as we upload different fragments to different hosts, while still allowing them to preview the file.

I had the Idea to always alternatingly upload junks from the beginning and either the first or second third of the file (randomly selected junks from the second or third third, but for one host only for the third third and for another only for the second third, till all those have been uploaded, though I realize, that the host must request that range itself, so it is up to the downloader to download sensibly).

Do you realize that, in this last sentence, you are contradicting your own position in the first paragraph, because you admit delays when downloading large files?

I see no difference between distribution of files of various sizes and distribution of files with equal sizes (such as file fragments). The effect I was describing is similar to splitting large files and distributing them separately with equal access to the candidate upload queues. However large files in queues take too much time on uploaders sites, and uploader cannot guarantee equal access to all downloaders.

Swarming, if PFSP is enabled, allows implementing this equality policy, and differences of file sizes become much less relevant than with PFSP disabled (where it makes sense to make distinctions between uploader's upstream bandwidth for large files, but at a too expensive cost for the distribution of small files.
What just needs to be guaranteed is that large files swarmed from multiple sources that got only some fragments, will converge rapidly to create the whole file for all clients.

This is possible provided that the uploader queue keeps at least 1 upload slot per fragmented file. If everybody (at least Limewire agents) adheres to this policy, all uploaders will be able to redistribute their copy of the fragments, and will collaborate to help the initial uploader (that may be the first and single source for that file, placing it at risks to monitoring, as long as the mesh contains too few alternate locations with complete files). Rapidly the intial single source will be helped and will no longer to alone to distribute the needed fragments.

I do think this is the whole interest of PFSP: faster distribution of new files, which will become popular and accessible much faster without depending to much on the first source. For very popular files that get lots of incoming transfer requests, we could even imagine to reduce the fragment size depending on the number of known alternate sources. That this file will be large or medium will not make a big difference. But for some small files below 1MB, this could be an improvement so that uploaders will be less likely placed at risk of failure.

This is already a problem still today: I got some partial files for which I can no more find any other copy and the initial source is offline, however the file was already large and it took lot of time to download that large fragment from the initial source. But as nobody helped that source, there was no other copy available for the end of file. Result: the time and bandwidth to upload this fragment in uploader was lost, and my own time and bandwidth was also lost too as I can't recover a complete copy.

My own measure for large files (videos or collections of graphics or CD images) show that this event occurs quite often, for 2/3 of attempted transfers, and for which there will never be any other source (the initial uploader may have spent much time to upload the begining of files to many candidate downloaders, without having finally uploaded at least one copy of the end-of-file fragment). If that uploader decides to stop there after uploading that file for 3 or 4 days, all this bandwidth will be lost. That's where we need a better strategy to make sure that after 2 or 3 hours of uploads, for a file that could have been uploaded directly to a single downloader, this file will have at least 1 or 2 complete mirrors on many hosts collaborating each other to terminate their partial copies.

Yes, but only in the beginning, else I could as well say I download only with 1/3rd of the possible download-speed. THis is an issue for DSL-Users, but far more for Modem-Users. While a DSL-User often doesn't manage to max out the download-speed, the modem User does so most of the time, and when he doesn't download for 2/3rd of the time (because he's in queue) he's losing real speed (except if all others directly upload to him instead of uploading something else).

Do you mean, that every program should always keep some upload-slots reserved for currently or brief ago downloaded-files?

This is only for big files, though (but true, I think).

Three Upload Queues aim mainly at making Small files more avaible.

>Do you mean, that every program should always keep some upload-slots reserved for currently or brief ago downloaded-files?

We are looking into this option. Specifically, we're examining if it would be feasable/helpful for a downloader to reserve a slot for its uploader to download parts of the file from it, if the uploader requires some ranges that the downloader has.