Partial File Sharing in LW!
 

Wow, PFS work in LW! In the 3.1.0 jum170, build at least. It's great news, the only problem is that I upload mostly to Shareaza users, and I don't want to, but I guess it's normal, since they're supporting PFS. Looking forward for leechers' sharing! It should help the network.

Glad you like it. If you look closely, you'll notice the files in the incomplete folder have turned from red to black indicating they're being shared, and contain the same stats as shared files do. :)

Works well here too. While checking for upload stutter, PFS started an upload to a LW 3.1.0 of a file I was downloading. Congrats

the 0k incompletes are also shown in black. Would this be because they're sharing info like alternate sources?

Will it do any good to load old incomplete files into the incomplete folder if the original downloads.dat file was lost?

Technically the incomplete files are shared the second they're created (assuming the user has allowed the sharing of partial files and the queryhit has given us a hash of the complete file). For zero-length (and other small files) this allows alternate locations to be exchanged. Realistically, you will only receive requests for the file after approximately 25% of it is downloaded (or 5MB, whichever is larger). This is because you will not insert yourself into the 'download mesh' until that threshold is reached. We're looking at tweaking those numbers.

Not all files in the incomplete directory are shared -- only those with an associated entry in downloads.dat. This is because we limit partial file sharing to requests by hash in an effort to reduce corruption. You may also notice that if, for whatever reason, the "Your download is corrupted" message appears while downloading, then the incomplete file will be unshared (and hopefully turn back to red).

Ah--thanks for the info.

Could the T-number (the #of bytes to be downloaded?) of the incomplete file be used to lookup the original hash to be reinserted into the downloads.dat?

let me know if TSQ (that's a stupid/silly/shitty question)! :p

In order to generate the needed hashes, you need every bit (literally) of the file. The number after the T is simply the size of the completed file. The reason the hash is not included in the filename of the incomplete file is because some filesystems have limitations on how long a filename can be, and hashes are long.

OK. Trying to resume old incompletes would not be a good idea then or in the future, once the downloads.dat file is suspect or deleted.

So, anyone with really old incompletes should just be advised to save themselves and others grief--remove them.
. . . and to ignore any search results that start with the Tnumber?

I guess I was thinking PFS might also mean additional PF recovery. Of all p2p programs I've tried, LW is the best at resuming incomplete files. It's so easy to forget what a pain interrupted downloads used to be less than a year ago!

LimeWire does not match queries against incomplete files. (Meaning that the numbers in the 'hits' column should NEVER go above 0 when looking at the incomplete directory.) If you see a result with a T-# when doing a search, it is because someone has actually shared their incomplete directory.

Ah, our prayers have been answered. Although I don't have anything partially dled (I deleted all my incompletes), it sounds here as if it is working.

This is so fantastic. Leechers will be forced to at least partially share. The entire gnutella network could really get a huge boost from this, content wise and user wise.

PFS finally in LimeWire?

This is great news..great job LimeWire developers..that was one of the few things that NEEDED to be added..

Finally...PFS in Lime arrives!

BUG! Pour Sam!
 

Salut Sam! It happens often that my limewire is trying to get fragments of the file I'm dling of my own client.... Maybe it is because I'm behind a router: I get a try from 192.168.1.1 (linksys), my local network address is 192.168.1.3. (I've activated port forwarding etc... I'm not firewalled...). It's obviously a small error...

Bonne chance pour trouver LA solution...

I noticed that the other night. It seems to be more of a bug in the downloading logic as a whole (meaning that if somehow you were enterred in the download mesh, you'll try to download from yourself) rather than in PFS... but, PFS will cause it to happen at a higher frequency because you actively enter yourself into the mesh. I'm not too positive how to stop it in the NAT'd case though, especially in the setup you mention (which is basically the same as mine).

There's a few other bugs we have to work out as well.

Sam, looks like after a crash my incompletes won't resume. Have you seen downloads.dat become unusable after a crash or should I try to get more info on this?

The downloads.dat file becomes unusable if the crash happens while LimeWire is trying to save the downloads.dat file.

Hi trap_jaw--good to read you again. Makes sense. Might there be a way to resume the incompletes that may have been affected?

No. Your downloads.dat file does not only contain information on download locations but also all the data on which blocks of a file have already been written and which blocks are still missing. The latter is (unfortunately) not recoverable until LimeWire implements a feature to verify single chunks of a file (e.g. THEX/TigerTree).

Thanks once again for the info. Would saving a backup of the downloads.dat file allow the file to be resumed from the last saved entry in case of a crash? I noticed the Acquisition forum discussing a save preferences feature (haven't tried Acq for a while).

Yes, LimeWire could simply always create a backup copy of the downloads.dat file. There was a patch for the FileManager which among other things added a feature that allowed create backup copies of important files but it didn't make it into the official LimeWire version yet.

Thanks. Sounds like that patch is needed if PFS makes the downloads.dat file even more vulnerable and valuable. I'm guessing the emphasis is on avoiding the crashes that lead to the corruption.
[aside--hmm: is there a link to see who should get the credit for that patch?]

I don't have a link - but the credit goes to Kenneth Corbin and Philippe Verdy.

FYI,

I just merged in a change to provide better backup support for the downloads.dat file. It goes through the following steps:

Before writing downloads.dat, rename the existing one to downloads.bak. If writing the file fails at any time, delete the current downloads.dat and copy the old downloads.bak to downloads.dat.

When starting up, if reading the downloads.dat file fails for any reason, it will try to read downloads.bak. If that succeeds, it will copy downloads.bak to downloads.dat.

Thanks for the info and update.

I have lost my fileurns.cache file several times last week (due to Apple JVM bugs), how painful this is depends upon how many files you share. I have just submitted a patch to implement the same backup strategy for that file as well.

A crash wiped out a 219 MB dld at 76%. After reading sberlin's post, I used this method: started dld again and, at 40%, copied downloads.dat and named it downloads.bak (I left it in incomplete folder). Went on dlding, until a scheduled shutdown occurred. When I resumed dlding, to my horror, it started at 0%. So I went to incomplete folder, removed downloads.dat and renamed downloads.bak (the one with the dld at 40%) as downloads.dat. Started dld of the file, but it still started at 0% not the expected 40%. What did I do wrong? Did I have to do sthg with the partially dlded file?
I am using LW basic 3.2 on M%acOS 9.2.2

Manually copying it might not work -- LimeWire writes the file every 30 seconds, so chances are rather high that you copied it 'during' a write, so the file wasn't actually finished.

Some people have also reported problems where resuming a file does not display the correct percentage immediately. However, one a second result is found and the file starts to download again, the percentage goes back to where it was.

I have just lost another downloads.dat file even though nothing appeared wrong upon quitting LimeWire. I copied away the .bak file and looked what might caused the problem. Upon attempting to read in that file I have found that the following exception is thrown:

java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.limegroup.gnutella.messages.QueryReply$PushPro xyContainer

I would suspect that there is something wrong with storing downloads via push proxies.