As a followup to my previous reply I have posted details on magnet link tests with Limewire 4.1.1 Pro BETA, which fixes the issues with Bitzi magnets that I previously reported (below). However, the download never does start even though the magnet is accepted by Limewire. Details, and test materials, are provided at:

http://mantiscorp.com/people/aew/lim...agnets411.html

A description of the original problem reported to this forum is available in this thread:

Limewire 4.0.6 Breaks Magnet Links
http://www.gnutellaforums.com/showth...threadid=26254

Over the next few days I'll test the other features mentioned in previous reply. For the record, I've been running 4.1.1 beta with lots of uploads and downloads for several days straight, day and night, without any crashing or freezing -- it's good and stable on my system.

Aaron
----
System specs for LW Pro 4.1.1 beta test:

OS: WinXP Pro, SP 1, up to date with patches/updates
as of July 15th 2004

CPU: Pentium 4 2.8Ghz, Hyperthreading, 800MHz front side bus

RAM: 512MB

Motherboard: Asus P4P800-E Delux

Hard Drive: 72 GB Serial ATA (Raptor; Western Digital)

Video Card: PNY nVidia FX 5200 AGP 128MB ("Verto")
---

Wait a sec, have you taken a closer look at your magnets? They contain your IP (the xs parameter) and if you delete the file it is certainly no longer available. So how do you want download something that no longer exist? Keep in mind that LimeWire does not trigger auto searches...

Rkapsi, are you referring to the Magnet411.html test results page located at:

http://mantiscorp.com/people/aew/lim...agnets411.html

I believe so, and will explain what those test steps mean in more detail below with regard to your point about the IP address:

I downloaded the test files using Limewire specifically so that I could create the magnet links using Limewire's built-in feature (when a file's in the Library I right-click on it and select "Magnet Details for File...").

Then, I delete the file so that it's not in my own library -- the test files are widely available on Gnutella and so I simply delete my own copy so that I can test the magnet links that Limewire creates for them; the files are still available on Gnutella, by many users, so removing them from my own library doesn't make it impossible to find (doing so only forces Limewire to actually try and find it on the Gnutella network by using information contained in the xt magnet link attribute). I intentionally delete the file from my own library because if I try to use a magnet link (click on it from a Web page, for example) for a file that's already in my library Limewire won't even attempt to download it, which makes sense. By deleting it, I force Limewire to actually use the entire magnet link -- the files are still available from many users on the Gnutella network (I intentionally choose popular files for this test so that when I deleted them from my library Limewire would still be able to find many sources on the network).

In summary: After I've obtained a widely available file I then construct the magnet link as described and then *delete* the file from my library so I can test if the magnet link I just created actually works. Because the test files I'm using are widely available, Limewire should have no problem accessing them via the magnet link. Limewire should use this portion of the magnet link for the Macbeth.txt file, for example, to find sources for the file on the network:

xt=urn:sha1:EECI66YQYRH347K7S5O3NNFP5SSSQDUV

As you point out the xs attribute (not xt above) does contain my IP (or the IP of whatever machine the magnet link was created on if Limewire was used to create the link). This is how Limewire creates magnet links, so I didn't remove the xs attribute. However, a static IP address such as stored in the xs attribute should *not* be the *only* way the file is located. The xs attribute identifies a specific location for a file (http://###.###.###.etc) which is only one way to find the file, and that http location should be combined with the xt attribute to enable multi-source downloads (swarming).

If the file isn't available via the IP address specified in the xs attribute the download should still happen because xs contains a globally unique, location-independent identifier (urn:sha1:###) that provides Limewire with all of the information it needs to find and download the exact file specified. There's no open-ended search involved using xs. In other words, based on the urn:sha1 value stored in xt Limewire can locate and start downloading the exact file automatically even if the IP host specified by the xs attribute is not available.

This is especially important to do on a dynamic P2P network such as Gnutella where the host specified by the xs attribute can "go away" moments after the magnet link is created. Since I created these links on my own computer, for instance, if I quit Limewire the xs attribute is no longer valid since Limewire's no longer available to handle incoming those http request. Similarly, if I shut down my computer the IP specified by xs is no longer available. Even if I just restart my computer the IP address changes since I (like many other users) have a dynamic IP address that is assigned to me when I connect to my ISP: each time I connect I am assigned a different IP address. Therefore, the xs attribute can never be considered reliable. Limewire should use xs as one of many potential sources, but shouldn't rely on the xs host being available. Instead, xs should be combined with xt for multi-source downloads of a file that's precisely identified by the urn:sha1 value.

Currently Limewire 4.1.1. Beta fails to start the download after it receives and parses certain magnet links, but it does just fine with others. More precisely, it fails to start the download for magnet links that I create using Limewire itself as described in my Magnet411.html page. I can force it to do so by clicking on the "Find More Sources" (or Resume) buttons, as described in my Magnet411.html page, but the program should actually initiate the download itself and not require human intervention in this way. In these tests the magnets are created by Limewire for widely available files on the Gnutella network: it should have no problem finding them, and will do so if I manually click on the "Find More Sources" (or Resume) buttons, otherwise the download will never happen (it just waits and reports that it needs more sources although many are actually available).

Does that help clarify the testing procedures I used in the Magnet411.html file?

Aaron

Again, LimeWire does not search automatically for files on the network. Auto searches killed almost Gnutella and as long as no one has a sudden inspiration how to solve that problem it won't change.

Until then magnets rely on the IP provided via xs parameter.

I'm afraid that I don't understand what you mean in terms of auto-searching drains the network. Does Limewire currently only use the IP address specified by the xs attribute in the magnet link, and ignore xt altogether intentionally because using it would cause more traffic? Or do you mean that it intentionally processes the magnet link (and handles the xt info properly) and shows the file name in the download area, but doesn't actually start the download until the users clicks on "Find More Sources". That is, do you mean that enabling someone to click on a magnet link on a Web page to have Limewire search/download automatically based on the xt urn:sha1 part of the magnet is intentionally not fully implemented by Limewire and that human intervention (clicking on "Find More Sources") is intentionally required because that that extra step somehow puts less strain on the Gnutella network?

In any case, why would searching for files automatically via magnet links be any worse than the manual searching that we do now (i.e, typing in search terms)? Tons of traffic, both searching and downloads, are the result of people trying to find specific files that aren't easily identified on the network (and so trial/error to find them ensues).

A magnet link may require more hops to find the exact file since using the xt=urn:sha1 information forces a search for a precise file, but that shouldn't cause a great deal of traffic since large lists of "possible matches" don't need to be returned for vague search terms -- only when an exact match is found does a download start, otherwise return results should be ignored (or, ideally, not returned at all if an exact match isn't found for a specific sha1 hash). Perhaps Gnutella doesn't support searching for an exact file?

I would think that being more precise by specifying the exact file to download would be better for network traffic overall because users could then find exactly the files they want faster and with less searching than if using the usual keyword (manual) approach. The download traffic, in particular, should be much less: with precise magnet links I could get exactly the file specified by the magnet, whereas with a manual search I have to type in keywords and start the search, download potential matches from a large list of files that's displayed, and then preview a lot of files as they download to see if I got what I wanted. Most of the time I have to repeat this process over and over until I get what I want, meaning I generate a lot of traffic (both for search terms and downloaded files that are aborted after preview, or download entirely then deleted upon realizing they're not what I want). A good implementation of magnet links could eliminate a lot of wasted searching and downloads, which in turn would save a lot of bandwidth.

Is there some history on the auto-search problem that I can look at to get a better context for the reason Limewire doesn't support it?

Aaron

Here's a thread:

http://www.gnutellaforums.com/showth...ight=auto+find

Search for the term "auto find"

LimeWire does not ignore the various attributes of a magnet link but if xs fails it doesn't take further action to find the file automatically (e.g. by ds or xt).

Thanks for the link -- I took a first pass, and will study in more detail, but at first blush it seems that the issue described in the forum discussion you've pointed me at is about the negative impact of "auto-requeries" (auto searching) which I agree with and don't suggest be changed for magnets. The problem at the moment with regard to Limewire's handling of magnets is that it never actually sends out a first query in the first place (other than to the IP address specified by xs, which will almost always fail on a dynamic p2p network since IP addresses change frequently and host computers come and go continually). So I think we're saying the same thing in terms of the bandwidth issues and auto-requeries (auto search) -- don't use auto-requeries for magnets at all, but simply do a single initial query based on the xt=urn:sha1: info so that the process of clicking on a magnet and having it at least start the download (if possible) is seamless and reliable.

So rather than support auto-requeries it's only a matter of Limewire doing a single query in the first place using the xt=urn:sha1: info instead of relying on the http:// IP host info provided in xs. This would be the equivalent of doing a single search, but in this case for a specific file. If that search fails to turn up any files then don't do an auto-requeries, but instead require the user can click on "Find More Sources" as usual. The problem is that currently the first search doesn't take place at all, other than for the http:// on a specific IP addresss specified by xs.

Since the proposed fix doesn't require auto-requeries, won't create large amounts of traffic, and is only needed to properly process a magnet links would Limewire do that?

Aaron

I have finished reading the entire forum about auto finding that you pointed me at, and can confirm what I wrote previously: auto find (aka "auto-requeries" or "auto searching") is not needed to fix the magnet download problem. Although it would be nice if there was a solution to the auto find issue I wasn't suggesting it for fixing the magnets and it isn't a feature that I'm suggesting be implemented at all; my only concern was that Limewire breaks in terms of handling the magnet if the original http:// source isn't available (and it almost never is; in the month or so that I've been testing magnets with Limewire only a few have ever worked seamlessly -- I end up manually searching for the file once the magnet fails to download because the host specified by the xs attribute isn't available).

My reason for posting my magnet test results was to provide details so that developers could fix the download issue, which seems pretty straight-forward and doesn't require auto-finding. If Limewire attempts just a single search (and not more) for the file specified by a magnet's urn:sha1:### info that would solve the problem I've reported. If that single search comes up empty, then it's up to the end user to manually "Find More Sources".

This solution (a single search based on the urn:sha1:### info if the xs http:// host is not available, but *not* using auto-find/search to repeatedly do the search) won't create the bandwidth issues that resulting in the auto find feature being removed from Limewire in the first place and would fix the problem with magnet links.

I think we were talking about two different things entirely for most of this dicussion, and hope this clarificaiton helps (and thanks for the material on why auto-find was removed in the first place).

Aaron

Actually, I think we could use some kind of auto-requerying.

I've been using a auto-requery-enabled LimeWire at home and it really works a lot better if LimeWire does the find more sources queries fully automated, one more FMS per hour. BearShare is using automated requeries, too and I think sooner or later we will see LimeWire use limited automated requeries again.

I'm not opposed to auto-requeries being implemented in Limewire in some form if the implementation doesn't flood the network, but for this particular feature (fixing the broken magnet download process) I'd be just as happy with a single-search based on the urn:sha1:### info to start. If auth find makes a comeback as a result that'd be great, but it not even having the single search occur seamlessly would be a great step forward since currently using only the http:// host is only useful when the host is a stable system on the network (very few are -- almost every magnet link I've tested is not available in Limewire since it relies exclusively on the IP host, whereas they work just fine in other p2p programs that actually use the extra info to search.

Shareaza, for example, does a good job of handling magnets. When a magnet is used with Shareaza it displays a dialog box to the end user asking if they want to download the file immediately (via the http:// host) or start a search (using the hash info).

Simply fixing the broken magnet download process (so that it starts a search at least once) would be a big step forward for integrating Limewire with the Web. Over this summer we're testing Limewire Pro and other P2P clients on a medical network that uses a Web-based front end to present doctors and patients with media (owned by the hospitals; no copyright issues). Currently Limewire doesn't work because it doesn't integrate well with the enterprise or the Web. To integrate with the Web it requires seamless downloading of magnets, and also the ability to hand-back the content it downloads to the client (serve to the client after processing the request to download a file). To integrate better with the enterprise we need some form of control over which nodes a given Limewire client will interact with. All of these are fairly straight-forward updates, and would go a long way toward making Limewire Pro suitable for enterprise (business) applications, and so I'm more than happy to do the extensive testing needed to help make that possible.

Aaron