Freebase & Phex
 

Hello all,
I recently noticed that somebody (ZwiSter) added a magnet link to Phex on my website, which is great.

I also noticed that there are a few entries by Phex in my cache. This surprised me, as the cache only seemed to work for G2 clients so far (or MP clients, like some people like to call it here ;) ) because of the way headers are sent in Gnutella clients. I was just wondering if it really works for Phex clients, or if this is some weird fluke.

So I'm posting here to ask if some Phex users could try and download some files off freebase (just pick any file, it's all freeware without ads or spyware, and most of it is open source), and see if your IP and client shows up in my cache.

If you want an explanation on what this cache is and what it does, I wrote an explanation here: http://www.freebase.be/g2cache.php .

Well, it seems to work. I've seen a new Phex in there now.

So if you ever post a magnet somewhere for Phex and/or Shareaza/morph/GnucDNA users, you can append the cache url to the magnet link so the people clicking the magnet get instant alt sources without having to search the network (unless they're the first one using the link, ofcourse ). Instructions on how to use it are here.

Thanks to whoever downloaded that Bridge Building game.

Can you give more details how this works. I mean Phex wont parse out the cache list for sure. What headers are you sending and recognizing.

Which license do you use for your alt-loc script?

I linked to your site from:

http://phex.kouk.de/wiki/index.php/Hauptseite

which is the main page of the Phex wiki (as soon as I got enough translatioins done it will go to a dedicated subpage, but as of now, it is a guide for users who want to download legal content with Phex).

Best wishes,
Arne

some stuff
 

Hi,
I'm bumping this thread again because I have a few questions about Phex:
I've seen a lot of downloads for the Phex installer on my site, and I was wondering where you're using a magnet for it. (If it keeps going at this rate, you'll beat cd-burner xp pro as top download ;) ).

I've also seen that you use the cache on this page for some content, but that you use &as= instead of &xs= . Why are you using that? It doesn't seem to be supported by gtk-gnutella (at least the version I have installed now, i should recompile it to see if more recent versions support it). I haven't tested any other clients, but I just want to know what the difference is between &as and &xs.

I'd also like to know if there's an IRC channel you guys hang out in ;).

Hi,

It's nice to see you again in here :)

first as & xs:
"as" means: acceptable substitute, which means, you can't be completely sure, if the source really has the correct file, for example to add a http-url, which can have different content later (like an error-page, because the content is down or an updated version of the file). You can use it to serve first sources for a magnet. You can find an example of that useage on my webpage:
- Page with magnets (to songs): http://draketo.de/inhalt/lieder.html
- Magma-list: http://draketo.de/inhalt/lieder/drak...ielliste.magma

"xs" instead means exact source, which contains a definite hash, in most cases the sha1-hash, to identify the content, so it will always be exactly the content you request.

And now as I think of it, the link to your cache should be an "xs", as it identifies the file via its hash. Sadly Phex accepts only one xs-source yet, and I need that one to reference to my own client to deeliver a first source.

Normally a client should first use only "xs"-sources and fall back to as-sources if no xs-sources are avaible.

Exact procedure would be, as far as I know, to

1.) search the network for the file. If the program doesn't find it in time,

2.) try the exact sources, and if they aren't reacheable (or too slow),

3.) try the acceptable substitutes. But if you download from accepteable substitutes, you _must_ check the file after downloading, because it could be corrupt (you should also check other downloaded files, but there it isn't a must).

And for Phex being downloaded quite often: I followed one advice you gave on your page:
"Keep in mind that the cache's efficiency improves if it gets used more, so I'd encourage people to just send me code patches, instead of using it on your own. Because one full cache is better than lots of empty caches. This code doesn't work on a lot of hosts, as you need to be able to read http headers directly."

And since Phex does its internal updates via magnet-links and the gnutella-network (what's the use of creating a p2p-app, when you don't trust it enough to download its updates via its own network?), I added your cache to our update magma-lists.

That means, every Phex which is updated internally contacts your cache and gets additional sources through it.

Should the strain get too much for your server, please tell us at once, so we can take it out of the magnets.

In the meantime, I'd like to say a big "Thank you!"

Your cache is something I hoped to see for years!

And I think it is all that is needed to make Gnutella just as effective as (or even more effective than) BitTorrent, since it doesn't need the cache, but can use it and gets more efficient by doing so (and a Phex will contact it regularly while it downloads (as it seems at the moment, that's once every 6 hours, that's what it tells me for this download), so you can even see who (and how many) has/have been active during the last "half day").

Your cache can be used perfectly together with http-altlocs:
A client gets a magnet and tries the Gnutella network.
If it doesn't find anything, it tries your cache, and if there aren't working clients in there, it downloads via http and becomes a working first source for other clients, who then spare the http-source.
So this eliminates the need to have a seeder running all the time. All that's needed is a http-source (which can be as slow as you want it) and your cache (which I hope is low profile, and which others can include very easily), and voila, you have magnets which are almost guaranteed to work, and fast.

And that's something, which has been missing in Gnutella (since it doesn't have searching by hash, because that would break some of the rather efficient optimizations which are being used (mainy QRP and Dynamic Querying, which each save about 70-90% bandwidth)).

Maybe you could tell the folks over at LimeWire about it. I think they could be quite glad to know about it (they've been trying to make magnets more efficient for some time now, but their solution was a quite heavy customized client instead of a slim server), or better still, tell it to teh Gnutella Developer Forum:
http://groups.yahoo.com/group/the_gdf/

(I can also post there about it, if you'd like me to).

Best wishes,
Arne

xs & as: If I read that post correctly, the cache is best served with xs, even though it doesn't send any file chunks by itself? I guess you should try to fix the bug that limits Phex to only one xs then, just so you can move to magnets that are better supported by the other clients (I have no idea who supports 'as', but I have a feeling not a lot support it, since most of them verify the downloads anyway).

As for the procedure for handling a download: I think the network is best served if the client tries the cache first, before searching the network. If the cache has useable sources, there shouldn't even be a need to do a network search during the download.

The Phex magnet: the load isn't anything freebase can't handle. The cache really doesn't take a lot of load, it's just sending a few headers and doing 3 db queries. I certainly don't mind you using it (In fact, I wrote the thing for people to use it. And to count downloads ;) ). I'm just always curious to know where people come from (i don't see any referers for magnets).

Using it for autoupdate is an excellent idea. If you ever feel like extending the script in a way that wouldn't exactly work with freebase, like sending the first few users to a http download to get it going quicker (or anything else you can imagine), feel free to host it yourself then if it's for specialised use. (although i might add an upload/alternate location field to freebase myself, i see some of the less popular downloads just don't find sources by themselves, even with Shareaza).

About Limewire: I've tried to get in touch with the people from Limewire (well, the guys that hung out in #limewire, those probably weren't the right people to talk to), and they weren't exactly supportive (except the guy who runs magnetmix).

An extra problem is that Limewire doesn't send its ip address and port when it connects initially, so basicly Limewire users (like firewalled users) never get added to the cache (because i have no way of knowing what port the client runs on). They do get to 'leech' the alt sources out of the cache, so it's not a big problem unless they're the vast majority of users (which is the current situation, sadly).

If they're really serious about improving magnets so they actually work on gnutella, I'd welcome it, ofcourse. I guess you could mention it on the gdf, if you want to. You know better than I who to talk to, and how to address them.

Hi Jonne,

"as" _should_ be supported by all (as it is part of the magnet standard: http://magnet-uri.sf.net ), but in fact isn't, because for instance limewire often doesn't do http-downloads.

The autoupdate already uses http-urls as starting location, even though I assume they are often not really needed.

You can get more infos on that in my infotext about magma lists (lists of magnets, similar to collections in shareaza, but simpler, more readeable and most importantly: easy to create automatically):
http://www.phex.org/mambo/content/view/56/47/
Phex uses magma-lists which contain the magnet for the update.

Also there are some more infos on magnets in my "GnuFU: Gnutella For Users" Wiki:
http://basis.gnufu.net/gnufu/index.p...netMakingGuide
(and if you'd like to add infos about your cache into that guide, feel free to do so. It's wiki for a reason :) )

Maybe it would be a solution to add alternate sources to the magnets instead of having them inside your script. Feels somehow more elegant to me.

Maybe you'll also take interest in Decentral Content Distribution using Phex. Some of it might also apply for Shareaza.

PS: I'm not on IRC, I don't know how Gregor does it.

But you can contact me via icq (48355585), aim (erynniakh), yim (arne_bab) or jabber (arne@jabber.fsfe.org, drak@kdetalk.net).

I've been thinking of altering Freebase's RSS feed so a gnutella client can subscribe, and can queue up a new file as soon as it becomes available (it probably wouldn't be a good idea to do it fully automatically, but just displaying new files in adialog with a description and an option to start a download would be nice). But sadly no client supports that yet. Some BT clients exist that can do that, though (but with torrents, not magnets). I think a feature like that would make file distribution more efficient too (lots of people would be downloading the same file simultaneously).

Someone's writing an rss plugin for Shareaza, but i don't think it's going to be finished soon. Ofcourse, it's pretty trivial to start generating a magma-list and a collection too, in php, but I'd prefer rss, because that's designed for pushing content. (magma lists and collections are more 'pull': the user downloads the file, parses it, and looks at what he wants. With rss, the user only sees the new files, and can choose to download a file as soon as it becomes available).

And using RSS would save me from confusing users with too many choices. They can open the rss file with any reader, and if that reader happens to be a gnutella client, the client can take care of downloading the files as soon as they're available.

Do you mean alt sources or http sources?
If you mean alt sources, it would break the cache, as I'd have no way to track the people that click the magnet (unless I add the cache as a source too).

If you mean http sources: I'm not really sure about which one would be best. I'd like to be able to send the http sources selectively, because if I just send them with every download, you might aswell just do http downloads. The gnutella client will barely download anything from gnutella. I'd just send a http source if there are no or few alt sources available for that file. And if people copy the magnet to their own site, I can update the http source without breaking the copied magnet.

A drawback would be that it'd tie the cache more to the website. http sources will have to be stored in another table, and that would be the table where all the files on freebase are stored. With adding http sources to the magnet, I wouldn't have that problem (but the http sources would be there every time, making every download essestially a http download).

edit: i tried sending a http source through the cache, but Phex doesn't accept it. (probably a protection so http sources don't end up in the alt source mesh). In fact, I think I tried doing it before with Shareaza, and I removed it again when it didn't work. So I guess I'll have to add sources directly into the magnet then.

I'd also like to know where exactly Phex stores its debug log (on Linux).

Hi Jonne,

Phex can in fact interpret and output RSS-Feeds, they just aren't included in the subscriptions, because I was a bit vary about writing code which would interpret every xml-file. I fully trust my codings skills to write something which allows a malicious xml-file to do quite nasty things, so I kept my fingers away from making xml-readouts automatic.

But if you include magnets as <link>-tag, phex can already get the magnets from them.

Besides: With magma-subscriptions, it is "push" too: The user subscribes, and Phex gets him/her every new file automatically (I hope we'll have a file-selection dialog someday, but I gave up the try this weekend after several hours). Files which are already in the share should not be downloaded again.

I'd like it very much to have rss-feeds with magnets (though I still prefer Magma-lists, maybe because I wrote the specs ;) ... I have some neat ideas, how they could be used in the future, but I don't know if someone will take up that idea, or if I will be able to code well enough to realize it in due time).

About additional sources: I thought about http-downloads, sorry for the braintypo...

Using http-sources, you can still have the clients mainly use Gnutella, and you can use slow webspace to jumpstart the magnets. I get around 300kB/s for popular files via Gnutella without http-source, and since my line can't deliver more juice, a http-source would be mostly insignificant as soon as the download mesh starts to kick in (and if the clients are programmed correctly, they won't even touch the http-source, when they have enough altlocs and get the full speed from them).

Gnutella can definitely outperform http, at least if you don't have a highspeed webpage and serve popular files. And even if 50% of the file come from the http-server, that's 50% saved bandwidth. For a managed server you could also simply reduce the speed it gives users for the files, and Gnutella takes the lead again.

bummer,
I already use the <link> tag for the link to the page that contains the magnet (and ed2k) links. Any chance in adding support for the <magnet> tag (or any other tag, <magnet> seems most logical to me)?

That way other readers can keep handling the feed the way they currently do, while gnutella clients can look for the <magnet> tag. (and ed2k clients for the <ed2k> tag, etc.).

I already added a <magnet> tag to my feed, in case you want to play with it. (the &'s are encoded as &amp;'s, because otherwise it wouldn't be valid xml, btw).

Taking the obvious route certainly has some advantages: The rss-feeds work with Phex out of the box!

(I added readout for magnet-tags, just in case, when I dabbled with the magma-reader to also do rss ;) )

Time to add RSS-subscriptions into the next Version of Phex.

Yay!

Time's up :)

The SVN Version of Phex now integrates rss-download and rss-subscription.
I tested it with your page, and it works. *happy*

Still missing is recognition of your sha1 (which is lowercase and longer than the one Phex reads out; what's the difference between the two ways of encoding the sha1? The one I know from magents is uppercase... ).

actually, the lowercase is completely wrong. I changed it to lowercase because it takes up less room (horizontally), and looks nicer, because i thought it didn't matter (Shareaza converts them to uppercase internally or something). But apparently the case matters in hashes, so I'll have to fix it. (give me a few minutes ;) ).

edit: sha1 and bitprint are uppercase now.

great!

Base32 is explicitely case-insensitive. That's a feature, not a bug. Emitting uppercase only is a good practice but you have to handle them in either case.

Relevant wikipedia article

Although I'll try to make sure all sha1/tth hashes are uppercase on freebase, just to avoid problems with clients that haven't implemented it 'correctly'. If you see any that are still lowercase, let me know.

ok