SuperNode Beta
 

LimeWire has released the beta version of our supernode implementation,
available at:

http://www.limewire.com/index.jsp/download_beta

In our preliminary results, client nodes receive almost no traffic over
their single, supernode connection, while supernodes are not adversely
affected by the increase in connections. The supernodes and clients also
use query routing.

If people have trouble connecting, you can try connecting directly to
216.27.158.74 on port 6357. We encourage anyone who would like to connect
to give it a try. Unfortunatley, only LimeWires can currently connect to
supernodes or become supernodes themselves (due to query routing, switching
between client/supernode
state, etc). The implementation is, as always, available from our CVS. The
branch is called supernode-branch. Credit for the supernode implementation
goes to Anurag Singla.

I am wondering: what happened to the Sparky Project? Is it dead? Is development continuing? Or is this an evolution of Sparky?
Thanks.
:-)

Hey TruStar, how's it going?

The Sparky project is still around, and it's actually incorporated into supernodes. We were having some reservations about releasing sparky given the current dynamism of the network (Chris can tell you much more than I on this part), but with the stability that supernodes add, it makes more sense. Clients communicate with supernodes using query routing (Sparky technology) that dramatically reduces the bandwidth they must use to communicate their files to the supernode.

This also makes it possible for the supernode to forward searches to the correct client nodes connected to it far more cheaply than if they had to search through a normal index, so it makes it much less costly to become a supernode (it doesn't slow down your machine as much as it otherwise would). In terms of communication between supernodes, that will likely not use the sparky technique.

So, the short answer is that it's still around and that it's getting incorporated into the client-supernode communication.

Hope that makes sense!

ATTENTION LIMEWIRE TEAM - supernode problem
 

Hey LimeWire Team,
U guys make a hell of a gnutella client. Its the best but the new supernode client is a serious pain in the **** to use,
I can't even get it to connect. It just sits there idle, i ve tryed adding the recommended IP address you mentioned in
this thread but no luck.I tryed to change the port through the options menu to the one u suggested but after i click "apply" and then hit "ok" it freezes up.
Could you perhaps elaborate on what to do. Im so used to just firing LimeWire up and having it autoconnect.

By the way, do u see supernodes as the future to gnutella scaling? Will the gnutella network be able to interconnect to the sheer size of Fasttrack? I can't imagine the amount of content being available with everyone connected to one another.
Keep up the good work.

Thanks,
Sweeppicker

SuperNode - what is it good for?
 

Dear Gnutella users, dear LimeWire team,

because I'm able to provide a connection with good speed, I somehow thought that someone would like it if I run a "SuperNode" of LimeWire. But I'm missing some kind of documentation to have a developer proved this to me. So I still wonder: What is a "SuperNode", compared to a normal client? What does it do, in contrast to a normal client? Who has an advantage of me running a SuperNode? - The first difference I discovered: I don't see a client/file statistic in the lower left part (status area), only the "Connect" button count of my shared files; my SuperNode client is running fine, though - at least many many clients are being checked.

It would be nice if I could find a hint on the LimeWire web page soon - today I could not, neither with the internal search egine nor with externals.

In respnse to the first user who has had trouble connecting, I think this is related to a more general connection problem, and not specifically related to supernodes. We took out some of the "quick connect hosts" from the supernode beta and added a supernode to that list, as we wanted to increase the likelihood that you would get connected to the supernode.

I would recommend restarting LimeWire, as the code for connecting to other users can sometimes sort of get "stuck" in a waiting state. We're going to look into this code again as soon as possible.

We definitely do see supernodes as the future of Gnutella. They allow the network to function for more efficiently, with far less messaging required across the network to search for files and to connect. With this technology, Gnutella will be able to "scale" to at least the size of FastTrack. It should significantly increase the number and quality of search results.

Thanks to everyone for trying the beta!

I've attached a diagram that gives a better picture of what supernodes do. In the diagram, each little oval is one computer, or "node," on the network, and each line is a connection between the nodes.

The larger ovals are the supernodes. The smaller nodes are the "client" nodes. Each client node only has a connection to the supernode.

I will attach a second quick diagram in the next message (I think I can only do one attachment per post) that depicts the current network.

As you will see, sending a message out on the current network makes traffic fly everywhere. With the supernodes, the client nodes also only receive searches if they actually have the requested file. Otherwise, the supernode does not forward the search to them.

Here's a depiction of the current network:

Impressive...
 

I think I got it. Of course, all supernodes need to communicate with each other and share search requests, to allow sharing between clients which are attached to different supernodes.

It's like a step towards the eDonkey2000 concept: There, servers only manage users and search requests, but don't host files. Here, a supernode is like a two-in-one request server and user client.

This concept I would expect to give a good boost to the Gnutella network - when many clients know about this technology.

do i have the beta version?
 

when I downloaded the beta version I received a message telling me "unable to install instashield". Does that mean that it did not install the new version or just the shield?:confused:

Thanks
 

Afisk-

I'm doing fine. Thanks for asking. I'm glad to hear Sparky will be used in later phases of the Gnutella evolution. The diagrams were especially helpful in visualizing the concept. When do you expect to see this supernode/client scheme to be realized?

Again, thanks for your reply.
:-)

I had a SuperNode running when I left the office on Friday - let's see if it is stable and still running after the weekend...

@natalie:

I found that the LimeWire installer may need to download additional components of the InstallShield system while preparing the installation - this may depend on the operating system you are running. If it is not able to download the proper components, and therefore is unable to complete the InstallShield system, then it most probably won't be able to complete the installation of LimeWire, too.

REPLT TO LIMEWIRE: attention Adam
 

Hi Adam,

I took your advice and restarted the limewire "supernode" client but still no luck.
The auto connect hosts just dont respond for some reason. It doesnt even appear to try and connect.
Its as if it forgets to. With Limewire 1.8 it autoconnects just fine. I had this same problem with "Sparky".
If I try to change the port via the options menu Limewire will just hang.
Im really eager to test out this beta. Sice I run Limewire 24/7 and am running off a cable modem I might have the resources to be a supernode.

Thanx again,
Mark P

Re: Thanks
 

> When do you expect to see this supernode/client scheme to
> be realized?

It was a long time ago that FreePeers (BearShare) created something called Defender (Before Defender was Reflector, by Clip 2). Defender too used the client/server model, where these "supernodes" handled most of the traffic, allowing low bandwidth users to connect to the supernodes and chat, search, save bandwidth, etc.

But a long time ago, expecting to implement the supernode/client model was a bit unrealistic.

Using Linux I encountered a little bug. When the download of a file is complete, it remains in the download-list although I checked the 'remove completed downloads' option. Moreover, the client tells me it is still downloading the file at 0k/s although the download has reached a 100% and the downloaded file has already been moved to the shared-folder.

Thanks for reporting this Linux errror. We'll have a look at it as soon as possible. If you're a programmer, you could also take a look at the source at www.limewire.org =). We'll fix it though, thanks!

I am not really sure why Vinnie never released Defender -- we have never really spoken with him about it at great length. The LimeWire "superpeers" are a similar concept with some slight changes here and there.

We expect to release our superpeer implementation quite soon -- sometime in the next month or so, if not sooner.

woohoo!
 

SUPER-PEER! SUPER-PEER! SUPER-PEER! She's SUPER PEERY!!


SUPER-PEER!!.....






get it? yeah....shutup.

vinnie
 

We didn't release it for a couple of reasons:

1) Proprietary implementation = bad

2) Inadequate discovery method for servers (relying on broadcast queries)

3) Uploading the entire index of files was expensive for modem users, and required a lot of extra work when the local file indexes changed (QRP solves this better, both technically and legally)

I think the LimeWire supernode implementation solves all of these problems and that's where we are going.

Hey Vinnie -- good to see you over hear on the LimeWire forums!

The proprietary implementation problem is definitely an issue. Hopefully the openness of the query routing proposal (the QRP that Vinnie alluded to, the underlying technology behind the old "Sparky" alpha), and of the supernode scheme and the LimeWire source code will help mitigate this problem.

As far as discovery of servers, that is one of the trickiest problems that we have faced as well, although I think that Anurag and Chris have worked out most of the issues there.

Best,

Discovery
 

Well, if both LimeWire and BearShare implement supernodes then discovery will be a much simpler.

The original Defender specifications called for searching for servers having certain languages or subjects but I think that the traffic it caused did not justify the feature.

After Limewire and FreePerre became best friends, maybe someone can explain some of the new ideas to the community? I have no idea of Sparky/Superpeer/Swarming implentation details, explanations or development discussion. Also I don't know who is QRP, Anurag, Chris?

Thx, Moak

The history of all of these ideas would be a bit too lengthy to go into in detail, but here's a general summary:

"Sparky" was the little codename for a LimeWire alpha release (released about 2 months ago or so) that was a test of the "Query Routing Proposal," or QRP.

The QRP is simply a way of doing a sort of distributed indexing of files on the network -- basically a way of efficiently letting the computers you're connected to know what files you have so that the computers around you only forward messages to you that they know are relevant to you (searches for files that you actually have). Just think of it as a way of routing messages on the network to the computers that the messages are relevant to as opposed to broadcasting those messages to everyone. The idea is to improve the efficiency of message passing on the network so that it doesn't take up so much bandwidth, leaving you with more bandwidth for transferring files. More information is available at:

http://www.limewire.com/developer/qu...%20routing.htm

Supernodes, or "superpeers," as discussed earlier in this thread are computers on the network that for purposes here index the "client" computers that are connected to them (see earlier diagrams). They actually use query routing for communication between clients and peers, but don't worry too much about that. Basically, the supernodes sheild client nodes from all of the messaging on the rest of the network, only forwarding searches to client nodes if they actually have the files. More information is at:

http://groups.yahoo.com/group/the_gd...upernodes.html

Swarming is simply the ability to download a single file from multiple sources. So, you download different parts of the file from different computers simultaneously. The idea here is to allow you to download files as quickly as possible (as you can try to max out your download bandwidth if the uploader is the bottleneck). Note that this doesn't help for low-bandwidth users.

So, QRP is a proposal and not a person. Chris and Anurag are superstar LimeWire developers and pillars of the community!

Links to all technical papers are at:

http://www.limewire.com/index.jsp/tech_papers

Woops -- looks link the lnks were too long. Just go to the last one -- it links to the others.

Thx a lot!

Maybe it would be a good idea and spread ideas to a larger community?

I think you assume that all developers or Gnutella interested users are inside the_GDF? I personally have some problems with the_GDF: The GDF is kind of hidden and still hard to find, I think no Gnutella webside has a link on it. The GDF files (e.g. as linked above) are only available together with an Yahoo account. A lot of traffic on the list are about client orientated details, while I was only interested in Gnutella ideas, concepts and development at whole.
Also it seems that development inside the_GDF is very slow and brake appart from the real world? I remembered when Xolox came up and showed the idea of segemented downloads, or details about FastTrack has been detected (especially the very interesting superpeer concept). I have seen no comment about this breakthrough (but Vinnie Falco saying how bad Xolox and there ideas are).

Personally I have been interested in Gnutella and new ideas. Months before (before I found the_GDF by acident) I was discussing with some other guys about swarming and future ideas... but the ideas were forgotten, because we didn't find other interested users.

Maybe in future you wanna read some postings in this forum (www.gnutellaforums.com, especially "General Gnutella Discussions") - or spread your research results to a bigger public. I hope a lot of interested user will now find a way to this new forum:

"General Gnutella Development Discussion"
http://www.gnutellaforums.com/forumd...?s=&forumid=42

I think dreams, ideas and knowledge are a key to an improved Gnutella (okay _hard_ work from the client developers is another!).

Greets, Moak

PS: Very interesting link collection!! Haven't checked your site a while. Btw, I'm sorry that I never could bring some ideas to Limewires research, at least the Limewire contest is not for Europeans.

hey!
 

Adam Fisk, what happened to the SuperNode beta? I really liked it. The reason I dont have it now is b/c I had to reinstall then I go check and its gone from limewire.com. But I have noticed in the limewire.props of 1.8b there are some strings containg the text 'SUPERNODE'. I'm making an assumption here, but does 1.8b has supernode features? But while I'm talking about the limewire.props file I would like to know what these values mean.


EVER_SUPERNODE_CAPABLE=false

SUPERNODE_MODE=true

SUPERNODE_PROBATION_TIME=300000

CLIENT_ID=(32 characters long)

The supernode beta basically got supplanted by our 1.8b beta, although you can still get it from:

http://www9.limewire.com/beta/supern...imeWireWin.exe

or

http://www9.limewire.com/beta/supern...eWireLinux.bin

or

http://www9.limewire.com/beta/supern...imeWireMac.bin

or

http://www9.limewire.com/beta/supern...imeWireOSX.sit


The values you noticed in your limewire.props file are there because we merged some of the supernode code into the main code base prior to releasing 1.8, although it is not active.

EVER_SUPERNODE_CAPABLE simply determines whether or not we have ever measured your machine to be capable of becoming a supernode or not. This is based on operating system, your measured upload and download bandwidth, the average amount of time you run the program on average, and your firewalled status. Note that this does not necessarily mean that you are incapable of being a supernode. Rather, it indicates whether or not we have ever determined that you are capable of becoming a supernode (specifically, whether you have done enough uploads and downloads for us to determine that you have high bandwidth).

I believe that the SUPERNODE_MODE key is a relic of the early code, and will not be included in the final supernode release. The SUPERNODE_PROBATION_TIME will also no longer exist in the actual release.

The CLIENT_ID is just a random number that uniquely (more or less) identifies your machine on the network (not to us, but to other Gnutella nodes). LimeWire knows nothing about this value. It is simply used to make messages work on the network.


As far as general Gnutella development goes, we do try to create as open an environment as possible by making all of our research documents publicly available on our web site and through forums like the GDF. The problem with participating in more general forums is that it starts to be very difficult to discuss technical details on a meaningful level unless the participants have a thorough understanding of the technology -- usually meaning that they have coded a Gnutella client themselves, or could easily do so. Basically, at some point it starts to become unproductive to open the discussion completely. Right now it's open, but you just have to be interested enough to find it, and technically savvy enough to understand it. If we had more resources, it might make more sense. Right now, however, we very much have our hands full!

Thanks.

For developers interested in getting involved, there's also the open source project at www.limewire.org. This includes a general discussion list on development issues.

I'm sorry to hear that you are not interested in a developer forum outside the_GDF.
For sure there are more highly skilled coders and network specialists which will not find a way to the GDF and therefor wasted. Do not expect a high amount of hits there and on the Limewire pages only. I personally are to busy to code my own client and don't want to participate at one special client, yet. However good luck with your research projects! Please post any documentation also outside Limewire and the_GDF.

PS: Please tell Vinnie not to change subjects without notice in the GDF, sometimes it's hard to follow threads. Someone will kill me for that notice yet. :)

Greets, Moak

Hi afisk!

No time?

There are developers having time to post useless threads!

But having no time to read a forum for improvements?

Ok all talented programmers know it is wasted time to think about Gnutella improvements, if you don´t want to hear them!

Morgwen

Hi Afisk,
yep I completely agree in this points! Thx for your feedback and I hope we find a bigger gnutella developer community to improve Gnutella and give ideas/concepts to the client developer! I also know that client programming needs a lot of time. So long, happy coding! Greets, Moak :)

Morgwen-

I'm not really sure what point you're trying to make. The client developers work very hard, and we basically participate in these forums whenever we have a chance.

Moak-

I too hope the Gnutella community expands and incorporates the many creative ideas of people like yourself to make the network as strong as possible. Are you working on a cleint? If you get a chance and haven't already, you might want to check out the source at limewire.org. It's meant partly as a reference implementation for other clients, so the code is very well commented with nice, complete javadocs.

Thanks for your work on all of this.

Hi afisk!

May be my post was a little to offensive!

But my point is!

There are many developers who work on Gnutella improvements, they are working also very hard for it!

So if you have only one or two hours a week to read this forum and comment the best ideas, it would help all people (you too!).

Not only the client developers are working hard for Gnutella!

All people who give people support for your products are working hard as well, please don´t forget this!

Morgwen

Morgwen-

I think I misinterpreted your message a bit. I did not mean in any way to discount the efforts of everyone involved in the Gnutella community. We all very much benefit from the hard work of people like yourself, and the client developers benefit a great deal from this larger community as well.

Hopefully we can all continue to work together to make the Gnutella protocol the web standard that we all hope it can become (it is now, to some extent but it could go much further)!

I've just skimmed through this thread and thought would it be feasible to implement a subsupernode feature in the normal LimeWire program.

The idea is that some of the ppl with 24/7 internet access might want to support the Gnutella network by appointing all available bandwidth to search queries and other Gnutella related network traffic when they're not downloading/uploading or sharing files.

Could this be an idea? Currently I'm on a ADSL 256/128 Up/Down line ,and I wouldnt mind letting LimeWire pass messages back and forth while I'm sleeping not at home i.e. not using the computer.

Idea?

That's definitely an interesting idea. The only problem here is that bandwidth is not typically the constraint on supernodes. The number of TCP/IP connections the machine can handle is more typically the bottleneck. In fact, supernodes need to devote surprisingly little bandwidth to messages -- not much more that normal nodes had to devote to bandwidth prior to the supernode implementation.

So, while this is an interesting and creative idea, it solves a problem that is thankfully not really there!

Thanks.

Hi afisk!

Trying to download the supernode beta with the mentioned link it always displays me the current 1.8c beta...

What is wrong?

Is the link incorrect?

Morgwen

Actually, that was just our general beta page, which was replaced by the 1.8c beta. The supernode beta is pretty much obsolete now, but it's code (a much improved version actually) will be included in the LimeWire 1.9 beta that is do out within the next week.

The supernode beta also had a built-in timer that should have timed it out by now as well.

Thanks.