New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
yacy - P2P Search Engine
I just found this on the internets - http://yacy.net/ .
It's a P2P Search Engine, so there is no central point. I just tested it, it's pretty amazing. At the moment I'm crawling LEB and LET from my home PC.
By default it's a community effort to index the whole www. But you can also setup a private node and only index the pages you want.
Might throw this thingy on a LEB. Just wanted to share. Might free us from the evil google.
Screenshot:
If you have an empty LEB, put it to some use and crawl the web :P
Thanked by 1ErawanArifNugroho
Comments
How much resources needed ?
M
@Maounique
At my desktop it uses 800MB, but then again i got 32gig ram in this box and don't really care. you'd need to play around with the max heap setting.
It's a nice idea, but it uses an unacceptable amount of resources (not surprising, considering it's written in Java...).
It managed to noticeably lag my desktop while just running, not doing anything special (2 x 2,4ghz AMD Athlon X2, 2GB RAM, OpenSuSE 12.1 + XFCE), to the point where I just terminated it and never started it back up out of sheer annoyance.
Why the hell would they write a crawler in Java instead of something like Python? Not only is it more lightweight, but (IMHO) the HTTP libraries in the Python standard library are way more usable than something like HttpComponents.
I have been wondering about this trend for quite a while. It seems that virtually all non-mainstream privacy/decentralization/etc. software is written in Java. I2P, Freenet, YaCy, and so on. The developers often come up with reasons like 'cross-platform' (uh, yeah, like every interpreted language is?), but noone has really been able to explain to me properly why they are all using Java. The only real exception I've seen is Anonet2, which seems to mostly use PHP-based stuff - not perfect, but definitely a lot more resource-conservative than Java.
The problem is that there do not seem to be enough Python devs that have the time to rewrite this stuff, and are familiar with pycryptopp and similar libraries.
Meh i'm sure this gives worse results than Google, so why use it?
If you had, like, taken the minimal effort to click the link provided in the very first post of this thread, you would have already had an answer to that question.
Of course its decentrzalized and better for privacy and blah blah, but all that is useless when it gives shittier results than Google.
That that is your opinion (or rather, your priorities) does not mean they're the priorities of someone else. Some value privacy over results accuracy - thereby answering the question of why someone would use it. And it won't get more accurate until more people start using it and contributing to it.