Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › Wanted: Feedback & testers for my Twitter Archive
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Wanted: Feedback & testers for my Twitter Archive

wlanboywlanboy Member
edited March 2013 in General

Hello,

I recently received a free BuyVM box (thanks: @liam / @Francisco) and a free MySQL account (thanks: @Fliphost).

First thought: Now I do have enough resources to start my twitter archive. My first version was a local backup of my own tweets. Soon some of my friends started requests to add different twitter timelines. So I created a little frontend to list/search through the growing list of tweets. After some weeks the whole system was ready.

Currently the system is split into three parts:
image

  • A lighttpd proxy running in front of a thin cluster
  • A offside MySQL Database holding the jobs (timelines, searchterms) and the tweets
  • A RabbitMQ server for command handling + ruby workers doing the jobs – running on a ipxcore vps (thanks: @Damian)

Every command issued by the web servers (excluding queries to the database) are forwarded through a RabbitMQ server (message driven) to ruby workers doing the Twitter API crawling – and storing the results into the MySQL server.

It is all about limiting the web frontend as much as possible. The only logic implemented in the web part are the queries every other “action” is handled through jobs sent by messages to the RabbitMQ server. On the other side there are workers listening on queues for new commands. The message routing / handlers within the RabbitMQ server do all the magic (filtering/delay/…).

Now I want to test the system as a whole. And therefore I ask for your support and your feedback.

For LET I have created a LEB related list of timelines (the system is supporting different lists to separate different user groups). So all major LEB hosts should already be added to the system. If anyone is missing -> feel free to add them to the system (add a request for a new timeline).

If Francisco and Fliphost agree I will post the IP of the web frontend. If not I will use “private messages” for everyone willing to test this tiny system.

Next steps:

  • RSS Feed support for search terms
  • Download of timelines (json)
  • Customizing initializr template

If you have any suggestions or change request – just post them.

Kind regards.

Comments

Sign In or Register to comment.