Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Small useful project ideas?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Small useful project ideas?

joepie91joepie91 Member, Patron Provider
edited September 2016 in General

I'm looking for some ideas for a small software project to work on; specifically, things that would be immediately useful to people here.

Emphasis on "small", though - I'm thinking of the kind of project that would be close to feature-complete after its first release, like utilities and monitoring tools and such. Naturally, the result would be open-source as usual :)

Any suggestions?

(EDIT: Things like "WHMCS modules" aren't really an option, though. I'd most likely be writing it in Node.js or, if the project warrants it, try my hand at doing it in Rust.)

«1

Comments

  • A proper todo list... with subtasks... and not using kanban bullshit.

    Thanked by 1Jonchun
  • Self-contained P2P network?

    Thanked by 1JasperNL
  • Awmusic12635Awmusic12635 Member, Host Rep

    Self hosted live chat?

    Bonus points for integrating with whmcs to verify client is who they say.

  • One tool I've seen people asking for that I'm fairly sure does not exist... grab a web page or a few from a website and see which CSS selectors in the stylesheets are redundant. The benefit being that it declutters the CSS file and saves on bandwidth.

  • Something that allows us to safely connect all our Kimsufi KS-3S together for a safe and fun porn sharing experience.

    Thanked by 2doughmanes netomx
  • Things like "WHMCS modules" aren't really an option, though.

    WHMCS style billing system that is known to suck but not as bad as WHMCS itself

  • @ricardo said:
    One tool I've seen people asking for that I'm fairly sure does not exist... grab a web page or a few from a website and see which CSS selectors in the stylesheets are redundant. The benefit being that it declutters the CSS file and saves on bandwidth.

    Then everything built with LESS would explode. :)

    Joking aside, I would definitely find use with something like this.

    Thanked by 1doghouch
  • joepie91joepie91 Member, Patron Provider
    edited September 2016

    @ricardo said:
    One tool I've seen people asking for that I'm fairly sure does not exist... grab a web page or a few from a website and see which CSS selectors in the stylesheets are redundant. The benefit being that it declutters the CSS file and saves on bandwidth.

    Hmm. Chrome already does this for single pages under the "Audit" tab, but not for entire sites.

    How would you envision this to work? You probably don't want to be scraping entire sites just to determine which selectors are unused, but if you only check a few pages, then it may report false positives, and you might end up removing selectors that you needed on a page you just didn't check.

    @GCat said:
    Self-contained P2P network?

    Self-contained in what sense? P2P networks are not trivial :)

    Awmusic12635 said: Self hosted live chat?

    Bonus points for integrating with whmcs to verify client is who they say.

    That's not a small project, though. There's lots of features that are needed before such a thing becomes useful. Not that I wouldn't do it as paid work, but it's definitely out of scope for the "small feature-completable projects" request :P

  • Chrome already does

    Interesting, hadn't known. I just recall someone asking for this kind of thing for years and years.

    How would you envision this to work?

    A few pages would be fine. Someone who's interested in this kind of tool would be fairly cognizant of how their sites works, e.g. for Wordpress you'd maybe choose the home page, a 'page' and a 'post'. Allow the user to enter a few pages of input and output selectors that aren't used.

    A web dev would find this useful if they've been rolling out a new page and improvising along the way, and looking to chop out the redundancy.

    Zooming out a bit, there's always a lot of interest in how to optimise a websites responsiveness, with the likes of YSlow being a useful tool. I think building great tools along these lines is great all round for being more conservative with resources, so it's a feel-good thing that people can get behind.

  • NodeJS? Make me a replacement for solus pwewse (not small, ik) :)

  • joepie91 said: That's not a small project, though. There's lots of features that are needed before such a thing becomes useful. Not that I wouldn't do it as paid work, but it's definitely out of scope for the "small feature-completable projects" request :P

    I would use XMPP as a backend - it's not that hard :)

    For something small but useful, how about an app that parses C code for #include <...>, locates the packages that would be needed on a given system and installs them, then compiles the C code?

  • joepie91joepie91 Member, Patron Provider

    ricardo said: A few pages would be fine. Someone who's interested in this kind of tool would be fairly cognizant of how their sites works, e.g. for Wordpress you'd maybe choose the home page, a 'page' and a 'post'. Allow the user to enter a few pages of input and output selectors that aren't used.

    A web dev would find this useful if they've been rolling out a new page and improvising along the way, and looking to chop out the redundancy.

    That seems like a potentially viable approach.

    doghouch said: NodeJS? Make me a replacement for solus pwewse (not small, ik) :)

    I'm not saying that I'm not still working on that, but... funding helps ;)

    jh said: I would use XMPP as a backend - it's not that hard :)

    Using XMPP as a backend would increase the complexity, not decrease it. Live chats aren't an inherently hard or complex thing to build, they're just too big and require too many features to fall within the specified boundaries.

    jh said: For something small but useful, how about an app that parses C code for #include <...>, locates the packages that would be needed on a given system and installs them, then compiles the C code?

    Few problems with that:

    1. I know very little about the C ecosystem.
    2. I don't really feel like encouraging people to use C for more stuff (look at Rust instead!)
    3. That'd require me to do a lot of integration work with package managers and conflict resolution to work out which dependencies are provided by which packages. There's no 1:1 mapping.
  • jhjh Member
    edited September 2016

    joepie91 said: Using XMPP as a backend would increase the complexity, not decrease it.

    I disagree. XMPP is remarkably efficient and easy to work with and eliminates the need for a database (that would create scalability and performance challenges). It could also eliminate the need for an admin interface.

    joepie91 said: Few problems with that

    joepie91 said: I know very little about the C ecosystem.

    You don't have to

    joepie91 said: I don't really feel like encouraging people to use C for more stuff (look at Rust instead!)

    You just said you know very little about C...? RUST is cool though.

    joepie91 said: That'd require me to do a lot of integration work with package managers and conflict resolution to work out which dependencies are provided by which packages. There's no 1:1 mapping.

    I do not think an MVP need conflict resolution.

  • How about a debian default install script that hardens the security and install basics software.

  • joepie91joepie91 Member, Patron Provider

    jh said: I disagree. XMPP is remarkably efficient and easy to work with

    I've worked with XMPP fairly extensively. It's not. It would be considerably easier to build your own internals, especially when you need custom features that aren't covered by a XEP (yet), which is precisely why basically every messaging company does that.

    I've had this discussion about XMPP many, many times. I've never seen the claims that XMPP is "easy to work with" hold any water when challenged, and I know from discussions with countless other developers that no, it is not "remarkably easy to work with" by any stretch of the imagination.

    I'm really tired of that discussion, and I don't really intend to have it again. Anything XMPP-based is out, simple as that.

    jh said: and eliminates the need for a database (that would create scalability and performance challenges).

    No, it doesn't. You're just using a database specific to the XMPPd you're using.

    jh said: It could also eliminate the need for an admin interface.

    It doesn't.

    jh said: You just said you know very little about C...?

    Yes. C is a bad idea for its lack of memory safety alone. That's all I really need to know about it, to draw that conclusion.

    (I also have my issues with the dependency model, but that's not a discussion I'd really want to have in this thread.)

    jh said: I do not think an MVP need conflict resolution.

    I'm not talking about an MVP, though. I'm talking about a feature-complete project. I pointed this out specifically in the first post.

    AlyssaD said: How about a debian default install script that hardens the security and install basics software.

    There's already quite a few scripts for this, and it's not really a simple project. Lots of edge cases to take into account. And honestly, if you want nicer system administration and 'hardened' systems, look into something like NixOS instead :)

    Thanked by 1GCat
  • @joepie91 how many cases of red bull do you need to work straight 24/7 until it's done? :P

    Thanked by 1GCat
  • joepie91joepie91 Member, Patron Provider

    @doghouch said:
    @joepie91 how many cases of red bull do you need to work straight 24/7 until it's done? :P

    I'm afraid I can only accept offers of Red Bull if you're also paying for the funeral :P

    Seriously though, if some providers (or anybody else, really) were to pitch in to get it done as regular hourly-paid work, it would probably not take too long. I just can't really allocate much time to it for as long as I still have to worry about generating an income...

  • #1: Self-hosted archive.is replacement.

    • Capture the page. There's tons of options: PhantomJS, Nightmare.js, etc... I'd prefer one based off chromium (Electron/NW.js) to allow better rendering. Run headless via xvfb-maybe.
    • Include core post-processing as a base plugin/s. Combine external resources, relative link parsing, strip JS, screenshot, archiving (mhtml and zip), etc...
    • Site based plugins for post-processing. This will allow plugins for individual sites (e.g. Expand Reddit comments).

    The basics could constitute as small. Just make the program extensible, and then others can add what they require... This could be the foundation for a side-by-side comparison utility, screenshot regression testing, or a content based diff tool.

    Let me know if this already exists ;).

    Thanked by 2GCat sayem314
  • @joepie91 said:

    @doghouch said:
    @joepie91 how many cases of red bull do you need to work straight 24/7 until it's done? :P

    I'm afraid I can only accept offers of Red Bull if you're also paying for the funeral :P

    Seriously though, if some providers (or anybody else, really) were to pitch in to get it done as regular hourly-paid work, it would probably not take too long. I just can't really allocate much time to it for as long as I still have to worry about generating an income...

    Well, if all the providers want a better panel, then it'd probably be a contribution thing. Providers who don't controlibute simply don't use the panel, and for the ones who do, it will be a plus for their own business.

    I wouldn't be able to chip in that much though, need to generate income too :/

  • joepie91joepie91 Member, Patron Provider

    @doghouch said:

    @joepie91 said:

    @doghouch said:
    @joepie91 how many cases of red bull do you need to work straight 24/7 until it's done? :P

    I'm afraid I can only accept offers of Red Bull if you're also paying for the funeral :P

    Seriously though, if some providers (or anybody else, really) were to pitch in to get it done as regular hourly-paid work, it would probably not take too long. I just can't really allocate much time to it for as long as I still have to worry about generating an income...

    Well, if all the providers want a better panel, then it'd probably be a contribution thing. Providers who don't controlibute simply don't use the panel, and for the ones who do, it will be a plus for their own business.

    I wouldn't be able to chip in that much though, need to generate income too :/

    Well, it's an open-source project, so anybody would be able to use the panel anyway. That's the point of the project :) But if it's not funded by anybody, then it's just going to take a good while before it exists.

    @telephone said:
    #1: Self-hosted archive.is replacement.

    • Capture the page. There's tons of options: PhantomJS, Nightmare.js, etc... I'd prefer one based off chromium (Electron/NW.js) to allow better rendering. Run headless via xvfb-maybe.
    • Include core post-processing as a base plugin/s. Combine external resources, relative link parsing, strip JS, screenshot, archiving (mhtml and zip), etc...
    • Site based plugins for post-processing. This will allow plugins for individual sites (e.g. Expand Reddit comments).

    The basics could constitute as small. Just make the program extensible, and then others can add what they require... This could be the foundation for a side-by-side comparison utility, screenshot regression testing, or a content based diff tool.

    Let me know if this already exists ;).

    That's actually surprisingly difficult to build. The closest things there are right now are wpull (which is a wget clone that supports youtube-dl and PhantomJS) and grab-site, which is pretty much a "job manager" around wpull to make it easier to grab entire pages or sites. Both are ArchiveTeam projects.

    Seeing as wpull is scriptable, you could probably already write extensions for those, along the lines that you're thinking of.

    Thanked by 1telephone
  • Another idea, something like PHPWHOIS, map the IPv4 WHOIS space and turn it into an API / downloadable thing. It's something I'm after that I could donate to (I'm interested in what entities are associated with the space, something more granular than ASNs)

    Thanked by 1GCat
  • @ricardo said:
    Another idea, something like PHPWHOIS, map the IPv4 WHOIS space and turn it into an API / downloadable thing. It's something I'm after that I could donate to (I'm interested in what entities are associated with the space, something more granular than ASNs)

    Maybe @GCat can donate some scanning power off his cluster

    Thanked by 1GCat
  • There's two things I wish existed that would be relatively simply to get a first version out. If I had the time I'd work on something in Golang for them.

    1. A DNS tool to manage DNS at multiple providers. ie, a single point where you add a record, and it gets added to Cloudflare, dns.he.net, and Route53. So you can use multiple DNS providers with a single management point.

    This would be pretty easy to get a base version out. If you made it modular, adding providers could be a community thing. Syncing providers and maintaining consistency would be important. In its core though, it would just be an interface for the various APIs.

    1. A smart SMTP load balancer. It would let you add multiple relays with unique information for each, and use all/some to send email. So you could take a handful of those 10k/m free SMTP providers and pool them together. Bonus points if it can track how many allowed emails and how many have been sent.

    This one also would be easy I feel. Using premade SMTP libraries, just make multiple objects for different relays, round robin them (or whatever), and track basic stats. This one could be used for spam though, but I'm not sure how big an issue that is since there are providers that give you 100k email a month for free.

    Thanked by 1GCat
  • joepie91joepie91 Member, Patron Provider

    ricardo said: Another idea, something like PHPWHOIS, map the IPv4 WHOIS space and turn it into an API / downloadable thing. It's something I'm after that I could donate to (I'm interested in what entities are associated with the space, something more granular than ASNs)

    This sounds viable, though I'm not sure whether the relevant WHOIS servers do any rate-limiting. I'm also not sure what kind of "refresh rate" you're thinking of?

    HackedServer said: A DNS tool to manage DNS at multiple providers. ie, a single point where you add a record, and it gets added to Cloudflare, dns.he.net, and Route53. So you can use multiple DNS providers with a single management point.

    That would be an interesting one. I'd have to think carefully about implementation, though - not all services provide an API, and it would need to be designed to deal with weird APIs as well.

    HackedServer said: A smart SMTP load balancer. It would let you add multiple relays with unique information for each, and use all/some to send email. So you could take a handful of those 10k/m free SMTP providers and pool them together. Bonus points if it can track how many allowed emails and how many have been sent.

    This one also would be easy I feel. Using premade SMTP libraries, just make multiple objects for different relays, round robin them (or whatever), and track basic stats. This one could be used for spam though, but I'm not sure how big an issue that is since there are providers that give you 100k email a month for free.

    Heh. I'm staying far, far away from anything to do with e-mail, I've been burned by that too many times. It looks easy, until you find that there's a weird edge-case that your SMTP library doesn't deal with, and now you spend the next week debugging shitty mailservers...

    Honestly, you might as well use something like nodemailer for that.

  • joepie91 said: This sounds viable, though I'm not sure whether the relevant WHOIS servers do any rate-limiting. I'm also not sure what kind of "refresh rate" you're thinking of?

    I think LACNIC does. A small pool of IPs would get around it and besides, I think mapping the entire space constitutes the low tens of thousands of lookups. Refresh rates, every 3 months or so would be good but that's just a ballpark... I imagine after grabbing the data a couple of times, you can get a good feel of how often it changes.

  • joepie91joepie91 Member, Patron Provider

    ricardo said: I think mapping the entire space constitutes the low tens of thousands of lookups.

    I don't know. How about ReferralServer entries or SWIPed IPs in general?

  • ZappieZappie Member, Host Rep, LIR

    ricardo said: Another idea, something like PHPWHOIS, map the IPv4 WHOIS space and turn it into an API / downloadable thing. It's something I'm after that I could donate to (I'm interested in what entities are associated with the space, something more granular than ASNs)

    Shameless plug:
    http://docs.bgpview.apiary.io/
    Example of whois + parsed: https://api.bgpview.io/prefix/8.8.8.0/24?with_raw_whois=true

    ^The abuse gets all whois things and stuff every for allocated/assigned IP space (both v4 and v6) and also whatever is seen on the global BGP table. Refreshes every 2 weeks, looking at reducing that to a few days soon enough

    If you want, more than happy to provide a DB dump of all the whois records


    @joepie91
    Would love to see a WebUI front end made to manage gDNSd (https://github.com/gdnsd/gdnsd)

    Thanked by 2ricardo GCat
  • joepie91joepie91 Member, Patron Provider
    edited September 2016

    Zappie said: Shameless plug: http://docs.bgpview.apiary.io/ Example of whois + parsed: https://api.bgpview.io/prefix/8.8.8.0/24?with_raw_whois=true

    ^The abuse gets all whois things and stuff every for allocated/assigned IP space (both v4 and v6) and also whatever is seen on the global BGP table. Refreshes every 2 weeks, looking at reducing that to a few days soon enough

    Any plans of retaining/providing historical data?

    Zappie said: If you want, more than happy to provide a DB dump of all the whois records

    I'm not sure whether this is on-demand or spidered data, but perhaps it would be an idea to automatically generate a dump of all the data every day, under some kind of open database license?

    Zappie said: @joepie91 Would love to see a WebUI front end made to manage gDNSd (https://github.com/gdnsd/gdnsd)

    Hm. That seems like too big of a project for now, especially given that I'm not familiar with gDNSd at all. Perhaps in the future...

    Thanked by 1GCat
  • Awmusic12635Awmusic12635 Member, Host Rep

    joepie91 said: That would be an interesting one. I'd have to think carefully about implementation, though - not all services provide an API, and it would need to be designed to deal with weird APIs as well.

    Might be helpful: https://github.com/AnalogJ/lexicon

    Thanked by 1qrwteyrutiyoup
  • ricardoricardo Member
    edited September 2016

    Zappie said: Shameless plug

    Not at all, looks great. Do you have a data download, though (I see you said you do). Would be great to have a look at the data :) It'd also be interesting to know how many records there are, since we've covered that in the thread.

    I do notice that it's not it's not returning the most granular data, though, e.g. whois --verbose 104.245.233.179 versus https://api.bgpview.io/ip/104.245.233.179

    Exactly what I was thinking of in my suggestion, though, thanks for sharing. Not looking to steer the thread OT. I'd like to start a thread about your tool after I have a better look if that's OK @Zappie

Sign In or Register to comment.