All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Batch Management System - looking to find or write
raindog308
Administrator, Veteran
I'm looking for (or will write) a batch management system, similar to control-m, autosys, and such.
What is a BMS?
- central master(s) have jobs that are run on different servers (or databases, etc.)
- masters control schedules, windows, dependencies, alternatives, etc.
- so you can say "these 30 inter-related jobs on these 4 systems are a unit". You can pause that unit, decide to run it on a different set of servers, etc.
- you can setup dependencies so that job 4 on server A doesn't proceed until job 1, 2, and 3 on servers X, Y, and Z complete, etc.
- you can setup complex chains of actions - e.g., submit this torrent, when it's complete, download it and convert it, then stage it, then notify these people if it matches this string or these if not, etc.
- you can do things like set windows and priorities - e.g, these jobs can run anytime between midnight and 4am, but jobs 15 and 16 are high priority
- fancier systems have event-related options, or the ability to carve up power (e.g., in the 12am-4am window, allow 75% of power to job 15 because it's high priority and then let the rest run as they may), though that's more "render farm" and not my particular need.
- and of course there are reports, alerts when things fail, escalation rules, etc.
Mainly I'm tired of doing some manual things, and also logging into many servers to modify crontabs.
I haven't found anything in FOSS that is comparable, and most of the products have no qualms about saying "requires 2GB of RAM, Java 7, etc." which make them unsuitable for LEBs. This is different than ansible or something because it's not around configuration but rather around productive work.
Before I start coding...anyone seen anything like this? There's GNU Batch which I haven't had a chance to look at yet. It's also possible that maybe I should be looking at some event-driven architectures, but my needs definitely include dependencies that cross servers/systems.
Comments
It's probably not the right language but hangfire.io for .net but runs on mono would tick a lot of boxes. We use it for the majority of our backend queue and batch processing
If cost isn't an issue, BMC bladelogic may be a good fit.
I read "Bitch Management System".
What platform/programming language were you thinking of using for this project?
Aka the back of your hand?
What is the target market for such a tool and why isn't one of the existing offering worth it? I assume you're not building it just for the fun of it but for your day job.
Nope, for my private use. In my day job we use big expensive things like autosys, control-m, flux, etc.
Right now I do have a surprisingly complex set of scripts that do things...
Unfortunately, this is all managed out of cron and the cron part of it is a pain to manage. Also, there's a fair amount of coordination...I could move it all to an event driven architecture, but a fair amount of it isn't truly event. e.g., I want to backup every night - that's not event driven, though once a linux iso completes, I want to kick off a chain of events (today I just poll, which works OK given the frequency).
I admit this is at least 50% pretension on my part. I'm the sort to spend two days writing a script rather than spend an hour doing something by hand. I'm also the sort that likes to spend my day with a couple reports that tell me everything going on in my world. But yeah, it's mostly pretension.
I found some software packages...alas, they all seem to be Java based. I'm going to guess that's not very 128MB-LEB-friendly, but maybe I just need to man up and upgrade those guys. I really like the rundeck tour - it looks awesome. I don't care if the master takes 4GB/4c, just don't want every node to also require 4GB because today most of my world is <1GB and runs fine.
http://rundeck.org/tour.html
There's Schedulix...again...Java-based:
http://www.schedulix.org/en/screenshots
OpenLava also looks good. It's LSF-compatible. Amusingly, in the late 90s I admin'd a huge LSF farm of Sun E450s where engineers submitted compute jobs...indeed, I remember it being more "compute queues" than "complex interrelated batch" but I'll check it...
http://www.openlava.org/features/features.html
My core languages are perl, python, and shell. I'm OK with several others. I find I can code almost an order of magnitude faster in perl than python, but that's just because I've been doing perl since 3.x and I know all the "underlying' stuff (once you know sed, awk, etc. perl is just a cleanup).
However, python tends to force me into better code, and comes batteries included more often. Also I like PyCharm. So probably python here.
I'm also tempted to do C with the justification that it'll take less RAM on the nodes, but really because it's fun to get down and dirty...
There's Huginn, an IFTTT clone. They have a large library of automated tasks, but you can script your own in Ruby.
I was with you right up to that point...
Nothing against Ruby but perl/python/php is enough for me :-)
I have used control-m as well before and I can't say I know of a solution that doesn't cost an absurd amount of $$ for the same functionality
Not sure if this fits exactly what you are looking for but it could be a possibility https://stackstorm.com/
You can create your own in python too: https://docs.stackstorm.com/reference/packs.html