Scheduling and queues for solo automations

There is a particular kind of automation that a single person builds in an afternoon and then quietly depends on for the next two years. It resizes images, posts a digest, refreshes a cache, syncs two tools that refuse to talk to each other directly. It works perfectly until the day it does not — a burst of input, a slow downstream API, an overlapping run that steps on its own toes — and then it fails in a way that is annoyingly hard to reproduce, because the conditions that broke it were a moment in time that has already passed. The whole craft of running automations alone is about anticipating those moments cheaply, before they cost you a debugging evening, without building infrastructure so heavy that maintaining it becomes its own second job.

This guide is about the two decisions that determine whether a solo automation stays calm under pressure or falls over: how it gets triggered, and whether it gets a queue. Those two choices, plus three reliability knobs that fall out of them, cover most of what keeps a small system honest. None of it requires a platform team or a cluster. The point throughout is to add exactly as much machinery as the workload demands and not one component more, because every piece you add is a piece you alone will have to understand at eleven at night when it misbehaves.

Almost every automation starts with one of two trigger styles. Cron is time-driven: the job runs on a schedule — every five minutes, every night, every Monday — regardless of whether there is anything to do. Event-driven is reaction-driven: the job runs when something happens — a webhook fires, a file lands, a row changes — and stays asleep otherwise. The honest way to choose between them is to ask what the work is actually a response to. If the work is genuinely periodic — a nightly report, a daily cleanup — cron matches the shape of the problem and you should not fight it. If the work is a response to a discrete thing happening, an event trigger matches the shape and cron is a poor imitation of it.

The trap is using cron to fake event-driven behavior because cron is easier to set up. Polling a source every minute to see if anything changed is cron pretending to be event-driven, and it carries two hidden costs: latency, because the average wait is half your polling interval, and waste, because most runs find nothing to do and burn work checking. Sometimes polling is genuinely the right call — when the source offers no events, or when the polling cost is trivial and a real event pipeline would be more fragile than it is worth. But choose it knowing it is a tradeoff, not because setting up a webhook felt like more effort in the moment. The effort you save at setup you pay back in latency and noise for as long as the automation lives.

There is also a quieter third style worth naming: the manual trigger. A surprising number of solo automations should be a button you press, not a schedule or an event, because they run rarely, benefit from a human deciding the moment, and do not justify any standing machinery at all. Reaching for cron on work that happens twice a month is over-automation. Part of choosing a trigger honestly is being willing to conclude that the right trigger is your own hand.

A queue is a buffer between the thing that asks for work and the thing that does it. Most solo automations do not need one: the trigger fires, the work runs inline, it finishes, done. You need a queue when that inline model starts breaking down, and there are a few specific symptoms that tell you the moment has arrived rather than a vague sense that queues are "more professional." The first is bursts: input arrives faster than the work can be done, so requests pile up and either get dropped or overwhelm whatever is processing them. A queue absorbs the burst and lets the worker drain it at a sustainable rate, turning a spike that would crash you into a backlog that merely takes a while.

The second symptom is slow or unreliable downstream work. If the job depends on an external service that is sometimes slow or sometimes down, doing the work inline means the trigger is held hostage to that service — a webhook handler that has to wait thirty seconds for a flaky API is a webhook handler that will time out and lose the event. A queue decouples accepting the work from completing it: you acknowledge the trigger immediately, put the work on the queue, and let a worker take its time, retry, and survive a downstream outage without losing anything. The third symptom is the need to retry safely, which is hard to do inline and natural with a queue, because a queue gives a failed job somewhere to wait and a clear place to be tried again.

If none of those three symptoms is present — no bursts, no flaky dependencies, no retry needs — you very likely do not need a queue yet, and adding one is premature infrastructure that you will maintain for a benefit you are not receiving. The discipline is to add the queue when the first symptom appears, not before and not long after. Before is over-engineering; long after is the debugging evening you could have avoided. The skill is noticing the symptom early and acting on it while the change is still small.

Adding a queue raises a question people often skip: what happens when the queue itself fills up faster than it drains? A queue with no answer to that is a leak waiting to happen — it grows without bound until something runs out of memory, disk, or patience, which is a worse failure than the inline crash you were trying to avoid because it fails later and more confusingly. Backpressure is the mechanism that pushes back when the system is overloaded, and having an explicit answer to overload is the difference between a queue that protects you and a queue that just delays the catastrophe.

For a solo automation, backpressure usually takes one of a few simple forms. You can cap the queue length and reject or shed new work once it is full, which keeps the system bounded and signals upstream that it is overwhelmed rather than silently swallowing more than it can handle. You can slow the rate at which you accept work so it never outpaces the drain rate. Or you can let the queue grow but watch its depth and treat sustained growth as the alert that the worker can no longer keep up — a backlog that only ever grows is telling you the arrival rate has permanently exceeded the service rate, which no amount of patience fixes. The wrong answer is the implicit one: an unbounded queue with no monitoring, which works right up until it does not and then fails in the least debuggable way possible.

The mental model that makes this easy is to think of the queue as a sink with a tap and a drain. If the tap runs faster than the drain for long enough, the sink overflows no matter how big it is. Backpressure is deciding, in advance, what happens at the overflow line — turn the tap down, widen the drain, or accept that water spills somewhere chosen rather than everywhere. Choosing that on purpose, while calm, is far better than discovering the system chose for you, badly, under load.

Once a queue exists, a tempting next move is to drain it as fast as possible by running many jobs at once. This is where a lot of solo automations hurt themselves, because unbounded concurrency turns the worker into the source of the very overload it was supposed to manage. Fire fifty jobs at a downstream API simultaneously and you will likely trip its rate limit, get throttled or banned, and end up slower than if you had run a few at a time. Open a connection per job with no cap and you can exhaust connections, memory, or file handles on your own machine. The queue protected the front door; an unbounded worker kicks down the back one.

A concurrency limit is the cap on how many jobs run simultaneously, and for a solo system it is one of the highest-leverage settings you have. The right number is not the largest your machine can technically launch; it is the largest the slowest thing downstream can comfortably absorb. If the bottleneck is an API that tolerates a handful of parallel calls, your concurrency limit is a handful, full stop — running more does not get the work done faster, it just gets it rejected faster. Setting this limit deliberately, tuned to the real bottleneck rather than to your machine, is what makes a queue-plus-worker actually reliable instead of merely buffered.

A useful habit is to start the concurrency limit low and raise it only if you have evidence the bottleneck can take more. Low and steady almost never causes a problem; high and hopeful causes the exact failures the queue was meant to prevent. When in doubt, one at a time is a perfectly respectable concurrency limit for a solo automation, and it removes an entire category of self-inflicted overload at the cost of some throughput you probably were not going to get anyway.

The moment you have a queue and retries, you also have the possibility that the same job runs more than once. A worker crashes after doing the work but before marking the job done, so the job is retried and runs again. A network blip makes you unsure whether a job completed, so you re-enqueue it to be safe. Duplicate delivery is not an edge case in queued systems; it is the normal weather, and the only robust defense is to make your jobs idempotent — safe to run more than once with the same result as running them once. An idempotent job that gets delivered twice does the right thing twice and leaves the world in the same correct state. A non-idempotent one double-charges, double-sends, or double-creates, which is how a reliability feature becomes a data-integrity bug.

Making a job idempotent usually comes down to giving each unit of work a stable identity and checking it before acting. Before sending the email, record that this exact message was sent and skip it if the record already exists. Before creating the record, key the creation on something unique to the request so a second attempt finds the first one rather than making a twin. The pattern is always the same shape: derive a key that is the same across retries of the same logical work, and use it to make the second run a no-op rather than a repeat. This is the single most important property in the whole system, because it is what lets you retry freely without fear, and retrying freely is most of what makes a queued automation resilient.

It is worth being honest that perfect idempotency is not always free, and for some operations it takes real thought — but the cost is almost always worth paying, because the alternative is a system where every retry is a gamble. If you take one idea from this guide into your next automation, take this one: assume every job can run twice, and build it so that running twice is fine. Everything else — the queue, the concurrency limit, the backpressure — is easier and safer once that assumption holds.

A real temptation, once you accept that a queue is warranted, is to reach for a heavyweight message broker and a fleet of workers, because that is what the architecture diagrams show. For a solo automation that is usually a mistake, because the operational cost of running serious queue infrastructure is real and it lands entirely on you. A managed queue you have to provision, monitor, secure, and upgrade is a standing commitment, and if your workload is modest, that commitment buys you very little while costing you a meaningful share of the attention you have. The goal is reliability proportionate to the workload, not maximal robustness regardless of need.

For most one-person systems, the smallest thing that gives you a buffer, a retry, and a concurrency limit is enough — often something already present in the tools you use, a lightweight library, or a simple persistent list you drain with a bounded worker. The test for whether your queue is appropriately sized is whether you can explain its entire behavior to yourself in a minute and recover it from scratch in an afternoon. If you cannot, it is probably bigger than your workload justifies, and that excess is not a safety margin — it is surface area for the next confusing failure. A modest queue you grasp entirely is more reliable in practice than a sophisticated one you grasp partially, because when it breaks — and everything breaks eventually — the thing that gets it running again is your understanding, not the platform's feature list. Small and fully understood beats large and partially understood every single time you are the only one on call.

It is easy to frame cron and queues as competing answers, but in practice they pair naturally and the strongest small systems use both. A common and durable pattern is a cron trigger that does nothing but enqueue work, handing off to a queued worker that does the actual processing under a concurrency limit with retries. The cron job stays trivial — it fires, it enqueues, it finishes in milliseconds — which means the schedule never gets held up by slow work, while the queue absorbs whatever the cron produces and the worker drains it safely. The schedule decides when work is created; the queue decides how work is consumed, and separating those two concerns is what keeps each one simple.

This separation is more powerful than it first looks. Because the cron job only enqueues, an overrunning batch can never cause overlapping runs to stomp on each other — the next tick just adds more to the queue, and the worker, bounded by its concurrency limit, processes the combined backlog at a safe pace. The classic cron failure, where a slow run is still going when the next run starts and the two collide, simply cannot happen when the schedule feeds a queue instead of doing the work directly. You get the predictability of a schedule and the resilience of a queue, and neither has to compromise for the other.

So the real architecture of a reliable solo automation is rarely "cron or queue." It is usually a clear trigger — time, event, or hand — feeding a small queue, drained by a bounded, idempotent worker, with an explicit answer for what happens when the queue fills. Each piece does one job, you can hold the whole thing in your head, and when something goes wrong the failure is contained to one understandable component. That is the shape worth aiming for: not the most machinery, but the least machinery that survives the bad day. The companion guide on error handling and alerts covers what that bad day looks like and how to hear about it before your users do.

Before a solo automation goes live, a few minutes of deliberate questions prevents most of the failures that would otherwise find you later. The questions are not exotic; they are exactly the decisions this guide has walked through, asked plainly so that none of them gets answered implicitly by default. An automation where each of these has a real answer is one you will rarely have to think about again, which is the entire point of automating it in the first place.