Queues, visibility timeouts, and dead messages

Queues are the quiet infrastructure behind agents, schedulers, and background work. app.nz queues follow the familiar SQS-shaped model: send a message, receive it with a lease, ack it when done, and retry it if the worker disappears.

The important concept is visibility timeout.

Receive is a lease

When a worker receives a message, the queue does not delete it. It hides the message until the visibility timeout expires. If the worker finishes, it acks the message and the row is removed. If the worker dies, the message becomes visible again.

That gives background work a basic recovery story without requiring every worker to implement its own retry table.

Max receives and dead messages

Some jobs will never succeed. A bad payload, missing secret, or broken downstream API should not retry forever. app.nz queues track receive count and move a message to dead state after the configured max receives.

Dead messages are not success. They are preserved failure. The operator can inspect them, fix the cause, and decide whether to replay.

Queue stats

A useful queue needs visible counters:

• total,
• visible,
• in flight,
• dead.

Those numbers tell you whether workers are keeping up, whether jobs are stuck, and whether retries are turning into poison messages.

Dispatch modes

Some queues are plain application queues. Others dispatch into app.nz systems such as agents. The queue record can carry dispatch configuration so a message becomes a task for the right subsystem.

That keeps the primitive general while still letting product features build on it.

Why not just run cron plus HTTP?

HTTP is a poor durability layer. If a worker request fails halfway through a long job, the caller may not know whether it should retry. A queue gives the job an identity and a state.

For agents, this matters because a single job can outlive a browser tab. For media tasks, it matters because a model run may take minutes. For CI, it matters because capacity may be temporarily unavailable.

The invariant

Every background job should be in exactly one of these states:

visible leased done dead

Once that invariant exists, features can share the same operational vocabulary. "Why did my task not run?" becomes answerable from queue state instead of logs alone.