Lifecycle and events

What changes a bench without anybody clicking anything.

Who this is for. Somebody wondering why a row changed, or adding a job that
changes one.

Before you start. BenchPress converges. A click is a request, not a
guarantee, and several passes run behind it to make the database agree with
Docker. Read Deploy pipeline for the click
path first.

The five bench states

Bench Instance.status holds one of five values.

Two more fields carry state of their own.

lease_state is only written while credits are on. It is separate from status
because a bench can be Running and already claimed for a stop.

The transitions

benchpress/lifecycle.py owns every transition, and each one carries side
effects that must not be skipped.

Call these rather than setting status directly. A bench stopped by a direct
write leaves its sites active, and a bench deleted by a direct write leaks a
container, a tunnel address and a route.

torn_down records each removal that failed rather than aborting. A teardown
that gets halfway is worse than one that reports which half it finished.

Incidents

benchpress/docker_events.py consumes the Docker event stream instead of
polling it. It observes and never acts. No stop, no restart, no route write.
deploy_manager owns those.

The stream is filtered server side, on the BenchPress label and on three
actions. Without the filter it carries three exec_* events for each health
check on each bench, multiplied by the fleet.

Each one is written as a Bench Event.

Two incidents inside the settle window become one. The window is
bench_event_settle_seconds, 15 seconds by default. An OOM outranks the die
that follows it by milliseconds, and a death outranks a health verdict.

The precedence is fixed: bench_healthy, bench_unhealthy, bench_died,
oom_killed, lowest first. A site that stopped answering on its way out is one
incident, and the death is the half the owner has to be told about.

A health verdict also writes container_health on the bench, so that field is
fresh in seconds rather than after a poll interval.

The listener writes a heartbeat every second. After 60 ticks with no beat the
heartbeat is no longer believed, and the */5 reconcile pass is what keeps
convergence going. A listener that stays down degrades BenchPress to what it
was before the listener existed, not to silence.

The stats collector

benchpress/stats_collector.py runs on the */1 cron and samples CPU, memory
and health for each running bench.

It polls at most 50 benches in one pass, set by stats_poll_max_benches. The
call costs about two seconds on the Docker socket for each container, so the cap
is what keeps the job inside its window.

The collector also stops a bench it finds dead, which is the one thing on this
page that acts rather than records.

The scheduled jobs

Eleven jobs. Diagnostics
carries the full table with each schedule, read off a live host. This page says
what each one is for.

Every minute

The scheduler tick on this host is four minutes, so */1 fires every four. The
entry stays */1 so shortening the tick needs no edit.

Every five minutes

Six jobs. Each is a net under a faster path, not the primary path.

Three of these need queue-long. reconcile.enqueue_run and
docker_events.enqueue_reconcile need the Docker socket, and the route
convergence needs the Traefik mount. queue-short has neither.

drain.sweep_expired_leases makes no Docker call, so it is safe on either
worker. It is the net under the lease warden, not the primary path. The
scheduler tick is four minutes here, so no cron entry can promise better than
that. The warden claims within seconds, and both use the same conditional
claim, so running together cannot stop a bench twice.

enforce_limits is deliberately not on the */1 cron. That job spends about
two seconds for each container on the Docker socket, and a decision queued
behind Docker input and output arrives late.

Daily and weekly

What the reconcile pass does

benchpress/reconcile.py is the only pass that looks in both directions. Every
other sweep converges outwards from a row, so it cannot see a thing that has no
row.

A deploy looks exactly like an orphan while it runs. _deploy_bench creates
the container before it writes container_id, so every deploy passes through
that state. The 15 minute grace window from orphan_grace_minutes is what keeps
the pass from reaping a live deploy.

Container ids are compared on their first twelve characters. Docker reports the
full id and a row may hold either form, so a live bench is never called an
orphan over a difference in notation.

Deploy Logs are capped at 50 for each bench, on top of Frappe's own seven day
sweep. The cap is the second bound, for the bench redeployed in a loop that
reaches thousands of rows inside a week.

Verify

Read a bench's recent incidents:

bench --site <site> execute frappe.client.get_list \
  --kwargs "{'doctype':'Bench Event','fields':['bench','event_type','severity','occurred_at'],'limit_page_length':10}"

Confirm the jobs are not stopped:

bench --site <site> execute frappe.client.get_list \
  --kwargs "{'doctype':'Scheduled Job Type','fields':['method','stopped'],'limit_page_length':0}"

Troubleshooting

Reference

Related

• Deploy pipeline — the click path these jobs correct.
• Data model — the row an incident becomes.
• Diagnostics — the job table, and what diagnostics does not cover.
• Settings reference — every number named above.