Xiaobai
Developer · Builder
Building AI engineering systems, developer tools and long-term digital assets at XBSTACK.
About Xiaobai & XBSTACK →
LangGraph First Checkpoint Crash: Why an Accepted Run Can Disappear
Reproduce a LangGraph first-checkpoint crash that leaves zero checkpoints, raises EmptyInputError on resume, and can hide an already accepted background run.
LangGraph First Checkpoint Crash: Why an Accepted Run Can Disappear
Your API accepts a background job and returns a job_id. The worker enters LangGraph, but the process dies before the first checkpoint reaches durable storage. After restart, you try to resume the original thread_id with invoke(None, config) and get:
EmptyInputError: Received no input for __start__
The difficult part is not the exception itself. If the application owns no separate admission record, the job can have no success record, no failure record, and no durable proof that it was ever accepted.
That exposes a boundary that exists before ordinary checkpoint recovery: did the business system turn admission into a durable fact before graph persistence began?
XBSTACK independently reproduced the failure shape from upstream LangGraph Issue #8764. On LangGraph 1.2.11 with langgraph-checkpoint-sqlite 3.1.1, we killed the child process immediately before the first SqliteSaver.put(). The baseline ended with zero durable checkpoints and zero user effects. A fresh process using the same thread_id raised EmptyInputError on invoke(None, ...).
We then ran the same crash with an application-owned acceptance ledger written before graph execution. LangGraph still had zero checkpoints after the crash, but the application could see that the job was accepted, recover its original payload, classify the state as recovery_required, and explicitly replay it.
This article answers one question: if a LangGraph run dies before its first durable checkpoint, how do you prevent an already accepted background job from disappearing from operational truth?
First, Separate Three Different Failure Shapes
This problem is easy to confuse with two other LangGraph recovery failures.
The first is streamed output visible in the UI but missing after cancellation. In the earlier LangGraph state-lost-after-cancel experiment, XBSTACK showed that UI stream delivery can lead authoritative graph state. If the node never returned, visible partial output may never have become a checkpointed state update.
The second is a checkpoint exists, but recovery reruns code or duplicates a side effect. That belongs to idempotency, pending writes, node replay, and external-system reconciliation. For that boundary, see LangGraph error recovery, retry, and timeout patterns; if the question is specifically about saver choice and persistence behavior, use the LangGraph Checkpointer comparison.
This article covers an earlier boundary: the first checkpoint has not been persisted at all when the process dies. There is no prior graph state for the recovery process to read.
Upstream Issue #8764 describes exactly this admission-visibility gap for background or fire-and-forget invocation shapes: a caller can consider the run accepted, yet a crash before the first durable checkpoint can leave no checkpoint and no durable failure marker. The issue is still open at the time of this article, so the containment pattern below is not presented as an upstream fix.
Test Environment and Success Criteria
The fixture intentionally avoids model latency, provider streaming, and network variance. It makes no LLM or external API calls.
| Item | Tested environment |
|---|---|
| Python | 3.10.2 |
| LangGraph | 1.2.11 |
| langgraph-checkpoint-sqlite | 3.1.1 |
| Checkpointer | SqliteSaver |
| durability | sync |
| Crash injection | SIGKILL before first SqliteSaver.put() |
| External model/API calls | 0 |
| Comparison | no acceptance ledger / application acceptance ledger |
PyPI lists LangGraph 1.2.11 as the current stable release at publication time. The local fixture therefore tests a current version rather than a historical pre-1.0 edge case.
The success criteria are operational, not merely “does the script exit cleanly?”:
- Can the system prove the job was accepted?
- Can it determine whether the graph has a durable checkpoint?
- Can it decide whether replay is appropriate without process-memory state?
- After replay, does the controlled fixture produce exactly one user effect and reach
completed?
Experiment 1: Kill the Process Before the First SqliteSaver.put()
The fixture subclasses SqliteSaver and pins the crash at the first persistence call:
class CrashBeforeFirstPut(SqliteSaver):
def put(self, *args, **kwargs):
global seen_first_put
if not seen_first_put:
seen_first_put = True
os.kill(os.getpid(), signal.SIGKILL)
return super().put(*args, **kwargs)
The graph contains one minimal user node. If that node executes, it appends one effect, giving us a concrete side-effect counter.
def node(state):
with open(effect_file, "a", encoding="utf-8") as f:
f.write("effect\n")
return {"done": True}
The initial invocation is ordinary:
app.invoke(
{"done": False},
{"configurable": {"thread_id": "accepted-run"}},
durability="sync",
)
The important variable is not sync; it is that the process is killed before the first persistence operation completes.
Baseline Result: Zero Checkpoints, Zero Effects, No invoke(None) Resume
After SIGKILL, a fresh process opens the same SQLite checkpoint database and tries:
app.invoke(None, config, durability="sync")
Observed baseline:
| Metric | Result |
|---|---|
| Child exit code | -9 |
| Durable checkpoints before recovery | 0 |
| User effects | 0 |
| Fresh-process resume | EmptyInputError |
| Error text | Received no input for __start__ |
The thread has no historical state from the checkpointer’s perspective. None means “continue from prior state,” but there is neither prior state nor new START input.
This is different from an incomplete checkpoint. There is no first checkpoint at all.
Why durability="sync" Does Not Remove This Window
LangGraph persistence and durability semantics define when formed state updates are persisted. sync is useful because completed state changes are persisted before the next execution step proceeds.
But persistence still has a physical operation boundary. A process can be killed, a container can be terminated, or a host can disappear before the first write actually completes. This experiment deliberately crashes at the entrance to the first SqliteSaver.put() to isolate that earliest window.
Production systems therefore need to distinguish two facts:
- Business accepted: your API told the caller that the job was accepted.
- Graph durable: LangGraph has at least one recoverable persisted state.
If business acceptance happens first and there is no independent admission record, the interval between those facts is an accepted-but-not-yet-durable window.
Experiment 2: Persist an Application Acceptance Ledger Before LangGraph
The second arm leaves LangGraph’s checkpoint semantics unchanged. It adds one application-layer step before graph execution: persist a minimal job record.
The fixture uses a separate SQLite table:
create table jobs (
job_id text primary key,
status text not null,
payload text not null
)
Before entering the graph it writes:
job_id = accepted-run
status = accepted
payload = {"done": false}
The child is then killed at the same first SqliteSaver.put() boundary.
After restart, LangGraph still has zero checkpoints, and invoke(None, ...) still raises EmptyInputError. That is the point: the acceptance ledger does not repair LangGraph or fabricate a checkpoint.
It only gives the application four durable facts:
job exists
status == accepted
checkpoint_count == 0
original payload exists
Recovery can therefore move the job from accepted to recovery_required and explicitly start graph execution again with the stored input.
Controlled Result: From Invisible Loss to Detectable, Replayable Work
The two arms produced:
| Case | Checkpoints after crash | invoke(None) | Ledger | Replay | Checkpoints after replay | Effects | Final ledger |
|---|---|---|---|---|---|---|---|
| baseline | 0 | EmptyInputError | none | no | 0 | 0 | - |
| acceptance ledger | 0 | EmptyInputError | accepted | yes | 3 | 1 | completed |
The containment arm ended with three durable checkpoints, one user effect, and a completed ledger state.
Do not generalize that result into “a jobs table prevents lost work.” In a real system, replay safety depends on whether an external effect already happened, whether it can be queried, whether a stable idempotency key exists, and whether attempts are reconcilable.
In this fixture the user effect occurs after the first checkpoint boundary, so the injected pre-checkpoint crash produces zero effects and the replay produces exactly one. That controlled order is not guaranteed in every application.
Design Durable Admission as an Explicit Lifecycle
If the LangGraph call is synchronous and the caller waits until the graph has established durable state, the failure is easier to observe. Background jobs, queues, async APIs, fire-and-forget execution, and worker pools need a more explicit admission lifecycle.
A minimal happy path can be:
accepted
↓
starting
↓
checkpointed
↓
running
↓
completed
Recovery branches should include at least:
accepted + zero checkpoint -> recovery_required
checkpointed + interrupted -> resume/reconcile
side_effect_unknown -> manual_reconcile
replay_failed -> failed
Minimum Acceptance-Ledger Fields
A production record should usually include:
job_id: application job identity;thread_id: LangGraph thread identity;status: accepted / recovery_required / completed / failed;- minimal replayable input rather than the entire sensitive conversation;
idempotency_keyfor external writes;- execution
attempt; accepted_at,started_at, andcompleted_attimestamps;- external object IDs or reconciliation references.
Sensitive payloads should be encrypted, minimized, or replaced with references to data that can be reloaded under the application’s authorization boundary.
Do Not Auto-Replay Every Zero-Checkpoint Job
This is the most important safety boundary in the entire pattern.
The wrong policy is:
if checkpoint_count == 0:
replay()
A safer decision looks more like:
if job.status == "accepted" and checkpoint_count == 0:
if side_effect_status_is_known_safe(job.idempotency_key):
mark_recovery_required(job.id)
replay(job.payload)
else:
mark_manual_reconcile(job.id)
Some applications can produce an external effect before the checkpoint that proves local completion. The remote system may commit the write while the local acknowledgement is lost. Blind replay can turn a lost-job problem into duplicate charges, duplicate emails, or duplicate orders.
The acceptance ledger solves visibility and decision evidence. Idempotency and reconciliation solve replay safety.
What This Experiment Does Not Prove
The tested boundary is deliberately narrow:
- no
PostgresSaver, Redis, or remote checkpointer; - no LangGraph Platform / Agent Server durable-execution conformance test;
- no Kubernetes graceful-shutdown path, only uncatchable
SIGKILL; - no “external effect happened before checkpoint failed” duplicate-effect case;
- no concurrent worker fencing on the same
job_id; - no claim that upstream Issue #8764 is fixed.
The supported conclusion is therefore specific: on LangGraph 1.2.11 with local SqliteSaver, a controlled crash before the first durable checkpoint can leave a reproducible zero-checkpoint window; an application-owned acceptance ledger can convert an invisible accepted run into work that is detectable, classifiable, and explicitly replayable when replay is safe.
When I Would Require an Acceptance Ledger
I would treat durable admission as required rather than optional when at least two of these are true:
- the API returns 202 / accepted before background execution is durable;
- users poll results minutes or hours later;
- execution crosses processes, containers, or workers;
- external side effects cannot be repeated safely;
- the job has billing, audit, or SLA requirements;
- “the job vanished with no failure record” is unacceptable.
A short, synchronous, locally retryable tool with no external effect may not need a separate ledger.
Reproduction Assets and Upstream Status
XBSTACK published the minimal reproduction and acceptance-ledger comparison:
- XBSTACK LangGraph first-checkpoint acceptance-ledger repro
- LangGraph upstream Issue #8764
- LangGraph persistence documentation
- LangGraph on PyPI
A successful local run emits:
LANGGRAPH_FIRST_CHECKPOINT_REPRO_PASS
Machine-readable evidence is stored in results/verification.json and results/checks.json in the public reproduction repository.
Conclusion
The most useful result is not the EmptyInputError. It is the distinction between two meanings of “accepted.”
Your API saying accepted does not mean LangGraph is already durable.
If a background run dies before its first checkpoint, a system with no separate admission record may be unable to resume and unable to decide whether the job is failed, lost, or never started. An application-owned acceptance ledger fills that visibility gap: persist what the business accepted, let LangGraph persist graph state, and reconcile the two during recovery instead of treating the checkpoint store as the only proof that a job existed.
For production agent systems, the reliable chain is: durable admission → checkpointed execution → idempotent effects → reconciliation. No single layer replaces the others.
Continue through the production LangGraph learning path
The LangGraph hub organizes state isolation, checkpointing, human approval, retries, observability, supervisors, subgraphs and memory into one reviewable path.
More to Explore
Topic hub →AI Engineering Weekly
Production changes, real failures, experiments and new XBSTACK assets.
DISCUSSION
Questions, verification and corrections
Sign in to comment. Every new comment is reviewed before publication; while pending, it is visible only to you and the administrator.