Recover Streaming LLMs

Most applications need no streaming-recovery configuration. The Runtime retries eligible failures before output begins and continues a retryable interruption after partial output. Add a StreamCheckpointStore only when an in-flight model turn must also survive process replacement.

Keep the two durable records separate

flowchart LR
    P[Provider stream] --> A[In-flight accumulator]
    A --> C[Stream checkpoint]
    A --> M[Complete assistant message]
    M --> T[Thread commit]
    T --> D[Delete checkpoint]
    C -. partial text, tools, retry count .-> A

The Thread commit remains the authority for messages and RunState. A stream checkpoint contains only the unfinished turn: Run and Thread ids, model, partial text, partial Tool calls, and retry count. It is not another conversation store.

Choose a checkpoint backend

use std::sync::Arc;
use awaken_agent_contract::store::stream_checkpoint::StreamCheckpointStore;
use awaken_store_fs::FsStreamCheckpointStore;

let checkpoints: Arc<dyn StreamCheckpointStore> =
    Arc::new(FsStreamCheckpointStore::open("/var/lib/awaken/stream-checkpoints")?);

let context = context.with_stream_checkpoint(checkpoints);

Use the in-memory implementation for deterministic tests. Use the filesystem implementation only when one process owns the directory. It writes through a temporary file, sync, and rename. For several processes, provide one shared implementation of the same contract; do not synchronize a second checkpoint format beside it.

get, put, and delete return Result<_, StreamCheckpointError>. The backend must report storage and fencing failures. The Runtime decides at the call site to log the failure and continue best-effort, so telemetry remains visible without turning a checkpoint outage into a second Run lifecycle authority.

What happens after an interruption

Request-only continuation messages are never committed. A Tool call is executed only when its accumulated arguments parse as complete JSON. After a normal return, the completed assistant message is committed through the usual Thread boundary and checkpoint deletion is attempted.

sequenceDiagram
    participant R as Runtime
    participant S as Checkpoint store
    participant L as Model provider
    participant T as Thread commit
    R->>S: get(run_id)
    S-->>R: saved partial or none
    R->>L: request or continuation request
    L-->>R: text and Tool deltas
    R->>S: put(partial, retry_count)
    alt retryable interruption
        R->>L: R1, R2, R3, or R4 recovery
    else complete response
        R->>T: commit complete assistant turn
        R->>S: delete(run_id)
    end

Provider retry hints are honored only when the adapter supplies them and remain capped at 60 seconds. See Errors for the retryable classification; do not infer it from display text in application code.

Confirm cross-process recovery

If you attached a durable checkpoint store, interrupt one representative Run after partial output, replace the process, and read the committed Thread. The setup is working when the replacement produces one complete assistant turn and does not execute an incomplete Tool call. Detailed fault-injection coverage belongs in the store and Runtime tests; see the testing strategy.

No external repair is needed for a single warning: the Runtime continues or finishes the turn. If persistent checkpoint warnings violate your restart requirement, correct the backend’s reachability, permissions, or fencing before claiming cross-process recovery. A completed Thread commit remains valid even if checkpoint cleanup warned.

Boundaries

• This mechanism does not retry permanent provider errors.
• It does not execute an incomplete or malformed Tool call.
• It does not promise byte-for-byte continuation; the provider generates the continuation from the saved prefix.
• It does not replace committed Thread history or Tool-effect recovery. See Tool recovery for external effects.