OpenClaw agent execution history, separated into three layers to determine which log to open first for each incident

The first problem you encounter after deploying an agent pipeline to production is this: something went wrong, but you have no idea which reasoning path called which tool, or why. You dig through the logs and find infrastructure events in abundance, yet the agent's internals feel like a black box.

OpenClaw records agent execution history across three distinct layers. A JSONL session transcript that serializes the entire reasoning chain; a ShieldEvent audit log that links every tool call in a hash chain; and a Gateway audit ledger that stores only metadata with no personal information. Understanding when to open each layer lets you handle incident post-mortems, reproduction attempts, and compliance audits all within a single observability framework.

As of 2026, built-in OpenTelemetry support has been added on top of this, making it possible to feed agent traces directly into your existing Prometheus and Grafana infrastructure. In this article we will examine the structure of each layer, how to combine them for specific scenarios, and the pitfalls that are easy to miss.

Role Separation of the Three Layers

Layer 1: JSONL Session Transcript

The session transcript is a JSONL file stored under ~/.openclaw/workspace/[job-name]/logs/. Because each line is an independent JSON object, it supports line-by-line streaming with jq or standard parsers.

A single line looks like this:

{
  "timestamp": "2026-09-13T04:21:07.312Z",
  "role": "assistant",
  "content": "I will check the file list and then run the build script.",
  "tool_call_name": "bash",
  "tool_call_arguments": { "command": "ls -la ./dist" },
  "tool_result": "total 48\ndrwxr-xr-x ...",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7",
  "parentSpanId": "00f067aa0ba902b6",
  "traceFlags": "01"
}

Starting with the 2026 release, lines that have a valid diagnostic trace context include traceId, spanId, parentSpanId, and traceFlags as top-level keys. This makes it possible to join local file logs with OTEL spans using a single traceId, letting you view distributed traces and local debugging in the same context. Note, however, that keys are attached only to lines where a trace context was generated — keep this in mind when writing join queries.

The debug level can be controlled in order: error → warn → info (default) → debug → trace. The common pattern is to leave it at info in production and raise it to debug or trace only when an issue occurs to capture detailed reasoning paths.

Layer 2: ShieldEvent Hash Chain

ShieldEvent records every tool call as a versioned record containing the tool name, session ID, policy decision (allow/deny), and a hash chain link to the previous record. Several lifecycle hooks — before_tool_call, tool_result_persist, before_model_resolve, before_prompt_build, and others — are registered automatically; refer to the official documentation's hook reference for the full list.

The hash chain structure makes it possible to detect whether an intermediate record has been tampered with. When a compliance audit requires you to prove "was this policy decision really made at this point in time," this tamper-evident chain is the key evidence.

The session transcript write and the ShieldEvent hash chain link fire sequentially after tool execution. The key point of the diagram is that these two paths do not branch in parallel — they proceed in order within a single lifecycle.

Layer 3: Gateway Audit Ledger

At the Gateway level there is a third layer. It is a ledger that records only agent execution metadata (identity, timing, tool name, normalized outcome) in a shared state DB — it is closer to an upstream view that aggregates the traces ShieldEvent leaves per session at an organizational level. The prompt body and raw tool arguments are not stored.

This design felt like a limitation at first, but it turns out to be an advantage in practice: it allows aggregated auditing while complying with privacy regulations. However, because the retention period and row limit for ledger records depend on environment configuration, organizations that require long-term auditing should check the defaults in the Gateway config-observability documentation at deployment time and separately set up a pipeline that periodically exports data to external storage.

⚠️ Secret masking is best-effort. All three layers apply automatic masking before writing to disk, but this is best-effort processing based on known patterns. Masking is not guaranteed for binary payloads, custom encodings, or non-standard identifier formats. When exporting logs to external systems or attaching them to tickets, it is strongly recommended to add an additional scrubbing pipeline on top.

Usage by Scenario

Scenario 1: Post-Mortem Analysis of a Production Incident

When you receive an alert that the agent called an unexpected tool, reconstruct the sequence of policy decisions for that session using the session_id in the ShieldEvent logs, then join the session transcript using the same session_id to trace the model's reasoning path.

# Extract deny events for a specific session from ShieldEvent logs
jq 'select(.session_id == "sess_abc123" and .policy_decision == "deny")' \
  ~/.openclaw/workspace/my-agent/logs/shield_events.jsonl
 
# Inspect the reasoning chain at that point in the session transcript
jq 'select(.timestamp >= "2026-09-13T04:00:00Z" and .role == "assistant")' \
  ~/.openclaw/agents/main/sessions/sess_abc123.jsonl

# Query distributed traces and local logs together using traceId
TRACE_ID="4bf92f3577b34da6a3ce929d0e0e4736"
jq --arg tid "$TRACE_ID" 'select(.traceId == $tid)' \
  ~/.openclaw/agents/main/sessions/sess_abc123.jsonl

Scenario 2: Reproduction Workflow to Restore State Just Before an Issue

One thing to clarify upfront: the /session save|load command that the community has long requested (official issue #13700) is not yet an official feature. A full snapshot capability that checkpoints context token count, model, and compaction state together is on the roadmap, and once it lands, A/B testing with a different model will also become seamless.

So what can you do right now? There is a workaround using the session transcript JSONL file itself. You load the ~/.openclaw/agents/main/sessions/<sessionId>.jsonl file directly and restart the conversation from just before the issue occurred. This is closer to a "message-history-based restart" than a full context restore, so sessions where a compaction point is involved may not reproduce 100%.

If a session is corrupted or in an inconsistent state, you can inspect and recover it using the openclaw doctor family of diagnostic subcommands (for the specific subcommands, check the CLI --help output for your deployed version — this is a conceptual example).

Scenario 3: Connecting a Grafana Dashboard via OpenTelemetry

To enable the built-in OpenTelemetry support available since the 2026 release, add a diagnostics.otel configuration entry. It follows the OpenTelemetry GenAI Semantic Conventions and supports event keys such as gen_ai.input.messages and gen_ai.output.messages (for the stability status of other content keys, refer to the stability notation on the spec page).

// openclaw.json (conceptual configuration example)
{
  "diagnostics": {
    "otel": {
      "enabled": true,
      "endpoint": "http://localhost:4318",
      "protocol": "http/protobuf",
      "content_keys": ["gen_ai.input.messages", "gen_ai.output.messages"]
    }
  }
}

Data exported by the diagnostics-otel plugin can be visualized immediately in the Grafana Labs public dashboard (ID: 25067). You can monitor token counts, latency histograms, active conversation gauges, and error counters in real time.

Warning: Enabling all content keys on a high-volume pipeline can cause trace storage costs to spike due to token and payload size. A practical approach is to start with only metadata keys enabled and selectively add the content keys you actually need.

Scenario 4: Setting Up a Compliance Audit Stream

Exporting the Gateway audit ledger to an external SIEM or separate stream allows you to aggregate tool execution patterns. The design of storing only metadata without message bodies provides a structure that simultaneously satisfies audit requirements while complying with privacy regulations such as GDPR.

# Export the Gateway audit ledger (conceptual example — check the deployed version for actual CLI)
openclaw audit export \
  --format jsonl \
  --since 2026-08-01 \
  --output ./audit_export_2026_08.jsonl
 
# Aggregate tool execution patterns with jq
jq -r '.tool_name' audit_export_2026_08.jsonl | sort | uniq -c | sort -rn

Trade-offs and Common Pitfalls

There is one easy-to-miss bug. A known case exists (issue #74640) where the agent:main:main pointer flips when openclaw.json is rewritten, causing session history to be lost. It is good practice to back up the session transcript before modifying the configuration file.

Hook API changes also need to be checked. In the mid-2026 release cycle, the before_model_resolve and before_prompt_build hooks were added and before_agent_start was deprecated (confirm exact release numbers in the official changelog). The latest line of observability plugins targets the new hooks, so existing plugins may need to be upgraded or migrated.

Which Layer to Open First When an Incident Occurs

Once you understand that the three layers cover different information scopes, the fastest approach in a real situation is to decide the order based on "what do I want to know." The decision flow looks something like this:

The indefinite accumulation issue with session transcripts, the best-effort limitation of secret masking, and the history-loss bug when openclaw.json is rewritten are all points where you can easily stumble during an actual incident response — building awareness of these into your pipeline design phase is worthwhile. Until /session save|load becomes an official feature, maintain your reproduction workflow via the workaround of backing up and loading transcript files, and approach OTEL by selectively opening content keys while watching the cost curve — this is the most realistic starting point.

References

• Logging · OpenClaw Official Documentation
• Gateway Logging · OpenClaw Official Documentation
• Audit Records · OpenClaw Official Documentation
• Session Management Deep Dive · OpenClaw Official Documentation
• Agent Harness Sessions and Results · OpenClaw Official Documentation
• Session State Awareness · OpenClaw Official Documentation
• Configuration — audit, logging, diagnostics, and telemetry · OpenClaw
• OpenTelemetry GenAI Semantic Conventions
• OpenClaw Tool Call Audit Log: How to Capture Every Agent Action – Zedly AI Blog
• OpenClaw Immutable Audit Log: Build a Tamper-Evident Event Chain – Zedly AI Blog
• OpenClaw Logging and Debugging: Troubleshoot Agent Behavior | SFAI Labs
• Openclaw Audit Logging: Compliance and Monitoring Configuration | SFAI Labs
• OpenClaw Logging Best Practices: Keep It Clear – ClawGo
• Instrument Your OpenClaw Agent with OpenTelemetry – LangWatch
• Tracing OpenClaw with OpenTelemetry and Orq.ai
• OpenClaw OTel Observability Dashboard – Grafana Labs
• OpenClaw Observability Plugin (henrikrexed) – GitHub
• GitHub Issue #25373: orphan transcript .jsonl files accumulate
• GitHub PR #84708: fix(agents): recover message-tool mirror replay poison
• GitHub Issue #13700: Session snapshots — save and load context checkpoints