# Observability tools This guide walks through Moat's observability commands for inspecting what happened during a run. You will learn how to view container logs, network traces, execution spans, and audit data -- and how to query raw data files directly. For architecture details on how this data is captured, see [Observability](/moat/concepts/observability). ## Prerequisites - A working Moat installation with Docker or Apple container runtime - At least one completed run (use `moat list` to check) ## Viewing logs `moat logs` displays container stdout and stderr with timestamps. View logs for the most recent run: ```bash $ moat logs [10:23:44.512] Starting server on port 3000 [10:23:44.789] Connected to database [10:23:45.123] Ready to accept requests ``` View logs for a specific run: ```bash $ moat logs run_a1b2c3d4e5f6 ``` Show the last N lines: ```bash $ moat logs -n 50 ``` Follow logs in real time with `--follow`: ```bash $ moat logs -f my-agent ``` See [CLI reference](/moat/reference/cli) for the complete list of `moat logs` flags. ## Network traces `moat trace --network` shows all HTTP and HTTPS requests that passed through the proxy during a run. View network requests for the most recent run: ```bash $ moat trace --network [10:23:44.512] GET https://api.github.com/user 200 (89ms) [10:23:45.001] POST https://api.anthropic.com/v1/messages 200 (1.2s) [10:23:46.234] GET https://api.github.com/repos/org/repo 200 (45ms) ``` Each line shows the timestamp, HTTP method, URL, response status code, and request duration. View network requests for a specific run: ```bash $ moat trace --network run_a1b2c3d4e5f6 ``` ### Verbose mode Add `-v` to include request headers, response headers, and response bodies: ```bash $ moat trace --network -v [10:23:44.512] GET https://api.github.com/user 200 (89ms) Request Headers: User-Agent: python-requests/2.28.0 Authorization: Bearer [REDACTED] Response Headers: Content-Type: application/json; charset=utf-8 X-RateLimit-Remaining: 4999 Response Body (truncated): {"login": "your-username", "id": 1234567, ...} ``` Injected credentials are redacted -- the actual token is replaced with `[REDACTED]`. ## Execution spans `moat trace` (without `--network`) displays execution spans showing the hierarchy and timing of operations within a run. ```bash $ moat trace 1. run/start (0s) 2. proxy/start (12ms) 3. container/create (234ms) 4. container/start (45ms) 5. container/wait (5.2s) Parent: run/start 6. container/stop (123ms) ``` Each span shows a sequence number, name, and duration. Spans with a parent are indented. Use this to identify which phase of a run took the longest. For example, a slow `container/create` span indicates image build or pull time, while a slow `container/wait` span reflects the agent's own execution. View spans for a specific run: ```bash $ moat trace run_a1b2c3d4e5f6 ``` ## Audit verification `moat audit` verifies the integrity of a run's tamper-proof audit log. The audit log records credential usage, network requests, console output, and other events in a cryptographically linked hash chain. Verify a run's audit log: ```bash $ moat audit run_a1b2c3d4e5f6 Auditing run: run_a1b2c3d4e5f6 =============================================================== Log Integrity [ok] Hash chain: 47 entries, no gaps, all hashes valid Local Signatures [ok] 1 attestations, all signatures valid External Attestations (Sigstore/Rekor) - No Rekor proofs found =============================================================== VERDICT: [ok] INTACT - No tampering detected ``` The verification checks hash chain integrity (each entry links to the previous), sequence continuity (no gaps), and Ed25519 signature validity. The audit log entries are stored in `audit.db` (SQLite). To inspect individual events, query the database directly or export a proof bundle with `--export` and examine the JSON. For details on the hash chain structure and trust model, see [Observability](/moat/concepts/observability). ## Exporting proof bundles Export a self-contained proof bundle for offline verification or sharing: ```bash $ moat audit run_a1b2c3d4e5f6 --export proof.json ``` The bundle contains: - All audit entries with their hashes - Signatures and attestations - Run metadata (run ID, timestamps) Verify an exported bundle on any machine -- the original run data is not required: ```bash $ moat audit verify proof.json Verifying bundle: proof.json =============================================================== VERDICT: [ok] INTACT - No tampering detected ``` This is useful for sharing audit evidence with reviewers or storing proof bundles in version control. ## Data locations and retention All observability data is stored per-run under `~/.moat/runs//`: | File | Contents | |------|----------| | `metadata.json` | Run metadata (name, state, timestamps) | | `logs.jsonl` | Container stdout/stderr | | `network.jsonl` | HTTP requests through proxy | | `traces.jsonl` | Execution spans | | `audit.db` | Tamper-proof audit log (SQLite) | When a container exits, Moat removes the container but retains these artifacts. ### Removing run artifacts ```bash # Remove a specific run's artifacts $ moat destroy run_a1b2c3d4e5f6 # Remove all stopped runs $ moat clean # Preview what would be removed $ moat clean --dry-run ``` Check disk usage with `moat status`: ```bash $ moat status Runtime: docker Active Runs: 0 Summary Stopped runs: 3 156 MB Images: 2 1200 MB Total disk: 1356 MB ``` ## Querying raw data The JSONL files are line-delimited JSON, so standard tools like `jq` work directly. ### Log queries Each log entry has two fields: `ts` (timestamp) and `line`. ```bash # Search for lines containing "error" $ jq 'select(.line | test("error"; "i"))' ~/.moat/runs/run_a1b2c3d4e5f6/logs.jsonl ``` ### Network queries Each network entry has `ts`, `method`, `url`, `status_code`, `duration_ms`, and header/body fields. ```bash # Count requests by status code $ jq -r '.status_code' ~/.moat/runs/run_a1b2c3d4e5f6/network.jsonl | sort | uniq -c # Find slow requests (over 1 second) $ jq 'select(.duration_ms > 1000) | {method, url, duration_ms}' \ ~/.moat/runs/run_a1b2c3d4e5f6/network.jsonl # List all hosts contacted $ jq -r '.url' ~/.moat/runs/run_a1b2c3d4e5f6/network.jsonl | \ sed 's|https\?://||' | cut -d/ -f1 | sort -u # Find all 401 responses across runs $ jq 'select(.status_code == 401)' ~/.moat/runs/run_*/network.jsonl ``` > **Note:** Request and response bodies in `network.jsonl` are captured up to 8 KB. Larger bodies are truncated. ## Troubleshooting ### No output from `moat logs` The run may not have produced any output. Verify the run exists with `moat list`. If the run is still active, use `moat logs -f` to follow live output: ```bash $ moat logs -f run_a1b2c3d4e5f6 ``` ### No network traces Network traces are only recorded for requests that pass through the proxy. If the agent makes requests that bypass the proxy (for example, to `localhost` inside the container), those requests are not captured. Verify the run used grants (which activate the proxy) by checking the audit log: ```bash $ moat audit run_a1b2c3d4e5f6 ``` ### Audit verification fails If `moat audit` reports a broken hash chain, the audit log file may have been modified. The output indicates which sequence number has a mismatch. For stronger guarantees, export proof bundles immediately after runs and store them in append-only storage. See [Observability](/moat/concepts/observability) for the trust model. ### Missing `traces.jsonl` Execution tracing may be disabled in the run's `moat.yaml`: ```yaml tracing: disable_exec: true ``` Remove or set this to `false` to re-enable execution tracing. Network request logging is separate and always enabled. ## Related pages - [Observability](/moat/concepts/observability) -- Architecture, data flow, hash chain structure, trust model, and attestation details - [CLI reference](/moat/reference/cli) -- Complete flag listings for `moat logs`, `moat trace`, and `moat audit` --- [← Moat documentation index](https://majorcontext.com/moat/llms.txt)