See what happened inside one answer
An agent answers in steps: a model plans, hands work to helpers, the helpers call tools, and the answer comes back. Pick two recorded runs of the same question and see where the time and the tokens went, and which step went wrong.
- Recorded runs, not live
- No model, no key
- A trace you paste is not sent anywhere
The run behind these traces
One real run from the book AI Engineering, Measured. An orchestrator model hands questions to two worker models: one looks up stock and prices, one searches operations documents. The same six questions were asked many times while two settings changed.
- Worker step cap: how many model calls a worker may make before it must stop.
- What happens when a worker reaches its cap. Honest: the worker says it did not finish. Silent: its last words go back as if they were the answer.
Model recorded in the trace: claude-opus-4-6
The run was recorded in English: its questions and the models' words are shown as recorded.
1. Pick two runs
Try a comparison
The question askedWe are short on mice. What would it cost to bring them to 10 units, and what is the change process for ordering hardware?
The question askedWe are short on mice. What would it cost to bring them to 10 units, and what is the change process for ordering hardware?
Look at your own trace
Paste one JSON object per line. It is read inside this page: nothing is sent anywhere, and nothing is kept when you leave.
What a line can hold
A line needs span_id. It may also have parent_span_id, session_id, name, seconds or duration_ms, input_tokens, output_tokens, stop_reason, truncated, error, tool_calls and text. The book's handoff and final records are read too. Lines are grouped by session_id.
One line from the recorded run, as an example:
{"run_id": "2026-09-05T21:46:23+00:00", "kind": "span", "span_id": "s0001", "parent_span_id": null, "session_id": "2026-09-05T21:46:23+00:00|cap2|honest|r1|Q1", "prompt_id": 1, "handler": "honest", "worker_max_steps": 2, "rep": 1, "loop": "orchestrator", "step": 0, "input_tokens": 684, "usage_input_tokens": 684, "usage_output_tokens": 62, "n_messages": 1, "tools_available": ["ask_inventory", "ask_docs"], "tool_calls": [{"name": "ask_inventory", "input": {"task": "How many headsets are currently in stock?"}}], "stop_reason": "tool_use", "text": "", "truncated": false, "seconds": 2.513}2. What changed between the two runs
Every number is counted from the two traces on this page.
| Measure | Run A | Run B | B minus A |
|---|---|---|---|
| Model calls | 9 | 7 | -2 |
| Model time, summed (seconds) | 39.222 | 29.779 | -9.443 |
| Tokens read | 8,810 | 6,854 | -1,956 |
| Tokens written | 1,415 | 1,268 | -147 |
| Calls cut off by the step cap | 2 | 0 | -2 |
| Worker answers reported missing | 2 | 0 | -2 |
| Unfinished answers passed on as finished | 0 | 0 | 0 |
Where the model time went
Each bar is one run, split by the loop that made the calls. The bars share one scale.
Orchestrator
ARun A: 16.814 s, 3 callsBRun B: 11.198 s, 2 callsinventory worker
ARun A: 11.414 s, 4 callsBRun B: 7.35 s, 3 callsdocs worker
ARun A: 10.994 s, 2 callsBRun B: 11.231 s, 2 calls
3. Step by step
One line per model call, in the order they happened. Indented calls were made by a worker for the call above. Bars share one scale across both runs.
Run AQ4 · cap 2 · honest · repeat 1
Orchestrator, call 1Asked for ask_inventory, ask_docs
Time3.644 sTokens705 read, 145 writtenWhat happened, and how it is measured
- What the model wrote
- Let me look into both of those questions for you simultaneously.
- Tools it asked for
- ask_inventory {"task":"How many mice are currently in stock, what is the price per unit, and how much would it cost to bring the stock level up to 10 units?"}
- ask_docs {"task":"What is the change process or procedure for ordering hardware?"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 23 of the trace
inventory worker, call 1Asked for lookup_stock
Time2.363 sTokens723 read, 66 writtenWhat happened, and how it is measured
- What the model wrote
- Let me look up the stock information for mice first.
- Tools it asked for
- lookup_stock {"query":"mouse"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 24 of the trace
inventory worker, call 2Cut off by the capAsked for calculate
Time3.175 sTokens834 read, 110 writtenWhat happened, and how it is measured
- What the model wrote
- Now I know the current stock is **0 units** and the price is **$19.50** per unit. Since we need to bring the stock level up to 10, that means purchasing 10 units. Let me calculate the cost.
- Tools it asked for
- calculate {"expression":"10 * 19.5"}
- Why it stopped
- This was the last call the cap allowed. The model was still asking for a tool, so the work stopped here.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.Cut off: the loop writes this into the record when it stops a worker at its cap. It is recorded when it happens, not guessed later.
Line 25 of the trace
Back to the orchestrator: reported as not finished
- What the worker passed back
- Nothing.
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
Line 26 of the trace
docs worker, call 1Asked for search_docs, search_docs
Time2.574 sTokens620 read, 112 writtenWhat happened, and how it is measured
- What the model wrote
- Let me search for information about the change process for ordering hardware.
- Tools it asked for
- search_docs {"question":"change process or procedure for ordering hardware"}
- search_docs {"question":"hardware ordering procedure"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 27 of the trace
docs worker, call 2Answered
Time8.42 sTokens1,571 read, 332 writtenWhat happened, and how it is measured
- What the model wrote
- The search results contain information about **change management windows** and **hardware replacement procedures**, but they do not contain any specific procedure or process for **ordering hardware** (e.g., procurement, purchase orders, vendor ordering, or supply chain processes). Here is what the documents **do** cover that may be tangentially relevant: 1. **Change window classifications** for hardware-related changes (like line card replacements), which are based on device tier: > *"Tier 1 devices are core routers and core switches. Tier 2 devices are edge routers, aggregation switches, and route reflectors. Tier 3 devices are access switches, out-of-band console servers, and lab equipment. Changes to Tier 1 devices require a four-hour approved window, submitted no later than ten business days before the change date. The window must be approved by the change advisory board and by the on-call network lead for the region. Changes to Tier 2 devices require a two-hour window submitted five business days ahead. Tier 3 changes may be made during the standing weekly window with no separate submission."* 2. **Hardware replacement procedures** (e.g., MOP-021 — Line Card Replacement) and **chassis inventory states** (REF-050), which deal with the operational side of swapping hardware. However, **the documents do not contain a procedure specifically for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
- Why it stopped
- It finished and gave its answer.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 28 of the trace
Back to the orchestrator: an answer
- What the worker passed back
- The search results contain information about **change management windows** and **hardware replacement procedures**, but they do not contain any specific procedure or process for **ordering hardware** (e.g., procurement, purchase orders, vendor ordering, or supply chain processes). Here is what the documents **do** cover that may be tangentially relevant: 1. **Change window classifications** for hardware-related changes (like line card replacements), which are based on device tier: > *"Tier 1 devices are core routers and core switches. Tier 2 devices are edge routers, aggregation switches, and route reflectors. Tier 3 devices are access switches, out-of-band console servers, and lab equipment. Changes to Tier 1 devices require a four-hour approved window, submitted no later than ten business days before the change date. The window must be approved by the change advisory board and by the on-call network lead for the region. Changes to Tier 2 devices require a two-hour window submitted five business days ahead. Tier 3 changes may be made during the standing weekly window with no separate submission."* 2. **Hardware replacement procedures** (e.g., MOP-021 — Line Card Replacement) and **chassis inventory states** (REF-050), which deal with the operational side of swapping hardware. However, **the documents do not contain a procedure specifically for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
Line 29 of the trace
Orchestrator, call 2Asked for ask_inventory
Time3.425 sTokens1,312 read, 102 writtenWhat happened, and how it is measured
- What the model wrote
- Let me retry the inventory question.
- Tools it asked for
- ask_inventory {"task":"What is the current stock level of mice and what is the price per unit for a mouse? I need to know how much it would cost to purchase enough mice to bring the total stock to 10 units."}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 30 of the trace
inventory worker, call 1Asked for lookup_stock
Time3.036 sTokens733 read, 67 writtenWhat happened, and how it is measured
- What the model wrote
- Let me start by looking up the stock information for mice.
- Tools it asked for
- lookup_stock {"query":"mouse"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 31 of the trace
inventory worker, call 2Cut off by the capAsked for calculate
Time2.84 sTokens845 read, 125 writtenWhat happened, and how it is measured
- What the model wrote
- Here are the current details for the mouse (SKU: MS-02): - **Current Stock Level:** 0 units - **Price Per Unit:** $19.50 To bring the stock to **10 units**, you'd need to purchase **10 mice**. Let me calculate the total cost:
- Tools it asked for
- calculate {"expression":"10 * 19.5"}
- Why it stopped
- This was the last call the cap allowed. The model was still asking for a tool, so the work stopped here.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.Cut off: the loop writes this into the record when it stops a worker at its cap. It is recorded when it happens, not guessed later.
Line 32 of the trace
Back to the orchestrator: reported as not finished
- What the worker passed back
- Nothing.
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
Line 33 of the trace
Orchestrator, call 3Answered
Time9.745 sTokens1,467 read, 356 writtenWhat happened, and how it is measured
- What the model wrote
- Here's what I was able to find: --- ### 🖱️ Mice Inventory & Cost Unfortunately, the inventory system wasn't able to return the current stock level or pricing for mice at this time. You may need to check the inventory system directly or contact the procurement team for current stock counts and unit pricing. --- ### 📋 Change Process for Ordering Hardware The documentation doesn't outline a specific **procurement/ordering** procedure for hardware. However, it does cover the **change management process** that applies when hardware changes are involved. Here's a summary: | Device Tier | Examples | Change Window | Lead Time | Approval | |---|---|---|---|---| | **Tier 1** | Core routers & core switches | 4-hour approved window | 10 business days | Change Advisory Board + on-call network lead | | **Tier 2** | Edge routers, aggregation switches, route reflectors | 2-hour window | 5 business days | Standard submission | | **Tier 3** | Access switches, console servers, lab equipment | Standing weekly window | No separate submission required | — | Mice would likely fall under **Tier 3 (or general lab/access equipment)**, meaning changes can be made during the standing weekly window with no separate submission needed. > **Note:** For the actual **purchasing/procurement process** (e.g., purchase orders, vendor ordering), you may need to consult a separate procurement or supply chain policy, as it wasn't covered in the network operations documentation. Would you like me to try looking up anything else?
- Why it stopped
- It finished and gave its answer.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 34 of the trace
Run BQ4 · cap 5 · honest · repeat 1
Orchestrator, call 1Asked for ask_inventory, ask_docs
Time3.844 sTokens705 read, 145 writtenWhat happened, and how it is measured
- What the model wrote
- Let me look into both of those questions for you simultaneously.
- Tools it asked for
- ask_inventory {"task":"How many mice are currently in stock, what is the price per unit, and how much would it cost to bring the stock level up to 10 units?"}
- ask_docs {"task":"What is the change process or procedure for ordering hardware?"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 324 of the trace
inventory worker, call 1Asked for lookup_stock
Time2.485 sTokens723 read, 67 writtenWhat happened, and how it is measured
- What the model wrote
- Let me start by looking up the stock information for mice.
- Tools it asked for
- lookup_stock {"query":"mouse"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 325 of the trace
inventory worker, call 2Asked for calculate
Time2.246 sTokens835 read, 76 writtenWhat happened, and how it is measured
- What the model wrote
- Now let me calculate the cost to bring the stock level up to 10 units.
- Tools it asked for
- calculate {"expression":"10 * 19.5"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 326 of the trace
inventory worker, call 3Answered
Time2.619 sTokens926 read, 76 writtenWhat happened, and how it is measured
- What the model wrote
- Here's the full breakdown: - **Product:** Mouse (SKU: MS-02) - **Current stock:** 0 units - **Price per unit:** $19.50 - **Units needed to reach 10:** 10 - **Total cost to restock to 10 units:** **$195.00**
- Why it stopped
- It finished and gave its answer.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 327 of the trace
Back to the orchestrator: an answer
- What the worker passed back
- Here's the full breakdown: - **Product:** Mouse (SKU: MS-02) - **Current stock:** 0 units - **Price per unit:** $19.50 - **Units needed to reach 10:** 10 - **Total cost to restock to 10 units:** **$195.00**
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
Line 328 of the trace
docs worker, call 1Asked for search_docs, search_docs
Time2.802 sTokens620 read, 116 writtenWhat happened, and how it is measured
- What the model wrote
- Let me search the documentation for information on both the change process and hardware ordering procedures.
- Tools it asked for
- search_docs {"question":"change process or procedure for ordering hardware"}
- search_docs {"question":"hardware ordering procedure"}
- Why it stopped
- It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 329 of the trace
docs worker, call 2Answered
Time8.429 sTokens1,575 read, 411 writtenWhat happened, and how it is measured
- What the model wrote
- Based on the search results, the documents describe a **change management process** related to hardware work (such as line card replacements), but they **do not contain a specific procedure for ordering hardware** (e.g., procurement, purchase orders, vendor selection, etc.). Here is what the documents **do** cover regarding the **change process** for hardware-related work: ### Change Window Classes (by Device Tier) > *"Window class is determined by device tier, not by the size of the change."* - **Tier 1 devices** (core routers and core switches): Require a **four-hour approved window**, submitted **no later than ten business days** before the change date. The window must be **approved by the change advisory board and by the on-call network lead** for the region. - **Tier 2 devices** (edge routers, aggregation switches, and route reflectors): Require a **two-hour window** submitted **five business days ahead**. - **Tier 3 devices** (access switches, out-of-band console servers, and lab equipment): Changes may be made during the **standing weekly window with no separate submission**. ### Emergency Changes > *"Emergency changes are exempt from the notice periods above. An emergency change requires verbal approval from the on-call network lead and a written record filed within twenty-four hours."* ### Hardware-Specific Notes > *"The NX-4200 does not support hot insertion on slot 0. Slot 0 requires a chassis reload, which converts this into a Tier 2 four-hour change."* --- **In summary:** The documents cover the change management process for performing hardware changes in the network, but they **do not contain a procedure for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
- Why it stopped
- It finished and gave its answer.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 330 of the trace
Back to the orchestrator: an answer
- What the worker passed back
- Based on the search results, the documents describe a **change management process** related to hardware work (such as line card replacements), but they **do not contain a specific procedure for ordering hardware** (e.g., procurement, purchase orders, vendor selection, etc.). Here is what the documents **do** cover regarding the **change process** for hardware-related work: ### Change Window Classes (by Device Tier) > *"Window class is determined by device tier, not by the size of the change."* - **Tier 1 devices** (core routers and core switches): Require a **four-hour approved window**, submitted **no later than ten business days** before the change date. The window must be **approved by the change advisory board and by the on-call network lead** for the region. - **Tier 2 devices** (edge routers, aggregation switches, and route reflectors): Require a **two-hour window** submitted **five business days ahead**. - **Tier 3 devices** (access switches, out-of-band console servers, and lab equipment): Changes may be made during the **standing weekly window with no separate submission**. ### Emergency Changes > *"Emergency changes are exempt from the notice periods above. An emergency change requires verbal approval from the on-call network lead and a written record filed within twenty-four hours."* ### Hardware-Specific Notes > *"The NX-4200 does not support hot insertion on slot 0. Slot 0 requires a chassis reload, which converts this into a Tier 2 four-hour change."* --- **In summary:** The documents cover the change management process for performing hardware changes in the network, but they **do not contain a procedure for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
Line 331 of the trace
Orchestrator, call 2Answered
Time7.354 sTokens1,470 read, 377 writtenWhat happened, and how it is measured
- What the model wrote
- Here's a combined summary for you: --- ## Restocking Mice | Detail | Value | |---|---| | **Current Stock** | 0 units | | **Price per Unit** | $19.50 | | **Units Needed** | 10 | | **Total Cost** | **$195.00** | You're completely out of stock, so you'd need to order all 10 units at a total cost of **$195.00**. --- ## Change Process for Hardware The network operations documents cover the **change management process** for performing hardware-related changes (not procurement/ordering specifically). Here's how it works: - **Tier 1 devices** (core routers & switches): Require a **4-hour approved window**, submitted **10 business days** in advance, approved by the **Change Advisory Board** and the **on-call network lead**. - **Tier 2 devices** (edge routers, aggregation switches, route reflectors): Require a **2-hour window**, submitted **5 business days** ahead. - **Tier 3 devices** (access switches, console servers, lab gear): Can be done during the **standing weekly window** with no separate submission. - **Emergency changes**: Exempt from notice periods but require **verbal approval** from the on-call network lead and a **written record within 24 hours**. > ⚠️ **Note:** The documents don't include a specific *procurement/ordering* procedure for hardware. You may need to consult a separate procurement or supply chain policy for that step. --- Let me know if you'd like to dig deeper into either topic!
- Why it stopped
- It finished and gave its answer.
- How it is measured
- Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.
Line 332 of the trace
4. What went wrong, and how to fix it
Each finding is worked out from the trace on this page.
Run A
A worker ran out of steps
- What happened
- The inventory worker used its 2 calls and stopped before it finished. It reported that it did not finish.
- Why it matters
- The answer for that part of the question is missing. Because the worker said so, the orchestrator knows, and can retry or say what is missing.
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
- How to improve it
- Compare with a run at a higher cap on the same question: if the worker finishes there, the cap is too low for this kind of question.
A worker ran out of steps
- What happened
- The inventory worker used its 2 calls and stopped before it finished. It reported that it did not finish.
- Why it matters
- The answer for that part of the question is missing. Because the worker said so, the orchestrator knows, and can retry or say what is missing.
- How it is measured
- The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
- How to improve it
- Compare with a run at a higher cap on the same question: if the worker finishes there, the cap is too low for this kind of question.
The orchestrator asked the same worker again
- What happened
- It asked the inventory worker 2 times. The later attempts cost 2 more model calls.
- Why it matters
- A retry can rescue an answer, or repeat the same failure at extra cost and time.
- How it is measured
- Counted from the handoffs: more than one handoff from the same worker in one session.
- How to improve it
- If the retry fails the same way, retrying is not the fix. Look at why the first attempt stopped.
The link holds your text, the settings and the result, so anyone who opens it can run the same check again. It is made in your browser: nothing is sent. Anyone you give it to can read your text.
Run B
Nothing in this trace was cut off, missing or retried.
The link holds your text, the settings and the result, so anyone who opens it can run the same check again. It is made in your browser: nothing is sent. Anyone you give it to can read your text.
The whole recorded run
All 48 sessions, grouped by their settings. Counted on this page from the same file.
| Worker cap | When the cap is reached | Sessions | Model calls | Calls cut off | Answers reported missing | Unfinished answers passed on |
|---|---|---|---|---|---|---|
| 2 | honest | 18 | 103 | 10 | 10 | 0 |
| 2 | silent | 18 | 98 | 11 | 0 | 11 |
| 5 | honest | 6 | 33 | 0 | 0 | 0 |
| 5 | silent | 6 | 32 | 0 | 0 | 0 |
| All sessions | 48 | 266 | 21 | 10 | 11 | |
The run's own script printed these totals when it finished: 266 model calls, 21 cut off, 83 handoffs, 10 with an empty answer. This page counts 266, 21, 83 and 10 from the same file.
These counts describe this one run: six questions, repeated three times at cap 2 and once at cap 5. They are not a rate for any model.
Not in these traces
These were not recorded, so this page shows no value for them.
How long the person waited
UNMEASURED
Only model calls were timed. Tool runs and the gaps between calls were not, so the summed model time is not the waiting time.
What a run cost
UNMEASURED
Token counts are recorded. A cost needs a price table checked against the provider's current page, and none has been checked here.
What the document search returned
UNMEASURED
The trace records the search request, not the passages that came back, so this page cannot show which passages the docs worker read.
Where this data comes from
- File
- runs/ch9-caps-2026-09-05.jsonl
- Repository
- sanovinci/dev-to-ai-engineer (private)
- Commit
- eb44af88398717da4a6433ba032c59c9d85519aa
- Script that recorded it
- scripts/ch9_run.py
- sha256
- b83f2757662ee768e02dfb9e8fe4bdaa7206220fe1f432c799a2dce99f50001d
The file is copied byte for byte, and the build stops if its sha256 differs from the one shown here.