Evidence at a glance
From Model Answers to Application Interfaces
Simon Willison released llm-echo 0.5 on October 18, 2026. It is a debugging plugin for the LLM command-line tool, providing an echo “model” that neither calls an external API nor runs a local LLM. Instead, it returns the content an application sent to the model. Its purpose is not to generate better answers, but to show what the application actually passed along.
The update came out of a specific development task: Willison was preparing a new LLM release and revisited the plugin while testing pull requests. Version 0.5 expands simulated usage reporting to account for system prompts, tool calls, and reasoning text. The change is about testing more of the model integration surface without involving a real model every time, not about improving answer quality.
Echo Makes the Input Inspectable
Echo returns information such as the prompt, system prompt, attachments, and conversation history as JSON. That shifts testing away from “Did the model produce a good answer?” toward more basic, diagnosable questions: Did the client send the intended content? Did the history and attachments make it into the request? Was the system prompt preserved? With no real model involved, output variation cannot obscure these interface problems.
The plugin also offers a way to test tool-call handling: tool_calls can be supplied in the input JSON so an application can exercise that flow without relying on a model to generate a call on demand. For teams changing client code or prompt assembly, fixed inputs help narrow regression tests to application behavior. This does not replace end-to-end checks against a real model, but it can prevent a wiring problem from being mistaken for model instability.
Usage Fields Enter the Test Surface
Another change in 0.5 is that simulated token usage now takes tool calls, system prompts, and reasoning text into account. The author’s example command is `uvx --with llm-echo llm -m echo 'hi there' -o thinking 1 --usage`. The `-o thinking 1` option produces two preset reasoning passages, after which the plugin echoes the input and reports usage: the example shows 2 input tokens and 24 output tokens.
The example demonstrates a test passing through prompt handling, reasoning output, and usage reporting rather than merely returning ordinary text unchanged. For application developers, reading and displaying usage correctly, and keeping the relevant code paths working when tool calls or system prompts are present, can be part of automated tests. Version 0.5 requires LLM 0.36 or later, so compatibility between the plugin and the command-line tool also belongs in the test environment.
Simulated Counts Are Not a Billing Forecast
The easiest part of this release to misread is the example’s numbers. The 2 input tokens and 24 output tokens are simulated figures, not a bill for a real model call. They do not establish that the plugin’s count matches any model’s tokenizer. The public notes provide neither a precise counting formula nor an error range against real usage.
So version 0.5 broadens what can be tested, not the precision of measurement. It can show that an application receives and processes a usage field under test conditions, but it cannot establish that production token accounting is accurate or estimate costs on its own. If the engineering goal is to validate budgets or billing, real model calls and their usage data are still needed.
Use It as a Regression-Test Double
For a technical lead, llm-echo is best understood as a test double for model interfaces. It can run in pull-request or automated tests to catch regressions in prompt assembly, tool-call handling, reasoning text, and usage fields. Because it neither contacts an external API nor runs a local model, these tests can isolate client integration problems from variation in model output.
A test double validates contracts and flows, not model behavior. It cannot tell you whether a real model will choose to call a tool, whether its reasoning output will meet expectations, or how a real service performs on latency, cost, and response quality. A practical approach is layered: use echo for fast, stable coverage of application interfaces, then use real models to test questions that only real models can answer. That gives llm-echo 0.5 a clear role: it reduces dependence on models in integration testing without making model testing optional.