Project page Developer tool

Ryan

A browser-based debugger for LLM inference: it runs one conversation through two runtimes, lines up what each recorded and reports where the evidence first differs.

At a glance

Ryan is a browser app that runs one short chat conversation through two LLM execution stacks, records a trace of each run, lines the traces up stage by stage and reports the earliest point where the recorded data differs, with what it could not conclude because a side did not record it. The default pair is WebLLM against Transformers.js, each running a SmolLM2-360M-Instruct build with greedy settings. Ryan reports divergence as evidence and attributes no cause to a runtime.

Question: On the same prompt, where do two inference stacks first differ: template, prompt, tokens, settings or output?
Method: One normalized trace per run, compared on 13 aspects and shown as 9 stages. A missing observation is never read as agreement.
Runs: WebLLM and Transformers.js in the tab; Ollama through an adapter and a local bridge.
Reports: The earliest observable divergence, its evidence, and what cannot be concluded.
Limits: No benchmark, ranking or root cause. Real model runs are not covered by the automated tests.

The problem

Two stacks that are supposed to do the same thing give different output for the same prompt. Before blaming a runtime you have to inspect the path from prompt to first different token. Each step of it can differ:

What can differ, and what each browser runtime exposes
Can differ	Ryan records	What the runtimes expose
Chat template	Template source and dialect	Transformers.js, when the tokenizer has one. WebLLM: not observable.
Rendered prompt	The exact string given to the tokenizer	Transformers.js only. WebLLM does not expose it, and Ryan does not reconstruct it.
Tokenization	Token ids, text, special flags, tokenizer metadata	Transformers.js only. WebLLM: not observable.
Artifact and lineage	Format, quantization, declared upstream checkpoint	Both declare it; neither records a content hash.
Generation settings	What each runtime applied, the request kept apart	Transformers.js 4.2.0 has no top-p and no seed; WebLLM has no top-k.
Greedy decoding	The recorded settings	Transformers.js turns sampling off at temperature 0. WebLLM has no greedy branch: it clamps temperature to 1e-6 and still samples.
Generated tokens	Per step: id, text, probability, top candidates	Transformers.js: ids, probabilities, top 8. WebLLM: text, no integer ids; probabilities and top 5 only above temperature 0.

Why “run both and diff the text” is not enough. A text diff cannot say where on the path two outputs came apart. A stack that does not expose a stage leaves nothing to diff, and “nothing recorded” must not be read as “the same”. So Ryan records each stage on its own, says whether it was observed, and compares only what both sides observed.

How traces are aligned

A run produces one trace, ryan.trace/v1: model, artifact and lineage; messages, chat template, rendered prompt and input tokens; the applied settings; one record per generated token; runtime and environment; and a capabilities table. In the schema, every field a producer may not have recorded is optional and is left absent, not filled with a default. The comparison covers 13 aspects. Each trace marks each aspect observed, not_observable or incomparable, and the comparison shows them as 9 stages in causal order: model, lineage, chat template, rendered prompt, tokenization, generation config, decode timeline, first divergence, runtime.

Aligned means every aspect is compared on both traces under a rule for its kind of data. A guardrail comes first: unless both traces observed an aspect, its result is unknown, with a sentence saying which side did not record it, even when both fields are absent or equal. A stage is differ if any aspect differs, else unknown if any is unknown, else agree.

How each kind of data is compared
Data	Rule
Rendered prompt	Strict equality; if it fails, a character-level longest-common-subsequence diff with the first differing UTF-16 code unit and a flag for whitespace or invisible-character differences. Above 500,000 character pairs it diffs by line, and above 100,000 line pairs it keeps only the common prefix and suffix.
Input tokens	The aspect: deep equality of the token records and the tokenizer metadata. The alignment: longest common subsequence over ids, or position by position above 2,000,000 cells (below).
Generated tokens	Position by position: step i of A against step i of B, no re-synchronization.
Candidates and scores	Candidates: position by position over the depth both sides recorded. Probabilities and logits: equal within 1e-6 × (1 + the larger absolute value), where both sides recorded them.
Everything else	Deep equality with object keys sorted; array order matters. For greedy runs temperature, top-k and do_sample are dropped from the settings first.

Input tokens are aligned by id with a longest-common-subsequence table, so one inserted token does not turn every later token into a mismatch. Equal runs become match rows; an A-only run followed directly by a B-only run is paired position by position into mismatch rows, and the surplus becomes only A or only B rows. A row’s position is its number in the alignment, not an index into either prompt. When the token counts multiplied exceed 2,000,000 the table would be too large, so Ryan aligns position by position instead, by the same id rule.

Unknown ids. A runtime that does not expose token ids records -1 rather than inventing one. Input tokens are compared by id alone, and -1 counts as an id like any other; alignment runs only when both sides observed tokenization, otherwise the stage is unknown with no rows. Generated tokens use a different rule (below).

How the first divergence is determined

The 9 stages run in causal order, so the earlier ones show whether the two runs were already apart before generation began. The first divergence stage itself reads only the generated tokens and the order of the top candidates. Ryan keeps three separate indices, all zero-based generation steps, and says which one it means:

First emitted-token divergence. Step i of A is compared with step i of B. A step is a mismatch if one side has no step there, if both sides have a real token id and the ids differ, or, when either side has no id, if the decoded text differs. Equal ids with different text are a mismatch; the special flag is not read. The first step that is not a match is the divergence, and the number of leading matches is the agreeing prefix.
First candidate-ordering divergence. Where both sides recorded top candidates, the lists are compared position by position over the depth they share: a step differs if, at a shared position, the ids differ (where both are real) or the texts differ. A step whose shared part agrees but whose depths differ is unknown, never agreement. Ordering can drift before the emitted text does, and that alone does not mean the output will differ.
First score difference. If the emitted token agrees and no difference in candidate ordering was observed, but a probability or logit that both sides recorded differs beyond the tolerance, the step is marked “scores differ”. The outputs agree there. That is not a divergence and does not change the first-divergence stage, although the Decode timeline stage shows it.

Two gates come first. If the generated tokens were not observed on both sides, or one side has no steps field, the result is unknown, with no divergence and no agreeing prefix, and the finding names the side. A trace without steps recorded nothing about what was generated, so that is not reported as a divergence at step 0; a trace with an empty list recorded that nothing was. The result is differ if the emitted tokens or the candidate ordering differ, unknown if the ordering could not be compared, else agree. So in the default pair, where WebLLM records no candidates at greedy settings, the same decoded text at every step gives unknown, no divergence, and an agreeing prefix as long as the run.

Ryan reports the step, a pane for each side and an explanation whose lines are tagged observed, concluded, cannot conclude or upstream. Steps are not re-aligned after an insertion, so one inserted token can turn every later step into a mismatch: read the rows after the first as consequences.

Why evidence and not a cause. The earliest observable divergence is the earliest difference in the recorded data, which can be later than the first real difference if something was not captured. A different token can come from the template, the tokenizer, the quantization, the settings or the runtime, and the sides can differ in several at once. So the report lists what could not be concluded, and unless the pair is exact (same artifact hash, rendered prompt, input token ids and settings) it says Ryan cannot attribute a divergence to a runtime cause. Ryan rates the default pair behavioral-only: same upstream checkpoint, declared and not hash-verified.

Why it runs in the browser

Both browser runtimes execute in the tab that has the page open: WebLLM on the main thread and Transformers.js in a module Web Worker. Ryan’s own code has no application server, database or analytics, and uploads no trace. A search of the source for network calls finds one use of the global fetch in Ryan’s own code, the client for the local bridge. The libraries do the downloading; the Transformers.js adapter only wraps the library’s download function so that Cancel can stop it.

Both stacks run on the same machine in the same tab, so a comparison needs no second host. That does not make them use the same backend: WebLLM requires WebGPU, while Transformers.js uses WebGPU when it can get an adapter and WebAssembly otherwise. Each trace records its runtime and, where exposed, its environment, and the runtime stage compares them.

What does leave the browser: the app’s own files; model files from Hugging Face hosts, which the browser then caches; for WebLLM, a compiled model library from GitHub’s raw-content host; ONNX Runtime files from a public CDN on the first Transformers.js run; and, while the hosting zone’s automatic setup is on, a Cloudflare Web Analytics script that the platform adds to the Ryan app’s pages, not to this site. The model library, ONNX Runtime and analytics requests are not made by Ryan’s own code.

A share link keeps the comparison in the URL fragment, which browsers do not send to a server, but the link contains the trace itself and Ryan has no redaction. A link longer than 8,000 characters is refused, and the app offers a file export instead. This page makes no claim about speed, hardware or repeatability: Ryan does not rank models or runtimes, and I have no measurement of how repeatable live browser runs are.

Runtimes and how each adapter observes a run

The three runtimes and what each observes
Detail	WebLLM	Transformers.js	Ollama
Library	@mlc-ai/web-llm 0.2.85, on the page’s main thread	@huggingface/transformers 4.2.0, with ONNX Runtime Web, in a module Web Worker	An adapter in the page and a local bridge process
Observes	Streamed text, token usage, timing; probabilities and top 5 only above temperature 0	Rendered prompt, template when exposed, input token ids; per step the id, text, probability and top 8	Streamed output, one step per chunk
Not observable	Template, rendered prompt, tokenization, integer token ids	Template if the tokenizer exposes none; probabilities and candidates unless captured at every step	Token ids, probabilities, candidates, rendered prompt

Ollama is reached through a local bridge: an adapter in the page and a separate Node process that listens on the loopback interface only, checks the page’s origin and a session token, and talks to Ollama. The page contacts it only when its address carries a pairing fragment. I say no more than that: I did not verify an end-to-end run against a live Ollama daemon.

Architecture diagram

The schema depends on nothing else in Ryan; the normalizer, the compare package and the fixtures each depend only on it. The compare package has no UI dependency: the UI renders its report and explanation and does not re-derive findings.

Modules and data flow of Ryan. A coloured edge groups boxes by kind. On a narrow screen the picture scrolls sideways.

The diagram as text

In the browser tab.
- Runtime adapters: each runtime returns a trace and a table of what it observed; the Ollama adapter reaches Ollama through a separate local bridge process, outside the tab.
  - Arrow to Trace schema: each run becomes a trace.
- Normalizer: detects an imported format by its structure and adapts it.
  - Arrow to Trace schema: adapted traces are validated.
- Share link codec: a comparison in the URL fragment; links over 8,000 characters are refused.
  - Arrow to Trace schema: a link opened on the compare page is validated.
- Trace schema: the contract; it depends on nothing else in Ryan.
  - Arrow to Compare package: two traces in this schema are compared.
- Compare package: plain TypeScript with no UI dependency.
  - Arrow to App UI: the report and explanation, shown by the UI.
- App UI: the React app: runs, imports, share links and the views.
Delivery.
- Static hosting: static assets; no application server.
  - Arrow to App UI: delivers the app’s files.

How it is tested

As counted on 30 October 2026, the live code has 45 Vitest files and 18 Playwright spec files. These are file counts, not cases run, and this page quotes no pass counts, coverage figures or CI results. CI runs a lockfile check, the type check, the unit tests, the build and the Playwright suite in Chromium only.

Trace packages: Node unit tests of the schema validator, the import adapters and the comparison engine: validity levels, the id-then-text rule, unknown steps never counting as agreement, the score tolerance, token alignment on both sides of the 2,000,000-cell limit, third-party claims kept out of Ryan’s conclusions, and that explanations avoid a set of blame phrases.
Adapters: Unit tests with fake workers and fake libraries pin ordering and state-machine properties: exclusive run slots, cancel, late messages from an earlier run. Contract tests pull the real validation and sampling code out of the installed library versions, so an upgrade that changes behaviour fails there. Neither runs a model.
Fixtures: Two hand-built demo documents with the expected divergence steps as exported constants (13 and 15, and 0), and one trace recorded in a real browser, from which tests derive variants.
End to end: Playwright in Chromium drives the real React screens, router and comparison pipeline with stand-in adapters, without model downloads or a GPU: routing, demos, share and export, errors, cancel and remount, layout in two fonts, keyboard use.
Reintroduced bugs: For several fixes made after review, the bug was reintroduced by hand to confirm that the new tests fail. The record is outside the repository, and no mutation-testing tool is used.

What is not covered

No automated test runs a real model: the end-to-end tests use stand-in adapters, and real in-browser execution is not verified in CI.
Real runs of both engines were done in a real desktop browser with WebGPU in audits in October 2026, outside the automated tests. Safari and iOS were not validated, and neither was a run against a live Ollama daemon.
The two bundled examples are hand-built demo documents, not recordings of live runs.
Accessibility evidence is a small Playwright file on keyboard use and page structure and an audit of an earlier build (September 2026); no conformance is claimed.