Methodology

Jev and every LLM get the same state, the same instructions and the same options or levels for every item. The LLM is asked to reply with a fixed JSON object. Jev returns typed answers by design.

Each item is one call. Every racer gets 4 parallel lanes. Each call is sent once, with no retries. Latency is timed from sending the request to receiving the full response. Replays in Beginner mode play at the recorded latency.

Every LLM runs at its provider's default settings. Claude Opus 5.5 cannot turn its thinking off, so it runs at low effort, and its thinking tokens are counted in its cost. The Claude models are claude-opus-5-5, claude-sonnet-5-5 and claude-haiku-4-5-20251001. Every Claude call has a 16,000-token output cap, and a reply cut off by the cap counts as a miss.

An answer that cannot be parsed counts as a miss and is shown with a "couldn't parse" note, never hidden.

Choice

Correct when the answer equals the correct option.

Noul

Jev's probability at or above 0.5 counts as yes. Correct when that matches the right answer.

Score

Jev is correct when it is within 0.5 of the correct level. An LLM is correct when its level equals the correct level.

Several questions in one request

Accuracy is the share of questions answered right.

Finding lines

The F1 of the lines found against the correct lines. Correct at 0.8 or more.

Text writing

Shown to you, not scored.

Cost is tokens times the stored price per million tokens. Jev's output tokens are free. A call that returns an error costs $0. A model with no stored price shows "price unknown" and is never estimated.

Developer mode uses the same runner and the same math, with your own keys. Jev, Anthropic, OpenAI and Google models use the table below, matched by the exact model ID, and so does Jev reached through an OpenRouter key; any other model from OpenRouter is priced from OpenRouter's published list; any other model shows "price unknown". With a TypeSafe key, Jev's time in Developer mode is the time our server measured for its call to TypeSafe, so the hop from your browser to our server is not counted. With an OpenRouter key the call goes straight from your browser, so its time is what your browser measured, and it includes OpenRouter's routing.

OpenAI and Google charge more for very long prompts (over 272,000 tokens at OpenAI, over 200,000 at Google). This site never sends a prompt that long, so the table holds only the standard price. A discount a provider gives for a repeated prompt (cached input) is not applied, so a cost can read slightly high, never low. A promotional price shows its last day; after that day the model shows "price unknown" until the table is checked again.

Items and their correct answers are written for this site and checked by hand before recording. Some items are written to show a weakness TypeSafe documents. The same items are never re-run to get a different result. A recording interrupted by a rate limit, overload or network failure is discarded and recorded again, because that measures the account, not the model. When content changes, it is recorded again and old recordings stop showing.

Level 8 (Trick Jev) was rewritten once. The first set of six tricks did not fool Jev at all: it answered all 12 messages correctly, as did Haiku 4.5 and Opus 5.5, and Sonnet 5.5 had two replies that did not parse. We then wrote new tricks aimed at documented weaknesses (wording that points at someone else, a planted instruction, a question about cancelling rather than a request) and recorded that set once. Only the second set is shown in the level.

The Number Crunch Showdown game was rewritten once too. Its first set of counting and arithmetic problems was easy for Jev (it answered all 12 correctly), so we wrote harder ones (longer words, multi-digit arithmetic) and recorded that set once. Only the second set is shown in the game.

Recordings are real outputs and are never edited.

We check that the site holds up when many people use it at once. A k6 load test signs in a test user and has simulated people each walk 11 pages (Home, Path, level 1, Games, a game, Arena, Sandbox, the start quiz, Leaderboard, Profile and this page), pausing 2 to 5 seconds on each one, as a reader would. Beginner mode replays come from files and Developer mode calls go from your browser straight to the provider, so serving pages is the server's main work.

It ran on 2026-10-06 against the production build on one Windows PC, which ran the server and all the simulated people at the same time. The target is no failed requests and pages under 2.0 s for 95% of loads.

What it shows: no request failed in any run. At 1,000 people at once, one server process on one PC runs out of processor time, so pages queue and slow down instead of failing. A deployment spreads people over several server instances, which this local test does not cover. The first 1,000-person run also found a real bottleneck: every page waited for one of only 10 database connections. We raised that to 40, which nearly tripled the pages served per second.

One Node process served up to about 99 pages a second on this PC. That is comfortably up to 400 people at once; beyond that pages queue.