15 probes·
6 from contributors·
7 groups·
every probe is one file, reviewed before it merges
A probe asks a model something a serving stack cannot easily fake, and records the
answer as a plain value. Two endpoints that return the same values are running the same thing.
Every probe is one small file. Anyone can write one, and an accepted probe is credited here and
in the repository.
tokenizer
How many tokens a fixed text costs. Different labs train different tokenizers, and the count is very hard to fake.
english pangram
tok-english
prompt_tokens for a fixed 212-char text
prompt_tokens for a fixed English text; tokenizers are unique per lab
chinese paragraph
tok-chinese
CJK segmentation differs most between labs
prompt_tokens for fixed CJK text; segmentation differs most between labs
code snippet
tok-code
indentation + symbol handling
prompt_tokens for fixed source code; indentation and symbol handling vary
emoji + rare unicode
tok-emoji
byte-fallback behaviour
prompt_tokens for emoji and rare codepoints; byte-fallback behaviour differs
template offset
template-offset
hidden system-template tokens
hidden serving-template tokens, from two prompts of known size difference
errors
The exact wording and shape of validation errors. This prose is written by the lab's own engineers, not by a router.
temperature: 2.0
err-temperature
edge value passes gateways; a lab with a lower ceiling answers in its own words
send temperature 2.0; gateways accept it and forward, labs with a lower ceiling answer with their OWN validation prose (the AllThingsIntel trick)
max_tokens: 10^9
err-maxtokens
the refusal names the lab's real output limit
ask for a billion output tokens; the refusal names the lab's real limit, and that number is vendor DNA (a gateway forwards this, unlike -1)
error code family
err-code-family
numeric codes are vendor DNA (GLM's 1301 gave Ox away)
the SHAPE of error responses: numeric code families vs string codes vs typed objects
shape
The vocabulary a model uses to say why it stopped.
finish vocabulary
finish-vocab
finish_reason values the endpoint emits
which finish_reason value a forced cut-off produces; vocabularies differ per lab
network
What the routing layer and the response headers reveal about who served the call.
the routing layer names its region and the provider that answered
which edge region and which provider actually served each call, from the router's own opt-in metadata snapshot
the router's own ledger: provider_name + data_region + native tokens
OpenRouter's /generation endpoint names the serving provider, its data region and the NATIVE token counts for one call
header presence/format is stack DNA (Bedrock, Vertex, Cloudflare…)
which response headers exist and in what format — serving stacks expose different header families, and CORS hides the rest
timing
Where the real upstream provider sits, measured through refusal paths rather than through a GPU.
names the upstream provider and the payload it refuses; RTT lives in raw
which upstream provider answers a lane and which payload it refuses, plus a GPU-free router->provider RTT measurement in raw
capability
Hard ceilings such as the largest prompt a host will accept.
64k, 128k, 200k and 256k+ window classes help date the variant
bisect the accepted prompt-size bucket; window classes help distinguish model variants without probing all the way to 1M
logits
The shape of the probability numbers, when a host exposes them.
δ_norm ≈ 0.32 is universal; the residual spread is a personal signature
the normalized 3rd top-logprob gap on pinned continuations. EVT predicts δ≈0.318 for Gumbel-class models; deviations and gap vectors are model-specific (fallrisk.ai protocol)