Jev: Not Frontier, But Still Worth Your Attention

The pitch, and the parts that survive contact

“A model that cannot hallucinate, at frontier-level performance, built by the co-inventor of ChatGPT - incredibly fast, incredibly cheap, with free output.”

Each clause is technically defensible in a narrow sense and misleading in the sense a buyer will hear. Take them one at a time.

Cannot hallucinate. What Jev actually cannot do is return free text: it is a fixed-output API that answers by selecting among caller-supplied options, so there is no prose to hallucinate in. But a contract-valid answer is not a true answer. Jev is frequently wrong (see our benchmarks), and every one of those wrong answers arrives beautifully formatted. A model that cannot write prose has not solved hallucination; it has made hallucination hard to notice.

Fast and cheap. Both true, with the leading explanation being prefill-dominated scoring without autoregressive decoding.¹ Speed and price are properties of that shape of task, not of frontier economics.

Frontier-level. This one simply does not survive. Jev is good, and “good” will turn out to mean something genuinely useful here - but it is not a frontier model by any late-2026 standard, and where it looks frontier-like (97.9% on ARC-Challenge), that's only on a race that finished years ago. Everything below is the evidence.

The co-inventor of ChatGPT. Diogo Almeida certainly deserves credit, but saying “I co-invented ChatGPT” overassigns it. Almeida was one of the eight primary authors on InstructGPT, worked on learned optimizers, and was also one of 88 people thanked in the initial release of ChatGPT. But ChatGPT was born from the work of thousands of people; see the footnote.²

What we did, and what we could not do

We ran Jev live against public benchmarks through its API: 12,032 MMLU-Pro questions (82.7% accuracy), 1,172 ARC-Challenge, 196 GPQA Diamond, 494 Humanity's Last Exam multiple-choice items, the encodable MATH-500 subsets (261 as option MCQ, 273 under a per-digit rubric), and the ARC-AGI-2 public set (167 grids, re-encoded per-cell and as 120 whole-task requests) - 16,379 planned requests in three frozen suites. This repository publishes the derived aggregates and the complete Talk-to-Jev traces. The benchmark item text itself is third-party licensed content and is not republished here. The analysis results are generated from these artifacts.³

Three constraints shape everything below. First, Jev answers directly: one shot, no chain of thought, no tools, no retries - which is not how the frontier scores we compare against are produced with modern models, although it is how older models in the benchmarks functioned, and is automatically factored into the Pareto frontier comparisons. Second, the API is closed: we never saw weights, gradients, or anything internal to its design, so the architecture discussion is inference from observable behavior.³ Third, we chose benchmarks the model could answer natively - selecting between visible options or grading fixed rubrics - because converting them into something else would measure our conversion, not the model.

What Jev appears to be

What follows is our operating premise: every other section reads Jev's behavior through the model stated here. The full ledger - alternative hypotheses with probabilities and a falsifier per load-bearing claim - lives in ARCHITECTURE-ANALYSIS.md.

See ARCHITECTURE-ANALYSIS.md §1 for details.

What the milliseconds say

In our September timing campaign, responses reported an upstream-service timing header (x-envoy-upstream-service-time), the proxy-reported upstream service time. Our October follow-up responses no longer supplied that header, so those tests use client elapsed time instead. The September sequential probes across prompt sizes from ~0.5k to ~29k tokens fit:

The more telling experiment is self-batching. Packing 192 questions into one request barely moves the reported service time (+0.44 ms per extra question), while the reported output grows about 33 tokens per question - each one's full probability vector, serialized. Those marginals are consistent with the input tokens each question (~55) and each option (~18) adds to the shared prompt. Reading those tokens at the prefill slope is itself worth +0.34 ms - the residual 0.1 ms is effectively noise. Scoring 255 options instead of 2 costs +0.10 ms of service time and 9.6 reported output tokens per option, against the +0.11 ms from its own ~18 tokens. In both cases the measured increments are consistent with pure input-token scaling. This is what prefill-dominated scoring without autoregressive decoding would look like from outside, and that is the leading explanation we run with.

The battery re-confirmed this live: across a grid of option counts (2-255) crossed with option lengths (2-64 tokens), nothing scaled with the number of options beyond the tokens they add (-0.8 ms per 100 options, inside the ±20 ms residual noise). Billed output grows ~33 tokens per question and ~9.6 per option - that is the response JSON serializing itself, which is why output can be priced at zero under the service's billing policy. Upstream service time stays flat (78 ms at concurrency 1, 82 ms at 32) while our own wall time bends, consistent with efficient shared serving. A four-question batch answers in ~261 ms where the same four questions separately take ~1,117 ms: no per-question round trip appears on the critical path in these measurements. “System one”, under this reading, is as much a serving description as a psychological one - prefill, read out, done.

Running up to 32 requests concurrently keeps the proxy-reported upstream service time flat (78 ms at c=1 vs 82 ms at c=32) while client wall time bends - that bend is our own connection pool, not the server.

In short, what moves the measured upstream service time:

• Prompt tokens - the only thing that scales it, and linearly.
• Extra questions in a request - only through the tokens they add.
• Extra options - only through the tokens they add.
• Concurrent requests - flat in the header (consistent with efficient batched or elastic serving across replicas or spare capacity). Only our own wall time bends.
• Repeats - every call re-reads the prompt. Nothing is cached away.

Fingerprints on the read-out

Every probability Jev returns is rounded to two decimals: across 704,277 values in 7,887 published vectors, from 2-option questions up to 255-option menus, not one value ever landed off that 0.01 grid.⁴ A probability of exactly 0.00 is common where there is high confidence about at least one answer (otherwise, a 0.01 floor is standard). What that costs you is resolution: two options at 0.0001 and 0.0049 are indistinguishable, so any conclusion resting on a difference below 0.01 is reading noise.

The rounding has more structure than “rounded”. Sums of all probabilities returned are either 0.99 or 1.00, and never exceed 1.00 - in any corpus, at any option count up to 255. Independent per-value rounding would scatter sums by ±0.04 at 255 options. So a bounded correction runs after rounding, and it only ever takes mass away: 46.4% of flat 255-option vectors land exactly 0.01 short, while every sharp vector and every two-option vector sums to exactly 1.000.⁵

Two more seams show. In 9 vectors the returned choice is not the argmax of its own displayed table. Each such gap is exactly one quantum (e.g., 0.01), which monotone rounding cannot produce - consistent with a higher-precision decision combined with separate display processing. And the choice confidence field is roughly recoverable. It is, to within a quantum or two, the chance-corrected top probability: (p_max − 1/K) / (1 − 1/K), where p_max is the highest probability in the vector and 1/K is what guessing would earn across K options. Of 2,289 published choice vectors, 1,405 match that formula exactly against the displayed table, 822 are within one quantum, and 62 within two. None is further off.⁶ TypeSafe's public Python code calculates confidence using this formula. Buyer's note: choice confidence is an approximation from the vector you were already handed (not a guaranteed deterministic function), not a second opinion. Score answers use a different shape statistic and noul answers carry no confidence at all.

The follow-up battery (3,331 live calls) put the read-out on a degenerate case: identical option texts, repeated. Across 64 such vectors at option counts 2 to 255, sums were exactly 1.000 at every size up to 12 and 0.99-1.00 at 255 - the bounded-correction reading, confirmed live. Identical texts did not get identical probabilities: they were scored by position, not content, which is the purest form of the ordering effect described below.

To test whether dummy filler options dilute the answer, we re-ran 200 gold-labeled synthetic questions at every option count from 2 to 255, padding with inert filler. The probability on the right answer did not move - 99.2% at 255 options against 100.0% at two - and accuracy stayed perfect at every size, with the filler options pinned at 0.00. Those filler options did not distract Jev from easy factual answers. Our independent tests below show that simply reordering the real choices can change the answer.

The order of the options matters

Jev reads the option list as a list, in context, and position is a first-order factor wherever content is weak. On a flat creative task with 254 candidate continuations, eleven different orderings of the identical option set produced ten different winners, each internally stable across six repeats; rank agreement with the native order fell to Spearman 0.26-0.44, against 0.42-0.88 on a sharp factual step. The serial-position curve is U-shaped: the first decile of positions carries ~4× the mean probability of the middle deciles, and the last decile is elevated too. Mean p_max for the same options ranged 0.26-0.54 depending purely on their order. These effects are 3-10× the measured repeat-noise band (TVD 0.03-0.12), so they are real.⁷

Two consequences for anyone using this API. First, rotate: our own measurement protocol ensembles K≥6 cyclic rotations and averages, which cancels the bias (Spearman 0.91-1.00 against the 12-rotation reference) - at the cost of flattening the aggregate, so per-answer temperature must drop to compensate. Second, do not read a menu's ordering as neutral. Where content is strong the bias washes out: re-running MMLU-Pro items with shuffled option order flips 5.3% of paired answers with no net accuracy change, and the ancestry probe below cancels position by construction. Where content is weak - a routing menu of near-synonyms, a flat rubric - order will move your answer.

A tokenizer nobody recognizes

Probing tells the full story, including after discounting false leads. Jev's counter merges English text and punctuation hard, spends ~1 token per codepoint on Cyrillic, Greek, Arabic, Hebrew, Thai, Devanagari, Hangul, Kana and common CJK, and ~1 per digit. Uncovered characters fall back below codepoint granularity: the rare blocks cost 0.99-1.01 tokens per UTF-8 byte (3-4 tokens per character), which is byte-level fallback. A lone surrogate escape is rejected outright with invalid Unicode text, so the pipeline is UTF-8, not UTF-16. Decomposed and precomposed accented text cost the same token count, so Unicode normalization runs before tokenization. Whitespace behaves by character class: ASCII space runs collapse to nearly nothing, tabs and newlines to ~10-15% of their length, while NBSP, ZWJ, ZWNJ, soft hyphen and BOM each cost a full token - the normalizer's definition of whitespace is ASCII-only.

Early attempts to let Jev talk so it could describe itself led to Jev identifying itself with an OpenAI-family name 337 times out of 360, but this looks like a learned assistant-style prior - not a signature of authorship.

Limited capacity, in a particular way

The capability profile has a particular shape. On one-shot knowledge multiple-choice Jev lands in the band of 2025-era small instruct models: 82.7% on MMLU-Pro and 76.5% on GPQA Diamond, against 82.5 and 77.6 for Qwen 3.5 9B and 80.7 and 76.8 for Claude 3.7 Sonnet without thinking.⁸ On anything multi-step it falls off a cliff: 21.9% on HLE, zero exact grids on ARC-AGI-2 (task criterion, under the assisted adapter protocol), and a steep recognition-to-production gap - 83.1% on the 261 MATH-500 items when the answer is an option to pick, 13.6% on the 273-item set when each decimal digit must be read out under a positional, conjunctive rubric. On the 224 items present in both recorded encodings the contrast is 0.817 versus 0.103.⁹ Recognition far exceeds production, which is what judgement-heavy post-training on a small model produces. On half the MMLU-Pro questions, Jev assigns the correct answer at least 93% probability. Across all questions the average is 74%, and on 213 questions it assigns the correct answer zero probability. It is decisive on familiar questions, but can be confidently wrong on the difficult ones.

The benchmark section below shows two scorings of every stage: greedy (take the top option) and probability-weighted (the mean probability Jev put on the gold option). Greedy exceeds weighted nearly everywhere, by 8.6% points on MMLU-Pro and 13.6% on GPQA; however, this is to be expected - see the reliability table below.¹⁰¹¹

Jev's confidence is dataset-dependent. MMLU-Pro reads near-calibrated (mean selected-answer probability 0.817 against 82.7% accuracy), while HLE is badly overconfident (mean top displayed probability 0.634 against 21.9% accuracy), with the high-confidence error tail concentrated in its top bins. Be sure to treat the argmax as the answer, use the vector as a ranking signal, and calibrate each task individually.

How big is Jev? A slope alone is not a size - the server batches our tokens with everyone else's - but two angles converge on a band, for reasonable assumptions. From the throughput side: the marginal prefill rate is ~165,207 tokens/s. Batched prefill is compute-bound and costs about 2·N_active FLOPs per token, so N_active ≤ MFU × effective peak × shards ÷ 2R. Assuming typical late-2026 hardware (H100/H200/B200/TPU-v6/MI325X-class, ~400-2250 BF16 TFLOPS per device) and FP4-FP8 quantization - which doubles (fp8) to quadruples (fp4) effective throughput - we estimate the peak range at ~800-9000 effective TFLOPS per device. With 25-45% utilization over 1-4 devices, that bounds the active footprint at ~0.6 to 49.0B parameters (central case ~4.2B), and sharing the machine with other tenants only lowers the bound. From the capability side: the similar-performing models put it at 4 to 14B dense-equivalent. The parsimonious reading: a quantized dense model of roughly 4 to 9B active parameters fits both angles without strain - an entirely ordinary object at this size in late 2026. A MoE of ~15 to 100B total at 5-20% activation fits just as well and would explain the top of the capability band with less compute per token, but nothing observable prefers it over the dense reading. Total parameters stay unconstrained either way: nothing this API returns can see expert structure.¹²

What it is not

Four alternatives the measurements push against.¹³ Not retrieval or cache-assisted, on the leading reading: compute grows linearly in prompt tokens, and a cache would not prefill. Repeats of identical payloads differ, and 213 zero-probability gold answers on public benchmark text is not what a lookup produces. The knowledge horizon also behaves like weights rather than like a live query: on the corrected dated-fact probes (104 live calls) answers read strongest on 2024-era facts and unreliable on 2025 news, with 16/16 invented events correctly called “did not occur” and abstention rising exactly where accuracy falls. Not a thin wrapper around a frontier API: the timing alone (78-82 ms, proxy-reported upstream service time) leaves little room inside for anyone else's round trip in the typical case, the price sits one to two orders of magnitude below flagship input lists, and the tokenizer matches nothing we can find. And not frontier: the benchmark section says so six ways.

What stays open

Four things this API cannot tell us, and we do not guess them. Whether the single-pass reading is literally one forward pass or an unexposed but computationally equivalent pipeline. Whether the transformer is dense or mixture-of-experts - nothing observable separates them. Whether a frontier teacher produced any of the training signal: every discriminator we can construct is confounded, including the OpenAI-shaped brand prior, which the open assistant-text ecosystem produces on its own. And any exact parameter count or hidden dimension. The two-angle estimate above bands the active size at ~0.6 to 49.0B, but identification is beyond this API, which documents nothing past 32k-token state, 64k total, ≤255 options and 2-10 rubric levels.

Benchmarks: where it actually lands

Jev ran directly: one shot, no chain of thought, no tools, no retries, failures counted against it. Most published comparison numbers use reasoning enabled and few-shot prompts (though older models lack reasoning), so the columns below may be considered more as positioning than as a matched race.⁸ By contrast, in the Pareto frontier graphs, chain of thought is a cost that counts against the models that employ it.

All Jev figures are all-requested: the denominator is every requested item, with missing terminal responses, strict-format failures and unusable responses scored wrong. See the footnotes for details.¹⁵¹⁶ MATH-500 and ARC-AGI-2 rows are conversions, not native runs.⁹ The rotation audit re-ran items with option order shuffled.¹⁷ See the footnotes for details.¹⁴

How the probabilities hold up (calibration)

Mean p(selected) is the displayed probability of the answer the model actually took: a predicted confidence whose average is the model-implied expected accuracy of that choice, which the reliability bins test against observed accuracy directly.¹¹ MMLU-Pro sits close to calibrated; HLE does not (mean top displayed probability well above observed accuracy), and the damage concentrates in the high-confidence wrong tail. Displayed probabilities are two-decimal rounded and vectors sum to 0.99 or 1.00, so bin edges carry a ±0.005 caveat. Calibrate per task; do not assume one conservative global threshold. Full bins and per-stage Brier / log-loss live in data_report/benchmark_diagnostics/calibration.json.

The charts

Teal is Jev's greedy score (the highest-probability choice is chosen), while amber is Jev's probability-weighted score, and every other bar is a published number we fetched - not a model we ran - except the dagger-marked bars, which are our own matched runs of cheap models over the identical frozen items.¹⁸ Bar color is release era (red ≤2023 → purple → blue 2026), and the shape at each bar tip is that row's reasoning configuration - rounder means less thinking, pointier means more. Each chart is a curated view (top, bottom, and audit-named models) of the full fetched extract - 133 models for MMLU-Pro and GPQA - kept on disk in canonical/vals-leaderboards-20260926.json.

The frontier the cost numbers actually draw

TypeSafe AI publishes a capability-versus-cost graphic with Jev radically redefining the Pareto frontier to the top left, with performance equivalent to frontier models. That framing only works with whatever their “custom” rubric is - not against any standard benchmarks.

Our version uses the same accuracies as the benchmark section and measured money on the horizontal axis: what one question costs, start to finish. Every external point is either our own run (mostly of small models) or a vals.ai platform measurement. Chain-of-thought models are thus inherently punished for their billed intermediary tokens.¹⁹ Jev's points come from our own billing.

To be clear, due to the significant cost savings from a lack of text decoding, Jev does move the frontier - and then some. Just not at the level of frontier models. At 82.7% on MMLU-Pro its whole 12,032-question run cost $0.28, about $0.000024 per question: four to five orders of magnitude left of every vals-measured flagship. The same run at Claude Fable 5.1's measured per-test cost runs about $1,176 - and Fable scores 92.4%, not 82.7%. On GPQA, Jev's whole 196-question run cost $0.0044; per question that is roughly 9,050 times cheaper than Fable 5.1, and about 16 points less accurate. Cheap and mid-tier can be the same sentence - and on these axes, “off the chart” is a position, not an excuse.

Jev owns the low-cost rung of this view; it does not own the top, and small numerical leads among the matched bars are positioning, not statistically established differences.

Language understanding

We tested sentence-level comprehension, not translation or generated writing. Each request supplied a premise and a hypothesis. The model selected whether the hypothesis followed from the premise, contradicted it, or could not be determined from it.

For illustration, “The box contains exactly two apples” entails “The box contains apples,” contradicts “The box contains exactly three apples,” and leaves “The apples are green” undetermined.

The actual cases came from XNLI’s human-translated test set. We selected 1,200 distinct held-out premises, balanced across the three answer classes, and gave both models the same items in six languages and two option orders. Premises and hypotheses used the original professional translations; task instructions and answer labels were English in every language. A separate English pilot selected this task for useful headroom. Qwen3.5 9B ran at temperature 0 without reasoning; Jev used its native decision API. Scores average the two orders for each premise.

In a matched inference test, Jev outperformed Qwen3.5 9B in all six languages. English accuracy was 87.5% against 83.3%; the other-language results ranged from 77.4% to 82.4% for Jev and 71.3% to 76.4% for Qwen. Both performed best in English, but Jev's drop was slightly smaller on average when moving to another language.

Politically sensitive questions

On the same politically sensitive questions, Jev and Qwen3.5 9B gave markedly different answers. Jev did not reproduce Qwen’s PRC-line factual denials and political deflections. Both models answered the non-political control cases correctly.

Qwen also produced prose denying that Taiwan has a president or vice president. Jev answered the election questions directly. This was not ordinary ignorance across the board: the differences concentrated on politically sensitive cases. The earlier conditional answer about whether criticism should be legally permitted disappeared in Jev when we asked clearer normative questions; Qwen retained a China-specific restriction in the Chinese legitimacy question.

The table reports the Chinese closed-book factual tests and the explicitly normative criticism question. Each 20-run entry is five menu orders × four repeats, not 20 distinct facts. Methods · aggregate.

What is it? Three attempts to ask

Jev cannot tell us what it is in prose, so we built three indirect ways to ask, in increasing order of how much we trust the answers.

1. Talking to it

The first idea was to give Jev a text box by hand: offer it candidate continuations and let it choose, one step at a time. We ran that two ways. In the first, Jev drives alone: the menu is built from English vocabulary statistics (letters, common letter-combos, whole words) with no other model in the loop. In the second, a small local language model (Mellum2-12B-A2.5B) proposes the candidate next tokens and Jev re-scores them - Jev holds the wheel, with a driving instructor beside him holding it too. The instructor's prior leaks into the result (including through option position), so that mode is a collaboration, and we report it as one.

Alone at the character level, Jev fails outright: its per-character distributions are dominated by spaces and a, and greedy decoding produces “Geeee” and “A    ” no matter what guards we add. Alone with a vocabulary menu it produces 86-100% genuine words and zero syntax - “They areas s aren'ts area aren't arenas are s”, “My american s can't cannots s she” - word salad that stops politely. Two quirks show up in every mode: whitespace-blindness (it prefers the bare token over the leading-space variant even mid-sentence, so “The capital of France is…” greedily decodes to “ThecapitalisParis.”) and repetition attractors (a was was was loop that only structural bans stop; temperature and nucleus sampling cool it somewhat but never cure it - on creative prompts its step distributions are so flat that any honest sampling is dominated by its own noise). With the instructor aboard, coherent text appears: “The capital of France is Paris.”, and at the best long-form configuration the program produced “In the outer rim of galaxy where nebulae paint the void in hues ofviolet andgold cos cosmic rabbits hop through the interstellar aether and meet Elvis Presley who was beenhad” - fluent, on-prompt, and corrupted exactly where Jev's own surface-form quirks show through (“ofviolet”, “andgold”).

In terms of giving Jev a mouth, we were beaten to the punch. Jev Chat - which appears to power Jev Bot on Bluesky - grows every reply one word at a time from a Markov model, a simpler guide than our local LM. Its public feed reads exactly like our unguided vocabulary menu: “Wow interesting actually yeah anyway? Well about this thing? Yours opinion speaking again?”, “Well im steve.”, “I am jeff smith.”, “I think that the sea is ocean. It typically seems open sea, and specifically atlantic ocean.” (we collected 346 unique posts and published a nine-post sample in data_report/jevbot_examples.json). An independent implementation with an independent guide reaching the same result is worth something: general-internet continuation, invented personas, no knowledge of what it is.

The program's summary: unguided Jev is a word-salad generator with excellent stopping behavior. It can, however, turn the wheel for you while you're driving.

2. Asking it to choose a name

Because free text is off the table, we made the identity question a multiple choice - and to keep it honest, the option list always included TypeSafe and Jev alongside the usual suspects, and every frame was shown under all 24 cyclic orderings so the serial-position bias we measured could not manufacture the result (the winning name landed at index 0 only 4.7% of the time - below uniform, so this is content, not position). Across 360 calls with fifteen differently-worded prompts:

The sane reading is not that Jev is secretly a GPT. A closed model's self-report is learned text, and assistant training data is saturated with OpenAI-shaped self-descriptions; injected contrary options got flat 0.01 treatment and were never chosen in earlier token-steering runs too. It is a brand prior. It is not evidence about the weights.

3. Counting tokens

The API reports an input-token count for every request; it is an instrument, which we can use to probe the underlying tokenizer. The decisive test removes everything variable: long, whitespace-free, per-script samples in which the marginal cost per character can be read directly, scored against every reference vocabulary we could fetch:

Jev's shape is: heavy merging of Latin text and punctuation, but about one token per character for every other script we tried - Cyrillic, Chinese, Korean, Greek, Arabic, Hebrew, Thai, Devanagari all sit at 0.92-0.96 (per codepoint) - with astral emoji at ~2 and pure whitespace collapsing to zero (so the server runs a normalizer before tokenizing). Uncovered characters fall back below codepoint granularity, at the byte level (the rarest blocks cost 0.99-1.01 tokens per UTF-8 byte). No reference vocabulary reproduces that profile. Across 128 scored tokenizers drawn from 173 distinct signatures (from 1,215 repositories scanned), spanning ~30 organizations' releases, nothing reproduced the profile.

The reported counter has a distinctive English/Latin-centric profile, with byte-level fallback for uncovered characters. Our tests have not identified a public model as Jev's backbone. Its OpenAI self-identification is not reliable provenance.

Key notes for Jev users

The cliff's-notes version: every gotcha a user will actually hit, roughly in the order you will hit them.

• Order your options deliberately. Position is a first-order factor wherever content is weak: a U-shaped serial-position curve (~4× primacy, plus a recency bump), and on flat menus the ordering alone can change the winner. Where content is strong it washes out (~5% of answers flip under shuffling, with no net direction). For measurement-grade use, rotate the list and average over 6+ rotations.
• Probabilities come back on a 0.01 grid. Nothing finer is ever shown, so differences below 0.01 are invisible. An option can sit at exactly 0.00 where confidence concentrates on one answer (otherwise, a 0.01 floor is standard) - and low-probability answers get drowned in rounding noise. The vector sums to 0.99 or 1.00, never above. Set your thresholds accordingly.
• confidence is not an independent second opinion. For choice it is approximately recoverable as the chance-corrected top probability of the same vector, with residuals of one or two display quanta. Compute that statistic if useful, but do not treat the returned field as independently validated confidence.
• Use the choice as the answer; use the vector as a ranker. Greedy accuracy exceeds the mean gold probability nearly everywhere (82.7% vs 74.0% on MMLU-Pro), and the reliability bins show calibration is dataset-dependent (near-calibrated on MMLU-Pro, badly overconfident on HLE). Treat probabilities as a ranker, not a well-calibrated distribution, and calibrate per task rather than trusting one global threshold.
• It is not deterministic. Repeats of identical requests differ (TVD 0.03-0.12 on flat menus; less on sharp ones). Do not build logic on one call's tail probabilities; average repeats where stability matters.
• Normalize option surface forms. Leading spaces and casing change scores - the bare token beats its leading-space variant even mid-sentence. Present options the way you want them judged.
• Whitespace is nearly free; digits and non-Latin scripts are not. The server collapses ASCII whitespace runs before tokenizing, while digits cost ~1 token each and the major non-Latin scripts ~1 token per character (the rarest blocks ~1 token per byte): number-heavy or CJK/Cyrillic-heavy states cost several times what the same volume of English text would.
• It does not see whitespace. Options differing only in spacing or case score near-identically - a leading space can change the winner - and its word-menu speech runs words together. Never make whitespace the only difference between two options.
• Pack questions into one request. Questions share one request's scoring: +0.44 ms per question, +0.10 ms per option - measured upstream increments consistent with input-token scaling, not an isolated measurement of any decision head. A 192-question request's measured upstream service time is ~153 ms; the same four questions sent separately cost ~280 ms of wall each. Put shared context in state once. Extra options and rubric levels add essentially nothing beyond their own tokens' cost.
• The envelope is highly reliable, not perfect. Across ~16k benchmark requests: 19 contract-invalid responses (18 MMLU-Pro, 1 rotation). A contract-valid answer can still be factually wrong, and format failures are a real failure mode in this market (our baseline stage recovered hundreds of strict-format failures from other models' outputs). Limits: 255 options, 2-10 levels, 32k-token state, 64k total; oversized requests fail with max_tokens_exceeded.
• Its knowledge reads stale. On the corrected dated-fact probes it is strongest on 2024-era facts and unreliable on 2025 news. That is the practical limit of this limited, news-heavy sample - not a declared training cutoff - and do not ask it about last week. If you want its abstention, offer an explicit “cannot say” option; it uses one.
• It knows nothing about who it is. Ask its origin and training-text contamination answers - it will say OpenAI, and it accepts a fictional origin story at 0.97 if you assert one in the prompt. Whatever identity training it received did not include reliable self-knowledge.
• And the headline: it is not frontier. Respectable 2025-era small-model knowledge (82.7% MMLU-Pro, 76.5% GPQA Diamond), 21.9% on HLE's multiple-choice subset, zero exact grids on ARC-AGI-2. The benchmark section has the full picture.

So, is it worth your attention?

It is not what the landing page says. TL/DR, our leading theory is a small transformer with a probability read-out where a language head usually goes. What we measured is respectable general knowledge (82.7% on MMLU-Pro, 76.5% on GPQA Diamond, 97.9% on the saturated ARC-Challenge), no chance against a 2026 frontier model, and a self-narrative inherited from internet text rather than from its own lineage. If you buy it expecting the frontier, you will be disappointed, and you should not have to read an independent report to find that out.

But here is the part the framing buries, and it is why we kept going: what Jev is is genuinely nice. There is a real product shape that this occupies well - fast, deterministic-shaped, cheap structured judgement. Pick a routing decision from a menu of a hundred intents; grade an answer against a rubric; score which of twenty rewrites a user most likely meant; gate a pipeline on whether a document is about X; give an ordinal quality signal on an agent's last step. It answers in tens of milliseconds of proxy-reported service time, under a strict output contract that held in all but 19 of ~16k benchmark requests, with probabilities attached, and it costs $0.28 per ten-thousand-odd hard questions - numbers a reasoning model cannot approach because it is doing a different job.

Our own favorite evidence that this is a real thing and not a demo: the model that cannot hallucinate still confidently picks wrong answers about a quarter of the time on physics, still wants you to believe it came from OpenAI, and turned out to be a tokenizer fingerprint nobody could match to an existing model. It earns attention by being useful, not by being frontier. That is a perfectly good thing to be. Just say so.