I became curious about Jev because of a JSON inefficiency that appears everywhere in production LLM systems, from threat intelligence to compliance monitoring.

We often send a model thousands of tokens of context so it can return something like:

{
  "risk": "high",
  "route": "security",
  "requires_review": true
}

The valuable computation is deciding high, security, and true. The rest is a serialization protocol. The model generates braces, keys, quotes, commas, and values token by token, then ordinary software parses those tokens back into typed data.

I think of this as a verbalization tax.

Jev is interesting because its core purpose appears to begin one level below language generation: if the consumer is software and the output space is already known, the model can expose decisions directly.

Chopping the verbalization tax

A surprisingly small Qwen experiment captures the first step:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B").eval()
x = tok(f"{state}\n{question}\nA:{a}\nB:{b}\nC:{c}\nAnswer:", return_tensors="pt")
z = m(**x).logits[0, -1]
ids = torch.tensor([tok.encode(c, add_special_tokens=False)[0] for c in "ABC"])
p = torch.softmax(z[ids], dim=0)

No JSON generation is required. Qwen already calculated the relevant judgment before it generated the first answer token.

That observation gives us a useful starting point for thinking about Jev from first principles.

What does a structured decision actually require?

Consider one field in a schema:

{
  "risk": ["low", "medium", "high"]
}

The underlying problem is simply to estimate:

\[P(y=k \mid s,q,C)\]

where:

  • $s$ is the state or document,
  • $q$ is the question,
  • $C={c_1,\ldots,c_K}$ is the set of allowed choices,
  • $y$ is the selected decision.

For this task, the useful final object is the distribution

\[\mathbf{p} = \left[ P(y=c_1), P(y=c_2), \ldots, P(y=c_K) \right].\]

A normal decoder LLM reaches that decision indirectly.

Given a sequence $x_{1:t}$, the transformer produces a hidden representation

\[h_t = F_{\theta}(x_{1:t}),\]

then projects it across the entire token vocabulary:

\[z_t = W_{\text{vocab}}h_t,\]

and converts those logits into token probabilities:

\[P(x_{t+1}=v) = \frac{\exp(z_{t,v})} {\sum_{u\in V}\exp(z_{t,u})}.\]

One token is produced, appended to the sequence, and the process repeats.

For JSON extraction, this means the model repeatedly performs language-model inference to serialize a decision it may already have represented internally.

If the output requires $M$ generated tokens, a rough runtime decomposition is

\[T_{\text{LLM}} \approx T_{\text{prefill}} + \sum_{t=1}^{M} T_{\text{decode},t}.\]

Structured generation systems can make this reliable with grammar constraints, JSON schemas, or constrained decoding, but the autoregressive loop remains.

Jev appears to change the target computation itself.

The first thing to remove: autoregressive verbalization

The simplest Qwen reproduction takes the ordinary next-token distribution and reads only the legal answer tokens.

If the prompt maps three semantic choices to the tokens A, B, and C, the model gives us logits

\[z_A,\;z_B,\;z_C.\]

We can immediately compute

\[P(c_i \mid s,q,C) = \frac{\exp(z_i)} {\sum_{j=1}^{K}\exp(z_j)}.\]

The program then constructs the typed result.

This already eliminates the decode loop:

\[T_{\text{decision}} \approx T_{\text{prefill}} + T_{\text{readout}}.\]

SemIf implements this idea with frozen Qwen models. It performs one native forward pass, reads the final-position logits corresponding to fixed answer slots, and skips answer-token decoding entirely.

This reproduction does not tell us Jev’s internal architecture. It demonstrates something more basic: a normal language model already contains enough semantic machinery to act as a zero-shot decision model if we change how its output is consumed.

That interpretation also lines up with TypeSafe’s public discussion. When Jev was described as a large zero-shot classifier whose output vocabulary is replaced by prescribed decisions, TypeSafe’s founder said that characterization was very accurate.

The second thing to remove: repeated understanding of the same state

The larger inefficiency appears when one document produces many fields.

Suppose we want twenty decisions from the same incident report:

\[q_1,q_2,\ldots,q_{20}.\]

A straightforward LLM system effectively computes

\[F_{\theta}(s,q_1), F_{\theta}(s,q_2), \ldots, F_{\theta}(s,q_{20}).\]

Most of the input $s$ is identical.

Current Qwen reproductions improve this through prefix caching. They calculate the state once, preserve its attention state, then evaluate the question suffixes from the shared prefix.

Conceptually, we can write the expensive state computation as

\[H = E_{\theta}(s).\]

Each decision then becomes

\[p_j = D_{\phi}(H,q_j,C_j).\]

The desired runtime shape becomes

\[T(N) \approx T_{\text{state}} + T_{\text{decisions}}(N).\]

The important architectural goal is

\[T_{\text{decisions}}(N) \ll T_{\text{state}}\]

for a reasonably large $N$.

That is one of the strongest behavioral clues about Jev. TypeSafe recommends grouping many questions over one state because adding questions has relatively little effect on latency. In one documented long-document example, thirteen questions issued together were dramatically faster than issuing thirteen separate requests.

SemIf reproduces part of this behavior with Qwen prefix caches. Its public benchmark moves from repeated full prompts to shared-state inference and then to parallel suffix evaluation, producing a large throughput improvement without changing the underlying model.

A purpose-built architecture can go further because the questions themselves no longer need to look like normal causal-LM continuations.

The third thing to remove: the full language vocabulary

The stock-Qwen experiment still carries another legacy of language modeling.

A decoder model computes

\[z=W_{\text{vocab}}h\]

where

\[W_{\text{vocab}} \in \mathbb{R}^{|V|\times d}.\]

For modern models, $V$ can exceed one hundred thousand tokens.

If the application has four possible decisions, projecting $h$ across the entire vocabulary and retaining four entries performs unnecessary work.

A decision-native architecture could instead represent each candidate semantically:

\[c_i = G_{\phi}(\text{description}_i)\]

and score it directly against the state representation:

\[s_i = f_{\phi}(H,q,c_i).\]

The probability distribution becomes

\[P(y=c_i \mid s,q,C) = \frac{\exp(s_i)} {\sum_{j=1}^{K}\exp(s_j)}.\]

Now the size of the language vocabulary has disappeared from the output computation.

NanoJev explores this direction using Qwen3-0.6B with explicit decision heads. Its implementation evaluates multiple candidate paths and multiple questions through a shared backbone and produces decision distributions without text decoding.

jevlike explores another plausible version in which candidate representations query the state tokens through attention.

Neither project reveals Jev’s private architecture. Both point toward the same computational objective: preserve the semantic representation learned by a language model while giving software a much cheaper readout mechanism.

What Jev appears to add

Removing generation explains only part of Jev’s usefulness.

The other half is adding structures that ordinary LLMs were never trained to expose cleanly.

The first addition is typed decision primitives.

For a categorical choice:

\[\mathbf{p} = \operatorname{softmax} \left( s_1,\ldots,s_K \right).\]

For a Boolean decision:

\[P(y=\text{true}) = \sigma(s).\]

For an ordered score with levels $r_1,\ldots,r_K$:

\[P(y=r_i) = \frac{\exp(s_i)} {\sum_j \exp(s_j)}\]

and the expected score can be calculated as

\[\mathbb{E}[y] = \sum_{i=1}^{K} r_i P(y=r_i).\]

These objects are immediately usable by software.

The second addition is parallel readout over shared state. If we have $N$ questions with varying numbers of candidates, the model can conceptually evaluate all candidate scores together:

\[S = \left\{ s_{j,k} \mid 1\le j\le N,\; 1\le k\le K_j \right\}.\]

Each question then receives its own normalization:

\[P(y_j=c_{j,k}) = \frac{\exp(s_{j,k})} {\sum_{m=1}^{K_j}\exp(s_{j,m})}.\]

This maps naturally onto GPU matrix operations. The model can treat many decisions as one batch of semantic reads from the same state representation.

The third addition may be the most consequential: training for the meaning of the probabilities themselves.

Calibration changes what software can do with the model

The probabilities produced by slicing Qwen logits are useful scores, but their numerical values are not automatically calibrated estimates of correctness.

Suppose the model returns

\[P(\text{security})=0.97.\]

For operational use, we would like decisions assigned approximately $0.97$ confidence to be correct roughly $97\%$ of the time under the relevant deployment distribution.

A standard language model was trained mainly through next-token prediction:

\[\mathcal{L}_{\text{LM}} = -\sum_t \log P_{\theta}(x_t\mid x_{<t}).\]

That objective rewards language prediction.

TypeSafe says Jev uses Reinforcement Learning for Calibrated Decisions, or RLCD, as part of its post-training. The details remain private, so any exact reconstruction would be speculation. The public goal is clear enough: optimize a model whose output distributions can act as useful decision probabilities.

That changes how the result can participate in software.

A system can map confidence ranges to different actions, request human review for uncertain cases, combine several probabilistic judgments, or explicitly trade recall against precision.

The probability distribution becomes part of the API rather than an incidental byproduct of token generation.

My current best guess of the Jev architecture

The public evidence supports a broad computational model more strongly than any particular attention mechanism.

The state is encoded once:

\[H=E_{\theta}(s).\]

Questions and candidates produce decision representations:

\[u_{j,k} = G_{\phi}(q_j,c_{j,k}).\]

A lightweight readout compares those representations with the shared state:

\[s_{j,k} = R_{\phi}(H,u_{j,k}).\]

Each primitive applies its appropriate normalization:

\[\mathbf{p}_j = \operatorname{Normalize}_{\text{type}(j)} \left( s_{j,1},\ldots,s_{j,K_j} \right).\]

The entire call can therefore be summarized as

\[s \xrightarrow{E_{\theta}} H \xrightarrow{\{R_{\phi}(q_j,C_j)\}_{j=1}^{N}} \{\mathbf{p}_1,\ldots,\mathbf{p}_N\}.\]

This explains several public observations at once: one state can support many cheap questions, arbitrary natural-language candidate descriptions can define new decisions at runtime, no output prose has to be decoded, and post-training can focus directly on the quality of decision distributions.

The exact form of $R_{\phi}$ remains the interesting unknown. It could involve candidate-conditioned attention, specialized decision tokens, a learned classifier over shared hidden states, or something more novel. The current open reproductions show that several implementations can reproduce pieces of the observed behavior.

Why I liked Jev and why the idea is useful

A large amount of enterprise LLM work has the same underlying structure to convert unstructured information to small typed state. Routing, policy checks, ranking, moderation, risk assessment, document classification, workflow branching, and many parts of information extraction all fit this pattern.

When the answer space is known, generating language creates an avoidable intermediate representation.

The same idea extends naturally to knowledge extraction. Arbitrary spans such as names, dates, or invoice numbers still benefit from span-oriented models such as GLiNER or dedicated extraction models. Once candidate entities and facts exist, a Jev-like decision layer can classify relationships, resolve ambiguous candidates, validate proposed facts, and attach uncertainty.

This gives me a different way to think about the relationship between LLMs and structured software.

Language modeling gave us the semantic representation. Jev explores what happens when the final interface is designed around the program that consumes that representation.

The efficiency gain seems to come from changing where the model spends its computation. Instead of repeatedly decoding a decision into tokens, the model can expose the decision directly. Instead of reprocessing the same state for every question, it can build a shared representation once and reuse it across many decisions. And instead of projecting each hidden state across the entire language vocabulary, it can score only the candidates that matter for the task.

At the same time, Jev appears to add the pieces that structured software actually needs: typed outputs, many decisions read from the same state in parallel, and probabilities trained to carry useful information about uncertainty.

This is my current best guess at the core of Jev. Its significance comes from redesigning the path from model understanding to software action, rather than spending more effort on making autoregressive structured generation faster.

References

  1. TypeSafe AI, Introducing System One Models and Jev. TypeSafe AI announcement

  2. TypeSafe AI documentation, Parallel Questions, demonstrating shared-state batching across 13 questions, and Jev 1.13 Jaggedness, documenting strengths and failure modes. Parallel Questions cookbook · Jev 1.13 Jaggedness

  3. TypeSafe / Hacker News launch discussion, including discussion of Jev as a zero-shot decision or classification model and comments from the TypeSafe team about the unpublished architecture. Hacker News discussion

  4. Theo Lee, SemIf (formerly OpenJev), an open Qwen-based reproduction using direct option-logit readout, shared-state prefix caching, and parallel decision evaluation. SemIf on GitHub

  5. TianyuCodings, NanoJev, a Qwen3-0.6B experiment with explicit decision heads, dynamic candidates, zero output-token decoding, and probability-learning experiments. NanoJev on GitHub

  6. vinnylarouge, jevlike, an independent experiment using candidate-conditioned attention to build a one-pass scorer over a variable set of text options. jevlike on GitHub

  7. Eric Zhang, openjev-sglang, a Jev-compatible API implemented with Qwen and SGLang, using radix caching, prefill-oriented serving, and constrained probability readout. openjev-sglang on GitHub