We often send a model thousands of tokens of context so it can return something like:

{
  "risk": "high",
  "route": "security",
  "requires_review": true
}

The valuable computation is deciding high, security, and true. The rest is a serialization protocol. The model generates braces, keys, quotes, commas, and values token by token, then ordinary software parses those tokens back into typed data.

I think of this as a verbalization tax.

Jev is interesting because its core purpose appears to begin one level below language generation: if the consumer is software and the output space is already known, the model can expose decisions directly.

A surprisingly small Qwen experiment captures the first step:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B").eval()
x = tok(f"{state}\n{question}\nA:{a}\nB:{b}\nC:{c}\nAnswer:", return_tensors="pt")
z = m(**x).logits[0, -1]
ids = torch.tensor([tok.encode(c, add_special_tokens=False)[0] for c in "ABC"])
p = torch.softmax(z[ids], dim=0)

No JSON generation is required. Qwen already calculated the relevant judgment before it generated the first answer token.

That observation gives us a useful starting point for thinking about Jev from first principles.

What does a structured decision actually require?

Consider one field in a schema:

{
  "risk": ["low", "medium", "high"]
}

The underlying problem is simply to estimate:

where:

• $s$ is the state or document,
• $q$ is the question,
• $C={c_1,\ldots,c_K}$ is the set of allowed choices,
• $y$ is the selected decision.

For this task, the useful final object is the distribution

A normal decoder LLM reaches that decision indirectly.

Given a sequence $x_{1:t}$, the transformer produces a hidden representation

then projects it across the entire token vocabulary:

and converts those logits into token probabilities:

One token is produced, appended to the sequence, and the process repeats.

For JSON extraction, this means the model repeatedly performs language-model inference to serialize a decision it may already have represented internally.

If the output requires $M$ generated tokens, a rough runtime decomposition is

Structured generation systems can make this reliable with grammar constraints, JSON schemas, or constrained decoding, but the autoregressive loop remains.

Jev appears to change the target computation itself.

The first thing to remove: autoregressive verbalization

The simplest Qwen reproduction takes the ordinary next-token distribution and reads only the legal answer tokens.

If the prompt maps three semantic choices to the tokens A, B, and C, the model gives us logits

We can immediately compute

The program then constructs the typed result.

This already eliminates the decode loop:

SemIf implements this idea with frozen Qwen models. It performs one native forward pass, reads the final-position logits corresponding to fixed answer slots, and skips answer-token decoding entirely.

This reproduction does not tell us Jev’s internal architecture. It demonstrates something more basic: a normal language model already contains enough semantic machinery to act as a zero-shot decision model if we change how its output is consumed.

That interpretation also lines up with TypeSafe’s public discussion. When Jev was described as a large zero-shot classifier whose output vocabulary is replaced by prescribed decisions, TypeSafe’s founder said that characterization was very accurate.

The second thing to remove: repeated understanding of the same state

The larger inefficiency appears when one document produces many fields.

Suppose we want twenty decisions from the same incident report:

A straightforward LLM system effectively computes

Most of the input $s$ is identical.

Current Qwen reproductions improve this through prefix caching. They calculate the state once, preserve its attention state, then evaluate the question suffixes from the shared prefix.

Conceptually, we can write the expensive state computation as

Each decision then becomes

The desired runtime shape becomes

The important architectural goal is

for a reasonably large $N$.

That is one of the strongest behavioral clues about Jev. TypeSafe recommends grouping many questions over one state because adding questions has relatively little effect on latency. In one documented long-document example, thirteen questions issued together were dramatically faster than issuing thirteen separate requests.

SemIf reproduces part of this behavior with Qwen prefix caches. Its public benchmark moves from repeated full prompts to shared-state inference and then to parallel suffix evaluation, producing a large throughput improvement without changing the underlying model.

A purpose-built architecture can go further because the questions themselves no longer need to look like normal causal-LM continuations.

The third thing to remove: the full language vocabulary

The stock-Qwen experiment still carries another legacy of language modeling.

A decoder model computes

where

For modern models, $V$ can exceed one hundred thousand tokens.

If the application has four possible decisions, projecting $h$ across the entire vocabulary and retaining four entries performs unnecessary work.

A decision-native architecture could instead represent each candidate semantically:

and score it directly against the state representation:

The probability distribution becomes

Now the size of the language vocabulary has disappeared from the output computation.

NanoJev explores this direction using Qwen3-0.6B with explicit decision heads. Its implementation evaluates multiple candidate paths and multiple questions through a shared backbone and produces decision distributions without text decoding.

jevlike explores another plausible version in which candidate representations query the state tokens through attention.

Neither project reveals Jev’s private architecture. Both point toward the same computational objective: preserve the semantic representation learned by a language model while giving software a much cheaper readout mechanism.

What Jev appears to add

Removing generation explains only part of Jev’s usefulness.

The other half is adding structures that ordinary LLMs were never trained to expose cleanly.

The first addition is typed decision primitives.

For a categorical choice:

For a Boolean decision:

For an ordered score with levels $r_1,\ldots,r_K$:

and the expected score can be calculated as

These objects are immediately usable by software.

The second addition is parallel readout over shared state. If we have $N$ questions with varying numbers of candidates, the model can conceptually evaluate all candidate scores together:

Each question then receives its own normalization:

This maps naturally onto GPU matrix operations. The model can treat many decisions as one batch of semantic reads from the same state representation.

The third addition may be the most consequential: training for the meaning of the probabilities themselves.

Calibration changes what software can do with the model

The probabilities produced by slicing Qwen logits are useful scores, but their numerical values are not automatically calibrated estimates of correctness.

Suppose the model returns

For operational use, we would like decisions assigned approximately $0.97$ confidence to be correct roughly $97\%$ of the time under the relevant deployment distribution.

A standard language model was trained mainly through next-token prediction:

That objective rewards language prediction.

TypeSafe says Jev uses Reinforcement Learning for Calibrated Decisions, or RLCD, as part of its post-training. The details remain private, so any exact reconstruction would be speculation. The public goal is clear enough: optimize a model whose output distributions can act as useful decision probabilities.

That changes how the result can participate in software.

A system can map confidence ranges to different actions, request human review for uncertain cases, combine several probabilistic judgments, or explicitly trade recall against precision.

The probability distribution becomes part of the API rather than an incidental byproduct of token generation.

My current best guess of the Jev architecture

The public evidence supports a broad computational model more strongly than any particular attention mechanism.

The state is encoded once:

Questions and candidates produce decision representations:

A lightweight readout compares those representations with the shared state:

Each primitive applies its appropriate normalization:

The entire call can therefore be summarized as

This explains several public observations at once: one state can support many cheap questions, arbitrary natural-language candidate descriptions can define new decisions at runtime, no output prose has to be decoded, and post-training can focus directly on the quality of decision distributions.

The exact form of $R_{\phi}$ remains the interesting unknown. It could involve candidate-conditioned attention, specialized decision tokens, a learned classifier over shared hidden states, or something more novel. The current open reproductions show that several implementations can reproduce pieces of the observed behavior.

Why I liked Jev and why the idea is useful

A large amount of enterprise LLM work has the same underlying structure to convert unstructured information to small typed state. Routing, policy checks, ranking, moderation, risk assessment, document classification, workflow branching, and many parts of information extraction all fit this pattern.

When the answer space is known, generating language creates an avoidable intermediate representation.

The same idea extends naturally to knowledge extraction. Arbitrary spans such as names, dates, or invoice numbers still benefit from span-oriented models such as GLiNER or dedicated extraction models. Once candidate entities and facts exist, a Jev-like decision layer can classify relationships, resolve ambiguous candidates, validate proposed facts, and attach uncertainty.

This gives me a different way to think about the relationship between LLMs and structured software.

Language modeling gave us the semantic representation. Jev explores what happens when the final interface is designed around the program that consumes that representation.

The efficiency gain seems to come from changing where the model spends its computation. Instead of repeatedly decoding a decision into tokens, the model can expose the decision directly. Instead of reprocessing the same state for every question, it can build a shared representation once and reuse it across many decisions. And instead of projecting each hidden state across the entire language vocabulary, it can score only the candidates that matter for the task.

At the same time, Jev appears to add the pieces that structured software actually needs: typed outputs, many decisions read from the same state in parallel, and probabilities trained to carry useful information about uncertainty.

This is my current best guess at the core of Jev. Its significance comes from redesigning the path from model understanding to software action, rather than spending more effort on making autoregressive structured generation faster.

References

1. TypeSafe AI, Introducing System One Models and Jev. TypeSafe AI announcement

2. TypeSafe AI documentation, Parallel Questions, demonstrating shared-state batching across 13 questions, and Jev 1.13 Jaggedness, documenting strengths and failure modes. Parallel Questions cookbook · Jev 1.13 Jaggedness

3. TypeSafe / Hacker News launch discussion, including discussion of Jev as a zero-shot decision or classification model and comments from the TypeSafe team about the unpublished architecture. Hacker News discussion

4. Theo Lee, SemIf (formerly OpenJev), an open Qwen-based reproduction using direct option-logit readout, shared-state prefix caching, and parallel decision evaluation. SemIf on GitHub

5. TianyuCodings, NanoJev, a Qwen3-0.6B experiment with explicit decision heads, dynamic candidates, zero output-token decoding, and probability-learning experiments. NanoJev on GitHub

6. vinnylarouge, jevlike, an independent experiment using candidate-conditioned attention to build a one-pass scorer over a variable set of text options. jevlike on GitHub

7. Eric Zhang, openjev-sglang, a Jev-compatible API implemented with Qwen and SGLang, using radix caching, prefill-oriented serving, and constrained probability readout. openjev-sglang on GitHub

The most interesting part was that several things I saw on my phone were genuinely produced by Google. The account recovery prompt was real. The verification request was real. The later security email also came through Google’s infrastructure.

The attacker was exploiting the meaning I would assign to those legitimate messages.

I would describe this technique as trusted-channel laundering. An attacker deliberately triggers legitimate security mechanisms from a trusted service, then uses a phone call or another social channel to provide a false explanation of what those mechanisms mean. Instead of constructing a convincing imitation of Google, the attacker gets Google to produce much of the convincing material.

There is an important defense that makes this particular attack much easier to stop. Google’s own current security guidance says very clearly:

“Google will never call you about your account security.”

Google also says it will never call and ask you to read a verification code or approve a device prompt. Even messages from legitimate Google domains should not be interpreted as proof that a caller represents Google.

That simple rule turns out to be extremely relevant to what happened.

Step 1: “Google” calls me

At around 8 PM Pacific time, I received a call from:

650-215-XXXX

The caller ID displayed Google.

The person said that someone had recently changed the contact phone number associated with my Gmail account. He told me that Google’s Trust and Safety team would contact me shortly to help secure the account.

A few minutes later, another call arrived from:

347-329-XXXX

This time a man speaking fluent English said he worked for Google’s security team. He explained that somebody had changed the contact information on my Gmail account and that he needed to verify my identity and help me recover it.

I am including both phone numbers because they were the numbers displayed during this incident and may be useful to someone searching for the same scam.

They should not be treated as durable identities for the attackers.

Caller ID names and numbers can be spoofed. A scammer can make a call appear to originate from a different number and can manipulate the displayed caller name. Even if these particular numbers were controlled by the attackers during my call, they can switch numbers at any time. Carriers may block them, Google may take action if its name is being abused, and the numbers themselves may later be reassigned.

So the useful indicator here is:

These were the numbers used in this incident.

The stronger security signal is the behavior that followed.

The timing also felt strange. It was around 8 PM in California and 11 PM in New York. More importantly, I had never asked Google for support.

Why did they use two calls?

The first call established the story before the second caller asked me to do anything sensitive.

By the time the supposed Google security engineer appeared, I had already been told that there was an account incident and that another Google employee would contact me.

The second call therefore arrived as an expected event.

That sequencing matters. The attacker was creating context first, then asking for authorization.

Step 2: They made Google send me a real recovery prompt

While “Google Security” was talking to me, my Gmail app displayed a Google account recovery notification.

A device in New York was attempting to recover my account.

That was a real Google prompt.

The caller immediately explained that this was expected because he was working on my recovery case. He told me to click Yes because the New York device belonged to the Google security process he had just initiated.

This is where the attack became much more interesting.

Google prompts can be used during account recovery. If Google sees an unusual sign-in or recovery attempt, the prompt can contain information such as the device and approximate location so that the account owner can reject an attempt they did not initiate.

The attacker had triggered exactly the warning that was supposed to protect me, then used the phone call to reinterpret it.

The notification was effectively asking:

Someone in New York is trying to recover this account. Is that you?

The caller’s explanation was:

Yes, that is me in New York helping you recover your account.

The same event now had two possible meanings, and the security outcome depended on which explanation I believed.

Why did they use Google’s own recovery prompt?

Because a genuine Google notification carries much more credibility than a fake webpage or a phishing email.

Once the victim accepts the premise that the person on the phone represents Google, suspicious information inside the notification can start working in the attacker’s favor.

The location New York could have warned me that somebody elsewhere was trying to access my account.

Under the attacker’s explanation, it became evidence that the supposed Google security engineer was doing exactly what he had promised.

That is the core pattern of this attack.

Step 3: “Please enter my job ID, 48”

Then came my favorite part of the attack.

The caller told me that his Google employee ID was 48 and instructed me to use 48 in the authentication prompt.

Of course, 48 was functioning as part of Google’s authentication challenge.

Google can use number matching as part of an authentication flow. A number displayed during one part of the transaction must correspond to the trusted device authorizing it.

The attacker gave the number a new meaning.

The security system was effectively saying:

Verify that this authentication session is the one you intended to authorize.

The caller told me:

This is my Google employee ID. Enter 48 to verify that I am helping you.

That is a subtle and effective semantic substitution.

For the victim, typing 48 feels almost harmless. There is no password being disclosed, no long OTP being read aloud, and no suspicious website being opened.

It sounds like entering an employee identifier into Google’s own application.

Why did they call 48 an “employee ID”?

People have been trained for years not to share passwords or verification codes.

A two-digit employee ID feels very different.

The attacker did not need to hide the authentication token. He changed the victim’s understanding of what the token represented.

This is one of the most interesting parts of the attack from a security-design perspective.

I refused to enter it.

Step 4: When the first path failed, they pivoted

After I questioned the first request, the caller immediately changed approaches.

An unfamiliar Gmail account:

{something with letters and numbers}@gmail.com

attempted to establish a recovery relationship involving my email address. Google then sent me another legitimate request related to that recovery process.

There was another verification code involved, and the caller wanted me to provide or confirm it.

Again, I refused.

I am including the Gmail address because it was directly involved in this incident and may help other people identify the same campaign.

Like the phone numbers, however, this email address should be treated as a temporary incident indicator rather than a permanent attacker identity.

The scammer can create another Gmail account at any time. Google may also suspend or remove the account after receiving abuse reports. By the time someone reads this article, {something with letters and numbers}@gmail.com may already be inactive, or the same group may be using completely different accounts.

The behavior is more durable than the identifier.

Why did they switch to another Gmail account?

The first route depended on getting me to authorize recovery of my own account.

Once I refused, the attacker needed another Google workflow capable of generating trusted messages to my inbox.

A recovery-email relationship provides exactly that.

The attacker can initiate an action involving an attacker-controlled Google account, cause Google to send the victim a legitimate message, then explain that message over the phone as part of the victim’s supposed security incident.

This also showed that the person on the phone was not committed to one carefully scripted exploit. He appeared to have several Google account workflows available and was choosing among them interactively based on what I accepted or rejected.

Step 5: A real Google security email appears to confirm “James Wilson”

The next step was the most convincing.

A security alert associated with the unfamiliar account was copied to my email address. The alert concerned an:

“App password created to sign in to your account”

event.

Inside the message was text along the lines of:

If you didn’t generate this password for [my email], then please proceed with the current call with Google security agent, {a common US male name}, Case ID: {some number}. More than likely, someone might be using your account.

At almost exactly the same time, the caller, who had introduced himself as {a common US male name}, told me that he had escalated my security case.

He repeated:

Case ID: 37518

This was a much stronger social-engineering artifact than an ordinary phishing email.

From the victim’s perspective, two apparently independent channels now agree.

The person on the phone says:

{a common US male name}, case {some number}.

A security email delivered through Google’s infrastructure also contains:

{a common US male name}, case {some number}.

The natural interpretation is that Google has authenticated the caller.

That conclusion does not follow.

I have not independently reproduced exactly which attacker-controlled Google field allowed that particular text to appear in the security notification, so I would treat that implementation detail as an open question. It may involve an account-controlled label, application name, metadata field, or another value that Google inserts into a legitimate notification.

The observable security problem does not depend on the exact field.

The attacker found a way to cause legitimate Google infrastructure to deliver a message containing information coordinated with the story being told on the phone.

Why did they do this?

Because agreement between independent channels normally increases trust.

If somebody gives you a case number over the phone, and then an apparently independent email from Google contains exactly the same case number, that correlation feels difficult to fake.

The attacker was manufacturing the correlation.

He controlled the action that caused Google’s system to generate the second piece of evidence.

This is where the term trusted-channel laundering becomes useful.

Trusted-channel laundering

Traditional phishing often requires an attacker to imitate something trusted.

They may create a fake login page, register a confusing domain, spoof a support email, or reproduce a company’s branding.

This attack has a different structure.

The attacker initiates an action inside a legitimate service.

Google then produces a genuine security artifact.

The attacker remains on another communication channel and explains what the artifact supposedly means.

The flow looks roughly like this:

Attacker action → legitimate Google workflow → genuine Google notification → attacker-provided interpretation

The trusted service supplies authenticity to the artifact.

The attacker supplies the semantics.

The victim sees the two together.

That is what I mean by trusted-channel laundering.

The attacker gets some of the credibility of Google’s infrastructure transferred to a story that Google itself never authenticated.

The deeper problem is semantic binding

From a security research perspective, the most interesting issue is what I would call a semantic binding failure.

An authentication system can accurately record that a trusted device approved an action.

The harder question is whether the human operating that device understood the action in the same way the security protocol intended.

During this call, the attacker repeatedly tried to replace the protocol’s meaning with his own.

An account recovery attempt became:

Google Security is helping recover your account.

A number-matching challenge became:

Enter my employee job ID.

A recovery-email verification became:

Verify your security case.

A security notification associated with another account became:

Google has confirmed your security representative.

The security mechanisms can all execute correctly while the human authorizes something based on a false mental model.

This is a useful reminder that authentication involves both protocol integrity and user interpretation.

Then I asked for his Google LDAP

At this point, I decided to test whether the caller could handle something outside his script.

I asked:

“What is your Google LDAP?”

He hung up.

No explanation. No argument. The call simply ended.

For this particular scammer, LDAP turned out to be a rather effective magic word.

I would not treat this as a real defense technique.

The next scammer can learn the terminology. They can prepare a believable LDAP identifier, invent one confidently, or develop a response explaining why they cannot provide it.

The interesting part was that the question moved the interaction away from the protocol the attacker had prepared.

Until that moment, he controlled what happened next. He initiated a Google workflow, knew what I was about to see, and immediately supplied an explanation for it.

An unexpected question reversed that asymmetry for a few seconds.

For someone analyzing the scam, that was useful.

For a normal user, there is a much easier defense: hang up.

Google has a help page that describes this exact class of scam

Afterward, I looked through Google’s security documentation.

Google has a help page specifically about Google Account Security Scam via Phone Call.

The most useful sentence is also the simplest:

“Google will never call you about your account security.”

Google additionally warns users that scammers may send messages from legitimate Google domains to make their claims appear authentic.

That warning is particularly important here.

A message can genuinely originate from Google’s infrastructure because the attacker triggered a legitimate Google workflow.

The origin of the email tells you who delivered the message.

It does not automatically establish who initiated the underlying event, what that event means for your account, or whether the person currently talking to you represents Google.

This is exactly why checking only the sender domain would have been insufficient during this attack.

What I think Google could strengthen

Google’s security mechanisms gave me enough information to stop the attack. I still think this incident exposes several places where security UX could make the attacker’s semantic manipulation harder.

1. Make user intent explicit in recovery prompts

Instead of only asking:

Are you trying to recover your account?

the prompt could emphasize:

Did you personally initiate this account recovery process on another device?

It could also include:

Google employees will never ask you to approve this prompt during a phone call.

The goal is to bind the approval to an action the user remembers initiating.

2. Explain where number matching values should come from

The weakness of 48 was not simply that the number was short.

The problem was that a caller could plausibly assign another meaning to it.

A stronger prompt might say:

Select this number only if it matches the number displayed on the device where you personally initiated sign-in. Never select a number given to you by someone on a phone call.

This tells the user about the expected provenance of the number.

3. Treat new recovery relationships carefully

If my email has suddenly become associated with another Google account as a recovery address, subsequent security messages about that account should make the relationship very clear.

Something like:

This security alert concerns another Google account that listed your email as a recovery address. It does not mean that your own Google Account has been compromised.

That one sentence would make this attack significantly harder.

4. Visually isolate attacker-controlled text

Any user-controlled field inserted into a security email should be considered potentially adversarial.

Account names, app labels, device names, recovery information, project names, and similar metadata should be visually separated from Google’s own authoritative security instructions.

Otherwise an attacker may be able to place social-engineering text inside a message that inherits the visual authority of a Google security alert.

5. Think beyond device possession toward intent verification

Modern authentication systems are increasingly good at confirming that a trusted device participated in an authorization.

Social engineering creates another question:

Did the user understand exactly what operation they were authorizing and who initiated it?

That is a much harder security problem.

This incident provides a clean example because the attacker’s main tool was repeatedly changing the perceived meaning of legitimate authentication events.

What should a user actually do?

You do not need to understand trusted-channel laundering, inspect email headers, know Google’s internal terminology, or ask anyone for their LDAP.

Google provides a much simpler rule:

Google will never call you about your account security.

If somebody claiming to be Google Account Security calls you unexpectedly, end the call.

Then open your Google Account security page yourself. Navigate there independently through Google’s app or website and inspect recent security activity.

If an authentication prompt appears that you did not personally initiate, reject it.

Do not read verification codes to someone on the phone.

Do not type a number into a security prompt because a caller tells you that it is their employee ID, case number, job number, or anything else.

If an email from Google arrives while the person is talking to you, inspect the event independently. The fact that Google genuinely delivered the email does not authenticate the caller.

And if you encounter the specific indicators I saw:

• 650-215-5242
• 347-329-3627
• fredwright726@gmail.com
• “James Wilson”
• Case ID 37518
• “job ID 48”

they may help connect your incident to the same scam campaign.

Do not rely on them as a blocklist.

Scammers can change phone numbers, caller ID names, email accounts, employee names, case numbers, and scripts very quickly. Google or telephone carriers may also disable some of these identifiers after abuse is reported.

The reusable defense is to recognize the mechanism.

What made this incident interesting to me as both an AI researcher and someone working in security was how little fake infrastructure the attacker actually needed. He could trigger real Google security workflows, observe what I would see next, and provide a carefully prepared interpretation over the phone.

For a few minutes, Google itself was producing many of the artifacts that made his story believable.

That is why a legitimate security message still deserves one more question:

Who caused this message to be sent, and what action am I actually being asked to authorize?

I wanted to approach the same landscape from a complementary direction. With my physics background, I naturally ask whether a complicated collection of methods can be reconstructed from a small set of primitives, like Coulomb’s law for classical electromagnetism. One can begin with a simple interaction such as $F = k_e \frac{q_1 \times q_2}{r^2}$, then introduce additional structure only when the current description becomes insufficient. To be clear: Maxwell’s equations are not derived from Coulomb’s law alone, but we haven’t had Maxwell’s equations for LLM, right?

Start from SFT and cross-entropy, ask exactly what information it contains, identify what it cannot express, then add one new degree of freedom at a time. Following that path, much of modern post-training can be organized around four conceptual transitions: from likelihood to preference, from offline projection to on-policy optimization, from sequence generation to environment interaction, and from scalar reward to distributional feedback. The individual algorithms then become branches of these transitions rather than isolated inventions.

This post works like an equation heavy note, but don’t be afraid. I will use a small set of annotations consistently: $x$ denotes the input or prompt, $y$ a generated response, $a_t$ the token or action taken at step $t$, and $s_t$ the corresponding model state or interaction context. The policy is $\pi_\theta$, while $p_{\mathcal D}$ denotes an external data distribution. Rewards are written as $R$ or $r$, advantages as $A_t$, and $\pi_{\mathrm{ref}}$ denotes a reference policy when one is needed. For agentic settings, $\tau$ denotes a full trajectory and $P_{\mathrm{env}}$ the environment transition dynamics. I will use $D_{\mathrm{KL}}(p\Vert q)$ in the standard direction, with the first argument defining the sampling distribution inside the expectation.

Disclaimer: this post is edited with LLM assistance.

I. From likelihood to preference

Cross-entropy as the primitive interaction

Let an autoregressive policy be (as in many GPT-style models)

Given demonstrations sampled from a data distribution,

supervised fine-tuning minimizes

At token level,

The objective contains a remarkably simple form of supervision. At each training state, the data identifies a target token and increases its probability. If

then the gradient with respect to a logit is (p-y) in numpy or:

The target token is pushed upward in probability, while alternatives are suppressed through softmax normalization. This is enough to support imitation, instruction following, and large amounts of behavioral shaping because the pretrained model already contains a rich distribution of latent capabilities.

The limitation follows from the same equation. Cross-entropy identifies a target but carries no explicit representation of relative quality among alternatives. Suppose two complete responses $y_A$ and $y_B$ are both plausible, while one is more factual, more concise, safer, or better calibrated. If the dataset contains only $y_B$, the optimization increases its likelihood, but the loss does not directly encode the relation

This becomes important whenever post-training is less about teaching an unseen capability and more about shifting probability between behaviors that already exist in the pretrained model. That missing relational structure motivates preference optimization.

Preference introduces a relative coordinate

A preference pair adds the ordering

A common latent-variable model represents this ordering through a scalar reward

with pairwise preference probability

The corresponding loss is

The important change is that supervision now constrains a difference between behaviors rather than simply assigning one behavior positive probability. SFT provides an absolute target $y^*$, while preference learning constrains the relative direction $y_w-y_l$. This is a more natural representation for many alignment problems because the pretrained model may already support both candidate behaviors. The remaining task is to move the decision boundary.

That distinction applies to properties such as factuality, verbosity, tone, refusal boundaries, confidence calibration, formatting, and tool-use style. Preference data is useful precisely because these dimensions often live inside an existing behavioral manifold rather than outside it.

RLHF turns preference into a policy objective

Once a reward model provides a scalar estimate of preference, the natural next step is to optimize the policy against it:

For a large pretrained model, unconstrained reward maximization can move the policy into poorly modeled regions of behavior space. The reward model is an approximation, so aggressive optimization may exploit inaccuracies in the reward surface or damage useful capabilities inherited from pretraining. KL-regularized RLHF addresses this by adding a reference policy:

The reward defines a direction of improvement, while the KL term controls how far the policy can move away from a known distribution. More importantly for the framework developed here, the expectation is taken over samples from the policy being optimized. This begins a shift in the source of training experience, from externally supplied demonstrations toward behavior generated by the model itself.

The optimization problem also has a useful closed-form optimum:

Equivalently,

That relationship provides the starting point for DPO.

DPO as an analytical branch of the RLHF objective

DPO substitutes the optimal-policy relation directly into the preference model. Define

The DPO objective becomes

Its gradient has the structure

where

Compared with SFT, the essential difference is visible directly in the gradient. Supervised learning reinforces a demonstrated response, while DPO applies opposite relative pressure to the preferred and rejected responses. The update strength also depends on how much the current policy already satisfies the preference ordering. This makes DPO well suited to reshaping boundaries between behaviors that are already supported by the model.

Its place in the broader framework is also precise. The optimal-policy parameterization comes from KL-regularized RL, but the samples usually come from a fixed preference dataset:

DPO therefore inherits an RL-derived target geometry while retaining offline sampling. This combination explains much of its appeal and also why it behaves differently from online RL when exploration matters. Other “PO”s like SimPO, IPO, KTO, ORPO, and related methods can be viewed as nearby branches that modify the reference policy, margin, pair structure, normalization, or statistical form of this same preference-learning problem.

The first major transition can therefore be summarized as a progression from likelihood, to relative quality, to reward-shaped policy optimization. The next transition changes the source of experience itself.

II. From offline projection to on-policy optimization

Policy gradient changes who generates the data

Suppose the policy generates its own response,

and receives reward

The objective is

Using the log-derivative identity,

we obtain

For an autoregressive policy,

Subtracting a baseline gives

and therefore

This equation is one of the central roots of modern post-training. The log-probability gradient remains familiar, while the coefficient becomes an advantage rather than a supervised target. The deeper transition is in the expectation itself. Training now occurs on samples produced by the current or recent policy, so the model participates in constructing its own training distribution.

This on-policy structure becomes especially important when useful behaviors are difficult to write down as demonstrations. The model can explore trajectories that were never explicitly supplied by a teacher, and the feedback function determines which of those trajectories are reinforced.

PPO adds trust to an on-policy update

Rollout data is usually produced by a policy that becomes stale during optimization. If $\pi_{\mathrm{old}}$ generated the sample while $\pi_\theta$ is being updated, the relevant importance ratio is

A policy-gradient estimator then contains the factor

Large ratios can produce unstable updates because a sample that was reasonable under the rollout policy may become poorly representative of the current policy. PPO controls this through clipping:

At this point the post-training problem contains three distinct ingredients: the distribution that generates experience, the signal used to evaluate that experience, and the mechanism that controls how strongly stale experience can modify the current policy. Many later RL variants can be understood as modifying one of these ingredients rather than introducing a new foundation.

The PPO clipping mechanism should also be distinguished from the KL penalty used in classical RLHF. Clipping constrains local updates relative to the rollout policy, while the KL penalty controls global drift from a reference model. They solve different regularization problems inside the same broader optimization system.

RLVR and GRPO modify different parts of the same equation

RLVR changes the origin of reward. If a task admits an external verifier, reward can be defined directly from an observable outcome:

The policy-gradient structure remains

The scientific improvement lies in the measurement process used to construct the advantage. Learned reward models estimate quality, while verifiers can connect reward to an externally checkable state. This is why mathematics, coding, formal verification, and some tool-use tasks provide particularly productive environments for RL.

GRPO modifies a different component. PPO often uses a learned value function $V_\phi(s)$ to estimate advantages, while GRPO uses multiple samples from the same prompt to create a relative baseline. Given rewards

a group-relative advantage can be written as

The policy-gradient structure remains the same, while the estimator of $A$ changes:

This distinction is useful because it separates two kinds of innovation that are often discussed together. RLVR improves the source of the reward signal, while GRPO changes the way that signal is converted into relative credit. GSPO, CISPO, and related methods continue this pattern by modifying ratio granularity, clipping behavior, normalization, or other terms inside the same on-policy framework.

III. From sequences to dynamical systems

Agent RL introduces environment dynamics

A reasoning task can often be approximated as a generation followed by evaluation:

An agent interacts repeatedly with an external state. Its trajectory can be written as

with environment dynamics

The optimization objective becomes

The corresponding gradient remains familiar:

The major new object is the environment transition process. For a coding agent, the next state depends on files edited, commands executed, compiler errors, tests, and previous tool calls. For a research or computer-use agent, future states similarly depend on earlier actions. The model therefore helps determine the distribution of states on which future learning occurs.

This makes agentic post-training a coupled dynamical system. Policy quality alone no longer determines the training distribution. The reachable state space is jointly determined by the policy and the environment.

Environment design becomes part of the learning problem

Once trajectories are sampled from both $\pi_\theta$ and $P_{\mathrm{env}}$, the quality of the environment becomes a first-class training variable. This is where recent GLM work is especially instructive. GLM-5.3 keeps the GLM-5.2 base model while scaling post-training through more environments, greater task diversity, more rollout compute, stronger verifier construction, and better long-horizon infrastructure.

This suggests a useful analogy with pretraining. Pretraining capability depends strongly on the interaction among model, data, and compute. Agentic post-training increasingly depends on policy, environment, verifier, and rollout compute:

The environment determines which states can be reached, the verifier determines which trajectories become informative, rollout compute determines how much of the state space can be explored, and the optimization algorithm determines how this experience changes the model.

This view also explains why environment generation is more than a data-engineering detail. A model cannot learn a recovery strategy if the training system never generates failure states that require recovery. It cannot learn long-horizon planning if episodes terminate before such behavior matters. Environment design therefore shapes the set of learnable behaviors in much the same way that dataset composition shapes pretraining.

Long horizons couple systems and optimization

Long agent trajectories introduce temporal staleness into the training system. Suppose a rollout begins under $ \pi_{\theta_{t-k}}$ and finishes after the learner has advanced to $\pi_{\theta_t}.$ The resulting importance ratio is

The lag $k$ becomes a variable in the optimization problem. Synchronous group methods can reduce this mismatch but waste computation when rollout durations vary greatly. Asynchronous execution improves utilization by processing completed trajectories immediately, while creating larger policy lag and stronger off-policy effects.

SAO can be understood directly from this tradeoff. GRPO relies on several rollouts for the same prompt to construct a group-relative baseline. Long-horizon asynchronous workloads make waiting for the entire group expensive, so moving to

removes the synchronization barrier. The group baseline disappears as a consequence, creating a reason to reintroduce a value function or another advantage estimator. The resulting algorithmic branch therefore follows from a concrete systems constraint: variable trajectory duration changes the optimal data-generation architecture, which then changes the credit-estimation problem.

This is one reason agentic RL increasingly blurs the boundary between optimization and systems design. Throughput decisions alter policy staleness, policy staleness alters importance ratios, and those ratios directly alter the gradient.

Policy identity itself becomes a systems variable

Large-scale RL introduces another source of mismatch. Rollout inference and gradient training may use different implementations, so even nominally identical model parameters can produce slightly different distributions:

Numerical precision, kernels, MoE routing, batching, and distributed execution can all contribute. It is therefore useful to distinguish deliberate policy evolution,

from implementation disagreement,

The first reflects optimization progress. The second reflects disagreement between systems intended to represent the same policy. Both matter because probability ratios enter directly into modern RL objectives.

GLM-5.3’s emphasis on train-rollout numerical alignment and R3-style MoE consistency fits naturally into this picture. Once the loss depends on log-probabilities and importance ratios, infrastructure quality becomes part of optimization correctness rather than a separate engineering concern.

IV. From scalar rewards to distributional feedback

On-policy distillation increases feedback bandwidth

A scalar reward compresses a high-dimensional trajectory into a small amount of information. Even a perfect binary verifier may provide only $ R\in{0,1}.$ A strong teacher model can instead provide an entire probability distribution over actions. At state $s$, let the teacher define $q_T(v\mid s)$ over the vocabulary.

Traditional distillation usually trains the student on states or outputs selected by the teacher. On-policy distillation changes this by allowing the student to generate the states:

The teacher then evaluates the states the student actually visits. A natural objective is

This construction combines the state distribution of on-policy RL with the dense feedback of knowledge distillation. For a sampled action, a teacher-derived learning coefficient can take the form

which again yields an update proportional to

The familiar policy-gradient-shaped structure therefore survives, while the feedback channel becomes much richer. Top-$k$, full-vocabulary, and multi-teacher variants mainly determine how much of the teacher’s distributional information is retained.

This suggests a broader progression in feedback bandwidth. Demonstrations provide one target behavior, preferences provide a relative ordering, rewards provide scalar evaluation, and teacher distributions provide dense local information over many possible actions.

V. Forward and reverse KL as the geometric unifier

After deriving the main branches from their local motivations, Lambert’s forward and reverse KL framing provides a clean geometric interpretation of the whole picture. This unified framing reminded me: if SFT is Columb’s law, then DPO is potential difference (or voltage) and KL-regularized RL is the free-energy minimization.

For SFT, let the target data distribution be

The forward KL is

Expanding,

The first term is independent of $\theta$, so minimizing the SFT cross-entropy is equivalent to minimizing

The expectation is taken under the data distribution. Training therefore occurs in regions selected by the target distribution, which produces the familiar mass-covering tendency of forward KL.

KL-regularized RL has the opposite orientation. Starting from

define the reward-tilted target

Then maximizing the RL objective is equivalent to minimizing

Here the expectation is taken under the policy itself. The policy therefore selects the regions in which optimization takes place, while the reward reshapes probability within those regions.

This gives a compact interpretation of the major transition from supervised learning to on-policy optimization. Forward KL naturally corresponds to externally sampled imitation, while reverse KL in KL-regularized RL corresponds to policy-sampled optimization toward a reward-tilted target.

The exact reverse-KL identity depends on the KL-regularized RL objective, so it should not be applied mechanically to every modern RLVR algorithm that weakens or removes explicit KL regularization. The more general lesson is the change in sampling geometry: one regime learns on states selected by an external distribution, while the other learns on states selected by the current policy.

This was one of the most useful ideas I took from Lambert’s treatment because it connects several branches that otherwise look unrelated. SFT and classical knowledge distillation naturally live on the forward-KL side. KL-regularized RL lives on the reverse-KL side. DPO derives its target structure from the reverse-KL RL solution while retaining offline preference data. On-policy distillation moves distillation itself onto the student distribution and therefore naturally takes a reverse-KL form.

The direction of KL is therefore closely related to a more fundamental question: which distribution determines where learning happens?

VI. A minimal physicist’s model of post-training

The previous sections suggest a generic update of the form

The variables in this expression correspond to the major conceptual branches developed above.

The distribution $\mu$ determines where training experience comes from. In SFT it is an external dataset, in offline preference optimization it is a preference dataset, and in online RL it is the current or recent policy interacting with a task or environment.

The coefficient $w_t$ represents the learning signal. It may come from a supervised target, a preference margin, an advantage estimate, a verifier, or a teacher distribution.

The factor $c_t$ controls how much the current policy should trust the sample. Importance weighting, clipping, asynchronous correction, and train-rollout consistency all enter here.

Agentic training adds the environment dynamics

which determine how actions change future states and therefore which experiences become reachable.

This provides a compact way to read new post-training papers. A method that changes the reward source primarily modifies $w_t$. A new advantage estimator changes the construction of $w_t$. A new clipping or importance-sampling scheme primarily modifies $c_t$. A new rollout strategy changes $\mu$. Environment generation changes the trajectory distribution itself. Asynchronous training and inference-training mismatch change the relationship among these quantities.

Seen this way, many modern algorithms become local deformations of a shared optimization structure rather than separate theories.

Closing thought

The Coulomb’s law analogy is useful to me because it imposes a discipline on how to study the field. Start from the simplest equation, identify the missing physical or statistical structure, then add only what is necessary.

Cross-entropy handles demonstrated likelihood. Preference learning adds relative quality. RLHF turns that relative quality into a constrained policy objective. DPO follows from the analytical form of the KL-regularized optimum. Policy gradients move the source of experience onto the model itself. PPO adds control over stale updates. RLVR improves the grounding of reward. GRPO changes the construction of advantage. Agent RL introduces environment dynamics. Asynchronous long-horizon training introduces policy age and off-policy effects. SAO responds to the resulting synchronization problem. Train-rollout alignment exposes numerical implementation as part of the optimization system. On-policy distillation increases the bandwidth of feedback from scalar reward toward a full distribution.

Lambert’s work provides an excellent engineering map of this territory. The first principle complement I find useful is to reduce that map back into a small number of recurring objects and ask how each new method changes them.

My current summary is therefore compact: modern LLM post-training is the study of where experience comes from, how quality is measured, how credit is assigned, and how safely probability can move. For agentic systems, the dynamics of the environment become an equally fundamental part of the problem.

Once those objects are explicit, the growing collection of post-training algorithms becomes much easier to place inside one framework. I highly recommend Lambert’s book and lecture series for anyone who wants to understand the engineering of modern LLMs. I hope this complementary perspective is useful for those who want to understand the underlying structure of the field.

I am getting tired of tracing all the so-called agent harness engineering techniques. You must feel the same.

First, the answer was better prompts. Then it was tool use. Then came memory, reflection loops, graphs, planners, evaluators, multi-agent teams, and swarms. Each technique has its own terminology, diagrams, frameworks, and success stories. Each fixes something, yet each also creates another class of failures that requires another layer of orchestration.

After a while, the field begins to resemble a growing collection of patches whose names change faster than the problem underneath them.

I kept seeing the same pattern. A loop was presented as the solution to brittle one-shot agents. A graph was introduced to control an unreliable loop. A swarm was proposed to overcome the limits of one planning path. Then an evaluator, memory layer, or deterministic solver was added to control the failures introduced by the previous layer.

Rather than continuing to trace every new framework, I wanted to understand the first principle. What failure are all these harnesses trying to contain, and why does that failure exist in the first place?

The answer became clearer when I stopped thinking of the LLM as a weak state machine and started from what it actually does. An LLM continues trajectories, while an agent is expected to control them. The gap between those two functions explains much of modern agent engineering.

How a small mistake becomes a trajectory

Imagine asking a research agent to investigate a technical question and publish a report.

The original request contains one ambiguous sentence. The agent chooses a reasonable interpretation, generates search queries from that interpretation, and retrieves sources that appear to support it. It summarizes those sources into memory. A later agent receives the summary rather than the original evidence, and an evaluator reads the same framing and approves the final report.

No individual step needs to look absurd. The first interpretation may have been highly plausible. The retrieved evidence may have been real. The summary may accurately reflect the selected sources, and the evaluator may correctly judge that the report is internally coherent. The final result can still be wrong because of a discrepancy that entered five steps earlier.

This is the characteristic failure of LLM-orchestrated systems. A local discrepancy becomes part of the next agent state, the next decision is conditioned on that altered state, and the discrepancy gains persistence, confidence, and eventually consequence as the trajectory continues.

The same pattern appears in more consequential tasks. An agent misreads the intended recipient and drafts the correct message for the wrong person. Another agent verifies the wording without checking the recipient, and a final tool call sends it. A coding agent misunderstands one architectural constraint and writes tests that encode its own misunderstanding. The tests pass, so their success becomes evidence that the implementation is correct. A tool request times out after the operation succeeds, and the agent interprets the timeout as failure, retries, and performs the action twice.

In each case, a probable local continuation becomes future context, future context becomes accepted state, and accepted state eventually becomes an external consequence.

A simple way to describe this dynamic is:

Here, $\delta_t$ represents the discrepancy already present in the trajectory, $\epsilon_t$ is the new error introduced at the current step, and $L_t$ describes how strongly the surrounding system propagates the previous discrepancy.

Unrolling the recurrence shows how every local error is weighted by the transitions that follow it:

This makes an important point precise. A longer trajectory is not automatically less reliable. The horizon matters through the product of the later amplification factors. A long coding session can remain stable when compilers, tests, version control, and environmental observations repeatedly pull the agent back toward reality. A two-step payment workflow can be unstable when one transition has a large and irreversible consequence.

When the effective values of $L_t$ stay below one, discrepancies contract. A compiler reports the actual syntax failure, a constraint solver rejects an impossible plan, a transaction receipt establishes what happened, or an independent source challenges a mistaken premise.

When the surrounding system simply preserves previous outputs, discrepancies remain. When the model consumes its own summaries, agents repeat one another’s claims, or actions irreversibly alter the environment, discrepancies can grow.

Agent reliability therefore depends on more than local model quality. It depends on the discrepancy dynamics created by the orchestration around the model.

Continuation is a proxy for control

A language model learns a conditional distribution over plausible continuations. Given the trajectory represented in its context, it proposes what may come next:

The history $h_t$ may contain the user request, previous reasoning, tool calls, observations, retrieved documents, memory summaries, and messages from other agents.

Large-scale training gives the model compressed patterns of successful planning, expert decisions, software workflows, tool use, correction, negotiation, and explanation. Post-training further shapes those continuations toward useful, safe, and instruction-following behavior.

This makes continuation an extraordinarily powerful proxy for control. When an agent encounters a familiar problem, it can generate the next step associated with successful examples of that problem. It can produce a plan, call a tool, inspect the result, and continue in a way that resembles an effective problem-solving trajectory.

A controller faces a different objective. It must choose an intervention according to the future world states that intervention is expected to produce:

The distinction between $h_t$ and $s_t$ is central. The model conditions on a representation of the trajectory, while the action affects the actual world state. The representation may be incomplete, stale, ambiguous, or partly generated by the model itself.

A tool timeout may correspond to several different realities. The operation may have failed, succeeded before the connection closed, or succeeded twice after a retry. The visible trajectory may preserve none of these distinctions.

The model’s actions also change the environment, the changed environment produces the next context, and earlier outputs become later inputs. Once an error enters the trajectory, the model may be reasoning from a situation partly created by its own mistake.

This creates the learning–control gap. Learning rewards strong average predictions across a distribution of observed examples, while control requires stability along the particular sequence of states produced by the policy’s own actions. High average next-step accuracy cannot guarantee that one self-generated trajectory will remain close to the intended path.

A second gap appears at the boundary between probability and consequence. A small change in model output can select a different tool, recipient, branch, or commit operation. The probability distribution changes smoothly while the external world changes discontinuously. A model may be nearly indifferent between drafting and sending even though the consequences of those actions are radically different.

Model improvement reduces the frequency of local discrepancies. Harness engineering determines how far each discrepancy can travel and how much consequence it can acquire.

This gives us the first principle: an agent harness controls the conversion of conditional continuation into persistent state and causal consequence.

What the current harness techniques are doing

Once we see the root problem, loops, graphs, and swarms become easier to compare because each architecture changes how discrepancy moves through an agent trajectory.

The naive agent minimizes propagation surfaces

A simple agent creates a short path from model output to tool execution. Its simplicity limits handoffs, coordination failures, shared-memory contamination, and ambiguous responsibility, which makes it a strong design for short, bounded, easily verified tasks.

Its weakness appears when one incorrect continuation reaches the world without an opportunity for correction. The central measure for a naive agent is consequence sensitivity: how much can one model decision change the final outcome?

The loop creates opportunities for correction

A loop allows the agent to act, observe, and update its next decision. Its value comes from the possibility that reality can challenge the model’s prediction. A test failure, search result, tool receipt, or changed environment can introduce information that was absent from the original continuation.

The same structure becomes dangerous when the loop repeatedly feeds the model its own claims, retries actions without tracking previous effects, or continues without measurable progress. More iterations then produce more correlated continuations rather than better grounding.

The central measure for a loop is correction per iteration: how much uncertainty or discrepancy does each cycle remove?

The graph limits where discrepancy can travel

A graph makes some transitions possible and others unreachable. It can enforce that testing happens before deployment, approval happens before payment, and verification happens before completion. Its strength comes from controlling the topology of influence.

Its weakness appears when the edges carry semantically ambiguous content. A clean arrow can still transmit an unsupported claim, a distorted summary, or a false assertion of authority.

The central measure for a graph is invalid-path exclusion: which dangerous trajectories have become structurally impossible?

The swarm expands search and changes error correlation

A swarm explores several hypotheses, sources, plans, or implementations in parallel. It can improve coverage when agents search different regions or use genuinely different methods.

The benefit depends on independent information. Agents using the same model, context, evidence, and shared memory often reproduce the same misconception. Their agreement can increase confidence while adding little truth. A mistaken claim can also circulate through the swarm until repetition appears to be confirmation.

The central measure for a swarm is independent information gain: what does each additional agent contribute beyond another correlated sample?

Other agent variants fit into the same picture. Hierarchies reduce coordination entropy while concentrating risk in a manager. LLM planners explore longer candidate trajectories while leaving feasibility uncertain. Deterministic solvers remove paths that violate explicit constraints while remaining dependent on the truth of their premises. Memory preserves state while creating the possibility that an old model-generated summary becomes permanent reality. Evaluators regulate which outputs are accepted while risking the replacement of evidence with another model opinion.

Each technique acts on a different transition in the same process.

What understanding the first principle changes

Once an agent is understood as a sequence of state transitions, the architecture question changes. Instead of beginning with whether the system should use a loop, graph, or swarm, we begin with where discrepancy can enter, where it becomes persistent, what can amplify it, what can correct it, and how much consequence it can acquire before correction.

That shift leads to several practical changes.

Engineer the joint trajectory, not isolated decisions

An agent succeeds only when its interpretations, plans, tool calls, observations, and state updates remain jointly correct.

Let $C_t$ denote the event that transition $t$ remains correct relative to the task, its constraints, and the real world. The probability that the entire trajectory remains correct follows the chain rule:

Even under the optimistic assumption that transitions are independent and each is correct with probability $p$, complete trajectory reliability becomes $p^T$. If ten decisions were independently correct 95 percent of the time, the probability that all ten were correct would be only about 60 percent.

Real LLM trajectories have stronger dependencies. An early output becomes part of the next context, so a mistaken interpretation can lower the probability that every later decision is correct. A corrupted memory entry can influence several downstream agents, while a shared false premise can cause an entire swarm to fail in the same direction.

Once an early discrepancy changes the trajectory, it also changes the conditional distribution governing every later decision.

This dependency structure is the topology of the agent system. A loop determines whether an earlier discrepancy receives corrective evidence or returns as input to the same model. A graph determines which later decisions become conditional on that discrepancy. A swarm determines whether agents contribute independent information or inherit a common error through shared context and memory.

Harness design therefore shapes the joint probability of trajectory success even when the underlying model remains unchanged. The objective is to reduce unnecessary dependencies, prevent unverified outputs from becoming shared premises, introduce evidence that can restore a diverging trajectory, and keep any remaining discrepancy from reaching high-consequence transitions.

Preserve the meaning of state

A model-generated claim, an external observation, a verified fact, a user authorization, a proposed action, a committed action, and a confirmed outcome should remain different objects.

The model can say that a payment was approved, yet that sentence should remain a claim until it is connected to a valid approval event. The model can predict that an email was sent, while that prediction should remain separate from the tool receipt proving that the send operation occurred. A memory summary can help future agents work efficiently, although it should retain references to the evidence from which it was derived and should never silently replace that evidence.

This separation creates epistemic integrity because language can propose a change in state without automatically creating it.

Build restoring forces into the trajectory

A reliable harness needs mechanisms that pull the agent back toward the real world. A compiler error can correct generated code, a deterministic constraint check can reject an inconsistent plan, an independent source can challenge an interpretation, a transaction receipt can establish whether an action occurred, and a verified postcondition can establish whether the goal was reached.

These mechanisms introduce information that did not originate from the same continuation process. Every loop should therefore be evaluated by whether new evidence makes the next iteration better grounded than the previous one.

Bound the consequence of residual error

Even strong models retain residual uncertainty, so a safe architecture reduces the amount of the world that each stochastic decision can change.

A general email tool may allow arbitrary recipients, attachments, and message bodies. A narrow capability may allow one approved draft to be sent to one approved recipient before a fixed expiry time. A deployment agent may prepare changes freely while requiring a separate commit capability for production. A financial agent may analyze opportunities broadly while receiving permission to execute only within a specific amount, account, and time window.

The practical goal is to separate what the model may propose from what the system will permit it to execute. Training can reduce the probability of a bad proposal, while a capability boundary can keep unauthorized or excessively broad actions outside the executable action space.

This reduces consequence amplification. The model can still make an incorrect proposal, while that proposal cannot produce an unlimited external effect.

Measure trajectory stability

A successful run shows that one trajectory worked, although it does not reveal whether the system is stable.

A stronger evaluation perturbs the initial wording, context order, model sample, memory summary, tool response, or timing, then measures how often the final outcome changes materially. This reveals whether the harness corrects small discrepancies, preserves them, or amplifies them.

Task success measures capability. Trajectory sensitivity measures whether that capability remains reliable when the exact continuation changes.

Use deterministic machinery at explicit boundaries

LLMs are valuable when the system must interpret ambiguous intent, infer meaning, search broad possibilities, or generate candidate solutions. Deterministic systems become valuable when known constraints must hold simultaneously.

A planner or solver can enforce budgets, dependencies, scheduling rules, permissions, mutual exclusions, and completion requirements. It can project a probabilistic proposal back into a valid region, while the facts supplied to that solver still require grounding because a deterministic system can propagate a false premise with perfect consistency.

A useful division of responsibility gives probabilistic models the semantic uncertainty, deterministic machinery the explicit combinatorial constraints, and environmental verification the task of establishing actual consequences.

Add agents when they add information

A second agent should bring a different source, method, tool, model, search region, or adversarial objective. A new persona reading the same context often adds ceremony rather than diversity.

Shared observations and authoritative state can be common across agents, while intermediate interpretations should remain contestable. Agreement can indicate stability, yet truth promotion should depend on evidence.

This creates search breadth without allowing repetition to become authority.

From framework recipes to causal harness engineering

The major harness techniques become easier to understand once we place them inside one lifecycle. The model first interprets ambiguous intent and proposes possible actions. The harness then checks those proposals against known constraints and determines whether the required authority exists. It executes one bounded intervention, observes what actually happened, verifies whether the expected state was reached, and updates the persistent state used by future decisions.

Loops are useful when observation can correct the next decision. Graphs constrain which influences and transitions are permitted. Swarms broaden the search when additional agents contribute genuinely independent information. Planners generate candidate trajectories, solvers enforce explicit consistency, memory preserves grounded state, evaluators regulate which claims become accepted, and capability boundaries determine which proposals can change the world.

Each stage answers a different question about meaning, possibility, validity, permission, consequence, or evidence. Treating those questions as separate system responsibilities prevents one plausible model continuation from silently answering all of them.

The field has spent the last few years inventing shapes around the model: loops, graphs, hierarchies, and swarms. Those shapes matter because they change the joint conditional probability of the trajectory and determine how discrepancy travels.

A loop can introduce a restoring force or recycle the same misconception. A graph can block an invalid dependency path or hide semantic confusion behind clean arrows. A swarm can contribute independent evidence or multiply a shared mistake. A solver can enforce consistency while faithfully amplifying a false premise.

The common problem sits underneath all of them. We trained a model to continue trajectories, then connected those continuations to systems that change the world.

Agent harness engineering begins when we ask which state transitions remain grounded, which dependencies improve or degrade the joint probability of success, which discrepancies contract, and which decisions are allowed to acquire consequence.

That is how continuation begins to become control.

Main-conference poster sessions reward planning. Workshop sessions reward luck.

A workshop room can be three corridors away from the topic you intended to see, and the poster that changes your afternoon may be the one beside the coffee table rather than the one highlighted in the program. This third ICML reflection is limited to workshop papers I encountered through that kind of fortunate wandering.

The papers span theoretical physics, post-training, foundation models for simulation, Transformer inductive bias, model evaluation, interpretability, and multi-agent diagnosis. Their common question emerged only after I stopped grouping them by application:

When an AI system encounters an error or a new environment, what should change, where should that change be stored, and what evidence tells us that the change was useful?

Learning is often described as parameter optimization. These workshop papers suggest a broader view. An AI system can change its current answer, its dialogue state, its inferred model of the environment, a selected subspace of its weights, or the interaction protocol among several agents. Every update preserves some structure and overwrites something else.

A useful update therefore balances three quantities:

Here, (s) is the current state of the system, (\mathcal{L}{\mathrm{new}}) measures the unresolved error, (\mathcal{D}{\mathrm{preserve}}) measures damage to valuable existing structure, and (\mathcal{C}) represents the cost of the update.

The hard research problem lies in defining those terms. A critique can improve the current derivation while making the overall argument less coherent. Fine-tuning can raise benchmark accuracy while erasing general capabilities. A physics model can produce a plausible next frame while learning the wrong dynamics. A lower perplexity can reward confidence without rewarding correctness. A group of agents can agree because they share the same error.

The workshop papers became a study of selective change: changing the right variable, on the right timescale, while preserving the right invariants.

Critique changes the solution before it changes the model

When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR studies an Actor–Critic–Judge loop on graduate-level quantum field theory and string theory problems. An Actor proposes and revises a solution, a Critic provides feedback with access to a reference solution, and an independent Judge evaluates the successive attempts. The authors vary Actor personas, Critic styles, model families, and model scales. Multi-turn interaction generally improves on single-shot attempts, while the size and character of the gain depend strongly on the Actor–Critic pairing. Constructive feedback is particularly useful when a weaker Actor is paired with a stronger Critic; strict or adversarial feedback provides no universal advantage. (arXiv)

The important object is the response of the solution to feedback:

where (S_t) is the quality of the solution after revision (t). A capable Critic can identify the correct error and still produce little gain when the Actor cannot interpret or incorporate the feedback. A milder Critic may outperform a more aggressive one by locating a repair within the Actor’s current competence.

This resembles educational scaffolding more than debate. A useful teacher does not enumerate every flaw simultaneously. The teacher identifies the next misconception that the student can repair. The value of feedback therefore depends on its correctness, novelty, timing, and usability.

SCALAR also offers an instructive evaluation method: track the sequence of revisions rather than only the final score. Two systems with identical final performance can have very different internal dynamics. One may repair a decisive error after a single critique. Another may oscillate among alternative mistakes. A third may repeatedly rewrite the answer while changing little of substance.

For scientific agents, this suggests measuring critique susceptibility: how efficiently a system converts feedback into improved work. When gains flatten, the appropriate response may be to switch Critics, decompose the problem, invoke a symbolic tool, or ask for human input. Continuing the same dialogue merely generates more tokens.

The broader lesson is that many failures should be repaired in the current artifact before they trigger a model update. A wrong sign in a derivation calls for a local correction. A systematic inability to reason about gauge invariance may justify a deeper change. The skill lies in distinguishing the two.

Context can carry learning without changing weights

Towards a Physics Foundation Model moves adaptation from dialogue into latent state estimation. Its General Physics Transformer is trained on approximately 1.8 TB of simulation data covering several fluid, thermal, multiphase, and fluid–solid systems. Given a short history of physical states, the model infers enough of the local dynamics to predict future states, including zero-shot experiments on unseen physical systems and boundary conditions. (arXiv)

The task is naturally expressed through a latent dynamics variable (z):

The model observes a short trajectory, infers a local description of how this world evolves, and uses that inferred state to generate the next step. The weights remain fixed while the effective model changes through context.

This is closely related to system identification in control theory. An engineer observes inputs and outputs, estimates the hidden dynamics, and then predicts or controls the system. A physics foundation model amortizes that procedure across many simulated worlds.

The architecture also contains an important scientific design choice. The Transformer estimates a temporal derivative, while a numerical integration step advances the state. Spatial and temporal derivative features provide additional numerical structure. The learned component handles cross-domain inference; the integration interface supplies a stable computational scaffold. (arXiv)

This hybrid structure is more inspiring than a slogan about learning physics from raw data. It suggests that general scientific models may arise from broad learned representations connected through well-chosen numerical interfaces. A neural component can infer the regime, parameters, or operator, while established solvers enforce part of the update structure.

Variable time increments are another methodological gem. When all trajectories share the same sampling interval, visual displacement can become a shortcut for temporal dynamics. Varying the interval forces the model to infer time scale from the context. Similar experiments could be useful in language-model research: change surface format, naming, scale, or tokenization while preserving the underlying operation, then test whether the model infers the mechanism or recognizes the dataset.

The central unresolved question is what kind of latent model has been inferred. The context may identify a familiar simulation family, support a flexible local interpolation, or recover reusable abstractions such as boundary type, conservation structure, or effective dynamics. Carefully constructed pairs of visually similar trajectories governed by different hidden laws could distinguish these explanations.

For AI research agents, the analogy is direct. A mechanistic-interpretability agent should infer a local intervention law from a small set of experiments. It should estimate thresholds, dose–response shape, generalization across prompts, and failure of perturbative assumptions before proposing the next intervention. Most new experimental objects should be handled through contextual system identification long before they motivate weight updates.

Fine-tuning should protect the structure that future learning needs

Rotation-Preserving Supervised Fine-Tuning studies selective change in parameter space. Standard SFT improves the target task while often reducing out-of-domain performance. RPSFT penalizes changes within dominant singular subspaces of the pretrained weight matrices, using those subspaces as an efficient proxy for directions that are expensive to disturb. The complementary space remains available for task adaptation. Across math-reasoning fine-tuning experiments, the method improves the in-domain/OOD trade-off, better preserves pretrained representations, and produces stronger initializations for later reinforcement learning. (arXiv)

The method embodies the stability–plasticity dilemma from continual learning and neuroscience. A system needs enough plasticity to acquire a new capability and enough stability to retain its reusable organization. Uniformly constraining the entire model wastes available degrees of freedom. Unconstrained adaptation allows a local training objective to rotate broadly useful machinery.

RPSFT asks a more precise question: which subspaces should carry the new task?

The dominant pretrained directions are a structural proxy rather than a complete map of capability. Their value comes from being computationally tractable and empirically associated with Fisher-sensitive gradient energy. The methodological pattern is broadly reusable: identify an expensive behavioral object, propose a cheaper structural proxy, verify that the proxy captures the relevant signal, and constrain only the corresponding degrees of freedom.

The downstream reinforcement-learning result deserves special attention. A checkpoint can look good immediately after SFT and provide a poor starting point for the next training stage. Preserving pretrained geometry can improve the future learning trajectory even when the current checkpoint difference looks modest.

This expands the meaning of update quality. A good adaptation should solve the present task, preserve unrelated capabilities, and maintain useful plasticity for future stages.

For alignment research, the next question concerns which behavioral mechanisms occupy the protected space. Refusal, truthfulness, self-report, and reasoning may share dominant pretrained geometry, or they may rely on fragile structures introduced later through post-training. Average OOD retention cannot distinguish these possibilities. Causal and representational measurements across ordinary SFT and RPSFT could reveal which alignment mechanisms survive because their supporting subspaces remain stable.

Weight space can become an observable

Weight-Space Physics offers the most unusual interpretability idea in this workshop collection: treat trained network weights as scientific measurements of the system they model.

The paper studies normalizing flows trained for lattice field theories across different coupling constants. JEPAWG maps physical couplings through a learned latent space into flow-network weights. The latent weight geometry recovers the intrinsic dimensionality of the coupling manifold, identifies the phase-transition region, and exhibits a finite-size shift aligned with the expected two-dimensional Ising scaling exponent. The model also generates effective flow weights at unseen coupling values. (arXiv)

Normally, a learned sampler is a tool for producing configurations from which physical observables are estimated. This work asks whether the parameters of the tool have themselves absorbed the structure of the physical theory.

That reversal creates an unusually clean interpretability benchmark. Physics provides known coupling dimensions, phase structure, and finite-size scaling. An interpretability method can therefore be tested against independently established ground truth rather than judged by whether its visualizations appear meaningful.

The cross-disciplinary inspiration is statistical physics. A phase transition captures a qualitative change in macroscopic behavior. An order parameter summarizes a regime. Finite-size scaling helps distinguish a genuine structural transition from a small-system artifact. Renormalization describes relationships preserved across scales.

Model-family interpretability could borrow the same logic. Consider a controlled family of language models trained with varying amounts of safety data, preference strength, harmful fine-tuning, or reasoning supervision. The checkpoints may form a low-dimensional weight-space manifold. Behavioral transitions could appear as geometric changes along that manifold, while scaling across model sizes could test whether the transition reflects stable structure.

This shifts interpretability away from isolated checkpoints. A single neuron in a single model can be an implementation detail. A consistent manifold, transition boundary, or scaling law across a controlled model family provides a stronger scientific object.

The paper also complicates the usual distinction between learning and measurement. Training a network at each physical coupling creates a measurement instrument whose parameters reflect the underlying system. The hypernetwork then learns the geometry of those measurements. Weight space becomes a bridge between simulation, representation learning, and scientific inference.

Inductive bias determines which differences survive learning

Transformers Learn Low Sensitivity Functions studies the kinds of functions Transformers tend to learn. Across vision and language experiments, the authors find that Transformers learn functions with lower sensitivity to token- or patch-level random perturbations than several alternative architectures. Lower sensitivity correlates with robustness, flatter loss regions, and progress during grokking. (arXiv)

For a Boolean function, sensitivity can be written as

A low-sensitivity function maps many neighboring inputs to the same output. The model compresses away local distinctions that do not appear necessary for its learned rule.

This bias explains part of Transformer robustness. It also creates a boundary between useful invariance and harmful blindness. In sentiment classification, one irrelevant token should rarely change the answer. In parity, exact arithmetic, source code, or cryptographic verification, a single symbol can determine correctness.

A follow-up line of research shows how simplicity bias can trap Transformers in an incorrect low-sensitivity rule under noisy features, even when the intended target is learnable. The architecture’s preference becomes helpful only when its favored invariances match the causal structure of the task. (arXiv)

This connects to the broader theme of selective change. Learning requires the model to decide which input differences deserve a different output. Low sensitivity says that Transformers often require stronger evidence before introducing a distinction.

Average sensitivity can still hide sparse high-leverage directions. A safety behavior may be stable under thousands of random edits and flip under one authority cue, role token, encoding convention, or activation direction. This motivates conditional sensitivity measurements targeted at task-relevant transformations rather than isotropic noise.

The architecture should ignore nuisance variation and remain responsive to causal variation. Determining which is which remains a property of the task, the data, and the measurement procedure.

A learning system can optimize the wrong evidence

Perplexity Cannot Always Tell Right from Wrong asks whether the metric used to judge learning preserves the ordering that actually matters.

In a simplified binary setting with accuracy (a) and confidence (\gamma), the average negative log-likelihood is

Equal-perplexity curves in the accuracy–confidence plane show that a more accurate model can receive worse perplexity when confidence changes in the wrong proportion. The paper also uses continuity properties of compact decoder-only Transformers to prove the existence of sequences that the model predicts incorrectly while assigning very low perplexity. (arXiv)

A structurally decisive error can disappear inside an average over many easy token predictions. The model may copy almost an entire sequence confidently and fail at the one position that determines task success. Perplexity remains low because the error occupies little of the total sequence mass.

The result becomes more consequential when the same blind spot affects gradients. A metric that fails to reveal an error during evaluation may also provide little signal for correcting it during learning.

This paper supplies the logical foundation beneath the previous workshop results. Critique helps only when the Critic identifies the true error. Fine-tuning helps only when the objective values the capabilities we intend to preserve. A physics model learns useful dynamics only when one-step loss and rollout evaluation reward the right structure. An interpretability manifold means something only when the measured geometry corresponds to the physical or behavioral quantity of interest.

The cross-disciplinary connection is decision theory. Log-likelihood is a principled scoring rule for distribution estimation, while deployments usually assign highly unequal utility to different mistakes. One wrong tool call, medical diagnosis, proof step, or safety decision can dominate thousands of locally plausible tokens.

A useful metric must therefore match the structure of the claim. Exact reasoning needs exact or verifier-based outcomes. Physical simulation needs rollout stability and invariant preservation. Agent evaluation needs action correctness and consequence-weighted cost. Average likelihood remains informative, while its interpretation must stay tied to the task.

Agreement can preserve a shared mistake

When Agreement Becomes Unsafe studies multi-agent deliberation in diagnostic-style decisions and starts from a crucial observation: agreement does not certify safety. Agents can converge because they share evidence, reasoning, or bias. The work introduces loss-aware energy control so that the deliberation policy accounts for disagreement, asymmetric decision costs, and the risk of consensus on a costly alternative. The paper appeared as an oral at the ICML 2026 Workshop on Failure Modes of Agentic AI. (OpenReview)

This complements SCALAR. Critique studies whether one agent can improve another’s solution. Diagnostic deliberation asks whether interaction should continue, stop, or escalate when several agents appear to agree.

Consensus is valuable when errors are sufficiently independent and agreement reflects converging evidence. Consensus becomes dangerous when all agents inherit the same training bias, missing feature, or misleading observation. Adding more similar agents can increase confidence without adding information.

The correct mediator must reason about the loss landscape over possible decisions. In high-stakes classification, false positives and false negatives can have different consequences. An agreement on the costly error may deserve more scrutiny than a mild disagreement among low-cost alternatives.

The cross-disciplinary analogy comes from ensemble decision-making and safety-critical diagnosis. A panel of identical instruments does not create independent evidence. Diversity in model name, prompt wording, or sampling seed may leave the relevant blind spot untouched.

The design question is therefore deeper than how many agents should vote. A reliable system needs to know which evidence each agent contributes, which biases they share, and which outcomes justify abstention or escalation.

The center: learning across timescales

These workshop papers describe changes occurring at several levels.

SCALAR updates a current solution through interaction. A physics foundation model updates a latent description of the environment through context. RPSFT changes selected parts of weight space while protecting reusable structure. Weight-Space Physics studies the geometry created by whole families of trained models. Low-sensitivity research identifies an architectural bias governing which input distinctions survive learning. Perplexity analysis examines whether the objective can recognize improvement. Loss-aware deliberation asks whether a group’s agreement is trustworthy enough to act upon.

The timescales differ, yet the design questions remain aligned.

A local error should first be repaired in the current artifact. A changed environment may require a new latent model. A repeated capability gap may justify a weight update. A weight update should preserve the structure needed for generalization and future learning. Every level requires a metric that sees the consequential error, and every multi-agent system needs a way to distinguish new evidence from duplicated confidence.

This produces a hierarchy of adaptation:

Moving right increases cost and persistence. A capable AI system should use the cheapest level that can resolve the error while escalating when the lower level has reached its limit.

This framing also changes how I think about scientific AI. A research agent should not immediately fine-tune itself whenever an experiment disagrees with its prediction. It should first revise the derivation, update the local hypothesis, inspect the measurement, and run a discriminating experiment. Weight updates belong to capability gaps that persist across many contexts.

The workshops offered a quieter message than the main poster halls. Better intelligence may depend less on making every component stronger and more on assigning change to the correct place.

Learning is the art of changing the variables that caused the error while preserving the structure that will still be needed afterward.

Finding these papers required some luck. Recognizing that they were asking the same question required walking away from their application labels.

References

When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic–Actor Loop for Agentic Reasoning

Rotation-Preserving Supervised Fine-Tuning

Towards a Physics Foundation Model

Perplexity Cannot Always Tell Right from Wrong

Weight-Space Physics: Interpretable Hypernetworks for Lattice Quantum Field Theories

Transformers Learn Low Sensitivity Functions: Investigations and Implications

When Agreement Becomes Unsafe: Loss-Aware Energy Control for Diagnostic Deliberation

My first ICML reflection, The Model Chooses a Future Before It Says a Token, followed model behavior through time. It asked where a decision first becomes visible, when it begins shaping the trajectory, and how long an intervention can still redirect the outcome.

This second reflection stays with the main poster sessions and turns the view sideways. When text is rewritten, knowledge is unlearned, a model is fine-tuned, representations are rotated, or an internal state is compressed, what still counts as the same object?

Machine learning experiments often evaluate a snapshot: one prompt, one checkpoint, one output, or one detector score. Many properties become clearer when we study the family of states reachable from that snapshot.

Let $x$ represent a text, model, skill, activation, or internal representation, and let $\mathcal{T}$ be a family of transformations. The orbit of $x$ is

A useful notion of identity should keep benign transformations inside the same region,

while preserving separation from objects whose meaning, function, or origin has genuinely changed:

The difficult part is defining $\mathcal{T}_{\mathrm{benign}}$. A paraphrase may preserve the identity of an instruction. Refactoring may preserve the identity of a software component. Changing tool permissions can create a different security object even when most of the code remains identical. Fine-tuning may preserve model lineage while changing behavior that matters to users.

The posters in this reflection approach the boundary from different directions. Watermarking asks which ownership signal survives transformation. Unlearning asks whether information remains recoverable after an attempted removal. Representation alignment asks which structure remains stable after coordinates and models change. Compression asks which information may disappear while the required function survives.

The common question is simple to state and difficult to operationalize:

Which changes should preserve identity, and which changes should cross its boundary?

Provenance is a communication problem

The watermarking and provenance posters differed in technique, yet each one selected a channel through which identity was expected to survive.

LLM Watermark Evasion via Bias Inversion examines watermarks carried by token-selection statistics. Its Bias-Inversion Rewriting Attack estimates likely watermarked tokens from surprisal and suppresses them during rewriting. The paper reports that a small opposing logit bias can remove the detector signal across several watermark schemes while largely preserving semantic content. This exposes the fragility of identities attached closely to replaceable lexical choices. (arXiv)

dgMARK chooses a different carrier for diffusion language models. These models resolve masked positions iteratively, and practical systems remain sensitive to the order in which positions are unmasked. dgMARK steers that order toward positions whose likely tokens satisfy a keyed parity condition. The watermark therefore travels partly through the generation path. Its sliding-window detector is designed to retain signal under insertions, deletions, substitutions, and paraphrasing. (arXiv)

Neural Honeytrace treats provenance as communication through model extraction. The original model embeds a weak signal across ordinary interactions, while extraction and adaptation act as a noisy channel. Verification aggregates evidence over multiple queries, allowing a distributed statistical relationship to remain detectable even when individual outputs appear unremarkable. The paper’s information-theoretic framing makes the roles of signal strength, channel noise, and adaptive attackers explicit. (arXiv)

These methods suggest that provenance is fundamentally a coding problem. The owner chooses a carrier, the deployment process applies a channel, and the attacker searches for a low-cost transformation that destroys the signal. Token statistics, decoding trajectories, cross-query distributions, and internal mechanisms provide different capacities and different attack surfaces.

Rethinking Forgery Attacks on Semantic Watermarks makes the attacker’s cost explicit through rate-distortion geometry. A black-box attacker uses a proxy model to move an unwatermarked sample into a region accepted by a target detector. Structural mismatch between the proxy and target models creates global drift and local deformation, producing a distortion floor for successful forgery. (arXiv)

For a target identity region $\mathcal{R}$, define $\mathcal{A}(x,\mathcal{R})$ as the transformations that move $x$ into that region. The minimum forgery cost is

This formulation reaches beyond watermarking. A useful identity mechanism should tolerate inexpensive benign edits while forcing impersonation to pay a meaningful semantic, functional, or perceptual cost.

Before ICML: I tended to think of provenance as finding a sufficiently stable signature.

After ICML: provenance looks like channel design. The carrier, transformation family, attacker, detector, and distortion measure jointly determine whether identity survives.

This view connects naturally to my recent research focus on agent-skill identity, fuzzy fingerprints, and composition-aware security. An agent skill can change its wording, internal organization, or implementation language while preserving function and lineage. It can also retain nearly identical code while acquiring new permissions, side effects, or trigger behavior. A robust identity system therefore needs to distinguish surface variation from security-relevant change.

Unlearning is a recoverability claim

Unlearning asks the inverse question. A provenance signal should remain accessible after benign transformations. Forgotten information should remain inaccessible after recovery attempts.

Suppose an unlearning procedure changes model parameters from $\theta$ to $\theta’$. A decline in the original answer probability,

shows that the familiar query path has weakened. It leaves open how many alternative access paths remain.

A more informative object is recoverability:

where $K$ is the target knowledge or behavior and $\mathcal{A}$ is a declared family of recovery procedures. This family might include paraphrased queries, other languages, activation interventions, adapter training, limited relearning, model merging, or alternative decoding strategies.

Unlearning Isn’t Deletion shows that models can appear to forget under accuracy and perplexity measurements while recovering the original behavior after relatively little retraining. Its representation-level analysis uses PCA-based shifts, centered kernel alignment, and Fisher information to distinguish reversible suppression from deeper representational damage. (arXiv)

The result changes the meaning of a successful forgetting score. Forgetting is always relative to an access model. A user with prompts alone, a researcher with activation access, and an attacker with fine-tuning data face different recovery boundaries.

DualOptim+ approaches the forgetting-retention trade-off through optimizer state. The method maintains a base state for structure shared by the forget and retain objectives, together with delta states for objective-specific residuals. It adapts the balance between shared and separated states according to directional conflict between the gradients. (arXiv)

This suggests that forgetting depends partly on where optimization stores agreement and conflict. A fully shared state can entangle incompatible objectives, while complete separation can discard common structure that both objectives need.

A cleaner immediate trade-off still leaves the recovery question open. An optimizer can suppress the target response precisely while retaining a latent representation that is easy to reactivate. The strongest evaluation therefore needs two axes: collateral damage to retained capabilities and the adversarial cost of restoring the forgotten capability.

A recovery curve can express this distinction. Let $\mathcal{A}_b$ be the set of recovery procedures whose cost is at most $b$:

The shape of $R(b)$ reveals whether the target knowledge was deeply disrupted or hidden behind a thin access barrier. Rapid recovery under a small budget indicates suppression. Slow recovery under increasingly capable attacks provides stronger evidence of structural removal.

Before ICML: I read a large forget-set performance drop as strong evidence of forgetting.

After ICML: the stronger claim concerns the cost, breadth, and mechanism of recovery.

This perspective also clarifies why selective deletion is difficult in large models. A fact or behavior may share internal structure with broad linguistic and reasoning competence. Removing one behavior while preserving its neighbors resembles surgery on a distributed system with overlapping dependencies.

Representation identity must survive coordinate changes

A representation can preserve function while changing its coordinates. Hidden units can permute, bases can rotate, layers can redistribute computation, and independently trained models can implement similar behavior through different local features.

Multi-Way Representation Alignment studies alignment among three or more representation spaces. Pairwise mappings scale quadratically and can become mutually inconsistent. The paper uses Generalized Procrustes Analysis to construct a shared orthogonal universe that preserves internal geometry, then applies a geometry-correcting step to improve correspondence across models. (arXiv)

This creates an important distinction between preserving geometry and maximizing agreement. An orthogonal mapping may retain distances and angles within each representation while producing imperfect retrieval across models. A more flexible mapping may improve correspondence while distorting the structure one hoped to compare.

For mechanistic interpretability, geometric agreement alone remains incomplete. Similar representations can support different causal computations, and similar probe accuracy can emerge from different mechanisms.

A stronger test would map a behaviorally relevant direction from model $A$ into the shared universe, transfer it into model $B$, and measure whether the intervention changes the same behavior. A transferred refusal, language-routing, or reasoning direction that retains its causal effect would provide stronger evidence of shared mechanism than observational alignment alone.

This suggests a hierarchy of claims:

Output similarity is easy to observe and easy to imitate. Representation similarity adds evidence of internal correspondence. Causal transfer asks whether the transformed object still performs the same computational role.

The same issue appears in model lineage. Fine-tuning and merging can move individual parameters considerably while retaining higher-level dependencies. Behavioral fingerprints ask whether outputs remain similar. Representation alignment asks whether internal geometry remains corresponding. Mechanistic fingerprints ask whether inherited computations remain causally necessary.

The broader methodological lesson comes from geometry and physics: coordinates describe an object, while invariants identify what survives a change of coordinates.

Before ICML: I viewed cross-model alignment mainly as a way to compare probes and steering directions.

After ICML: shared coordinates look like a prerequisite for determining whether a mechanism or capability survived adaptation.

Compression asks what may safely disappear

Provenance asks which signal must survive. Compression asks which information can be removed while preserving the function that matters.

Let $c:X \to Z$ compress an input or internal state, and let $g$ operate on the compressed representation. The additional task loss introduced by compression can be written as

A task-preserving compression requires

while $Z$ uses fewer tokens, dimensions, channels, or computational steps than $X$.

When LLMs Develop Languages explores this idea through multi-agent communication. Agents develop compact symbolic languages with their own symbols, rules, and message-passing contracts. A router selects or combines these protocols according to task difficulty, reducing generated-token cost while preserving accuracy on the evaluated reasoning tasks. (arXiv)

The result suggests that natural-language reasoning contains substantial redundancy from the perspective of machine-to-machine communication. Human-readable prose supports explanation, negotiation, and repair. Agents sharing an interpreter can encode part of the same computational state in a smaller protocol.

The central question becomes what disappears during compression. Final-answer accuracy may survive while uncertainty, evidence provenance, alternative hypotheses, or safety constraints vanish from the message. A compact language can also become an opaque private channel whose semantics are difficult for humans or external verifiers to inspect.

A useful machine protocol therefore needs a richer preservation criterion than accuracy alone. It should carry enough state for downstream action, enough provenance for verification, and enough structure for local repair when the receiving agent detects an inconsistency.

BeaconKV asks a related question about inference memory. Its beacon queries represent global query clusters and guide which key-value cache entries should be retained. Cache compression must estimate which past states will matter for future generation before those future dependencies are fully known. (OpenReview)

This is temporal sufficiency. A state that appears unimportant at the current token may become decisive after a topic shift, delayed reference, or later tool result. An effective cache policy must preserve information whose value depends on futures that remain unresolved.

Robust Length Prediction adds a systems-level version of the same issue. The same prompt can induce a distribution of output lengths under a fixed model and decoding configuration, and this distribution may have a heavy tail. Predicting a single expected length hides the operational difference between a consistently moderate response and a prompt that usually ends quickly but occasionally triggers a very long trajectory. (arXiv)

Together, these papers point toward a shared principle: compression should preserve option value. The information worth retaining is partly determined by futures that remain possible.

Average-case objectives can remove rare evidence, delayed dependencies, or safety-critical state because these contribute little to typical performance. Compression for reasoning models and agents therefore needs stress tests built around low-frequency, high-consequence dependencies.

Transformation-aware evaluation

Every claim in this reflection depends on a metric. A detector says that a watermark survived. A benchmark says that knowledge disappeared. A similarity score says that representations align. An accuracy number says that compression preserved function.

Each metric should be evaluated under the transformation family that defines the claim.

A provenance method needs rewriting, paraphrasing, extraction, adaptation, and forgery tests. An unlearning method needs alternate prompts, languages, steering, relearning, and recovery tests. A representation-alignment method needs causal transfer across models. A compressed protocol needs rare-task, uncertainty-preservation, and safety tests. A cache method needs delayed dependencies and distribution shifts.

The general robust score can be written as

where $M$ is the relevant measurement and $\mathcal{T}_{\mathrm{claim}}$ contains the transformations under which the property is expected to hold.

The infimum matters because average performance can hide a cheap boundary crossing. A watermark may survive many local edits and fail under one strong rewriting method. An unlearned model may resist direct prompting and recover after a small adapter update. A compressed protocol may solve common tasks and lose one safety-critical distinction.

A static benchmark measures one location. A transformation suite maps the boundary.

This also provides a disciplined way to separate robustness claims. A method can be robust under paraphrasing, fragile under translation, and completely reversible under fine-tuning. Each statement can be true because each refers to a different transformation family.

The research object is an orbit

The first post ended with a temporal principle: find the earliest reliable decision point and intervene before the wrong future becomes difficult to reverse.

The second set of main-session posters adds a complementary principle. A checkpoint, output, skill, or representation is one sample from a larger orbit of transformations. The scientific object includes the orbit, the invariants that persist inside it, and the boundary at which identity or function changes.

For provenance, the orbit includes rewriting, extraction, and adaptation. For unlearning, it includes alternate access paths and recovery procedures. For representation analysis, it includes basis changes and model families. For compression, it includes smaller representations that preserve enough information for downstream use.

This perspective sharpens the meaning of robustness. A robust system should ignore variations that preserve the task and respond strongly to variations that alter its causal structure. Robustness therefore depends on knowing which transformations belong in each category.

My recent research focus on fuzzy identity, agent-skill provenance, and composition-aware security fits naturally into this view. Agent components rarely remain textually identical. Their meaningful identity lies in a combination of function, dependencies, permissions, side effects, and behavior under composition. The hard cases arise when surface similarity remains high while the security object has changed, or when surface similarity falls while lineage and function remain intact.

The resulting research program has three parts. We need to define the transformation family, identify the representation or mechanism that should remain stable, and search for the cheapest adversarial transformation that crosses the intended boundary.

That boundary is where identity, security, and interpretability meet.

The real object is the pattern that survives the transformations we are willing to call harmless, and the real attack is the cheapest transformation that changes the identity without paying the intended semantic or functional cost.

References

The Model Chooses a Future Before It Says a Token

LLM Watermark Evasion via Bias Inversion

dgMARK: Decoding-Guided Watermarking for Diffusion Language Models

Neural Honeytrace: A Robust Plug-and-Play Watermarking Framework against Model Extraction Attacks

Rethinking Forgery Attacks on Semantic Watermarks in Black-Box Settings: A Geometric Distortion Perspective

Unlearning Isn’t Deletion: Investigating Reversibility of Machine Unlearning in LLMs

DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models

Multi-Way Representation Alignment

When LLMs Develop Languages: Symbolic Communication for Efficient Multi-Agent Reasoning

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions

ICML is an excellent fitness program disguised as a machine learning conference. My phone recorded roughly 20,000 steps a day, so this review has a natural methodological limitation: it covers eight poster sessions and is bounded by my walking distance.

I filtered those sessions through the questions closest to my research: how aligned behavior is represented, how post-training changes model internals, how safety mechanisms fail under distribution shift, how agents decide when to act, and how interpretability can move from explaining behavior toward changing it.

One idea kept resurfacing:

A model often chooses a future before the final answer makes that choice visible.

The commitment may happen through an internal feature, an expert route, a reasoning branch, a retrieved passage, a social-response channel, or a tool action. I use margin loosely here to mean the model’s relative preference between such competing futures: one reasoning branch over another, refusal over compliance, correct evidence over a distractor, or a safe action over an unsafe one.

Before ICML, I often asked which feature, neuron, or direction controlled a behavior.

After eight poster sessions—and many corridors between them—the more useful question became:

Where does the decision first become visible, when does it become causal, and is there still time to intervene?

1. Reasoning, steering, and interpretability

Thinking helps when it improves the next decision

Several papers converged on a view of reasoning as a sequence of local decisions rather than a monolithic chain of thought.

How does Chain of Thought decompose complex tasks? models reasoning as the decomposition of a difficult classification problem into a tree of smaller ones. The analysis produces an optimal depth: shallow reasoning leaves the original decision too hard, while excessive depth accumulates new opportunities for error. SmartThinker reaches a related conclusion from the training side. A fixed length penalty can compress difficult solutions too aggressively, so the desired reasoning budget should depend on the problem and on the distribution of successful response lengths. (arXiv)

For a reasoning state ($h_t$), the object I care about can be written as

The branch might be a mathematical subgoal, a correction to an earlier assumption, a safe continuation, or a decision to verify rather than guess. A useful reasoning step improves the future decision landscape. Overthinking begins when new steps add low-margin branches without resolving the old uncertainty.

SafeThink made this temporal view particularly concrete. Its main empirical result is that many unsafe reasoning trajectories can be redirected with a short corrective prefix during the first few reasoning steps. The important object is therefore not merely whether the final answer is safe, but how early the trajectory remains recoverable. (arXiv)

This resembles a classical feedback-control problem. The intervention should arrive after enough evidence of drift has appeared, yet before the trajectory has settled into a harmful basin. Heavy intervention everywhere wastes control effort and increases collateral effects; late intervention edits the surface after the decisive branch has already been taken.

Sparse routes carry large behavioral effects

The same structure appeared inside model architectures.

Sparse Models, Sparse Safety shows that MoE safety can depend on a small set of router decisions: manipulating a few safety-critical routers can redirect computation toward unsafe routes. TraceRouter extends the routing view beyond explicit MoE routers by tracing harmful semantic influence through cross-layer feature paths. (arXiv)

Attention provided another version of the story. Surgery links harmful fine-tuning to changes in attention-sink divergence and regularizes that statistic during adaptation. The Structural Origin of Attention Sink traces sink formation back to variance discrepancies in value aggregation, their amplification by super neurons, and resulting dimensional imbalance. The interesting commonality is how a small statistical asymmetry can be amplified into a stable routing pattern. (arXiv)

This makes sparsity both attractive and dangerous. Sparse mechanisms offer interpretable handles, yet a safety property concentrated in a few routes can also be bypassed through small perturbations or post-training updates. The relevant question is therefore not simply whether a component has causal leverage. A useful control point needs enough behavioral reach, a tolerable side-effect profile, and a nontrivial operating window.

Steering has a geometry and a schedule

Two steering papers clarified why finding a direction is only part of the problem.

Spherical Steering replaces additive displacement with rotation on a representation sphere, separating angular movement toward a target behavior from radial norm distortion. Steer Like the LLM starts from another observation: prompting does not induce the same activation shift at every token. Prompt Steering Replacement learns state- and position-dependent coefficients that approximate the intervention pattern produced by a successful prompt. (arXiv)

Together with early trajectory correction, these papers outline three coupled questions:

• When is the trajectory still steerable?
• Where in the model is the target distinction represented and used?
• How should the intervention move the state without pushing it off-manifold?

Before ICML, I thought of steering mainly as direction discovery.

After ICML, steering looks more like a policy over layer, time, geometry, and dose.

Visibility, causality, and long-horizon influence

A useful hidden gem was Automatic Layer Selection for Hallucination Detection. Its FEPoID criterion selects intermediate layers where hallucination-related signals become especially detectable. The broader lesson is that “where to look” is already part of the mechanism: a signal can be absent from one layer, linearly visible in another, and causally committed somewhere else. (arXiv)

For each behavior, I now want to separate four questions:

1. Is the relevant information represented?
2. Is it visible in a usable coordinate system?
3. Is it causally used?
4. Is the outcome still changeable at that point?

Towards Long-Horizon Interpretability supplies an important complementary method. FlashTrace attributes multi-token spans and recursively propagates influence through intermediate reasoning tokens back toward source inputs. This is closer to the causal object needed for reasoning models: an early assumption may have little effect on the immediately following token while strongly shaping an entire later derivation. (arXiv)

Representation units and cross-model coordinates remain part of the same puzzle. Towards Atoms of Large Language Models asks whether neurons and standard SAE features are the right stable units. Multi-Way Representation Alignment maps multiple models into a shared representation universe rather than constructing inconsistent pairwise correspondences. A mechanistic claim becomes more convincing when the same causal quantity survives a change of layer, unit, model, or coordinate system. (arXiv)

External context is also a routing system

The First Drop of Ink extends the routing perspective beyond model internals. A small proportion of semantically plausible distractors causes a sharp initial loss in long-context accuracy, while adding more distractors produces diminishing additional damage. The first strong distractor captures enough attention to redirect the evidence path; the rest mostly compete within an already contaminated context. (arXiv)

This matters for RAG and agents with persistent memory. Retrieved context is not passive storage. Correct and misleading passages compete to become the basis of the answer. A useful evidence margin is therefore the model’s preference for the best supported source over the strongest plausible distractor.

2. Alignment, self-report, and evaluation

Saying is another behavior

The self-report and explanation papers challenged a common shortcut in alignment evaluation. A model’s account of its personality, beliefs, or reasons is still a generated output rather than privileged access to its internal computation.

The Personality Illusion finds that instruction tuning can stabilize self-reported personality traits even when those reports remain weak predictors of behavior. Persona prompts move questionnaire answers much more consistently than they move risk-taking, sycophancy, or other behavioral measures. What LLMs Explain Is Not What They Believe formalizes explanation sufficiency relative to a distribution of alternative inputs and finds that many explanations remain insufficient even under alternatives generated from the model’s own input beliefs. (arXiv)

A conversation at the second poster suggested a more precise follow-up question. Explanation failure can arise at several stages:

• the relevant feature is absent;
• it is represented but not used;
• it is used but inaccessible to verbalization;
• it reaches the verbal channel but is translated unfaithfully;
• a separate social-response circuit produces a plausible rationale.

Anthropic’s Natural Language Autoencoders make this distinction experimentally approachable by training an activation verbalizer together with a reconstructor that maps the verbalization back into activation space. The related global-workspace study asks which representations are available for report, flexible reasoning, and intervention. (Anthropic)

This gives a better framing than treating self-report as either honest introspection or empty performance. Self-report, refusal, sycophancy, evaluation awareness, and behavioral safety may share upstream representations while using different downstream readouts. Their natural activations may correlate even when causal interventions separate them.

Before ICML, I viewed self-report mainly as a questionable evaluation signal.

After ICML, it looks like a useful mechanistic target: one channel among several that may reveal how post-training organizes social and alignment-related behavior.

Judges and preferences are measurement systems

The same caution applies to the evaluators used to train and study models.

A Coin Flip for Safety audits LLM safety judges under shifts in attack, victim model, and data. Judge reliability can fall toward chance, and attack optimization can raise measured success by exploiting evaluator failure rather than producing more genuinely harmful content. (arXiv)

Measuring Human Preferences in RLHF Is a Social Science Problem moves one step further upstream. A pairwise annotation is an observed response produced by an elicitation procedure. It may reflect a stable preference, a preference constructed in context, a framing artifact, or no settled attitude at all. The paper argues that measurement validity should precede preference aggregation. (arXiv)

This is an important cross-disciplinary import. Behavioral science has spent decades separating latent constructs from the instruments used to measure them. Alignment pipelines often collapse those layers into one object called “human preference.”

For evaluation, the practical consequence is straightforward: a change in judge score is evidence only after the judge has been validated on the transformed output distribution. Otherwise, a steering method, jailbreak attack, or agent policy may simply be moving examples across the evaluator’s blind spots.

3. Agents and consequential actions

Agent safety becomes most concrete at the boundary between thought and action.

Learning When to Act or Refuse treats refusal as a first-class agent action within a plan–check–act loop and learns from pairwise comparisons of complete tool-use trajectories. Think Twice Before You Act introduces Thought-Aligner, which rewrites unsafe intermediate thoughts before they determine the next action. (arXiv)

For a state (s_t), the relevant quantity can be expressed as

A careful-looking rationale is insufficient when the action distribution remains unchanged. The consequential question is whether the safety mechanism shifts the probability of reading a private file, sending an email, entering credentials, modifying a database, or executing a command.

Safety engineering offers the useful analogy here. Interventions should be placed before irreversible transitions, with stronger checks for actions whose effects are difficult to undo. This turns agent safety from a wrapper around the final answer into a property of the state-transition policy.

Before ICML, I thought of agent safety mostly as guarding an autonomous system.

After ICML, I think of it as maintaining a safe action margin at each consequential boundary.

4. Skills, composition, and the new supply chain

The skill-based agent papers made composition impossible to ignore.

SkillTrojan distributes malicious logic across benign-looking skills and reconstructs the payload only under particular trigger and composition conditions. A Theoretical Game of Attacks via Compositional Skills studies the same structural problem at the level of linguistic and cognitive transformations: translation, roleplay, encoding, analogy, and indirection can each look harmless while their composition reveals a harmful objective. (arXiv)

The key security property therefore belongs to the composition graph rather than to any isolated component.

A skill can pass unit tests, contain plausible documentation, and remain useful on clean tasks while still participating in a triggered multi-skill payload. A meaningful SkillBOM needs to describe more than text similarity or code provenance. It should include tool permissions, memory access, side effects, trigger conditions, compatible composition partners, and observed execution traces.

The closest analogy is software supply-chain security, although agent dependencies are more varied. The graph can include executable code, natural-language instructions, retrieved documents, memory entries, API schemas, and private conventions shared among agents.

This also changes how fuzzy identity should be evaluated. A robust skill fingerprint should tolerate benign refactoring while remaining sensitive to changes in permissions, side effects, and compositional behavior.

5. Editable artifacts for reliable agents

The most inspiring agent-design pattern came from theorem proving and code security.

Editable Proof Sketch represents a proof plan as a dependency-aware artifact. When one subgoal fails, the system repairs the affected region while preserving proven parts that do not depend on it. CVE-Factory takes sparse vulnerability metadata and constructs executable environments, tasks, tests, and validation procedures at scale. (arXiv)

Both suggest a better substrate for agent reasoning than an ever-growing natural-language scratchpad.

A scientific agent should maintain an editable research object containing hypotheses, evidence, metrics, experiments, code artifacts, failures, dependencies, and revisions. A coding agent should preserve passing tests while localizing the cause of a failing one. A security agent should operate inside an executable environment whose success criteria can be independently checked.

The shared workflow is:

1. externalize the task into a structured artifact;
2. make dependencies explicit;
3. verify atomic parts with an external checker;
4. preserve validated components;
5. repair only the region affected by failure.

Formal methods and software testing have relied on this structure for decades. Their lesson for agents is deeper than “add a verifier.” The artifact must be organized so that verifier feedback identifies what should change and what should remain untouched.

This also produces more meaningful intervention points. A failed subgoal, unsupported claim, unsafe action, or invalid dependency is a better unit of repair than an arbitrary token in a long chain of thought.

A focused research agenda after ICML

Across the eight sessions, the recurring pattern was a decision becoming progressively more visible, more causally effective, and eventually harder to reverse.

The route may be an internal feature, an attention head, an MoE expert path, a reasoning branch, an explanation channel, a retrieved evidence path, a skill composition, or a sequence of tool actions. The competing futures may be correct and incorrect, refusal and compliance, grounded and distracted, truthful and sycophantic, or safe and unsafe.

The questions I am taking forward are:

• Where does a behavior first become visible inside the model?
• When does that signal become causally involved in the outcome?
• How do alignment and other post-training stages reshape it?
• Which apparently related behaviors share an upstream mechanism, and which only correlate at the surface?
• How can an intervention change the trajectory while preserving neighboring capabilities?
• How can agents externalize their state into editable and verifiable artifacts?
• How should evaluators be stress-tested before their scores are treated as evidence?

Twenty thousand steps a day was enough to leave me with one compact conclusion:

Find the earliest reliable decision point, determine whether it is causal, and intervene before the wrong future becomes difficult to reverse.

References

1. How does Chain of Thought decompose complex tasks?
2. SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning
3. Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away
4. Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
5. TraceRouter: Robust Safety for Large Foundation Models via Path-Level Intervention
6. Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink
7. The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
8. Spherical Steering: Geometry-Aware Activation Rotation for Language Models
9. Steer Like the LLM: Activation Steering that Mimics Prompting
10. Automatic Layer Selection for Hallucination Detection
11. Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs
12. Towards Atoms of Large Language Models
13. Multi-Way Representation Alignment
14. The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning
15. The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs
16. What LLMs Explain Is Not What They Believe: Evaluating Explanation Sufficiency Under Models’ Own Input Beliefs
17. Natural Language Autoencoders: Turning Claude’s Thoughts into Text
18. Verbalizable Representations Form a Global Workspace in Language Models
19. A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
20. Measuring Human Preferences in RLHF Is a Social Science Problem
21. Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use
22. Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
23. SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems
24. A Theoretical Game of Attacks via Compositional Skills
25. Editable Proof Sketch for Automated Theorem Proving
26. CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

Why Lean?

Lean 4 is a dependently typed theorem prover and programming language. Like Coq or Isabelle, it lets you write mathematical proofs that a type-checker verifies mechanically. Unlike those systems, Lean 4 also has a usable extraction/evaluation story and a growing standard library (Std4, Mathlib).

The key property we care about: Lean cannot lie about its own axioms. Every theorem Lean accepts is either provable from a known-good axiom set or contains sorry (Lean’s escape hatch, analogous to admit in Coq). Running #print axioms theorem_name after a successful compile reveals the full axiom dependency set. If sorryAx appears there, the proof is a placeholder — Lean “accepted” it the way a compiler accepts todo!() in Rust.

The trusted axiom set for computational theorems is small: {propext, Classical.choice, Quot.sound}. Anything beyond that is suspect.

A one-minute Lean example

Here is a simple function and its correctness theorem in Lean 4:

def addOne (n : Nat) : Nat := n + 1

theorem addOne_spec (n : Nat) : addOne n = n + 1 := by
  unfold addOne
  rfl

rfl closes the goal because both sides reduce to the same expression. The type-checker verifies this without trusting the programmer’s intuition. Now consider a more interesting statement:

theorem addOne_pos (n : Nat) : 0 < addOne n := by
  unfold addOne; omega

omega is a decision procedure for linear arithmetic. The proof is still machine-checked; omega is just a tactic that applies the decision procedure and either closes the goal or fails.

The leap from toy arithmetic to real code is what the pipeline attempts to automate.

The Motivating Example: HMAC Tag Comparison

Consider these two Python functions:

# vulnerable: early-exit byte loop
def token_verify_vulnerable(token: bytes, expected: bytes) -> bool:
    if len(token) != len(expected):
        return False
    for a, b in zip(token, expected):
        if a != b:
            return False
    return True

# fixed: constant-time comparison
def token_verify_fixed(token: bytes, expected: bytes) -> bool:
    return hmac.compare_digest(token, expected)

Both are functionally equivalent — they return True if and only if token == expected. A verifier that only proves functional correctness would green-light both.

But they differ in cost. The vulnerable implementation returns as soon as it finds a mismatched byte. An attacker can measure the comparison time and recover the correct token byte by byte: submit \x00..., then \x01..., etc. — the first byte that takes longer to compare is a match. This is a textbook timing side-channel.

The Lean model in RepoVerify/TokenVerify.lean makes the distinction formal:

-- Both implementations satisfy the functional theorem
theorem insecureEq_correct (xs ys : List Nat) :
    insecureEq xs ys = true ↔ xs = ys := by ...

theorem ctEq_correct (xs ys : List Nat) :
    ctEq xs ys = true ↔ xs = ys := by ...

-- Only the fixed one satisfies the cost theorem
theorem ctEqCost_eq_length_when_same_length
    (xs ys : List Nat) (h : xs.length = ys.length) :
    ctEqCost xs ys = xs.length := by ...

-- The leak: vulnerable cost depends on content, not just length
example : insecureEqCost [0, 0] [1, 0] = 1 := by decide
example : insecureEqCost [0, 0] [0, 1] = 2 := by decide

The lesson: a formally correct theorem can still miss the security property that matters. You have to ask whether you proved the right theorem, not just a theorem. Running python source/attack_demo.py demonstrates recovery of the full secret tag from the vulnerable implementation in deterministic polynomial time.

The Pipeline: Code → LLM → Lean → Five Gates

The code2lean pipeline generalizes this question. Given any Python function:

1. AST extraction — verify/extract.py pulls out the function body, argument types, and return type using Python’s ast module and packages them into a FunctionSpec.

2. LLM proposer — the function is sent to an LLM (GPT-5.5, Gemini 3.1 Pro, or Claude Opus 4.7) with a structured prompt asking it to write a complete Lean 4 file: the function definition, a correctness theorem, and a proof. The LLM picks the theorem statement freely; only the namespace and naming convention are fixed.

3. Five validation gates:

Gates A–D are mechanical. The only LLM judgment in the verification path is the critic (gate E), and its job is narrow: decide whether the theorem is vacuous.

Why the critic matters: the vacuous theorem problem

Consider bit_count8, which counts set bits in a byte. An LLM proposer might write:

theorem bit_count8_spec (b : Nat) (h : b < 256) :
    bit_count8 b ≤ 8 := by ...

This theorem is true. Lean accepts it. Gates A–D all pass. But a constant-zero implementation (bit_count8 b := 0) also satisfies result ≤ 8. The theorem proves nothing about what the function computes.

The critic prompt says: “Would this theorem distinguish a correct implementation from a buggy one? If not, return WEAK.” A good proposer writes instead:

theorem bit_count8_spec (b : Nat) (h : b < 256) :
    bit_count8 b = (List.range 8).countP (fun i => b &&& (1 <<< i) ≠ 0) := by ...

This is a functional specification. Any implementation that returns a wrong bit count will fail it.

A Full Walkthrough: insecure_compare

examples/01_insecure_compare/source.py contains:

def insecure_compare(a: bytes, b: bytes) -> bool:
    if len(a) != len(b):
        return False
    for x, y in zip(a, b):
        if x != y:
            return False
    return True

The fixture in fixture.py provides 6 test cases: equal pairs, different-length pairs, one-off pairs, empty inputs.

A one-shot GPT-5.5 proposal (from last_lean_openai.lean):

def insecureCompare (a b : List Nat) : Bool :=
  if a.length ≠ b.length then false
  else a.zip b |>.all (fun (x, y) => x == y)

theorem insecureCompare_correct (a b : List Nat) :
    insecureCompare a b = true ↔ a = b := by
  simp [insecureCompare]
  constructor
  · intro h
    exact List.zip_eq_iff_eq.mp (List.all_zip_eq_true.mp h)
  · intro h; subst h; simp [List.all_zip_eq_true]

Gate A passes (no forbidden tokens). Gate B passes on the first attempt. Gate C reports {propext, Classical.choice, Quot.sound} — clean. Gate D: all 6 fixture cases match. Gate E: the critic returns PASS — the ↔ a = b biconditional fully pins down the function’s behavior.

What this does not prove: that the comparison is constant-time. Exactly as designed. The pipeline proves what it can prove; the cost property requires a separate cost model, which is future work (see docs/roadmap.md).

Benchmarks: Which LLM Proposes Better Theorems?

We ran three proposers on the same 10 hard examples:

The table reveals a three-way trade-off:

Lean acceptance (did the proof close?) is mostly a Lean-tactics signal. Gemini won it, largely by leaning on Classical.choice for existential witnesses. GPT needed more repair rounds but got there on most examples.

Theorem strength (did the critic approve?) is mostly a proposer signal. GPT-5.5 naturally reaches for tight functional specs — it imports library lemmas (Nat.lcm, Nat.gcd), defines auxiliary helpers to model Python floor-division semantics, and writes biconditionals. Gemini and Claude default to easier targets: range bounds, definitional unfolds, set-membership instead of list-equality.

Speed is a latency signal. Claude’s ~4.5× wall-clock advantage comes from the absence of hidden reasoning tokens (GPT and Gemini burn 4–8k reasoning tokens per hard example; Claude does not in the standard Messages API).

Four examples — bit_count8, is_power_of_two, list_filter_even, list_unique — were WEAK across all three proposers. The WEAK reasons were nearly identical: bounds-only theorems, membership instead of equality, etc. This suggests the bottleneck on those four is the prompt’s theorem-shape guidance, not the model.

What This Is (and Isn’t)

It is: a concrete, runnable pipeline that chains LLM proposers with mechanical Lean verification and a structured critic. It demonstrates that for the class of pure, total, simply-typed Python functions, automated Lean translation and verification is feasible today.

It isn’t: a security scanner for arbitrary production code. The current scope is intentionally narrow — single functions, no I/O, no external dependencies, no floats or dicts, no concurrency. The cost/side-channel theorems that would catch timing leaks are not yet auto-derived; the HMAC demo uses a hand-written Lean baseline.

The path from here to vulnerability finding runs through two open problems: (1) auto-deriving cost models alongside functional ones, and (2) mutation-kill testing to mechanically verify that proposed theorems distinguish correct from buggy implementations. Both are on the roadmap.

The main message from the benchmarks: the gates work. Gate D (differential testing) catches mistranslations that Lean would otherwise silently accept. Gate E (critic) catches vacuous theorems that A–D miss. And gate C (axiom allowlist) ensures that a sorry-stuffed proof can’t sneak through as “verified.” The combination is more trustworthy than any single check — and considerably more trustworthy than asking an LLM whether its own output is correct.

Code, examples, and run artifacts: github.com/phunterlau/code2lean

References

Lean 4 and tooling

• de Moura, L. & Ullrich, S. The Lean 4 Theorem Prover and Programming Language. CADE 2021.
• Mathlib Community. Mathlib4.

LLM-assisted theorem proving

• Polu, S. & Han, J.M. Generative Language Modeling for Automated Theorem Proving. arXiv:2009.03393, 2020. (GPT-f)
• Jiang, A.Q. et al. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. arXiv:2210.12283, ICLR 2023.
• Yang, K. et al. LeanDojo: Theorem Proving with Retrieval-Augmented Language Models. arXiv:2306.15626, NeurIPS 2023.
• Xin, H. et al. DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data. arXiv:2405.14333, 2024.
• Trinh, T.H. et al. AI achieves silver-medal standard solving International Mathematical Olympiad problems. Google DeepMind, 2024. (AlphaProof)

LLM + code formal verification

• Misu, M.J. et al. Towards AI-Assisted Synthesis of Verified Dafny Methods. arXiv:2402.00247, FSE 2024.
• Pei, K. et al. Can Large Language Models Reason about Program Invariants?. ICML 2023.

Verified security-critical software

• Zinzindohoué, J.-K. et al. HACL*: A Verified Modern Cryptographic Library. CCS 2017.
• Klein, G. et al. seL4: Formal Verification of an OS Kernel. SOSP 2009.

Standard quantization (like INT4 or FP8) helps, but it introduces a fatal flaw: deterministic bias. When you round a vector down to a lower precision, the errors accumulate, distorting the delicate attention matrix and crippling the model’s reasoning capabilities at long contexts.

Enter TurboQuant. By leveraging a beautiful piece of high-dimensional geometry known as the Johnson-Lindenstrauss (JL) Lemma, TurboQuant compresses the KV cache down to 3 bits (or even 2 bits) with practically zero accuracy loss. It achieves a 5x memory reduction without retraining, turning that 8K context ceiling into a 40K playground on the exact same hardware.

The secret isn’t just compression; it’s unbiased compression. To understand why it works so flawlessly, we have to trace the math back to its bedrock.

The Anchor: Cauchy-Schwarz and the Geometry of Attention

To understand the elegant hack of TurboQuant, we have to stop thinking of Attention as a sequence of matrix multiplications and start looking at it as a measure of geometric similarity.

The core of the Transformer’s attention mechanism relies entirely on the inner product between a Query vector and a Key vector. This relationship is governed by the absolute dictator of inner products: the Cauchy-Schwarz inequality.

In a high-dimensional space (e.g., 4096 dimensions), Cauchy-Schwarz dictates that the dot product is maximized when the vectors are perfectly aligned, and becomes zero when they are perfectly orthogonal. This geometric alignment is exactly what the Softmax function turns into an “Attention Score.” If you destroy this geometric relationship by clumsily rounding the numbers, the model hallucinates.

The Chasm and The Bridge: The Polarization Identity

We want to squash our 4096-dimensional vectors down to a much smaller size (say, 256 dimensions) to save VRAM.

The Johnson-Lindenstrauss (JL) Lemma is famous for proving that you can project points into a lower dimension using a completely random matrix, and the Euclidean distances between those points will be preserved. But here is the catch: the Transformer doesn’t compute Euclidean distances. It computes inner products.

How do we bridge the gap between preserving distances (JL) and computing attention scores (Cauchy-Schwarz)? We use a beautiful piece of algebra called the Polarization Identity:

This identity is the “aha!” moment. It proves that a dot product is not some separate, mystical property—it is literally just a function of lengths and distances. It expresses the inner product entirely in terms of the squared distances of the sum and difference of the vectors.

The Leap: Applying the Random Shadow

Now the trap is sprung. We introduce a random projection matrix $\Phi$ that squashes our vectors from a high dimension $d$ down to a lower dimension $m$. Because this projection is a linear operation, we know that $\Phi(q+k) = \Phi q + \Phi k$.

The JL Lemma guarantees that our random shadow $\Phi$ perfectly preserves Euclidean lengths with exponentially high probability:

If JL preserves the lengths of $|q+k|_2$ and $|q-k|_2$, then according to the Polarization Identity, it mathematically must preserve the inner product $\langle q, k \rangle$. The deterministic bounds of Cauchy-Schwarz are perfectly protected by the probabilistic bounds of the JL Lemma.

The QJL Innovation: 1-Bit Unbiased Estimation

Standard quantization methods (like rounding to the nearest integer) introduce bias. They systematically shift vectors in a specific direction.

TurboQuant utilizes Quantized JL (QJL), which takes this random projection one step further. Instead of storing the exact projected values, it stores only the sign of the projection, turning the Key into a tiny 1-bit vector:

Because the projection $\Phi$ is completely random and data-oblivious, the errors introduced by taking the sign are evenly distributed. This makes QJL an unbiased estimator. In statistical terms, the expected value of our compressed dot product perfectly matches the true dot product:

TurboQuant achieves its “zero accuracy loss” by applying this 1-bit unbiased QJL code specifically to the residual error left over after a standard coarse quantization. The math preserves the exact geometry, which in turn preserves the model’s reasoning.

The Magic of the Logarithmic Scale

If you want to find the true elegance of this approach, look closely at the scaling law it produces. To preserve the geometric structure of $N$ tokens in the KV cache with an error tolerance of $\epsilon$, the target dimension $m$ must scale according to:

Notice what is missing from that equation: the original dimension $d$.

The memory footprint required to preserve the attention matrix is bottlenecked only by the number of tokens in your context window ($\log N$). It completely decouples the memory requirements of the KV cache from the width of the model’s architecture.

Because it scales logarithmically, jumping from an 8K context window to a 40K context window only requires a tiny, incremental bump in the projection dimension. By anchoring its logic in Cauchy-Schwarz and exploiting the dimensional shortcuts of the JL Lemma, TurboQuant turns an intractable hardware limit into a solved geometric puzzle.

Final Thoughts and the Future: Randomness is the Answer

If I have one takeaway from this work, it is that unbiased estimation is not optional for low-bit quantization. Standard quantization methods (rounding) are a blunt instrument that creates systematic biases, which destroy the delicate attention matrix. By combining coarse quantization for low variance and unbiased QJL residual coding for zero bias, TurboQuant achieves a flawless balance.

This highlights the true gift of high-dimensional geometry: the Concentration of Measure. When you have thousands of redundant dimensions, pure randomness ceases to be noise; it becomes a statistically reliable tool. The large dimension $d$ makes it mathematically probable that a random projection $\Phi$ works. The path forward for memory optimization isn’t just about shrinking the precision; it’s about squashing the dimension, while protecting the relative geometry.

Implications & Future Horizons

The immediate practical implication is the democratization of long-context LLMs. Massive models can now be run locally with large context (80k+) on a standard 16GB or 24GB consumer GPU. But this work can inspire even deeper changes:

Hardware for Randomness: Our current GPUs are optimized for deterministic floating-point arithmetic. Could we see hardware accelerators in the future with dedicated modules for fast, on-the-fly random projections?

Unbiased Everything: Is there a way to adapt this residual QJL framework to weights or activations? While those are dynamic and harder to compress offline, the focus on preserving unbiased statistical correctives could change how we approach low-bit model execution.

Ultimately, TurboQuant shows us that the Transformer is far more robust than we believed. Its internal representation is resilient to extreme compression, provided we don’t break the geometric rules that define its reasoning. We can let the model forget the absolute, as long as we help it remember the relationships. That is the final lesson of Cauchy-Schwarz and the JL Lemma.

References

• TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
• QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead

The Elegance of Mozart’s Attention Mechanism

In 1792, Mozart’s Musikalisches Würfelspiel (Musical Dice Game), K.516f, was published. The system is deceptively simple: 176 pre-composed musical measures arranged in a grid. The user rolls two six-sided dice ($2d6$) 16 times. Each roll corresponds to a specific measure for that column in the grid, generating a mathematically unique 16-bar minuet.

From a LLM mechanistic interpretability standpoint, the beauty of Mozart’s game is that it is a strictly autoregressive, discrete-token generator with a context window of zero.

In a standard Large Language Model (LLM), predicting the next token $x_t$ relies on the conditional probability of the entire past sequence:

Mozart bypassed the need for this computational overhead. In K.516f, the choice of Measure 3 has zero statistical dependence on Measure 2. The generation is completely memoryless. Instead, the model’s “attention” is 100% focused on its absolute positional encoding (the step $t$): \(P(x_t | \text{position } t, \text{dice roll})\)

How does it remain harmonically coherent without context? Mozart engineered the matrix as an aggressive, hardcoded attention mask. He ensured that every possible measure at $t$ smoothly resolves into every possible measure at $t+1$. Any dissonant, harmonically invalid transition was manually assigned a $-\infty$ pre-softmax penalty by the composer, effectively masking it out of the latent space.

Furthermore, the $2d6$ sampling acts as a physical temperature parameter. By using a triangular probability distribution ($P(7) = 16.7\%$, $P(2) = 2.7\%$) rather than a uniform one, Mozart lowered the entropy of the system. He statistically biased the model to generate the most “standard” harmonic progressions, reserving high-surprise edge cases for the extreme tails of the distribution.

Unifying the Grid: The Ramanujan Sum

If we were to code Mozart’s game today, we would use a simple for loop to force the piece to stop at $t=16$. But why does a 16-measure grid feel psychologically and harmonically complete? To understand this, we must abandon the discrete grid and apply the continuous mathematics of Srinivasa Ramanujan.

Ramanujan would not view Mozart’s matrix as a set of rules, but rather as the natural resonant frequency of a periodic equation. We can model the macro-structure of the minuet using a Ramanujan Sum ($c_q(n)$), which extracts periodic signals from noise:

By setting the fundamental period $q = 16$, the equation acts as a harmonic pendulum. Here is how Mozart’s attention mechanism unifies with Ramanujan’s math:

The Journey ($n = 1$ to $15$): As the measures progress, the complex exponentials point in various directions in the complex plane, causing destructive interference. Musically, this represents harmonic tension—the algorithmic wave is wandering through the latent space, seeking resolution.

The Half-Cadence ($n = 8$): When we reach the halfway point, the fraction simplifies to $\frac{a}{2}$. The vectors snap to the real axis. This momentary, symmetrical mathematical pause perfectly mirrors the structural “half-cadence” in classical phrasing.

The Resolution ($n = 16$): At the final measure, the fraction simplifies to an integer. Every term in the sum points in the exact same direction ($e^{2\pi i a} = 1$). The destructive interference vanishes into a massive spike of constructive interference.

The structure doesn’t resolve because of an arbitrary grid boundary; it resolves because $q=16$ ($2^4$, the fractal symmetry of classical phrasing) is the fundamental node where the equation naturally reaches maximum constructive harmony. Mozart’s positional attention mechanism is simply the geometric projection of this periodic equation.

Expanding Dimensions: Bach’s Deep Self-Attention

If Mozart’s dice game is a rigid, 1D loop locked to $q=16$, Johann Sebastian Bach’s beautiful Fugues (The Well-Tempered Clavier which has a beautiful Chinese name 赋格) represent the expansion of this mathematical framework into high-dimensional, deep-memory architectures. A fugue cannot be generated by a zero-context Markov chain like in Mozart’s dice game. It begins with a single “prompt” token sequence: the Subject. When the second voice enters, it must continuously look back at the Subject to generate valid counterpoint.

In LLM terminology, Bach implemented Multi-Head Self-Attention.

Each voice (Soprano, Alto, Tenor, Bass) acts as an independent attention head. They process the exact same context window but project it into different dimensional spaces. While Mozart relied on stochastic dice (sampling), Bach relied on deterministic linear algebra. The initial Subject vector is subjected to complex matrix transformations in the latent space:

• Transposition (Translation: $f(x) + c$)
• Inversion (Reflection: $-f(x)$)
• Augmentation/Diminution (Time Scaling: $f(2t)$ or $f(t/2)$)

Bach also utilized what we mechanistic interpretability researchers call Induction Heads. When the Alto voice enters with the “Answer,” it acts as an attention circuit specifically trained to recognize the sequence in the Soprano’s past and perfectly reconstruct it at the current time step. Meanwhile, the other heads calculate orthogonal vectors (the Countersubject) to ensure the dot product of the combined voices perfectly satisfies the vertical rules of harmony.

If we return to Ramanujan, Bach’s polyphony represents the full, unconstrained analytic continuation of the harmonic equations. While Mozart collapsed the variables into a degenerate case (a rigid loop in C Major), Bach allowed the variables to become complex numbers, unlocking all 24 keys and forcing the equation to expand dynamically across the complex plane.

The Convergence

Whether we are engineering modern Transformers, calculating Ramanujan sums, or analyzing 18th-century manuscripts, the computational goal remains identical. LLM and music generation are ultimately the search for mathematical symmetry across time. Mozart mapped it via hardcoded masking and stochastic geometry; Bach calculated it via deep contrapuntal attention matrices; and Ramanujan provided the equations that prove they are all navigating the exact same latent space.