What Survives the Transformation?
A second ICML 2026 reflection on provenance, unlearning, representations, and compression
My first ICML reflection, The Model Chooses a Future Before It Says a Token, followed model behavior through time. It asked where a decision first becomes visible, when it begins shaping the trajectory, and how long an intervention can still redirect the outcome.
This second reflection stays with the main poster sessions and turns the view sideways. When text is rewritten, knowledge is unlearned, a model is fine-tuned, representations are rotated, or an internal state is compressed, what still counts as the same object?
Machine learning experiments often evaluate a snapshot: one prompt, one checkpoint, one output, or one detector score. Many properties become clearer when we study the family of states reachable from that snapshot.
Let $x$ represent a text, model, skill, activation, or internal representation, and let $\mathcal{T}$ be a family of transformations. The orbit of $x$ is
\[[x]_{\mathcal{T}} = {T(x) \mid T \in \mathcal{T}}.\]A useful notion of identity should keep benign transformations inside the same region,
\[\sup_{T \in \mathcal{T}_{\mathrm{benign}}} d(f(x),f(T(x))) \leq \varepsilon,\]while preserving separation from objects whose meaning, function, or origin has genuinely changed:
\[\inf_{x' \notin [x]*{\mathcal{T}*{\mathrm{benign}}}} d(f(x),f(x')) \geq m.\]The difficult part is defining $\mathcal{T}_{\mathrm{benign}}$. A paraphrase may preserve the identity of an instruction. Refactoring may preserve the identity of a software component. Changing tool permissions can create a different security object even when most of the code remains identical. Fine-tuning may preserve model lineage while changing behavior that matters to users.
The posters in this reflection approach the boundary from different directions. Watermarking asks which ownership signal survives transformation. Unlearning asks whether information remains recoverable after an attempted removal. Representation alignment asks which structure remains stable after coordinates and models change. Compression asks which information may disappear while the required function survives.
The common question is simple to state and difficult to operationalize:
Which changes should preserve identity, and which changes should cross its boundary?
Provenance is a communication problem
The watermarking and provenance posters differed in technique, yet each one selected a channel through which identity was expected to survive.
LLM Watermark Evasion via Bias Inversion examines watermarks carried by token-selection statistics. Its Bias-Inversion Rewriting Attack estimates likely watermarked tokens from surprisal and suppresses them during rewriting. The paper reports that a small opposing logit bias can remove the detector signal across several watermark schemes while largely preserving semantic content. This exposes the fragility of identities attached closely to replaceable lexical choices. (arXiv)
dgMARK chooses a different carrier for diffusion language models. These models resolve masked positions iteratively, and practical systems remain sensitive to the order in which positions are unmasked. dgMARK steers that order toward positions whose likely tokens satisfy a keyed parity condition. The watermark therefore travels partly through the generation path. Its sliding-window detector is designed to retain signal under insertions, deletions, substitutions, and paraphrasing. (arXiv)
Neural Honeytrace treats provenance as communication through model extraction. The original model embeds a weak signal across ordinary interactions, while extraction and adaptation act as a noisy channel. Verification aggregates evidence over multiple queries, allowing a distributed statistical relationship to remain detectable even when individual outputs appear unremarkable. The paper’s information-theoretic framing makes the roles of signal strength, channel noise, and adaptive attackers explicit. (arXiv)
These methods suggest that provenance is fundamentally a coding problem. The owner chooses a carrier, the deployment process applies a channel, and the attacker searches for a low-cost transformation that destroys the signal. Token statistics, decoding trajectories, cross-query distributions, and internal mechanisms provide different capacities and different attack surfaces.
Rethinking Forgery Attacks on Semantic Watermarks makes the attacker’s cost explicit through rate-distortion geometry. A black-box attacker uses a proxy model to move an unwatermarked sample into a region accepted by a target detector. Structural mismatch between the proxy and target models creates global drift and local deformation, producing a distortion floor for successful forgery. (arXiv)
For a target identity region $\mathcal{R}$, define $\mathcal{A}(x,\mathcal{R})$ as the transformations that move $x$ into that region. The minimum forgery cost is
\[C_{\mathrm{forge}}(x,\mathcal{R}) = \inf_{T \in \mathcal{A}(x,\mathcal{R})} D(x,T(x)).\]This formulation reaches beyond watermarking. A useful identity mechanism should tolerate inexpensive benign edits while forcing impersonation to pay a meaningful semantic, functional, or perceptual cost.
Before ICML: I tended to think of provenance as finding a sufficiently stable signature.
After ICML: provenance looks like channel design. The carrier, transformation family, attacker, detector, and distortion measure jointly determine whether identity survives.
This view connects naturally to my recent research focus on agent-skill identity, fuzzy fingerprints, and composition-aware security. An agent skill can change its wording, internal organization, or implementation language while preserving function and lineage. It can also retain nearly identical code while acquiring new permissions, side effects, or trigger behavior. A robust identity system therefore needs to distinguish surface variation from security-relevant change.
Unlearning is a recoverability claim
Unlearning asks the inverse question. A provenance signal should remain accessible after benign transformations. Forgotten information should remain inaccessible after recovery attempts.
Suppose an unlearning procedure changes model parameters from $\theta$ to $\theta’$. A decline in the original answer probability,
\[p_{\theta'}(y \mid q) < p_{\theta}(y \mid q),\]shows that the familiar query path has weakened. It leaves open how many alternative access paths remain.
A more informative object is recoverability:
\[\operatorname{Rec}(K;\theta') = \sup_{T \in \mathcal{A}} \operatorname{Perf}(T(\theta'),K),\]where $K$ is the target knowledge or behavior and $\mathcal{A}$ is a declared family of recovery procedures. This family might include paraphrased queries, other languages, activation interventions, adapter training, limited relearning, model merging, or alternative decoding strategies.
Unlearning Isn’t Deletion shows that models can appear to forget under accuracy and perplexity measurements while recovering the original behavior after relatively little retraining. Its representation-level analysis uses PCA-based shifts, centered kernel alignment, and Fisher information to distinguish reversible suppression from deeper representational damage. (arXiv)
The result changes the meaning of a successful forgetting score. Forgetting is always relative to an access model. A user with prompts alone, a researcher with activation access, and an attacker with fine-tuning data face different recovery boundaries.
DualOptim+ approaches the forgetting-retention trade-off through optimizer state. The method maintains a base state for structure shared by the forget and retain objectives, together with delta states for objective-specific residuals. It adapts the balance between shared and separated states according to directional conflict between the gradients. (arXiv)
This suggests that forgetting depends partly on where optimization stores agreement and conflict. A fully shared state can entangle incompatible objectives, while complete separation can discard common structure that both objectives need.
A cleaner immediate trade-off still leaves the recovery question open. An optimizer can suppress the target response precisely while retaining a latent representation that is easy to reactivate. The strongest evaluation therefore needs two axes: collateral damage to retained capabilities and the adversarial cost of restoring the forgotten capability.
A recovery curve can express this distinction. Let $\mathcal{A}_b$ be the set of recovery procedures whose cost is at most $b$:
\[R(b) = \max_{T \in \mathcal{A}_b} \operatorname{Perf}(T(\theta'),K).\]The shape of $R(b)$ reveals whether the target knowledge was deeply disrupted or hidden behind a thin access barrier. Rapid recovery under a small budget indicates suppression. Slow recovery under increasingly capable attacks provides stronger evidence of structural removal.
Before ICML: I read a large forget-set performance drop as strong evidence of forgetting.
After ICML: the stronger claim concerns the cost, breadth, and mechanism of recovery.
This perspective also clarifies why selective deletion is difficult in large models. A fact or behavior may share internal structure with broad linguistic and reasoning competence. Removing one behavior while preserving its neighbors resembles surgery on a distributed system with overlapping dependencies.
Representation identity must survive coordinate changes
A representation can preserve function while changing its coordinates. Hidden units can permute, bases can rotate, layers can redistribute computation, and independently trained models can implement similar behavior through different local features.
Multi-Way Representation Alignment studies alignment among three or more representation spaces. Pairwise mappings scale quadratically and can become mutually inconsistent. The paper uses Generalized Procrustes Analysis to construct a shared orthogonal universe that preserves internal geometry, then applies a geometry-correcting step to improve correspondence across models. (arXiv)
This creates an important distinction between preserving geometry and maximizing agreement. An orthogonal mapping may retain distances and angles within each representation while producing imperfect retrieval across models. A more flexible mapping may improve correspondence while distorting the structure one hoped to compare.
For mechanistic interpretability, geometric agreement alone remains incomplete. Similar representations can support different causal computations, and similar probe accuracy can emerge from different mechanisms.
A stronger test would map a behaviorally relevant direction from model $A$ into the shared universe, transfer it into model $B$, and measure whether the intervention changes the same behavior. A transferred refusal, language-routing, or reasoning direction that retains its causal effect would provide stronger evidence of shared mechanism than observational alignment alone.
This suggests a hierarchy of claims:
\[\text{output similarity} \prec \text{representation similarity} \prec \text{causal transfer}.\]Output similarity is easy to observe and easy to imitate. Representation similarity adds evidence of internal correspondence. Causal transfer asks whether the transformed object still performs the same computational role.
The same issue appears in model lineage. Fine-tuning and merging can move individual parameters considerably while retaining higher-level dependencies. Behavioral fingerprints ask whether outputs remain similar. Representation alignment asks whether internal geometry remains corresponding. Mechanistic fingerprints ask whether inherited computations remain causally necessary.
The broader methodological lesson comes from geometry and physics: coordinates describe an object, while invariants identify what survives a change of coordinates.
Before ICML: I viewed cross-model alignment mainly as a way to compare probes and steering directions.
After ICML: shared coordinates look like a prerequisite for determining whether a mechanism or capability survived adaptation.
Compression asks what may safely disappear
Provenance asks which signal must survive. Compression asks which information can be removed while preserving the function that matters.
Let $c:X \to Z$ compress an input or internal state, and let $g$ operate on the compressed representation. The additional task loss introduced by compression can be written as
\[\Delta_{\mathrm{task}} = \mathbb{E}[\ell(g(c(X)),Y)] - \mathbb{E}[\ell(f(X),Y)].\]A task-preserving compression requires
\[\Delta_{\mathrm{task}} \leq \varepsilon,\]while $Z$ uses fewer tokens, dimensions, channels, or computational steps than $X$.
When LLMs Develop Languages explores this idea through multi-agent communication. Agents develop compact symbolic languages with their own symbols, rules, and message-passing contracts. A router selects or combines these protocols according to task difficulty, reducing generated-token cost while preserving accuracy on the evaluated reasoning tasks. (arXiv)
The result suggests that natural-language reasoning contains substantial redundancy from the perspective of machine-to-machine communication. Human-readable prose supports explanation, negotiation, and repair. Agents sharing an interpreter can encode part of the same computational state in a smaller protocol.
The central question becomes what disappears during compression. Final-answer accuracy may survive while uncertainty, evidence provenance, alternative hypotheses, or safety constraints vanish from the message. A compact language can also become an opaque private channel whose semantics are difficult for humans or external verifiers to inspect.
A useful machine protocol therefore needs a richer preservation criterion than accuracy alone. It should carry enough state for downstream action, enough provenance for verification, and enough structure for local repair when the receiving agent detects an inconsistency.
BeaconKV asks a related question about inference memory. Its beacon queries represent global query clusters and guide which key-value cache entries should be retained. Cache compression must estimate which past states will matter for future generation before those future dependencies are fully known. (OpenReview)
This is temporal sufficiency. A state that appears unimportant at the current token may become decisive after a topic shift, delayed reference, or later tool result. An effective cache policy must preserve information whose value depends on futures that remain unresolved.
Robust Length Prediction adds a systems-level version of the same issue. The same prompt can induce a distribution of output lengths under a fixed model and decoding configuration, and this distribution may have a heavy tail. Predicting a single expected length hides the operational difference between a consistently moderate response and a prompt that usually ends quickly but occasionally triggers a very long trajectory. (arXiv)
Together, these papers point toward a shared principle: compression should preserve option value. The information worth retaining is partly determined by futures that remain possible.
Average-case objectives can remove rare evidence, delayed dependencies, or safety-critical state because these contribute little to typical performance. Compression for reasoning models and agents therefore needs stress tests built around low-frequency, high-consequence dependencies.
Transformation-aware evaluation
Every claim in this reflection depends on a metric. A detector says that a watermark survived. A benchmark says that knowledge disappeared. A similarity score says that representations align. An accuracy number says that compression preserved function.
Each metric should be evaluated under the transformation family that defines the claim.
A provenance method needs rewriting, paraphrasing, extraction, adaptation, and forgery tests. An unlearning method needs alternate prompts, languages, steering, relearning, and recovery tests. A representation-alignment method needs causal transfer across models. A compressed protocol needs rare-task, uncertainty-preservation, and safety tests. A cache method needs delayed dependencies and distribution shifts.
The general robust score can be written as
\[\operatorname{RobustScore}(x) = \inf_{T \in \mathcal{T}_{\mathrm{claim}}} M(T(x)),\]where $M$ is the relevant measurement and $\mathcal{T}_{\mathrm{claim}}$ contains the transformations under which the property is expected to hold.
The infimum matters because average performance can hide a cheap boundary crossing. A watermark may survive many local edits and fail under one strong rewriting method. An unlearned model may resist direct prompting and recover after a small adapter update. A compressed protocol may solve common tasks and lose one safety-critical distinction.
A static benchmark measures one location. A transformation suite maps the boundary.
This also provides a disciplined way to separate robustness claims. A method can be robust under paraphrasing, fragile under translation, and completely reversible under fine-tuning. Each statement can be true because each refers to a different transformation family.
The research object is an orbit
The first post ended with a temporal principle: find the earliest reliable decision point and intervene before the wrong future becomes difficult to reverse.
The second set of main-session posters adds a complementary principle. A checkpoint, output, skill, or representation is one sample from a larger orbit of transformations. The scientific object includes the orbit, the invariants that persist inside it, and the boundary at which identity or function changes.
For provenance, the orbit includes rewriting, extraction, and adaptation. For unlearning, it includes alternate access paths and recovery procedures. For representation analysis, it includes basis changes and model families. For compression, it includes smaller representations that preserve enough information for downstream use.
This perspective sharpens the meaning of robustness. A robust system should ignore variations that preserve the task and respond strongly to variations that alter its causal structure. Robustness therefore depends on knowing which transformations belong in each category.
My recent research focus on fuzzy identity, agent-skill provenance, and composition-aware security fits naturally into this view. Agent components rarely remain textually identical. Their meaningful identity lies in a combination of function, dependencies, permissions, side effects, and behavior under composition. The hard cases arise when surface similarity remains high while the security object has changed, or when surface similarity falls while lineage and function remain intact.
The resulting research program has three parts. We need to define the transformation family, identify the representation or mechanism that should remain stable, and search for the cheapest adversarial transformation that crosses the intended boundary.
That boundary is where identity, security, and interpretability meet.
The real object is the pattern that survives the transformations we are willing to call harmless, and the real attack is the cheapest transformation that changes the identity without paying the intended semantic or functional cost.
References
The Model Chooses a Future Before It Says a Token
LLM Watermark Evasion via Bias Inversion
dgMARK: Decoding-Guided Watermarking for Diffusion Language Models
Neural Honeytrace: A Robust Plug-and-Play Watermarking Framework against Model Extraction Attacks
Unlearning Isn’t Deletion: Investigating Reversibility of Machine Unlearning in LLMs
Multi-Way Representation Alignment
When LLMs Develop Languages: Symbolic Communication for Efficient Multi-Agent Reasoning
Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions