KDD 2026 MILETS oral

Fantastic Manifolds and How to Catch Them

Aionoscope: Debugging Latent-State Accessibility in Time-Series Representations

Alexander Chemeris · Ming Jin · Randall Balestriero — KDD 2026 MILETS · Jeju · arXiv 2607.00956

blog.langotime.ai

Toto-2.0-2.5B · sine_phase · layer 4

Generator knob paired with the latent ring

phase knob
0.00
Representation manifold (PCA)

default instrument

We All Use Linear Probes to Check Decodability

The default check is: can a linear readout recover this factor?

word vectors

king − man + woman = queen

king minus man plus woman equals queen as two equal vector directions royalty woman − man + (woman − man) man king woman queen

time series

sine @10 Hz − sine @5 Hz + square @5 Hz = square @10 Hz

sine at ten hertz plus the square minus sine offset equals square at ten hertz frequency square − sine + (square − sine) sine @5 Hz square @5 Hz sine @10 Hz square @10 Hz

Schematics of the word-analogy intuition (word2vec / GloVe) and the same intuition transposed into time series.

Why is phase so bad?

20 plotted dense generative parameters · median R² across 37 model-plus-adapter systems

median of 37 model-plus-adapter systems, best layer, single linear probe (Aionoscope)

A low linear-probe score might mean the factor is gone, or that it lives on a curved manifold.

expected topology

Factor Shape
amplitude interval
frequency ordered curve
phase circle
event time line
regime separated components

Phase means 0 and 2π are the same place. A straight readout has no native way to wrap.

field motion

Geometry Is Eating Interpretability

Recent work treats representation spaces as manifolds rather than directions, across language and video models.

It reverse-engineers geometry from natural data, with no ground truth. In time series we can order the geometry in advance — and check the delivery.

language models

preprint

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior ↗

arXiv 2605.05115 · May 2026

preprint

Do Sparse Autoencoders Capture Concept Manifolds? ↗

arXiv 2604.28119 · Apr 2026

research series

The Neural Geometry blog series ↗

goodfire.ai/research/neural-geometry

video world models

preprint

Interpreting Physics in Video World Models ↗

Joseph, Garrido, Balestriero et al. · arXiv 2602.07050 · Feb 2026

motion direction is encoded with circular geometry; physical variables become accessible mid-stack and degrade toward the output layers

interactive lab

J-Lens for video models: uncovering steerable physical dynamics ↗

Jay Hack · blocket-league.vercel.app

motion direction wraps into a ring in activation space — geometry discovered, not ordered in advance

one source of truth

Aionoscope: An Instrument with Ground Truth

One latent state emits the stream and the labels. Nothing is annotated after the fact.

Process samples z Draws a latent state z: which components are active, plus their generative parameters — frequency, amplitude, phase, timing, trend coefficients, regime.
the same z
View renders x Turns that state into an observed stream x = V(z, ε) — mixing, noise, nuisance, sampling rate. observed stream
Exact labels from z Emitted by the very same z — categorical y_cat = C(z), which components are present, and dense y_dense = D(z), the parameter values. ground truth
14 component processes
34 dense parameters
k ∈ {1,2,3} mixtures
frequency · hz
3.00
amplitude
0.75
phase
0.00

a sine of exactly this frequency, a spike at exactly this time — the ground truth is the dial we turn

defining property

Process-to-View Latent state and observed signal are separated: the View renders the signal, and the categorical and dense labels are emitted by the very same state that rendered it. One source of truth — no natural dataset can offer that.
Fresh seeded streams, not a stored corpus Deterministic given benchmark version, config and seed, so streams are regenerated exactly rather than stored — new held-out evaluation streams are minted on demand instead of reusing a fixed set.
GPU-first, fast Generation runs on GPU: the process sampler and the view renderer are both GPU-first.
Arbitrary-resolution factor sweeps One factor swept on a fixed grid at any resolution while everything else is pinned — which is what makes geometry measurable.
Configurable mixture complexity k active components per sample, set per benchmark (k = 1, 2, 3 in the published sweep).
Nuisance control Nuisance variation and noise are configurable and can be switched off for clean single-factor curves.

We're looking for contributions of more signal generators, views, and parameters to generate.

Linear Probe Results for Various TSFMs

Run Over 2 Mixed Aionoscope Signals, with the bubble size showing model size

Live aionoscope.langotime.ai dashboard snapshot at k = 2 mixed signals: 43 model-plus-adapter systems, median over 10 validation seeds, each system at its own best layer.

Regression By Parameter Type (R2)

Same chart on the dashboard — phase is recovered badly by every model

Median across the 43 systems, each at its own best macro-R² layer; values clipped at 0, as the dashboard does. phase: 0.051 median · 0.248 best, against frequency_hz 0.870 · 0.970.

43 model-plus-adapter systems
22 model families
33 / 43 systems above 0.900 mean AUROC
0.648 best dense macro R²; oracle is 1.000
29 / 43 best dense layer is not the final layer

aionoscope.langotime.ai

live catch

Catching a Manifold, Live

Generator phase knob

The signal slides as phase sweeps

phase knob
0.00
Representation manifold (PCA)

See more models, tested at every layer at https://aionoscope.langotime.ai/manifolds

protocol manifold_v0

How We Catch One

A ring in a projection starts the suspicion. The score is computed in the full PCA-64 point cloud.

01
Sweep one factor on a fixed grid 1,024 values, one enabled factor, one repeat per point; every other factor is canonical non-degenerate, nuisance off, validation half-grid offset.
02
Encode with the frozen model Toto-2.0-2.5B, every evaluated layer, mean-pool valid tokens into one vector per grid point.
03
PCA to 64 dimensions The measured object is the PCA-64 cloud, not the 2-D or 3-D picture used on stage.
04
Score the full cloud, per layer Linear and k=8 geodesic Spearman, geodesic gain, 5-NN recall, scaled stress; cycle closure and circular order for periodic factors.
05
Keep projections as pictures only PCA-2/3 and Isomap are for the eye. No UMAP, no t-SNE, and no projection panel is ever used as a score.

Per-layer metric readout

Toto-2.0-2.5B · sine_phase

geodesic Spearman Sweep one factor, measure distance along the manifold. Does that order match the factor values? Near 1: order survives the curve.
linear Spearman Same question, straight-line distance in representation space. High: the factor runs along a straight direction. Lower: ordered but bent.
3-NN recall Neighbours in the factor sweep — do they stay neighbours in the representation? High: local structure survives. Low: near values land far apart.
geodesic gain The gap between the two Spearman scores. Above zero: following the curve orders the sweep better than a straight line — the manifold is curved.

protocol manifold_v0 · Toto-2.0-2.5B · sine_phase · real per-layer metrics

same instrument, other knobs

Beyond Phase: Event Time and Trend

We're looking for contributions of more planted signals to sweep, and more metrics to score the geometry they leave behind.

Event-time knob

Move a one-sample spike across the window

event time
0.00

Distilled from public dashboard plot-data JSON for offline preview.

Representation manifold (PCA)

Layer 0 jagged comb

Layer 21 ordered curve

probe vs manifold, literally

Linear probe vs geometry

A probe score answers whether a scalar is linearly readable. Geometry answers what shape the factor actually took.

geometry organised
geometry collapsed
probe high
Frequency decodable and structured — but the scalar hides a coil that folds at Nyquist
TBC decodable yet disorganised
probe low
Phase present but bent into a circle — the probe calls it noise
Amplitude genuinely gone — probe and geometry agree

The same quadrants, plotted

Probe R² up, geometry across, bubble size = model parameters.

No system in this snapshot sits in the high-probe / collapsed-geometry quadrant — the top-right region is empty.

next steps

Next Steps

a new benchmark

We propose to build a new manifold-reconstruction benchmark and a debugging tool to measure manifold quality and make it easy to track it and compare.

If phase is truly periodic, the representation should not need an arbitrary seam where 2π wraps to 0.

manifold steering

So far we have only read the geometry out. Recovering a structure does not prove the model computes with it — a readable direction can be a shadow on the wall rather than the mechanism. Recent video-model work makes the gap concrete: a linear-probe direction is readable but not writable, while a Jacobian-lens-oriented write is causal. The next step is to test our manifolds the same way — write along the manifold versus off it with a J-Lens-style intervention, and measure whether the model's own prediction follows.

expected topology

  • line
  • ordered curve
  • circle
  • torus
  • separated components

uses

  • training-progress proxy without downstream tasks
  • surprise and curriculum signals
  • steering along the manifold
  • transfer to video world models as an open question

Contact me if interested in collaboration!

contacts on the next slide

from craft to engineering

From Craft to Engineering

  • 01 Send me a model to test, I'll happily add it.
  • 02 The code is open-source! Contribute more signals (basic signals, physiology, telemetry, etc)
  • 03 Use Aionoscope to debug your own models!

blog.langotime.ai

generator knob

phase knob
0.00

latent ring

B1 backup

The Door Scale Can't Reopen

Scaling can reopen many doors. It cannot reopen one the tokenizer has already shut.

Toto-2 ladder

Amplitude foil

Mechanism note: per-window standardization erases amplitude before the backbone; capacity raises the ceiling a factor can reach — it does not decide which factors are in the room.

B2 backup

Debugging a World Model

A multi-component system fails downstream and the loss curve does not say where. Decompose the path.

“put probes into different parts of the system so you can monitor the path — good here, good here, oops, not good here.”

method

  • Test the encoder first.
  • Sweep planted factors.
  • Check the manifold.
  • Then move to the predictor path.

Today's diagnosis is that the encoder is the ceiling of our current architecture — Aionoscope is how we avoid blaming the wrong component.

No claim of a located predictor-path bug: that experiment has not been run.

B3 backup

Geometry vs the Loss Curve

The probe says “perfectly represented” at every checkpoint while the geometry is dismantled underneath it.

LeNEPA-CauKer2M · linear_trend_slope · layer 5

Arc spread squeeze

220 checkpoints, steps 1k→220k
R² = 1.00 probe pinned across the run
0.97 → 0.63 distance-order decay
2.2× arc spread squeeze
monotone order Sweep one factor step by step and see where each step lands in the representation. The score asks if the points keep the same order along the curve. 1.0 means order fully kept; low means the sweep folds back or is shuffled.
straight-line order The same question, but measured with straight-line distance in representation space, not distance along the curve. High means the factor lies along a straight direction; lower means it is ordered but bent, so a linear readout struggles.
probe R² How well a linear readout predicts the factor value from the frozen representation. 1.0 means a straight line recovers the factor exactly; 0 means no better than predicting the average. It says nothing about shape.
Arc spread squeeze How far apart the two ends of the sweep sit in representation space, tracked over training. When it shrinks, the model packs the whole factor range into a smaller region, so equal factor steps move the representation less.

Hypothesis: manifold smoothness and intrinsic dimension will be tested as a downstream-free training-progress proxy and surprise signal for curriculum. This is a research question no one has answered — we're testing it, not reporting it.

B4 backup

Geometry vs the Loss Curve — every layer

The same rise-then-decay runs at every depth, not only the layer we picked: all eight layers start near 0.5, peak between step 5k and 71k, and end below their own peak. Layer 0 — the tokenizer front-end — is excluded on purpose. Hover a line to read its layer, step and score.

monotone order · layers 1–8

straight-line order · layers 1–8

monotone order Sweep one factor step by step and see where each step lands in the representation. The score asks if the points keep the same order along the curve. 1.0 means order fully kept; low means the sweep folds back or is shuffled.
straight-line order The same question, but measured with straight-line distance in representation space, not distance along the curve. High means the factor lies along a straight direction; lower means it is ordered but bent, so a linear readout struggles.

aionoscope manifold corpus v20260619T143710Z · LeNEPA-CauKer2M · linear_trend_slope · 220 checkpoints. Deepest and highest: layer 5 peaks at 0.975 (step 48k) and ends at 0.609; layer 8 peaks earliest at 0.911 (step 5k) and ends lowest at 0.421.

Honest note: this slide is built from the published corpus, while the single-layer B3 slide is built from the earlier blog asset, so the values differ slightly — same peak and endpoints, layer-5 correlation 0.95.