# TLT12 — a Temporal Latent Traversal dataset TLT12 is a rendered video dataset for studying latent variable discovery, both invariance and equivariance, in predictive self-supervised learning. Unlike datasets that sample a random pose per image, every sample here is a *sequence* in which one or more latent factors sweep smoothly across their range while the others stay fixed. The full generative latent state is shipped alongside every frame, so what a model should be invariant or equivariant to is known exactly rather than assumed. The dataset is rendered from 12 object categories of [ShapeNetCore.v2](https://shapenet.org/) using Blender Cycles via [BlenderProc](https://github.com/DLR-RM/BlenderProc). It was introduced in [Gorgan Mohammadi, Halvagal & Zenke (eLife, 2026)](https://doi.org/10.7554/eLife.110895.1). | | | |---|---| | Categories | 12 ShapeNet synsets, 10 objects each | | Sequence length | 32 frames | | Resolution | 128 × 128 RGB | | Latent factors | 10, continuous, fully annotated per frame | | Format | [WebDataset](https://github.com/webdataset/webdataset) tar shards | | Splits | train / val / test (80 / 10 / 10 by sequence) + separate ood-test (held-out objects) | Authoritative per-split counts are in `stats.json` and in each split's `dataset_info.json`. ## Example traversals

airplane cap car chair earphone laptop mug piano

--- ## Contents ``` TLT12/ ├── README.md this file ├── download.sh fetch the dataset (resumable, checksum-verified) ├── DATASHEET.md datasheet: composition, collection, intended use, limitations ├── LICENSE CC BY-NC 4.0 — covers the shards, previews, and metadata ├── CITATION.cff machine-readable citation ├── requirements.txt dependencies for the provided loader ├── synset_labels.json synset ID → category name and class index ├── SHA256SUMS checksums for every file — also the download manifest ├── stats.json per-split and per-category sequence counts, object list ├── wds_dataset.py PyTorch loader for the shards ├── example_load.py runnable example: load a batch, print it, save a GIF ├── verify_release.py check your download is complete and intact ├── make_release_manifest.py (for the maintainer) regenerate SHA256SUMS and stats.json ├── generation/ │ ├── tlt12_commands.sh the exact pipeline that produced this release │ └── provenance.json generator version, render settings, packaging settings ├── examples/ animated previews of representative sequences └── shards/ ├── train/ shard-*.tar.gz + dataset_info.json ├── val/ shard-*.tar.gz + dataset_info.json ├── test/ shard-*.tar.gz + dataset_info.json └── ood-test/ shard-*.tar.gz + dataset_info.json ``` --- ## Quick start ### 1. Download TLT12 is distributed as loose WebDataset shards. If you have downloaded the entire dataset as an archive from Zenodo or elsewhere you can skip this step and directly go to Step 2. Otherwise you can download it from our server. Use `download.sh` to fetch the shards under `SHA256SUMS`. It pulls every file it lists, and verifies the result: ```bash ./download.sh --dest ./TLT12 ``` It is safe to interrupt and re-run: files already present with the correct checksum are skipped without contacting the server, and a partially downloaded file is resumed rather than fetched again. **Try a small piece first.** Each of `val` and `test` is about a tenth of the dataset, and a split download still includes the loader, docs, and metadata, so it is a complete working release: ```bash ./download.sh --split val --dest ./TLT12 # ~10% of the total ./download.sh --list # file list and exact total size ./download.sh --jobs 4 # 4 parallel transfers ``` `wget -c -r -np -nH --cut-dirs=1 /` works too if you prefer, but you will need to check `sha256sum -c SHA256SUMS` yourself afterwards. ### 2. Verify and load ```bash cd TLT12 pip install -r requirements.txt python verify_release.py # structure, checksums, decode test python example_load.py --shards shards/train --gif sequence.gif ``` `verify_release.py` uses only the standard library, so it runs before you install anything. If it reports a checksum failure, re-run `download.sh` (it refetches only the bad files). In your own code: ```python from wds_dataset import TraversalWebDataset, make_loader ds = TraversalWebDataset("shards/train/*.tar.gz") loader = make_loader(ds, batch_size=16, num_workers=4) for batch in loader: batch["frames"] # FloatTensor [B, 32, 3, 128, 128], values in [0, 1] batch["latents"] # FloatTensor [B, 32, 10] — latent state per frame batch["base_latent"] # FloatTensor [B, 10] — the frozen background state batch["traversal_velocities"] # FloatTensor [B, 10] — 0 = frozen, ±v = sweep speed batch["synset_id"] # list[str] — ShapeNet category batch["obj_id"] # list[str] — ShapeNet object instance batch["traversal_factors"] # list[list[int]] — which factors vary in this sequence ``` Two filters return new pipelines: ```python rotating = ds.filter_by_factor("rot_z") # sequences where rot_z sweeps mugs = ds.filter_by_synset("03797390") ``` `TraversalWebDataset(urls, transform=...)` accepts a per-frame callable (`[3, H, W] → [3, H, W]`) if you want to plug in a torchvision augmentation pipeline. ### Using it without PyTorch The shards are ordinary compressed tar files, so nothing here is required. Each sample is one sequence, keyed by an 8-digit index: ``` 00000042.frame_0000.jpg … 00000042.frame_0031.jpg raw JPEG bytes 00000042.latents.npy [32, 10] float32 00000042.base_latent.npy [10] float32 00000042.velocities.npy [10] float32 00000042.meta.json {"synset_id", "obj_id", "seq_idx", "traversal_factors"} ``` `tar tf shards/train/shard-00000.tar | head` will show you the layout directly (use `tar tzf` if the shards are `.tar.gz` — `webdataset` reads either without any change to your code). --- ## The latent space Each scene is described by 10 continuous factors. `latents[t]` gives the exact state of frame `t`. | Index | Name | Range | Meaning | |---|---|---|---| | 0 | `rot_x` | [−π/8, π/8] | object rotation about x | | 1 | `rot_y` | [−π/8, π/8] | object rotation about y | | 2 | `rot_z` | [−π, π] | object rotation about z (circular) | | 3 | `floor_hue` | [0, 1] | hue of the ground plane (circular) | | 4 | `spot_theta` | [0, π/4] | spot light elevation | | 5 | `spot_phi` | [0, 2π] | spot light azimuth (circular) | | 6 | `spot_hue` | [0, 1] | spot light hue (circular) | | 7 | `trans_x` | [−0.5, 0.5] | object translation along x | | 8 | `trans_y` | [−0.5, 0.5] | object translation along y | | 9 | `trans_z` | [−0.5, 0.5] | object translation along z | Rotations are Tait-Bryan (XYZ extrinsic) Euler angles. The camera is fixed; apparent viewpoint change comes from object rotation. **How a sequence is built.** A `base_latent` is drawn per sequence and holds the scene fixed. A subset of factors (listed in `traversal_factors`) is then swept across the sequence at the rate in `traversal_velocities`: `0` means frozen, `±v` means it moves `v ×` its full range in that direction. The practical consequence: within one sequence, the varying factors are the *only* thing that changes, which is what makes clean invariance and equivariance measurements possible. **At the edge of a factor's range.** Circular factors (`rot_z`, both hues, `spot_phi`) wrap, so those sequences tile seamlessly. The bounded factors (`rot_x`, `rot_y`, `spot_theta`, the three translations) instead *reflect*: the value turns around and travels back, like an elastic collision. Position stays continuous — there is no visual jump or freeze — but the instantaneous velocity changes sign. This matters if you use `traversal_velocities` for equivariance targets, because it records only the velocity a sequence *started* with. Do not assume `latents[t] == base_latent + t · v · span / 31`; past a turning point the true motion is the other way. For `rot_x` and `rot_y` at `|v| = 1` a bounce is the common case, not an edge case — a full-span sweep from an interior start point has to reach a boundary. `latents` is always exact, so take instantaneous velocity from the frames themselves: ```python d = batch["latents"][:, 1:] - batch["latents"][:, :-1] # [B, 31, 10] ``` (For circular factors, wrap `d` into `[−span/2, span/2]` before using it.) Category labels come from `synset_labels.json`, which maps each synset ID to a name and a contiguous `class_index` in `[0, 12)`. --- ## Which objects are in which split Every split records its own membership, so you never have to scan the shards: ```python import json info = json.load(open("shards/test/dataset_info.json")) info["split_by"] # 'sequence' or 'object' — what the held-out split measures info["split_seed"] # reproduces the partition info["n_objects"] # number of distinct objects in this split info["objects"] # {synset_id: [obj_id, ...]} — exactly which ones ``` `stats.json` carries the same lists for all splits at once, together with `splits_are_object_disjoint` and a pairwise `object_overlap_between_splits` count — both measured from the shards themselves rather than copied from the metadata. --- ## Known issues and differences from 3DIEBench **Splits share objects.** The train/val/test split is a random shuffle of *sequences*, not of objects, so the same 120 object instances appear in all three splits. Held-out performance therefore measures generalisation to unseen *latent configurations*, not to unseen objects. If you need object-level held-out evaluation, use the ood-test split which contains ten different objects per synset. **Duplicated ShapeNet faces.** Most ShapeNetCore.v2 meshes store every face twice on the same shared vertices. Under a path tracer every ray hit then ties on depth, and the winner flips with BVH traversal order as the object rotates — producing polygon-bounded patches that switch on and off between frames. This is a systematic property of the ShapeNet export (a corpus scan found ~98% of models affected, most of them at exactly 2× copies per face), and it is visible wherever the two copies carry different materials. The generator removes the duplicates at load time. See `generation/provenance.json` for whether this release was rendered with that repair enabled; it is the main respect in which these renders differ from [3DIEBench](https://github.com/facebookresearch/SIE), which was produced from the same corpus without it. **Small object count.** 10 objects per category is deliberate, the design budget goes into 1000 sequences per object rather than into instance diversity. Treat category-level classification results accordingly. --- ## License The dataset shards, previews, and metadata are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). See `LICENSE` following the non-commercial Terms of Use dictated by ShapeNet. **Code** — The code to generate the dataset resides in a separate repository (https://github.com/fzenke/3D_latent_traversal_dataset_generator/). This code is released under GPL v3.0, inherited from generator Code underlying from Meta's SIE release. TLT12 is a derived work of ShapeNetCore.v2. The 3D models themselves are **not** redistributed here, but your use of these renders is additionally subject to the [ShapeNet Terms of Use](https://shapenet.org/terms). Where the two licenses differ, the more restrictive condition applies. --- ## Citation If you use TLT12, please cite the paper that introduced it: ```bibtex @article{gorganmohammadi2026jepa, title = {Understanding neural circuit principles for representation learning through joint-embedding predictive architectures}, author = {Gorgan Mohammadi, Ashena and Halvagal, Manu Srinath and Zenke, Friedemann}, journal = {eLife}, year = {2026}, doi = {10.7554/eLife.110895.1}, url = {https://doi.org/10.7554/eLife.110895.1}, } ``` Please also cite ShapeNet, and, since the data generation process was derived from it, SIE / 3DIEBench: ```bibtex @article{chang2015shapenet, title = {ShapeNet: An Information-Rich 3D Model Repository}, author = {Chang, Angel X. and Funkhouser, Thomas and Guibas, Leonidas and Hanrahan, Pat and Huang, Qixing and Li, Zimo and Savarese, Silvio and Savva, Manolis and Song, Shuran and Su, Hao and Xiao, Jianxiong and Yi, Li and Yu, Fisher}, journal = {arXiv preprint arXiv:1512.03012}, year = {2015} } @inproceedings{garrido2023sie, title = {Self-supervised learning of Split Invariant Equivariant representations}, author = {Garrido, Quentin and Najman, Laurent and LeCun, Yann}, booktitle = {International Conference on Machine Learning (ICML)}, year = {2023} } ``` --- ## Generating your own variant The generator is open source: **https://github.com/fzenke/3D_latent_traversal_dataset_generator** It renders arbitrary latent-traversal datasets from ShapeNet: different categories, resolutions, sequence lengths, factor subsets, and traversal velocities. `generation/tlt12_commands.sh` records the exact pipeline used for this release; `generation/provenance.json` pins the generator commit. You will need your own copy of ShapeNetCore.v2, which requires free registration at [shapenet.org](https://shapenet.org/).