Pretraining and inference code for a large-scale depth-recurrent language model
-
Updated
Dec 29, 2025 - Python
#
recurrent-depth
Here are 10 public repositories matching this topic...
Awesome list of papers, code, models and blogs on Looped / recurrent-depth / weight-tied Transformers — depth as a third scaling axis
awesome-list universal-transformer large-language-models looped-transformers test-time-compute recurrent-depth latent-reasoning
-
Updated
Sep 5, 2026
Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.
ai reproducible-research benchmarks personal-assistant quantization external-memory gemma virtual-assistant distillation inference-optimization tool-use edge-inference int4 llm long-context small-language-models kv-cache-compression recurrent-depth 4gb-vram verifier-rl
-
Updated
Sep 8, 2026 - Python
Awesome list + deep-dive reports on looped / recurrent-depth transformers (2018-2026): the Loop technology behind GPT-6 Astra. 122 papers, ZH/EN reports, translated PDFs, survey, slides.
transformers awesome-list universal-transformer llm test-time-compute recurrent-depth latent-reasoning looped-transformer
-
Updated
Sep 8, 2026 - Python
Recurrent-depth transformer, fixed. Fork of kyegomez/OpenMythos with scatter-based MoE (2.94x faster), proper ACT halting, DeepSeekMoE load balancing, SDPA kernels, and a working training loop.
pytorch transformer moe claude mla adaptive-computation-time mixture-of-experts deepseek recurrent-depth
-
Updated
Apr 20, 2026 - Python
Loop pretrained language model layers for deeper reasoning without new parameters. Open training, evaluation, and export to standard Transformers and vLLM checkpoints.
pytorch reasoning llm vllm nanotron test-time-compute recurrent-depth latent-reasoning qwen3 looped-transformer
-
Updated
Oct 7, 2026 - Python
Does the Universal/Looped-Transformer trick (reuse one block N times instead of stacking N) work on Mamba? Weight-shared depth in a state-space model, with honest per-seed results.
deep-learning reproducible-research pytorch ssm mamba state-space-models weight-sharing universal-transformer sequence-modeling recurrent-depth parameter-efficiency looped-transformer associative-recall
-
Updated
Sep 15, 2026 - Python
A teaching-scale looped transformer (recurrent-depth) inspired by GPT-6 Astra, built from scratch in PyTorch. Features shared-weight loops, per-token depth policies, test-time loop scaling, and audited runs.
deep-learning transformers pytorch transformer educational language-models language-model from-scratch llm looped-transformers test-time-compute recurrent-depth latent-reasoning mixture-of-recursions gpt6 gpt6astra
-
Updated
Oct 10, 2026 - Python
Measured MLX/CUDA pretraining experiments: MoE, Muon, recurrent truncated backprop, fused loss and batching. Includes failures, raw scalar measurements and limits.
cuda muon language-models mlx mixture-of-experts pretraining negative-results apple-silicon recurrent-depth truncated-backpropagation
-
Updated
Oct 6, 2026 - Python
Recurrent-depth, P/R/C looped transformer with MoE, GQA, and stable recurrence
ml transformers pytorch mixture-of-experts gqa flash-attention looped-transformers recurrent-depth lti-stability
-
Updated
Oct 6, 2026 - Python
Add this topic to your repo
To associate your repository with the recurrent-depth topic, visit your repo's landing page and select "manage topics."