Repository navigation
tests: per-layer recompute under the exact walk takes the same step in a smaller chunk graph - #50
Merged
Merged
Conversation
* ci : add older, min and dry-run options to ccache-clear Assisted-by: pi:llama.cpp/Qwen3.8-27B * pi : add note about not wrapping lines in PR descriptions [no ci] Assisted-by: pi:llama.cpp/Qwen3.8-27B
* text: conversion * init impl * mtmd: conversion * impl mtmd cpp * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Revert "ci : disable ubuntu-rocm (ggml-org#26969)" This reverts commit 9558fa4. * ci: set ccache compiler_check=content for ROCm build The ROCm toolchain is pip-installed fresh on every run, so the clang binary's mtime changes each time. With ccache's default compiler_check=mtime that invalidates the whole cache and warm builds only reached ~70% hits. Hash the compiler contents instead so the cache survives toolchain reinstalls. * Update ccache size to 1GB We're waivering with so many architectures built, we need a bigger ccache limit. * merge fix --------- Co-authored-by: Jim Wu <ywu@xilinx.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* arg: remove -no-cnv from cli * clarify about not adding exccesive test cases
* fit: also take into account n_streams * server: make the draft context follow the target context With a non-unified KV cache the target context now holds n_ctx_train tokens per sequence, while the draft context was still created with n_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot filled beyond that point makes the draft batch fail to decode, and the server answers 500 on the request. The draft context now takes its size from the target context, so both hold the same number of tokens per sequence. Contexts that share their cells with the target no longer need the kv_size override. The memory reserved for the draft model before fitting is measured at the largest context the target can take, since the draft context grows with the target and a fixed byte margin cannot express that. * fit: take an optional second model into account Illustrates the alternative discussed on the draft context fix. The memory of a draft or MTP context is currently handed to the fit as a fixed byte margin, which cannot express a memory that grows with the context the fit is still deciding on. common_fit_params now takes an optional second model that shares the devices of the main one. Its context follows the main context and its memory is measured again whenever that context changes, so the reduce path stays exact instead of conservative. A model that cannot be measured on its own, such as a shared cell MTP context, is skipped with a warning and the main model is fitted alone. This drops the reservation block in the server, which no longer has to probe the trained context size of the target to guess an upper bound. --------- Co-authored-by: Pascal <admin@serveurperso.com>
* add common/json * migrate common * adapt jinja * migrate server * big wip * migrate tests * wip * revert some excessive changes * wip * wip 2 * revert redundant changes * fix server crash * various fixes * fix ci * harden a bit * clean up * rm json-shim * add some comments * rm redundant decl
* mtmd: use ggml_rope_set_offset * add comment
* vulkan : added PAD_REFLECT_1D operation Implemented the GGML_OP_PAD_REFLECT_1D operation for the Vulkan backend Changes: - pad_reflect_1d.comp: implemented the GLSL compute shader with reflection logic - vulkan-shaders-gen.cpp: register the shader for SPIR-V compilation - ggml-vulkan.cpp: pushed constants struct, pipeline creation, supports_op, dispatch function, compute switch and debug validation Tested the PAD_REFLECT_1D on Intel Iris Xe (Vulkan 1.4, Mesa 25.2.8): Correctness: PAD_REFLECT_1D(type=f32,ne_a=[512,34,2,1],pad_0=10,pad_1=9) = Pass PAD_REFLECT_1D(type=f32,ne_a=[3000,384,4,1],pad_0=10,pad_1=9) = Pass 2/2 tests passed - All test are passed Performance: ne_a=[512,34,2,1] -> 5.38 us/run, 24.55 GB/s ne_a=[3000,80,1,1] -> 30.09 us/run, 59.62 GB/s ne_a=[3000,384,4,1] -> 158.31 us/run, 54.39 GB/s * Update ggml/src/ggml-vulkan/vulkan-shaders/pad_reflect_1d.comp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* cuda : add POOL_1D support * fix: add missing trailing newline for editorconfig compliance
* ui : add browser-style conversation tabs store Track open conversation tabs in order, persisted to localStorage and pruned against the loaded conversation list on init. The chat layout syncs the route's tab on every navigation, so any way of reaching a conversation opens a tab for it. * ui : add temporary new-chat tabs New-chat tabs are unsaved conversations carrying a temporary id used directly as the route (#/chat/<id>). They live in memory and are only persisted to the database - keeping the same id so the route and tab stay stable - when the first message is sent. Deleting one drops it without confirmation, and deleting conversations now closes their tabs. * ui : render conversation tab bar in chat layout Desktop-only tab bar above the chat screen, one tab per open conversation or new-chat tab. The active tab follows the route id; clicking navigates, middle-click or the close button closes (switching to the left neighbor), and a trailing + starts a new chat. Tabs appear only on chat-id routes; the bare #/ new-chat view has none. The bare route stays put unless a prompt/model deep-link routes it to a new-chat tab. * ui : route new-chat entry points through tabs The sidebar New chat item, Cmd+Shift+O, the search page and the arrow-key fallback now open a new-chat tab instead of navigating to the ?new_chat URL, which is removed. New chat is no longer a special route but a tab like any other conversation. * ui : track sidebar expanded state in a shared ui store Move the desktop sidebar expanded/collapsed state out of deviceStore into a dedicated uiStore so the chat tab bar can react to it. Assisted-by: pi * chat : add opt-in conversation tabs setting Add a Display setting that turns browser-style conversation tabs on or off, enabled by default. Assisted-by: pi * chat : add browser-style conversation tabs with a new-chat screen Track open conversations as tabs above the chat, one per open chat, plus a single New chat tab for the bare `#/` route. New chat is just the `#/` screen - no temporary conversations - and its tab is dropped when navigating away. Sending the first message creates a real conversation and opens a tab for it. Assisted-by: pi * chat : turn tab bar into a horizontally scrollable carousel Make the tab bar a horizontally scrollable carousel with edge scroll buttons and active-tab centering, and align its styling with the sidebar. Assisted-by: pi * chat : restyle the scroll-to-bottom button to match tab styling Assisted-by: pi * chat : add close-tab keyboard shortcut Assisted-by: pi * chat : soften tab bar fade and dim inactive tabs Assisted-by: pi * feat: Add stop button to tabs * refactor: Componentize * ui : fix carousel scrollability detection Observe the content wrapper as well as the container, since adding overflowing items does not change the container's own box size. Also expose an onScrollableChange callback. Assisted-by: pi * ui : add unified ScrollCarousel component Single carousel component with top/center variants, gap and scroll options, and hover-revealed chevrons. Rename the HorizontalScrollCarousel accessibility story accordingly. Assisted-by: pi * ui : migrate carousels to ScrollCarousel Switch the settings mobile header, attachments list, thumbnail strip, and MCP resources to the unified component, and drop HorizontalScrollCarousel. Assisted-by: pi * ui : improve chat tabs carousel UX Scroll newly added tabs into view, fade overflowing tabs at the edges, and hide the New chat button while a new-chat tab is open. Assisted-by: pi * refactor: Naming * chat : add keyboard shortcut to jump between conversation tabs Shift+Cmd/Ctrl+Left/Right cycles the open tabs, mirroring the existing Shift+Cmd/Ctrl+Up/Down conversation navigation. Assisted-by: pi * chat : make the whole tab item act as a link The full tab is now a link instead of only the inner label button, while the stop and close buttons stay interactive by swallowing their clicks. Assisted-by: pi * chat : adjust tab bar width and use a shared offset variable Widen the tab bar for the expanded sidebar and rename the tab bar height variable to --chat-tabs-offset with a smaller value so the chat screen min-height accounts for the overlay without overshooting. Assisted-by: pi * chat : account for the tab bar offset in the assistant min-height Subtract the tab bar offset when it is shown so the last assistant message does not overflow the available viewport space. Assisted-by: pi * refactor: Post-review fixes * ui : restore deep links on the chat start page - handle ?model selection, with ?load=true eager router loading - ?q now creates a conversation, sends the prompt, and clears the params - show the not-available-model dialog for unknown models - never block mount on the conversation list Assisted-by: pi * ui : fix tab item link nesting and centralize tab constants - the tab anchor covers the whole item while stop/close stay siblings, so interactive elements are never nested inside the anchor - cmd/ctrl/middle clicks are left to the browser (new window) - extract the tab labels, the active-tab data attribute, and the sidebar-offset max widths into constants Assisted-by: pi * ui : tidy scroll carousel hook and keep mobile header arrows on - drop the dead scrollLeft/scrollRight helpers and the unused onScrollableChange/scrollBy props - init the carousel once instead of inside a derived - restore items-start on the center variant - always show the settings header arrows on touch Assisted-by: pi * ui : keep the new-chat tab across reloads and fall back on close - the new-chat sentinel is no longer pruned on init, so reloading on the bare new-chat route keeps the tab the user is on - closing the active conversation falls back to the new-chat screen when Conversation tabs are off Assisted-by: pi * ui : don't block startup on the conversation list - prune persisted tabs after the list loads in the background instead of awaiting it during init - openNewChat now returns void; its return value was never read Assisted-by: pi * ui: fix routing nits * chore: Update doc comments * refactor: Mark fire-and-forget openNewChat calls as `void` * chat: fix the deep-linked prompt, the tab width and the tab shortcuts The chat start page creates the conversation and hands the prompt over to the chat route, which still sees it in the query string. Sending it on both sides queues the second copy as a pending message, which shows up as a stray user bubble once the answer lands and vanishes on reload since it never reaches the database. The tab bar takes the max width of the collapsed sidebar while it is expanded, and the other way round. The tab list is pruned against a snapshot of the loaded conversations, so a conversation created while that list is still loading loses its tab even though the route just opened it. The active tab then falls out of the list and the cycling shortcut jumps to an edge on every keypress instead of moving one tab over. Tabs synced from the route are kept as they are, only the persisted ones are pruned. The rich chat input claims ctrl or alt with shift and an arrow for its badge-aware word jump, which now belongs to the tab cycling shortcut. Holding shift hands the key combination over, the plain word jump is unchanged. The close-tab shortcut consumes the event before checking whether the setting is on, and the logo background loses its importance flag. --------- Co-authored-by: Pascal <admin@serveurperso.com>
…7574) * ggml : fix meta tensor split state propagation * Add test-llama-archs to CI
* DeepseekV4: fix rollback with multi-seq * fix model loading * make pending rollback single use * only clear cache for seq_id for full load * add assert for compress ratio * make graph topology static * pass true instead of flags in clear_compressed * cont : clean-up + TODOs --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…org#26692) The device_info loop iterates over the discovered devices and gets the available and total memory counts. With the CUDA backend (and possibly others too) this requires creating a GPU context, which, in case of CUDA, results in a 550 MB VRAM allocation. For this information to be used in any way, the log verbosity must be set to LOG_LEVEL_TRACE. If it's not, including in the default configuration, the contexts get created, memory sizes get queried, then the log function quietly discards the data. In certain cases the user may not want to use any GPU resources. The device_loop iteration is the only place touching the GPU that cannot be skipped. Fix by checking the verbosity level and skipping the loop if there would be no output.
Run test-llama-archs with 1 to 4 GGML_METAL_DEVICES, mirroring the existing CUDA runs, and dispatch the job unconditionally since the per-backend guards now decide what to run. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
…gml-org#27594) * mtmd: use pillow-accurate resize algo, correct resize_algo for all models * speed optimization
* tests : fix tests for multi-gpu environment * cont : not needed
* readme : update links * readme : update maintainer PRs list Add the new members of the `ggml-org` `maintainers` team to the author filter of the maintainer PRs link (nikwen, marty1885, Titaniumtown), keeping the canonical team ordering. The list now matches the team exactly (35 members). Assisted-by: pi:llama.cpp/Qwen3.8-27B
* DSV4: sm tensor * set coarser granularity for head splits * fix dspark * add model saving for dsv4 + allow dflash to return on specific device * add comment about dsv4 seq_rm * simplify * add shared expert delayed allreduce * remove special test for dsv4
…gml-org#27513) * mamba2 : flatten mamba2 in/out projections to dispatch gemm instead of gemv * mamba2 : remove redundant output reshape
* ggml : fix ggml_clamp * cont : update ggml-alloc
…ml-org#27602) * ci : apply ccache-clear with older/min/dry-run to all ccache jobs Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : install gh in ccache-clear if missing (container jobs) The ccache-clear action relies on the gh CLI, which is not present in container-based jobs. Install it on demand so those jobs can clear caches. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : install gh via apt repo in ccache-clear The install.sh script used previously is no longer served (404). Switch to the official GitHub CLI apt repository, which is still available. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : pass --repo to gh cache commands in ccache-clear In container jobs gh cannot auto-detect the repository from git, so gh cache list/delete fail with 'failed to run git: not a git repository'. Pass the repository explicitly via --repo using GITHUB_REPOSITORY. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : drop -new suffix from vulkan ccache key The -new suffix was only needed to force a fresh cache. With ccache-clear now evicting stale caches, the original key can be used again. The old ccache-vulkan-ubuntu-24.04-arm-new entries still match the ccache-clear key prefix and are cleaned up automatically. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : fix ccache-clear date parsing on macOS (BSD date) macOS ships BSD date, which has no -d option. The older cutoff check was silently disabled there: 'date: illegal option -- d' errors in the log and the loop was only stopped by the min limit, risking deletion of caches not older than the cutoff (e.g. saved by a concurrent job). Parse the ISO-8601 timestamps with GNU date when available and fall back to BSD date otherwise (TZ=UTC, fractional seconds dropped). Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : extract ccache-clear logic into scripts/ccache-clear.sh The composite action now consists of a dedicated step that installs the GitHub CLI when missing (e.g. in container jobs) and a thin step that calls the new script. The script follows the make-release-checks.sh conventions (usage/env header, set -euo pipefail, CLI flags) and only checks that gh is available. The action inputs are unchanged, so the workflow steps are untouched. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : remove unused apple ccaches
* mtmd: video: fix moov at the end of file Co-authored-by: rkfg <rkfg@rkfg.me> * fix SIGPIPE * windows: handle broken pipe case --------- Co-authored-by: rkfg <rkfg@rkfg.me>
…d chunk at a smaller horizon A chunk graph the device budget refuses is refused AFTER its ubatch was applied: the chunk's cells already sit in the attention cache, and on a hybrid model the recurrent cell's position has moved to the chunk's end. The adaptive horizon (#47) retried train_chunk on top of that, so the retry could not prepare its ubatch at all, and the epoch stopped instead of shrinking to a horizon that fits. Measured on the 5090 (2026-10-10 04:34Z), Kimi's first exact-walk job on Qwen3.8-27B: "the graph needs 9236.1 MiB more on CUDA0, over the 3418.0 MiB it may add" -> "the chunk at 1155 did not fit with a gradient horizon of 1155 positions: retrying at 512" -> "init_batch: failed to prepare attention ubatches" -> the job failed in 8 s. The reverse pass's rewind (pop attention to [0, c0); restore the recurrent checkpoint and decode to c0) is now one lambda, run before the first try AND before every retry, with the walk flags cleared around the retry's rewind so its decode is the same plain forward as the first. test-walk-exact case 6 (2048-token window): a device budget that lets a chunk graph grow by nothing refuses every horizon; each retry must reach the device preflight again with zero ubatch failures, and the epoch then refuses by name with the adapter untouched. On the 5090: with the fix, 3 retries (1920 -> 896 -> 384 -> 128), 4 refusals, 0 ubatch failures, OK on Qwen3.5-0.8B (hybrid) and Qwen2.5-Coder-1.5B; without it, 1 retry and 1 "failed to prepare attention ubatches", FAILED, which is the production signature. Every other case is unchanged (exact vs one graph cosine 0.998 / 0.998). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
fix(train): the exact walk rewinds the cache before retrying a refused chunk at a smaller horizon
…n a smaller chunk graph Kimi's first exact run on the 5090 (2026-10-10 08:32Z, job 888ce9c6) was refused at chunk 256 needing 5.6 GiB more under every gradient horizon from 1155 down to 256: the chunk's activations, every layer's kept for the backward pass, not the horizon. Per-layer recompute (#37) is that memory, but it is opt-in on /train and the core never asks for it, and nothing tested it together with the exact walk. Case 7: the same masked window, exact walk in 4 chunks, with and without recompute. Measured on the 5090: Qwen2.5-Coder-1.5B Q4_K_M: cosine 1.0000, norm 1.0000; chunk graph 657.6 -> 286.1 MiB Qwen3.5-0.8B Q8_0 (hybrid): cosine 1.0000, norm 1.0000; chunk graph 847.4 -> 534.4 MiB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kimi's first exact run on the 5090 (2026-10-10 08:32Z, job 888ce9c6) was refused at chunk 256, needing 5.6 GiB more under every gradient horizon from 1155 down to 256. #49's retry ladder worked, but the horizon wasn't the memory. The chunk's own activations (every layer's, kept for the backward pass) were. Per-layer recompute (#37) removes exactly that memory, but it's opt-in on
/train, the core never sends it, and nothing tested it together with the exact walk.Case 7: the same masked window, exact walk in 4 chunks, with and without recompute (5090, CUDA):
Both models pass the full test (cases 1–6 unchanged). The core change that sends
recompute: truewith the exact walk follows in continuum.🤖 Generated with Claude Code
https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc