Repository navigation
ExLlamaV3: wire up LoRA loading, listing and unloading - #526
Open
DailyDisco wants to merge 1 commit into
Open
DailyDisco wants to merge 1 commit into
DailyDisco wants to merge 1 commit into
Conversation
load_loras, get_loras and unload(loras_only=True) were stubs, so the LoRA endpoints and the lora section of config.yml did nothing, though exllamav3 has had PEFT adapter support (exllamav3.model.lora) since v0.0.31. - Adapters attach under the load lock once running jobs finish (or are cancelled with skip_queue), loaded in a worker thread. Missing ones and ones that match no layer of the model are reported in failure; the rest still load. - Every change rebuilds the generator: it reuses K/V pages and recurrent checkpoints by token prefix, so after the weights change a shared prefix would reuse state computed under the old ones. - get_loras keeps the requested name and scaling for /v1/lora, since exllamav3's lora_scaling is the effective factor. - Refused with tensor parallelism (exllamav3 skips TP-sliced layers) and with disk_cache_dir (its fingerprint doesn't cover runtime adapters, and it outlives the process).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Is your pull request related to a problem? Please describe.
The ExLlamaV3 backend's
load_loras,get_lorasandunload(loras_only=True)are stubs ("LoRAs aren't hooked up tothe ExLlamaV3 backend yet"). So
/v1/lora/load,/v1/lora,/v1/lora/unloadand thelora:section of config.ymldo nothing, even though exllamav3 has had PEFT adapter support (
exllamav3.model.lora.LoRA) since v0.0.31. #434 madea start on this. This PR is a complete version that also covers listing, the prompt cache and the configurations
that can't work.
Why should this feature be added?
LoRA.from_directoryin a worker thread, under the load lock, afterrunning jobs finish (or are cancelled with
skip_queue), the same way a model load waits. An adapter that ismissing or unreadable, or that matches no layer of the model (an adapter for a different base model), is reported
in
failure. The others still load.create_generator(lock_held=True)). Thegenerator reuses K/V pages and recurrent checkpoints by token prefix. Once the weights change, a prompt that
shares a prefix with an earlier one (usually the system prompt) would otherwise reuse state computed under the old
weights.
get_loras()keeps the requested name and scaling for/v1/lora.get_active_lorasread exl2'slora_path, which exllamav3'sLoRAdoesn't have, and exllamav3'slora_scalingis the effective factor (alpha/rand rsLoRA folded in), not the requested scaling.
disk_cache_dir: the disk tier's fingerprint covers the model weights and cache layout but not runtimeadapters, and it outlives the process. It would serve base-model pages to adapter prompts and the reverse.
Examples
End-to-end on an RTX 3090, with each server started from config.yml and driven over HTTP. Run A loads the adapter at
startup through
lora.loras. Run B starts without it and then calls the endpoints. Greedy 64-token completions of thesame prompt, which is long enough to fill two cache pages:
/v1/lora/loadgives the same completion and logprobs as loading at startup/v1/lorasynthetic-r8, scaling 1.0qwen3.8-27b-pi, scaling 1.0failurefailuretests/test_lora.py(7 tests, CPU only, containers built with__new__liketests/test_generator_recovery.py)covers:
nameless config entry that is skipped instead of reported as
Nonetests/(minus thereq_*scripts that need a running server) gives 530 passed.ruff format --diffandruff checkare clean.Additional context
gated MLP and GDN layers. The adapter applies to prompt prefill but mostly not to generated tokens.
LoRA: keep adapters applied on the fused decode paths turboderp-org/exllamav3#607 fixes that. This PR works with any exllamav3 that has
exllamav3.model.lora, but theoutput only matches the adapter with that fix. The runs above used it.
common/model.py), so swapping adapters rebuilds thegenerator twice. I left that as it is.