Skip to content

ExLlamaV3: wire up LoRA loading, listing and unloading - #526

Open
DailyDisco wants to merge 1 commit into
theroyallab:mainfrom
DailyDisco:feature/exl3-lora-endpoints
Open

DailyDisco wants to merge 1 commit into
theroyallab:mainfrom
DailyDisco:feature/exl3-lora-endpoints

Conversation

@DailyDisco

Copy link
Copy Markdown
Contributor

Is your pull request related to a problem? Please describe.

The ExLlamaV3 backend's load_loras, get_loras and unload(loras_only=True) are stubs ("LoRAs aren't hooked up to
the ExLlamaV3 backend yet"). So /v1/lora/load, /v1/lora, /v1/lora/unload and the lora: section of config.yml
do nothing, even though exllamav3 has had PEFT adapter support (exllamav3.model.lora.LoRA) since v0.0.31. #434 made
a start on this. This PR is a complete version that also covers listing, the prompt cache and the configurations
that can't work.

Why should this feature be added?

  • Loading: each adapter is attached with LoRA.from_directory in a worker thread, under the load lock, after
    running jobs finish (or are cancelled with skip_queue), the same way a model load waits. An adapter that is
    missing or unreadable, or that matches no layer of the model (an adapter for a different base model), is reported
    in failure. The others still load.
  • Prompt cache: every LoRA load or unload rebuilds the generator (create_generator(lock_held=True)). The
    generator reuses K/V pages and recurrent checkpoints by token prefix. Once the weights change, a prompt that
    shares a prefix with an earlier one (usually the system prompt) would otherwise reuse state computed under the old
    weights.
  • Listing: get_loras() keeps the requested name and scaling for /v1/lora. get_active_loras read exl2's
    lora_path, which exllamav3's LoRA doesn't have, and exllamav3's lora_scaling is the effective factor (alpha/r
    and rsLoRA folded in), not the requested scaling.
  • Refused, with an error that says why:
    • Tensor parallelism: exllamav3 skips TP-sliced layers, so the adapter would apply partly or not at all.
    • disk_cache_dir: the disk tier's fingerprint covers the model weights and cache layout but not runtime
      adapters, and it outlives the process. It would serve base-model pages to adapter prompts and the reverse.

Examples

End-to-end on an RTX 3090, with each server started from config.yml and driven over HTTP. Run A loads the adapter at
startup through lora.loras. Run B starts without it and then calls the endpoints. Greedy 64-token completions of the
same prompt, which is long enough to fill two cache pages:

Check Qwen3.5-0.8B EXL3 4.0bpw, synthetic r8 adapter Qwen3.8-27B EXL3 5.0bpw, harness-hub/Qwen3.8-27B-pi-adapted
Runtime /v1/lora/load gives the same completion and logprobs as loading at startup yes (logprobs identical) yes (logprobs identical)
Adapter changes the output (mean abs. difference in chosen-token logprob vs base) different text; 0.36 over the first 8 tokens same text; 6.1e-3 over all 64 tokens
Second base request reuses the prompt cache 512 cached tokens 512 cached tokens
First request after load, and after unload, reuses nothing 0 and 0 cached tokens 0 and 0 cached tokens
/v1/lora synthetic-r8, scaling 1.0 qwen3.8-27b-pi, scaling 1.0
Missing adapter in failure in failure
After unload base completion and logprobs base completion and logprobs
  • tests/test_lora.py (7 tests, CPU only, containers built with __new__ like tests/test_generator_recovery.py)
    covers:
    • one generator rebuild per load request, with the old generator closed
    • partial failure: a missing adapter and an adapter that matches no layer (that one is detached again), plus a
      nameless config entry that is skipped instead of reported as None
    • no rebuild when nothing loaded
    • TP and disk-cache refusals that never touch the model
    • unload detaching every adapter without unloading the model, and staying a no-op when nothing is loaded
    • a request waiting on the load condition being released afterwards
  • Results: tests/ (minus the req_* scripts that need a running server) gives 530 passed. ruff format --diff and
    ruff check are clean.

Additional context

  • exllamav3 1.6.7 and earlier skip attached adapters on the fused and graph-captured decode paths of EXL3 attention,
    gated MLP and GDN layers. The adapter applies to prompt prefill but mostly not to generated tokens.
    LoRA: keep adapters applied on the fused decode paths turboderp-org/exllamav3#607 fixes that. This PR works with any exllamav3 that has exllamav3.model.lora, but the
    output only matches the adapter with that fix. The runs above used it.
  • A load request still unloads the current LoRAs first (common/model.py), so swapping adapters rebuilds the
    generator twice. I left that as it is.

load_loras, get_loras and unload(loras_only=True) were stubs, so the
LoRA endpoints and the lora section of config.yml did nothing, though
exllamav3 has had PEFT adapter support (exllamav3.model.lora) since
v0.0.31.

- Adapters attach under the load lock once running jobs finish (or are
  cancelled with skip_queue), loaded in a worker thread. Missing ones
  and ones that match no layer of the model are reported in failure;
  the rest still load.
- Every change rebuilds the generator: it reuses K/V pages and
  recurrent checkpoints by token prefix, so after the weights change a
  shared prefix would reuse state computed under the old ones.
- get_loras keeps the requested name and scaling for /v1/lora, since
  exllamav3's lora_scaling is the effective factor.
- Refused with tensor parallelism (exllamav3 skips TP-sliced layers)
  and with disk_cache_dir (its fingerprint doesn't cover runtime
  adapters, and it outlives the process).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant