Skip to content

feat(otel): add opt-in gzip compression for the default OTLP exporter - #1921

Open
DavidTraina wants to merge 3 commits into
langfuse:mainfrom
DavidTraina:feat/otel-compression
Open

DavidTraina wants to merge 3 commits into
langfuse:mainfrom
DavidTraina:feat/otel-compression

Conversation

@DavidTraina

@DavidTraina DavidTraina commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds an opt-in way to gzip the span batches sent by the default Langfuse OTLP exporter. It works through a new Langfuse(otel_compression=...) argument or a new LANGFUSE_OTEL_COMPRESSION env var.

Today the only way to compress is the standard OTEL_EXPORTER_OTLP_(TRACES_)COMPRESSION env vars. They work only because Langfuse never passes compression to OTLPSpanExporter. They also apply to every OTLP exporter in the process, and the SDK does not mention them.

Behavior:

  • Accepted values are gzip and none, trimmed and case-insensitive. Other values such as deflate log a warning and are ignored.
  • Precedence: otel_compression > LANGFUSE_OTEL_COMPRESSION > OTEL_EXPORTER_OTLP_TRACES_COMPRESSION > OTEL_EXPORTER_OTLP_COMPRESSION > none.
  • When neither Langfuse setting is set, the exporter gets compression=None. Existing setups that use the OTEL env vars behave as before.
  • The default is unset, so nothing changes for existing users. Open question for maintainers: should gzip become the default later? Every supported server (>= v3.63.0) decodes it.
  • Only the default exporter is affected. The span_exporter docstring now says a custom exporter must configure compression itself. Passing otel_compression together with span_exporter logs a warning.
  • The LANGFUSE_OTEL_MAX_BATCH_SIZE_BYTES cap is unchanged. It measures uncompressed bytes, which matches the server's limit after decompression.

Server support:

  • The OTLP endpoint has decoded gzip since v3.30.0 (fix: support gzip encoding on otel endpoint langfuse#5710), in web/src/pages/api/public/otel/v1/traces/index.ts.
  • It does not decode deflate, br, or zstd. Those requests get a 400, which the exporter does not retry, so the batch is lost. That is why deflate is not offered, even though the OTel Python exporter supports it.

Measured against a local Langfuse v4.50.0, with one batch of 11 spans carrying about 5 KB of chat-style input and output each:

Setting Bytes on the wire
unset 52,330
otel_compression="gzip" 15,584
LANGFUSE_OTEL_COMPRESSION=gzip 15,573

All three batches were ingested with input and output intact.

Companion JS PR: langfuse/langfuse-js#974

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Refactor
  • Documentation update
  • Tooling, CI, or repo maintenance

Verification

List the main commands you ran:

uv run --frozen ruff check .                                # All checks passed!
uv run --frozen ruff format --check <changed files>         # 6 files already formatted
uv run --frozen mypy langfuse --no-error-summary            # no errors
uv run --frozen pytest -n auto --dist worksteal tests/unit  # 714 passed, 2 skipped

ruff format --check . also flags tests/unit/test_media.py. That file is unchanged here and already fails on main.

The new tests in tests/unit/test_span_processor.py export through the real exporter to the local otlp_http_server fixture. They check Content-Encoding and decode the body. They cover the argument through the full client, the env var, and precedence against the OTEL env vars. With compression=None hardcoded, 4 of the 5 fail. The remaining one checks that the OTEL env vars still apply when nothing is set.

Not run: tests/e2e and tests/live_provider, which need a server and provider keys.

Checklist

  • I self-reviewed the diff using code_review.md.
  • I added or updated tests for behavior changes.
  • I updated docs, examples, or .env.template if needed.
  • I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
  • I did not commit secrets or credentials.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Comment on lines +78 to +84
if raw_value is None:
setting = LANGFUSE_OTEL_COMPRESSION
raw_value = os.environ.get(LANGFUSE_OTEL_COMPRESSION, "")

value = raw_value.strip().lower()
if not value:
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Blank argument skips environment setting

If otel_compression is empty or contains only spaces while LANGFUSE_OTEL_COMPRESSION=gzip, this return skips the valid environment setting and silently falls back to the standard OTEL settings. Treat a blank argument as unset, or reject it consistently with other invalid values.

Suggested change
if raw_value is None:
setting = LANGFUSE_OTEL_COMPRESSION
raw_value = os.environ.get(LANGFUSE_OTEL_COMPRESSION, "")
value = raw_value.strip().lower()
if not value:
return None
if raw_value is None or not raw_value.strip():
setting = LANGFUSE_OTEL_COMPRESSION
raw_value = os.environ.get(LANGFUSE_OTEL_COMPRESSION, "")
value = raw_value.strip().lower()
if not value:
return None
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/_client/span_processor.py
Line: 78-84

Comment:
**Blank argument skips environment setting**

If `otel_compression` is empty or contains only spaces while `LANGFUSE_OTEL_COMPRESSION=gzip`, this return skips the valid environment setting and silently falls back to the standard OTEL settings. Treat a blank argument as unset, or reject it consistently with other invalid values.

```suggestion
    if raw_value is None or not raw_value.strip():
        setting = LANGFUSE_OTEL_COMPRESSION
        raw_value = os.environ.get(LANGFUSE_OTEL_COMPRESSION, "")

    value = raw_value.strip().lower()
    if not value:
        return None
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. otel_compression is typed Literal["gzip", "none"], so a blank string is not a supported value for the argument. The blank check exists for LANGFUSE_OTEL_COMPRESSION set to an empty string in the environment. The JS SDK treats its compression option the same way.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That distinction is reasonable. Since otel_compression is typed as Literal["gzip", "none"], blank values are only expected from LANGFUSE_OTEL_COMPRESSION, where the existing empty-string handling is sufficient. I’ll withdraw this concern; no change is needed.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant