Covenant

Turn HTTP APIs, Redis data, TLS certificates and file checks into Prometheus metrics using YAML configuration.

Covenant is a configurable, multi-endpoint Prometheus exporter. It separates data collection from transformation and metric exposition, so you can adapt a source without writing a separate exporter for every service.

Prometheus evaluates alert rules; Alertmanager handles notification delivery, grouping and silences. Covenant does not implement a notification engine.

The optional CertLord collector exposes validated certificate observations, with sample Prometheus rules for expiry, retries and source freshness. It requires Python 3.8+ and is included from Covenant 0.0.71.

Why Covenant?

Your internal API already knows how many orders are waiting, how many workers are active or whether a background task completed. Covenant can extract those values and expose them to Prometheus.

It is useful for internal services and unusual data sources, particularly when you need several custom exporters with similar collection logic. Dedicated exporters remain a good choice when they already cover your service. For complicated business logic, a small purpose-built exporter may be easier to maintain than a long chain of configuration filters.

Quickstart: a JSON API to Prometheus metrics

This repository includes a local example with a small HTTP server and Covenant. It needs Docker with Compose and does not require Nginx, Redis or an external API.

git clone https://github.com/decryptus/covenant.git
cd covenant
docker compose -f examples/quickstart/compose.yml up -d
curl --fail http://localhost:9118/metrics/orders

The source JSON contains:

{"orders":{"pending":12},"workers":{"active":3}}

The output includes these samples, along with Prometheus HELP and TYPE lines:

orders_source_up 1.0
orders_pending 12.0
orders_workers_active 3.0

Edit examples/quickstart/data/status.json, then call the endpoint again to see the new values. Collection happens when the endpoint is scraped.

The example pins decryptus/covenant:0.0.71 and exposes port 9118 only on the local host. If the first request arrives before the API is ready, retry it.

# Inspect logs or stop the example.
docker compose -f examples/quickstart/compose.yml logs covenant
docker compose -f examples/quickstart/compose.yml down

How it works

1. A client requests /metrics/<endpoint> or /probe/<endpoint>.
2. The endpoint's plugin reads its configured source or sources.
3. Filters extract and transform values and labels.
4. Collectors update a dedicated registry and return Prometheus text.

Configuration term	Meaning
Endpoint	A named collection exposed through an HTTP route, such as orders
Plugin	The source adapter: http, redis, ssl or filestat
Target	A collection definition with a name, source config and collects
value_tasks	An ordered list of transformations applied to a metric value
Labels	Static or extracted dimensions attached to metrics
Template	An imported YAML/Mako file parameterized through endpoint vars

An endpoint defines either metrics or probes, not both. The shipped module files provide the routes; /probe/ and /probes/ are both accepted for probes.

Define your own API metrics

Here is the endpoint section from the quickstart, reduced to one metric. Place it under the endpoints mapping in your configuration:

orders:
  plugin: http
  metrics:
    - name: status
      config:
        url: http://api:8000/status.json
        format: json
        timeout: 2
      collects:
        - orders_pending:
            type: gauge
            documentation: Orders waiting to be processed.
            value_tasks:
              - '@filter': jmespath
                expression: orders.pending

See the complete configuration, including general and module imports. Change the URL and JMESPath expression to match your API. The HTTP plugin expects HTTP 200 and decodes JSON when format: json is set. In Docker, localhost means the Covenant container; use a reachable hostname or a Compose service name for other services.

Filters can be chained. For example, extract a string value and convert it:

value_tasks:
  - '@filter': jmespath
    expression: orders.pending
  - '@filter': builtins
    func: float

Other filters cover regular expressions, jq, built-in types, math, time and file paths. See the filter implementations and the shipped templates for their accepted arguments.

Choose metric types deliberately

A gauge represents a current value, such as queue length. Covenant's counter uses inc: it adds the supplied value at each collection. Do not feed an API's already cumulative total into a normal counter, or each scrape will add it again. The code also provides const_counter for setting a source-supplied counter value; see metric types.

Distinguish failure, missing data and zero

• on_fail handles a failed source collection.
• on_noresult handles an absent result from a transformation.
• By default, affected collectors are removed from exposition. You can configure a replacement value when it has a clear meaning.

For example, a source availability gauge can use:

orders_source_up:
  type: gauge
  documentation: Whether the source API request succeeded.
  value: 1
  on_fail:
    value: 0

For business metrics such as queue length, a missing field is not necessarily zero. Only add on_noresult: {value: 0} if zero is the intended meaning. A successful HTTP request also does not prove every expected field exists.

Reuse the shipped templates

Source	Plugin	Template	Source requirements
Apache	http	apache.yml	Accessible server-status endpoint
Nginx	http	nginx.yml	/nginx_status; optional /nginx_version
RabbitMQ	http	rabbitmq.yml	Management HTTP API and appropriate credentials
Redis	redis	redis.yml	Reachable Redis instance
TLS	ssl	secure-layer.yml	Reachable TLS target
File existence	filestat	file.yml	File visible to the Covenant process

Nginx

Add this endpoint alongside your existing endpoints:

nginx1:
  plugin: http
  vars:
    url: http://nginx:8080
    timeout: 2
    version_enabled: false
  import_metrics: metrics.d/nginx.yml

Expose Nginx's status page at /nginx_status. version_enabled: false disables the separate /nginx_version request, which a standard Nginx installation does not provide automatically.

curl --fail http://localhost:9118/metrics/nginx1

This template exposes connection counts, connection-state labels, a source up gauge and a scrape-failure counter.

TLS probe

secure-layer1:
  plugin: ssl
  vars:
    timeout: 10
  import_probes: probes.d/secure-layer.yml

curl --get --fail http://localhost:9118/probe/secure-layer1 \
  --data-urlencode 'target=https://example.com'

The template includes connection/probe success and certificate validity metrics.

File existence probe

application-file:
  plugin: filestat
  vars:
    state_include_paths:
      - '^/watched/app\.pid$'
  import_probes: probes.d/file.yml

curl --get --fail http://localhost:9118/probe/application-file \
  --data-urlencode 'target=/watched/app.pid'

In Docker, mount the required host directory read-only at /watched. This tests file existence; a PID file's presence does not prove its process is running.

Template paths are resolved relative to the main configuration directory. The Docker image already includes modules, metrics.d and probes.d under /etc/covenant. Mounting an entire directory there hides those bundled files; include them in your mount or mount only the main configuration, as in the demo.

Connect Prometheus

For a fixed endpoint, add a scrape job to your Prometheus configuration:

scrape_configs:
  - job_name: covenant-orders
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics/orders
    static_configs:
      - targets: ['covenant:9118']

Here covenant:9118 must be reachable from Prometheus. It works as a service name when the containers share a Docker network. For Prometheus running directly on the demo host, use localhost:9118 instead.

For one dynamic TLS target:

scrape_configs:
  - job_name: covenant-tls
    scrape_interval: 60s
    scrape_timeout: 15s
    metrics_path: /probe/secure-layer1
    params:
      target: ['https://example.com']
    static_configs:
      - targets: ['covenant:9118']

Enable the corresponding endpoint in Covenant before scraping it. Prometheus's own up metric describes its scrape of Covenant; source availability is reported by the metrics you configure inside Covenant.

Run and configure Covenant

Collection deadlines

Set a finite result wait in the main YAML configuration:

general:
  result_timeout: 30

result_timeout is expressed in seconds and defaults to 30 when omitted, so existing configurations remain valid. Positive fractional values are accepted; zero, negative, boolean, null, non-finite and platform-overflow values are rejected at startup. This applies to both metric and probe requests, including time spent waiting in the endpoint queue. An expired wait returns HTTP 504. Existing endpoint-not-found (404), endpoint-type (400) and plugin-error (500) responses remain unchanged. lock_timeout separately bounds acquisition of the module lock.

The deadline bounds the client's result wait; it does not interrupt a running plugin, cancel backend I/O, or remove an already queued job. Keep backend timeouts configured. The endpoint queues remain unbounded; this setting is not queue backpressure. Late callbacks are ignored, and completed/expired waits retain no result in a module-level dictionary. Collections that previously took more than 30 seconds now time out unless a larger value is configured. Set the Prometheus scrape timeout and backend timeouts consistently with this budget.

Collection dispatch, correlation and completion live in CollectionService, which accepts endpoint queues and the existing plugin job factory. It consumes plain parameters and returns results or domain exceptions, with no HTTP, CLI or TUI imports. HTTP modules validate requests and map these exceptions to responses. The CovenantEPTObject callback contract, plugin hooks, endpoint registries, routes, and metric names/labels are preserved. Historical module methods _push_epts_sync(endpoint, method, params, args=None), _set_result(obj) and _get_result(uid) remain available as compatibility facades. HTTP handlers still call them, so subclass overrides continue to participate. Submission returns the UID; retrieval preserves the {'error': ..., 'result': ...} shape, including plugin errors, and expiration raises HTTP 504.

results is now a synchronized mapping view of completed, unconsumed jobs. Lookup, membership, iteration, copy, pop and assignment for an active UID are supported. It is not a concrete dict; replacing the attribute or depending on its exact type is not supported. Expired, consumed and unknown callbacks/writes are ignored, so they cannot recreate orphaned results. Unlike the old dictionary, unconsumed results expire too. A finite deadline starts at submission; delaying _get_result does not reset it. One active expiry worker per module cleans up abandoned calls, is reused across adjacent calls, and exits when idle. New callers should use the service API; these methods are retained for transition without a scheduled removal.

Docker is the simplest way to use the tested runtime. To run your own configuration:

docker run --rm --name covenant \
  -p 127.0.0.1:9118:9118 \
  -v "$PWD/covenant.yml:/etc/covenant/covenant.yml:ro" \
  decryptus/covenant:0.0.71

For a Python installation, pip install covenant installs the version published on PyPI, which may differ from GitHub or Docker Hub. Installing from the repository requires its dependencies and native build tools; the Dockerfile provides the working build recipe.

# Foreground, with an explicit configuration.
covenant -f -c /etc/covenant/covenant.yml

# Daemon mode, with writable PID and log locations.
covenant -c /etc/covenant/covenant.yml \
  -p /tmp/covenant.pid --logfile /tmp/covenant.log

# Foreground debug logging.
covenant -f -l debug -c /etc/covenant/covenant.yml

The current Docker image uses Python 3.11. Do not assume Python 3.12+ compatibility: legacy dependencies import imp and asyncore, removed in Python 3.12.

Timeouts and access

The network templates expose vars.timeout. HTTP and TLS use a source timeout; the Redis template sets socket_timeout and socket_connect_timeout. A custom HTTP definition should set config.timeout explicitly. general.lock_timeout is not an overall collection deadline. Allow enough Prometheus scrape time for all sources configured in an endpoint.

Keep collection endpoints on a trusted network. Dynamic target parameters let callers select destinations or paths permitted by the plugin configuration. For HTTP endpoints with credentials, prefer a fixed configured URL. Do not commit credentials; see the credentials example and provide deployment-specific files securely. Imported Mako templates are trusted configuration, not a sandbox for untrusted input.

Tests and Docker releases

See tests/README.md for local test setup and coverage.

The Docker Hub workflow builds and tests on master and pull requests. After successful tests on master, a new stable version in VERSION and RELEASE automatically creates its missing vX.Y.Z tag and publishes the tested Linux amd64 image as:

• decryptus/covenant:X.Y.Z
• decryptus/covenant:vX.Y.Z

Publication requires the GitHub repository secret DOCKERHUB_TOKEN. The workflow does not update latest or overwrite existing Git tags. Ordinary commits on an already tagged version skip publication. Manual version-tag pushes remain supported. See Docker Hub setup and release instructions.

The container checks include a jq expression, the collected unittest suite covering collectors, templates, runtime integration and CertLord observations, the installed package version and CLI startup. They do not replace a full integration test against your real services.

Troubleshooting

Symptom	Check
Cannot reach port 9118	Container logs, port mapping and listen address
Endpoint not found	Module imports, endpoint name and /metrics/ versus /probe/ route
API connection failure in Docker	Use a reachable service hostname rather than container-local localhost
Metric absent	Source response, filter expression, and on_fail/on_noresult behavior
Nginx version collection fails	Disable version_enabled unless /nginx_version is configured
Files reported absent	Container mount path, permissions and include/exclude expressions
Image not published	Tag/version checks and Docker Hub credentials in the Actions log

License

GNU General Public License v3.

Publishing to PyPI

See PyPI publishing for Trusted Publisher setup and automated releases.

See configuration validation for YAML schema coverage and compatibility.