English | 中文
tRPC-Metrics-Prometheus is the Prometheus metrics plugin for tRPC-Python. Importing it registers MetricsPlugin and the client and server filters. In the worker process that runs business code it starts an HTTP exporter and exposes metrics for Prometheus to scrape. When a Pushgateway address is configured, it also pushes on an interval.
- Register on import:
import trpc_metrics_prometheusregisters the metrics plugin and the filters. RPC metrics are counted by the filters, so business handlers do not report them by hand. - Pull: By default this process serves its metrics at
http://{ip}:8080/metrics. If the port is already in use, the plugin logs a warning and the service still starts. - Push: After
push_gateway_addris set, the same metrics are pushed to the Pushgateway everyreport_intervalseconds. The job name isinterval_batch_push. - Custom metrics:
custom_counter,custom_gauge, andcustom_histogramrecord call counts, in-flight concurrency, and duration of business functions.
Python: 3.8+
Dependencies: trpc-python (trpc_sdk) and prometheus-client>=0.20.0.
Install with pip:
pip install trpc_metrics_prometheusThis assumes a tRPC project already exists. The plugin needs two things: import the package, and add plugins.metrics.prometheus to the framework config. The RPC filters are attached separately on server.filter or client.filter.
Import it at the top of the service entry point. Importing registers the plugin, so this must happen before the framework starts.
import trpc_metrics_prometheusip is both the instance label and the HTTP exporter bind address. port defaults to 8080 and path defaults to /metrics. Only that path returns metrics. Other paths return 404.
When app, server, namespace, env_name, set_name, container, and ip are left empty, they fall back to server.app, server.server, global.namespace, global.env_name, global.full_set_name, global.container_name, and global.local_ip.
Leaving push_gateway_addr empty serves pull only. The code default of report_interval is 0, and that value applies only when pushing is enabled. Set a positive number of seconds when pushing is on. 0 makes the push loop spin without sleeping. Basic auth is used when both username and password are non-empty. If only one of them is set, pushes are sent without auth, and the gateway must allow that.
plugins:
metrics: # 监控配置
prometheus: # prometheus monitor
report_interval: 5 # 推送间隔，单位秒。代码默认为 0；开启推送时请设为正数
push_gateway_addr: x.x.x.x:9090 # Pushgateway 地址，host:port。留空则不推送
username: 'xxxxx' # Basic 认证账号，可选
password: 'xxxxxxxx' # Basic 认证密码，可选
app: ${APP_NAME} # 业务名。可选，未配置则与 server.app 一致
server: ${SERVER_NAME} # 服务名。可选，未配置则与 server.server 一致
namespace: ${NAMESPACE} # 物理环境。可选，未配置则与 global.namespace 一致
env_name: ${DOCKER_ENV} # 用户环境。可选，未配置则与 global.env_name 一致
set_name: ${SET_NAME} # 用户 Set。可选，未配置则与 global.full_set_name 一致
container: ${POD_NAME} # 容器名称。可选，未配置则与 global.container_name 一致
ip: ${POD_IP} # 本机 IP，同时是 exporter 监听地址。可选，未配置则与 global.local_ip 一致
port: 8080 # exporter 端口，可选，默认 8080
path: /metrics # exporter 路径，可选，默认 /metricsClient-call metrics are attached on the client. Server-call metrics are attached on the server. Either side can be enabled on its own.
server:
filter:
- prometheus
client:
filter:
- prometheusAfter the service starts, scrape the address from the config. When ip is 0.0.0.0, use an address that can actually reach this machine.
curl http://127.0.0.1:8080/metricsexamples/pull_exporter.py calls init_metrics directly and writes sample Counter, Gauge, and Summary metrics into REGISTRY. It does not start a tRPC service.
python3 examples/pull_exporter.py
curl http://127.0.0.1:8080/metricsexamples/README.md describes the Greeter service and this standalone exporter. From the repository root, ./examples/run_examples.sh starts Greeter, sends SayHello, checks the exporter, and stops the server. ./examples/run_examples.sh pull runs the standalone exporter and then stops it.
The filters report the metrics below. rpc_client_* is a downstream call made by this service. rpc_server_* is a request received by this service.
rpc_client_started_total,rpc_server_started_total: number of calls startedrpc_client_handled_total,rpc_server_handled_total: number of calls finished, withcodeandcode_typerpc_client_handled_seconds,rpc_server_handled_seconds: handling-time histogram, in seconds
Each RPC metric carries these labels:
system_name: alwaystrpccaller_service,caller_method,callee_service,callee_methodnamespace,app,server,env_name,set_name,instance,container,pid
instance is the configured ip. pid is the current process id. An empty method name is recorded as unknown.
Streaming calls go through run_stream and are passed through unchanged. They are not counted in the metrics above.
The default code is chosen as follows. When the call fails, the exception's code is used, or TRPC_INVOKE_UNKNOWN_ERR when the exception has no code. When the call succeeds, the response object's retcode, ret_code, and code are tried in that order. If none of them is set, the code is "0".
The default code_type maps "" and "0" to success, and every other code to exception.
When the business code is not in those fields, replace the function the filter uses:
import trpc_metrics_prometheus.filter as prom_filter
def my_get_code(ctx, req, rsp, err) -> str:
if err is not None:
return str(getattr(err, "code", "1"))
return str(getattr(rsp, "biz_code", "0"))
prom_filter.DEFAULT_GET_CODE_FUNC = my_get_codeTo classify some codes as timeout or another type, replace the code_type mapping. The values include success, exception, and timeout:
from trpc_metrics_prometheus import CodeTypeException
from trpc_metrics_prometheus import CodeTypeSuccess
from trpc_metrics_prometheus import CodeTypeTimeout
from trpc_metrics_prometheus import set_code_type_func
def my_code_type(code: str) -> str:
if code == "101":
return CodeTypeTimeout
if code in ("", "0"):
return CodeTypeSuccess
return CodeTypeException
set_code_type_func(my_code_type)All three decorators take a metric name, a description, label names, and a function that computes label values from the wrapped function's arguments. They wrap both synchronous and asynchronous functions. Each custom metric also gets the common labels namespace, app, server, env_name, set_name, instance, container, and pid. The plugin must finish initializing before these decorators run, or those labels have no values.
custom_counter: increments{name}_startedbefore the call and{name}_finishedafter it. The scrape names arecustom_{name}_started_totalandcustom_{name}_finished_total.custom_gauge: increments before the call and decrements after it returns. The value is the current in-flight count. The scrape name iscustom_{name}.custom_histogram: records the function duration in seconds. The scrape names arecustom_{name}_bucket,custom_{name}_sum, andcustom_{name}_count.
from trpc_metrics_prometheus import custom_counter
from trpc_metrics_prometheus import custom_gauge
from trpc_metrics_prometheus import custom_histogram
def label_values(ctx, req):
return (req["method"],)
@custom_counter("order", "order calls", ["method"], label_values)
@custom_gauge("order_inflight", "in-flight order calls", ["method"], label_values)
@custom_histogram("order_seconds", "order call duration", ["method"], label_values)
async def create_order(ctx, req):
return {"code": 0}A metric name is created once in the process. Later reports that use the same name reuse the existing metric.
This project is licensed under Apache-2.0.