Skip to content

refactor: OpenTelemetry instrumentations record HTTP and process metrics - #388

Merged
HardMax71 merged 1 commit into
mainfrom
refactor/otel-http-metrics
Oct 14, 2026
Merged

HardMax71 merged 1 commit into
mainfrom
refactor/otel-http-metrics

Conversation

@HardMax71

@HardMax71 HardMax71 commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

The FastAPI OpenTelemetry instrumentation never worked. dishka_lifespan.py called instrument_app inside the lifespan, but Starlette builds the middleware stack on its first ASGI call, which is the lifespan startup itself, and instrument_app only patches build_middleware_stack without rebuilding. So no request produced an OTel span or HTTP metric, and MetricsMiddleware was the only HTTP telemetry. A probe that drives the lifespan the way uvicorn does recorded nothing with the old placement and all three HTTP instruments with the new one.

create_app now instruments the app before the stack exists, so the OTel middleware wraps everything, and MetricsMiddleware is deleted. The backend image sets OTEL_SEMCONV_STABILITY_OPT_IN=http (pytest's env does too), so requests land as http_server_request_duration_seconds, http_server_active_requests and the two body size histograms, labelled by FastAPI's route template instead of our regexes. Since the OTel middleware is outermost, the 403, 429 and 413 answers from CSRF, rate limiting and the size limit now show up in the metrics; before they didn't.

The psutil gauges go as well. The collector's host_metrics receiver already scraped host memory, and now host CPU utilization too, and every backend and worker process reports its own memory, CPU, threads and file descriptors through opentelemetry-instrumentation-system-metrics. setup_metrics moves to app/core/metrics/base.py, and psutil is no longer a direct dependency.

The HTTP dashboard's panels read the new series. The contract test builds the instrumentations' families on a bare app the way create_app configures them, and passes. The middleware, metrics reference and contract testing docs describe the new setup.

Checked with ruff, mypy, grimp, 564 unit tests, the 3 contract tests and a strict docs build. Against the real create_app with an in-memory reader, a request recorded http.server.request.duration with method, status and scheme, and the security headers still came back.


Summary by cubic

Fixes the FastAPI OpenTelemetry instrumentation, which ran inside the lifespan after Starlette had already built the middleware stack and so never wrapped a single request. HTTP and process metrics now come from OpenTelemetry instrumentations, and the hand-written MetricsMiddleware and psutil gauges are removed.

HTTP metrics

  • Requests record http_server_request_duration_seconds, http_server_active_requests and the two body size histograms, labelled by FastAPI route templates instead of regex-normalized paths.
  • The stable semantic convention names require OTEL_SEMCONV_STABILITY_OPT_IN=http, now set in the backend image and pytest.
  • The OTel middleware wraps CSRF, rate limiting and the size limit, so their 403, 429 and 413 responses show up in metrics where they were absent before.

Process metrics

  • Each backend and worker process reports its own memory, CPU, threads and file descriptors via opentelemetry-instrumentation-system-metrics; host CPU and memory come from the collector's host_metrics receiver.
  • setup_metrics moved to app/core/metrics/base.py, and psutil stops being a direct dependency.

The Grafana HTTP dashboard, the metrics reference and middleware docs, and the contract test now match the new series.

Written for commit a2c7eb6. Summary will update on new commits.

View guided diff Turn on auto-fix

…ics, and MetricsMiddleware is gone

FastAPIInstrumentor ran inside the lifespan, after Starlette had built the middleware stack on the lifespan's own ASGI call, so it never wrapped anything and the app recorded no HTTP spans or OTel HTTP metrics. create_app now instruments the app before the stack exists, and the hand-written MetricsMiddleware goes. The backend image opts into the stable HTTP semantic conventions, so requests show up as http_server_request_duration_seconds and friends, labelled by FastAPI's route template.

The psutil gauges go too. The collector's host_metrics receiver already scrapes host CPU and memory, with CPU utilization now turned on, and each process reports its own memory, CPU, threads and file descriptors through opentelemetry-instrumentation-system-metrics. setup_metrics moves to app/core/metrics/base.py and psutil stops being a direct dependency.

The HTTP dashboard reads the new series, the contract test builds the instrumentations' families the way the app configures them, and the middleware, metrics reference and contract testing docs describe the new setup.
@HardMax71
HardMax71 enabled auto-merge (squash) October 13, 2026 23:58
@github-actions github-actions Bot added documentation Improvements or additions to documentation area: api FastAPI routes, auth, rate limiting, settings area: execution Queue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes area: observability Metrics, dashboards, alerts, logs, traces, OTel collector area: deploy Compose, deploy.sh, images, release workflow, production host labels Oct 13, 2026
@coderabbitai

coderabbitai Bot commented Oct 13, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c88caccb-eb6c-4c8a-82f6-a0c89c830b4c
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 72.72727% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
backend/app/core/metrics/base.py 64.70% 6 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Flag Coverage Δ
backend-unit 72.39% <72.72%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
backend/app/core/dishka_lifespan.py 100.00% <ø> (ø)
backend/app/core/metrics/__init__.py 100.00% <100.00%> (ø)
backend/app/core/middlewares/__init__.py 100.00% <ø> (ø)
backend/app/main.py 93.44% <100.00%> (+0.22%) ⬆️
backend/app/worker.py 86.53% <100.00%> (ø)
backend/app/core/metrics/base.py 75.00% <64.70%> (-15.91%) ⬇️
Components Coverage Δ
Backend 91.15% <72.72%> (+<0.01%) ⬆️
Frontend 92.60% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HardMax71
HardMax71 merged commit bcbf249 into main Oct 14, 2026
23 checks passed
@HardMax71
HardMax71 deleted the refactor/otel-http-metrics branch October 14, 2026 00:01
HardMax71 added a commit that referenced this pull request Oct 18, 2026
…stead of straight to Jaeger (#391)

Every span now reaches the collector, which #389 needs so its connectors can turn spans into metrics before tail sampling drops most of them. Jaeger gets what the collector's existing tail sampling keeps: traces with an error, traces over 1 s and a 10 % sample, 10 to 20 s after they end. The tracing doc also no longer says FastAPI is instrumented in the lifespan, since #388 moved that into create_app.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: api FastAPI routes, auth, rate limiting, settings area: deploy Compose, deploy.sh, images, release workflow, production host area: execution Queue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes area: observability Metrics, dashboards, alerts, logs, traces, OTel collector documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants