Skip to content

Escape line feeds in Prometheus exporter label values - #4155

Open
varun3257 wants to merge 1 commit into
tensorflow:masterfrom
varun3257:prometheus-label-linefeed
Open

varun3257 wants to merge 1 commit into
tensorflow:masterfrom
varun3257:prometheus-label-linefeed

Conversation

@varun3257

Copy link
Copy Markdown

SanitizeLabelValue in prometheus_exporter.cc escapes backslash and double quote but not line feed, which the Prometheus text exposition format also requires to be written as \n inside a label value. The REST dispatcher percent-decodes the model name from the request path and records it as the model_name label of /tensorflow/serving/request_count for every request, including ones for models that do not exist, so an unauthenticated POST /v1/models/x%0Ay:predict leaves a metric cell whose label value contains a raw line feed. From then on GeneratePage emits that sample split across two lines and line-based scrapers reject the page: the Python prometheus_client parser fails with Invalid value: '{model_name="x', and the OpenMetrics lexer refuses a raw newline inside a quoted label value. The cell is never removed, so the metrics endpoint stays unparseable until the server restarts.

Escaping the line feed in SanitizeLabelValue, next to the two existing escapes, keeps each sample on one line and the label value round-trips as x\ny on the scraper side. A test next to the existing SanitizeLabelValue test checks the escaped output.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant