Repository navigation
Conversation
The basic idea is to treat unset (0) help text as unchanged. And let double qoutes "" represent clearing the value to empty string. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Make omitting unchanged help text optional and require Senders that do it to resend the full help text periodically on a configurable interval, so Receivers do not need to retain state and lost help text is recovered. Reword the stateless statements in the introduction and FAQ accordingly. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
bwplotka
left a comment
bwplotka
left a comment
There was a problem hiding this comment.
Nice, I love it! Some suggestions
| Senders that omit unchanged help text MUST: | ||
|
|
||
| * Send the full help text of a series whenever it changes. | ||
| * Also resend the full help text of every series periodically, even if it is unchanged, so that the time between two requests carrying the full help text of a series never exceeds the resend interval. For a series whose help text is empty, the full help text is `""` (see below). |
There was a problem hiding this comment.
Since metadata-only TimeSeries messages are invalid in stateless modes per line 378, a Sender can only resend help text when a new sample or histogram is sent for that series, so "never exceeds the resend interval" cannot be met when the scrape interval exceeds the resend interval or a batch is delayed or dropped on 429. Maybe phrase this as resending the full help text on the first request for a series and on the first request after the resend interval has elapsed since the last acknowledged full help text?
There was a problem hiding this comment.
Also I dont' understand the wording ", so that the time between two requests carrying the full help text of a series never exceeds the resend interval. " -- can we improve and mention MOTIVATION for periodic MUST? does it need to be MUST?
|
|
||
| Receivers are not required to retain help text between requests. A Receiver that has no help text for a series (e.g. after a restart, or because the requests of one Sender are spread across multiple Receiver replicas) obtains it with the next periodic resend, so the help text can be temporarily absent but never stays lost. | ||
|
|
||
| In the default `none` mode, an empty help text carries no such meaning and Receivers MUST NOT assume it indicates an unchanged help text. |
There was a problem hiding this comment.
Giving help_ref = 0 and "" different wire meanings in none vs stateless modes conflicts with lines 119 and 497, where modes only tighten optional Sender rules into MUST requirements and Receivers MAY ignore the mode parameter. Why not apply this help_ref encoding across all modes including none, or otherwise update lines 119 and 497 to state that Receivers using help text must inspect mode?
| * `Metadata.type` MUST be specified and MUST NOT be `METRIC_TYPE_UNSPECIFIED` (`0`). For series with an unknown metric type (such as `untyped` scraped metrics or recording rule outputs), Senders SHOULD assume: | ||
| * `METRIC_TYPE_GAUGE`, unless the metric name has a `_total` suffix, in which case Senders SHOULD assume `METRIC_TYPE_COUNTER`. | ||
| * `Metadata.help_ref` set to `"(type inferred from untyped)"` if empty, or with `" (type inferred from untyped)"` appended to the existing help text. | ||
| * The help text of the series (see [Help in `stateless` and `stateless-st` Modes](#help-in-stateless-and-stateless-st-modes)) set to `"(type inferred from untyped)"` if it is empty, or with `" (type inferred from untyped)"` appended to the existing help text. If the Sender omits help because it is unchanged, the previously sent help text MUST already have included this suffix. |
There was a problem hiding this comment.
Nesting a MUST inside the parent Senders SHOULD assume: block is contradictory if a Sender does not use this fallback, and it also misses the reverse case where a series changes from untyped to typed with the same raw help text and drops the suffix. It might be cleaner to state that Senders evaluate whether the help text changed after applying any inferred-type suffix.
| For the detailed rationales behind each 2.0 Remote-Write decision, see: https://github.com/prometheus/proposals/blob/alexg/remote-write-20-proposal/proposals/2024-04-09_remote-write-20.md | ||
| --> | ||
| The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB. | ||
| The Remote-Write protocol is designed to be stateless (with one optional exception for optimizing away repeating metric help texts), and otherwise there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB. |
There was a problem hiding this comment.
Let's kill this and not be so precise here IMO. It's ok to have optimizations. We already fail staleness with summaries and classic histograms. The INTENTION is to have it stateless.
| The Remote-Write protocol is designed to be stateless (with one optional exception for optimizing away repeating metric help texts), and otherwise there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB. | |
| The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB. |
| // help_ref is a reference to the Request.symbols array representing help | ||
| // text for the metric. Help is optional, reference should point to an empty string in | ||
| // such a case. | ||
| // such a case. In stateless and stateless-st modes an empty string means the help |
|
|
||
| ##### Help in `stateless` and `stateless-st` Modes | ||
|
|
||
| In [`stateless` and `stateless-st` modes](#modes), Senders MAY omit the help text of a series that is unchanged, by setting `Metadata.help_ref` to the empty string (`help_ref` not set, or set to `0`). A series is identified by its complete label set. Omitting help text is OPTIONAL; Senders that do not implement it, or have it disabled, send the full help text in every request. |
There was a problem hiding this comment.
Nice, this effectively makes it impossible for sender to request help "reset" or removal. I think this is 100% acceptable. Sender can send "<whitespace>" to do this or anything else
| Receivers that use help text MUST interpret `Metadata.help_ref` as follows: | ||
|
|
||
| * If `help_ref` resolves to the empty string (this includes `help_ref` not being set, as it then defaults to `0`), the request does not carry the help text of the series. Receivers that retain help text between requests MUST keep the last help text received for the series, or treat the help text as empty if none was received, and MUST NOT clear it. Receivers that do not retain state between requests MAY treat the help text as empty for this request. | ||
| * If `help_ref` resolves to the two-character string `""` (two ASCII double quote characters), the help text is the empty string. Receivers that retain help text MUST replace any previous help text with the empty string. |
There was a problem hiding this comment.
Wait. Ah that's the reset. I love it
Summary
statelessandstateless-stmodes, Senders MAY (optionally) omit the help text of a series by sending an empty/unsetMetadata.help_ref, to avoid resending unchanged text.help_refas: empty/unset means "not carried by this request" (Receivers that retain help keep the last value and MUST NOT clear it),""(two ASCII double quotes) means "clear to empty", anything else is the new help text.""cannot be represented in these modes.nonemode an empty help text carries no such meaning.2.0-rc.6to2.0-rc.7.Test plan
statelessandstateless-stModes" section and its anchor links.Signed-off-by: György Krajcsovits gyorgy.krajcsovits@grafana.com