Repository navigation
Scope runner pull, metrics and stop to the job submission - #4425
Merged
Merged
Conversation
The server could not tell the runner it submitted a job to from another one at the same address. A container restarted by the backend (e.g., Kubernetes' default `restartPolicy: Always`) answers /api/pull with no job states, which the server took for a healthy running job, while the new runner waited for a submission that never came, exited, and was restarted again. If the address went to another job's container, the server would collect that job's logs, states and metrics, and could stop it. The server now sends `job_submission_id` with /api/pull, /api/metrics and /api/stop, and the runner answers 409 if it runs another job submission or none. The server then terminates the job, collects no logs or metrics from that runner, and does not stop it. Old servers do not send the ID, and old runners ignore it, so both behave as before. Part-of: #4420 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The server could not tell the runner it submitted a job to from another one at the same address. A container restarted by the backend (e.g., Kubernetes' default
restartPolicy: Always) answers /api/pull with no job states, which the server took for a healthy running job, while the new runner waited for a submission that never came, exited, and was restarted again. If the address went to another job's container, the server would collect that job's logs, states and metrics, and could stop it.The server now sends
job_submission_idwith /api/pull, /api/metrics and /api/stop, and the runner answers 409 if it runs another job submission or none. The server then terminates the job, collects no logs or metrics from that runner, and does not stop it. Old servers do not send the ID, and old runners ignore it, so both behave as before.Part-of: #4420