Skip to content

[Bug]: A new service with replicas: 0..N starts a replica on creation, without any request #4419

Closed

Description

@vitostrokov

Steps to reproduce

  1. On dstack 0.22.5 with a gateway, apply a new service that scales to zero:
    type: service
    name: srv
    image: nginx
    port: 80
    replicas: 0..1
    scaling:
      metric: rps
      target: 1
  2. Send no requests to it.

Actual behaviour

A replica is provisioned straight away. In our case, a 0..1 RTX 5090 service submitted at 10:12:11 UTC had a job submitted and an instance created 14 seconds later, without any request. The replica then ran until scale_down_delay passed and was scaled down (SCALED_DOWN). After that the service behaved as expected: it stayed at zero until a request arrived.

An in-place update of an existing service does not do this. Only a new run does.

Expected behaviour

A new service with replicas: 0..N stays at zero replicas until the first request, as described in #3003.

dstack version

0.22.5 (server and gateway)

Server logs

No response

Additional information

The cause appears to be the placeholder replica count combined with the pre-0.20.7 compatibility branch:

  1. submit_run (server/services/runs/__init__.py) creates every run with desired_replica_count=1 # a relevant value will be set in RunPipeline, and leaves desired_replica_counts empty.
  2. When the pending run is first processed, _process_pending_service (background/pipeline_tasks/runs/pending.py) calls compute_desired_replica_counts (background/pipeline_tasks/runs/common.py).
  3. There, desired_replica_counts is empty and the service has the single default replica group. So the special case for services created by a pre-0.20.7 server applies: prev_counts = {DEFAULT_REPLICA_GROUP_NAME: run_model.desired_replica_count}, which is the placeholder 1.
  4. The gateway has no stats for a service that was just registered. So RPSAutoscaler.get_desired_count returns current_desired_count as is (if not stats: return current_desired_count), which is 1.
  5. total == 1, so the run is submitted with one replica.

Two possible fixes, either of which should be enough:

  • Apply the compatibility branch only to runs that already have jobs, for example prev_counts == {} and run_model.jobs and .... A service created by a pre-0.20.7 server had jobs by the time a 0.20.7+ server processed it, while a new run has none.
  • Or, for services, create the run with desired_replica_count set to the replica minimum (0 for 0..N) instead of the placeholder 1.

As hardening, the no-stats return in RPSAutoscaler.get_desired_count could also be clipped to [min_replicas, max_replicas], like the other branches.

Impact: every new scale-to-zero service costs one unrequested cold start. For a GPU service that means the full image pull, weight download and warm-up (about 20 minutes for us), plus the scale_down_delay.

I'm happy to send a PR if one of these fixes looks right to you.

Activity

  1. peterschmidt85 commented on Oct 27, 2026

    @peterschmidt85
    Contributor
  2. self-assigned this
    on Oct 27, 2026
  3. jvstme commented on Oct 29, 2026

    @jvstme
    Collaborator

    Hi @vitostrokov, thanks for reporting. I believe #4433 fixes the bug. The new dstack release with the fix will most likely be available later this week, please give it a try when you have a chance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions