[BigRoy]

Changelog Description

Retries after a 502/503 response or a connection error now wait a random time between 50% and 150% of the previous fixed delay: 1–3 s instead of 2 s after 502/503, and 0.05–0.15 s instead of 0.1 s after a connection error. Clients that failed at the same moment no longer retry at the same moment.

Context: ynput/ayon-launcher#129.

Additional info

When many clients start at once (e.g. 200 new farm machines running the AYON launcher) and the server answers 503, they all retried exactly 2 s later, hitting the server together again. Random delays spread the retries out.

The average delay stays the same, so the total wait time doesn't change. Exponential backoff would ease the load further, but it would also make failing requests wait longer, so it isn't part of this PR.

Testing notes:

1. pytest tests/test_request_retries.py (no server needed).

[iLLiCiTiT]

Not sure about the reasoning, but I don't need to fully understand.

[BigRoy]

I'm not too confident this actually solves a real world scenario. The chance that all 200 machines boot up and fail exactly at the same second is likely relatively small. So it's an "improvement" for a hypothetical scenario.

I'll close it until we can actually confirm with other optimizations we still hit this pain point in a real production scenario.