[manuaero]

Description

The unit-data and integration jobs fail on develop, and so on every pull request, on all five Python versions. The same tests fail each run. This fixes them. Tests only, apart from one error message in the SDK (last row of the table).

There are eleven separate causes, each read from the tracebacks in the CI logs of #2078 and checked against the code.

Tests	Cause	Fix
9 in tests/data/annotation_import/test_mea_prediction_import.py	The backend rejects every MEA prediction import: 9 FAILED_PRECONDITION: MEA prediction import has been deprecated.	Skipped, with that reason. The tenth test in the file only reads a model run; it passes and stays enabled.
3 in tests/integration/test_embedding.py, test_export_embeddings_custom, test_mmc_all (all ERROR at setup)	The staging organization is at the cap of 10 custom embeddings, and the shared embedding fixture could not heal itself. It sweeps stale embeddings and retries when it hits the cap, but only inside except LabelboxError. Since Client.create_embedding became a GraphQL mutation (#2074) the cap arrives as MalformedQueryException, which is not a LabelboxError, so the sweep never ran.	tests/embedding_cleanup.py catches both, for the create and for the delete of a stale embedding that another job has already removed. Unit tests cover both.
test_client_errors.py::test_semantic_error, ::test_query_complexity_error	Deployed API environments replace the message of every GraphQL validation error with GraphQL validation error. The tests expected the detailed text.	They accept the masked message, or the detailed one from an API that does not hide schema details. Nothing else passes.
test_data_rows.py::test_data_row_delete_and_create_with_same_global_key (2 of 5 jobs)	A global key is not free the moment its data row is deleted: the import checks keys against the search index, which drops the row a little later.	The test repeats the import while "duplicate global key" for that key is the only error, for up to 3 minutes. Any other error fails at once.
test_label.py::test_label_bulk_deletion (1 of 5 jobs)	Three labels are not listed the moment they are created.	The test waits for them with the helper the fixture already uses.
test_project.py::test_update_project_resource_tags (1 job, seen once the tests above passed)	The test counted every resource tag of the organization and expected two more than before. Five jobs run the suite against the same organization at once, so the count moves under it.	It checks that the two tags it created are on the project, by ID.
test_mea_prediction_import.py::test_model_run_project_labels (1 job, same)	The test exported the model run's labels once, straight after the fixture attached them, and one data row was not there yet.	It repeats the export until every data row has a label, for up to 3 minutes, then makes the same assertions.
test_ontology.py::test_feature_schema_is_archived (1 job, same)	Client.is_feature_schema_archived raised "Failed to get the feature schema archived status." for an answer that was neither 200 nor 404. The SDK dropped the status and the body, so the log cannot say whether it was a rate limit or a server error.	Not fixed: the cause is unknown. The error now carries the status and the body, so the next occurrence can be read. This is the one change under src/.
Any test, in setup or teardown (5 tests in 3 jobs of one run): ApiLimitError: You have exceeded 15000 requests in a 60 second time window	The API counts requests per user in a fixed 60 second window. Every job uses the same staging user, with 32 workers each. The SDK does not retry a 429, and the suite's one rerun after 5 seconds does not outlast the window.	The test client (tests/conftest.py) waits and resends a request answered with 429, up to four times, at the level of the HTTP session so that REST calls in teardown are covered. It prints when it waits.
Any test that creates a project (4 tests in 3 jobs): MalformedQueryException: Unsupported labeling frontend	createProject sometimes returns a project with no editor attached, when many are created at once. connect_ontology then tries to attach the first editor named "Editor", which this organization's API refuses. Seventeen tests took that path in one run and all seventeen needed their rerun.	The test client creates such a project again, twice at most, and prints the ID of each one it discards. See below: this is a backend defect the tests now work around.
test_model_run.py::test_model_run_delete, and the setup of test_model_run_status	The test took the newest model in the organization, not its own. Another worker's model can be deleted between listing it and reading its runs (TypeError in pagination), and the test deleted that other worker's model run.	It uses the model its own fixtures created, and also asserts that the run was listed before it was deleted.

test_issue_management.py::test_get_issues_with_status_filter errored in one job on a rate-limit response during setup. That is not a defect of the test and it is not changed.

What reviewers should know

Public API that no longer works. This PR does not touch it:

• ModelRun.add_predictions() and MEAPredictionImport.create(), create_from_objects(), create_from_file() and create_from_url() always raise the error above.
• ModelRun.upsert_predictions_and_send_to_project() calls add_predictions() first, so it fails the same way. Its tests were already skipped, without a reason.
• Eleven example notebooks call model_run.add_predictions: all eight under examples/prediction_upload/, and custom_metrics_basics, custom_metrics_demo and model_predictions_to_project under examples/model_experiments/.

These want a deprecation notice or a clearer error, and a changelog entry. That is a decision for the SDK's owners, so it is left out of a test fix.

test_query_complexity_error can no longer tell complexity apart. Where messages are masked, exceeding the complexity limit raises InvalidQueryError, like any other validation failure. ValidationFailedError is only reachable against an API that shows schema details. The test accepts either, each with its own message pinned.

A project can come back from createProject without an editor. This is on the backend, and no test change removes it. Project creation catches an error in its post-commit setup and returns the project anyway, with creationSetupFailed: true, a field the SDK does not request. Which setup step fails under concurrent creation is not established: the tests only see that the project has no editor a second later. The test client prints Project <id> has no editor, creating it again each time, so the API's logs can be searched for those IDs.

The rate-limit wait is back-pressure, not less traffic. If the budget of the shared user is short, the suite now slows down to it and says so in the log, where it used to fail. Worst case for one request is four waits of about a minute.

Whether the embedding tests go green depends on what holds the ten slots. The sweep only removes embeddings this suite created (sdk-int-ci-v2-…) that are more than 12 hours old. If something else in the staging organization holds them, the cap stays, and the manual cleanup workflow is needed.

How it was verified

No staging credentials were used. Each cause was read from the five logs of each job, and each fix was reviewed independently against those logs. tests/unit/test_embedding_cleanup.py passes (21 tests, three added). CI on this pull request is the real test: the first run with the first five fixes turned those tests green and showed the next three, each in one job out of ten. The run after that showed the last three. The two additions to the test client were also exercised against fakes: a 429 followed by a 200, a limit that never lifts, and a project that comes back without an editor once, and every time.

Type of change

• Bug fix (non-breaking change which fixes an issue)

All Submissions

• Have you followed the guidelines in our Contributing document?
• Have you provided a description?
• Are your changes properly formatted?

🤖 Generated with Claude Code

---

Note

Low Risk
Primarily test and CI harness changes; the SDK change only improves error diagnostics on a failure path.

Overview
Stabilizes staging CI for the Labelbox Python SDK by hardening integration test infrastructure and aligning assertions with deployed API behavior. Almost all changes are in tests; the only SDK tweak enriches the LabelboxError from is_feature_schema_archived with HTTP status and response body when the REST call is neither 200 nor 404.

Integration client (conftest) now retries on 429 (shared API key across parallel jobs) and re-creates projects when createProject returns one without a labeling editor. MEA prediction import tests are skipped because the backend rejects that flow as deprecated; the remaining model-run export test polls until labels appear in export.

Flaky / environment-specific fixes: embedding fixture healing catches MalformedQueryException for the custom-embedding cap (not only LabelboxError); GraphQL validation tests accept the masked "GraphQL validation error" message on staging; data-row tests wait for global-key reuse after delete; label/project/model-run tests avoid org-wide counts and use ID-based or wait/retry patterns under concurrent CI.

Unit coverage for embedding cleanup is extended for the new exception types and client path.

^{Reviewed by Cursor Bugbot for commit 737d0b6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{Reviewed by Cursor Bugbot for commit 737d0b6. Configure here.}

[cursor]

Missing editor deletes healthy projects

Medium Severity

IntegrationClient._create_project treats a missing labeling frontend as a failed create and deletes the project. That frontend is not in the create payload, so this is a second read with no wait. A project whose editor is not visible yet is destroyed, and the third create is returned without the same check, so the caller can still get a project with no editor.

 

^{Reviewed by Cursor Bugbot for commit 737d0b6. Configure here.}

[cursor]

Rate-limit retries wake up together

Medium Severity

wait_out_rate_limit sleeps a fixed time on every 429, with no jitter. Every CI job uses the same API key, so jobs that hit the cap together wait the same Retry-After or the same 20 seconds and resend together. That recreates the rate limit this wrapper is meant to ride out.

 

^{Reviewed by Cursor Bugbot for commit 737d0b6. Configure here.}