Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ uv run --no-sync python scripts/run_tests.py focused --feature native-descriptor
# The same suites and shard assignments used in CI.
uv run --no-sync python scripts/run_tests.py interpreter --shard 1 --shards 4
uv run --no-sync python scripts/run_tests.py prepare-browser
uv run --no-sync python scripts/run_tests.py browser-conformance --shard 1 --shards 4
uv run --no-sync python scripts/run_tests.py browser-conformance --shard 1 --shards 6
uv run --no-sync python scripts/run_tests.py browser-integration --shard 1 --shards 3
uv run --no-sync python scripts/run_tests.py python
```
Expand Down Expand Up @@ -68,6 +68,25 @@ Representative form submissions, all source-error/diagnostic checks, offline
reloads, service-worker/profile updates and every Stop/Restart cycle retain
their UI paths. The console report records UI/worker counts and elapsed time.

Shard weights prefer the checked-in `scripts/test_costs.json` inventory of
successful command timings. New checks retain the manifest's fallback estimate;
costs affect assignments only, never selection or preparation/packaging order.
To refresh from a passing Linux CI run, download its `interpreter-*`,
`browser-conformance-*` and `browser-integration-*` artifacts under one directory,
then run:

```sh
uv run --no-sync python scripts/update_test_costs.py build/ci-timings \
--source https://github.com/outside-labs/pocket-python/actions/runs/RUN_ID
```

The importer records report hashes/revisions, ignores removed IDs, deduplicates
reports and takes the median for repeated samples. Failed, incomplete or invalid
timings are rejected. Review the generated inventory before committing it; do
not mix different platforms or worker configurations. Runner reports now record
the host and worker configuration. The inventory is a scheduling estimate,
never a performance assertion or a substitute for running the complete gate.

## Fast development and merge policy

Run the smallest meaningful local checks for the behavior being changed. A
Expand Down
4 changes: 4 additions & 0 deletions scripts/run_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import json
import os
from pathlib import Path
import platform
import shlex
import subprocess
import sys
Expand Down Expand Up @@ -53,6 +54,9 @@ def run(selected, report_path, metadata):
report = {
'schema_version': 1, 'revision': git_output('rev-parse', 'HEAD'),
'working_tree_modified': bool(git_output('status', '--porcelain')),
'host': {'os': platform.system(), 'machine': platform.machine(),
'python': platform.python_version(),
'browser_workers': env.get('POCKET_PYTHON_BROWSER_WORKERS', '2')},
'started_at': datetime.now(UTC).isoformat(), **metadata, 'checks': [],
}
try:
Expand Down
Loading