Skip to content

Require an Artifacts section on PRs that change the CLI - #43

Open
awoods187 wants to merge 1 commit into
dualentry:mainfrom
awoods187:andyw/require-pr-artifacts
Open

awoods187 wants to merge 1 commit into
dualentry:mainfrom
awoods187:andyw/require-pr-artifacts

Conversation

@awoods187

Copy link
Copy Markdown

Summary

Unit tests mock the API client, so they check the CLI against the author's reading of the API, not the API itself. This adds a CI check that a PR changing src/ shows the change working against a real API in an ## Artifacts section of its description.

Running the open PRs against a live API turned up bugs their unit tests had passed. These included attachment uploads that failed on every call and a statistical-journals list crash on real rows. This check makes that step part of every PR.

Changes

  • scripts/check_artifacts.py: fails when the PR touches src/ (the same rule as the changelog check, reusing needs_note) and the description has no ## Artifacts section, or the section has no evidence. Evidence means a non-empty fenced code block, an image, a video, or a GitHub upload link. Template comments are ignored, so the untouched template and text like "N/A" fail.
  • .github/workflows/artifacts.yml: runs the check on pull_request, including edited, so adding artifacts to the description re-runs it without a new push. The body is passed through env, not interpolated into the script. A skip-artifacts label opts a PR out.
  • PR template: adds an Artifacts section that explains what to include (commands and output, JSON, before/after for writes, screenshots, and what couldn't be tested live) and to redact secrets.
  • CONTRIBUTING.md: adds the step to the PR checklist.
  • Tests: tests/test_check_artifacts.py.

Test plan

  • Unit tests pass (uv run pytest): 305 passed.
  • Linter passes (uv run ruff check src/ tests/ scripts/ and uv run ruff format --check src/ tests/ scripts/)
  • Manually tested: the script was run end to end on a throwaway commit that touches src/; see Artifacts.

Artifacts

Run against the live descriptions of #39–#42, the new template, and some failing cases. A throwaway commit touching src/ made each one count as a user-facing change:

#39:       Artifacts found.                         exit=0
#40:       Artifacts found.                         exit=0
#41:       Artifacts found.                         exit=0
#42:       Artifacts found.                         exit=0
template:  The `## Artifacts` section has no evidence: add command output in a ``` block, or a screenshot.   exit=1
empty:     This PR changes src/ but its description has no `## Artifacts` section.                       exit=1
N/A:       The `## Artifacts` section has no evidence: add command output in a ``` block, or a screenshot.   exit=1

Without the throwaway commit (this branch only changes .github/, scripts/, tests/ and docs):

$ python3 scripts/check_artifacts.py origin/main
No user-facing changes; artifacts not required.

Not exercised: the workflow itself on GitHub. This PR doesn't touch src/, so the job reports the exemption rather than checking a body.

Needs a maintainer

  • Create the skip-artifacts label.
  • Optionally, make Artifacts a required status check in branch protection. Until then it reports but doesn't block a merge.

The check confirms evidence is present, not that it's convincing; that's still the reviewer's call.

Unit tests mock the API client, so they check the CLI against the
author's reading of the API, not the API itself. A new CI check fails PRs
that touch src/ unless the description's Artifacts section holds real
output (a code block) or a screenshot. The skip-artifacts label opts out
for changes with nothing to run. The PR template and CONTRIBUTING.md
explain what to include.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant