Skip to content

ci: run the release-signoff metric lint whenever the skill changes - #39929

Draft
bosconi wants to merge 1 commit into
mainfrom
jc/skill-metrics-lint-inputs
Draft

bosconi wants to merge 1 commit into
mainfrom
jc/skill-metrics-lint-inputs

Conversation

@bosconi

@bosconi bosconi commented Oct 17, 2026

Copy link
Copy Markdown
Member

The mz-release-signoff metric lint never runs on PRs that only change the skill, so a broken skill reference is first caught on main.

Current failure mode

  • ci/test/lint-skill-metrics.sh runs inside the Lint docs step (ci/test/lint-docs.sh).
  • Lint docs declares inputs: [doc/user, src/adapter/src/catalog, test/sqllogictest/autogenerated]. On PR builds, ci/mkpipeline.py trims every step whose inputs didn't change. .agents/skills/mz-release-signoff/** is not an input, so a skill-only PR never runs its own lint.
  • main builds always run Lint docs (it's in the keep-list in mkpipeline.py), so the failure first appears after merge. Lint docs also gates deploy-website, so a broken skill reference blocks the docs deploy too.
  • Example: mz-release-signoff: notes from the v26.43.0 production US sweep #39134 touched only skill files. Its PR build (135898) had no Lint docs job. After it merged, main failed Lint docs for 11 builds (13:57–15:21 UTC on 2026-10-12, starting at 138591), until mz-release-signoff: pin reference-dashboards.md metric count to 26 #39769 fixed the pinned count.

Change

  • New step lint-skill-metrics in the Lints group runs ci/test/lint-skill-metrics.sh. Its inputs are .agents/skills/mz-release-signoff and doc/user/data/metrics.yml, the catalog the lint resolves names against.
  • Removed the call from ci/test/lint-docs.sh. A skill failure no longer blocks the website deploy, and a docs-only PR no longer runs the skill's pytest.
  • Added lint-skill-metrics to the main keep-list in ci/mkpipeline.py. Without it, webhook builds on main would skip the new step like they skip every other non-lint step.

Testing

  • ci/test/lint-skill-metrics.sh passes locally (4/4).
  • git ls-files on the new inputs matches mz-release-signoff: notes from the v26.43.0 production US sweep #39134's files; on the old Lint docs inputs it matches none of them.
  • The template parses and step ids are still unique. check-pipeline.sh only runs in CI, so this PR's own build is the first real pipeline check. Since it edits ci/, that build runs untrimmed.

🤖 Generated with Claude Code

The lint ran inside Lint docs, whose inputs don't cover
.agents/skills/mz-release-signoff, so PR builds trimmed it away for
skill-only changes and the failure first appeared on main. Give it its own
step whose inputs are the skill and the metrics catalog it checks against,
and keep it on main builds alongside the other lints.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant