From 422e49f540cfc8a2f97066a9a029cd5a50a20c40 Mon Sep 17 00:00:00 2001 From: yifu zuo Date: Thu, 29 Oct 2026 12:33:53 -0500 Subject: [PATCH] feat(property-based-testing): add property-based-testing skill plugin Testing methodology: default to property-based tests over example-based unit tests; coverage lands in the same commit as the production code it proves. Contributed from our own agent skill set. --- .factory-plugin/marketplace.json | 6 +++ README.md | 8 ++++ .../.factory-plugin/plugin.json | 8 ++++ plugins/property-based-testing/README.md | 9 ++++ .../skills/property-based-testing/SKILL.md | 46 +++++++++++++++++++ skills/property-based-testing | 1 + 6 files changed, 78 insertions(+) create mode 100644 plugins/property-based-testing/.factory-plugin/plugin.json create mode 100644 plugins/property-based-testing/README.md create mode 100644 plugins/property-based-testing/skills/property-based-testing/SKILL.md create mode 120000 skills/property-based-testing diff --git a/.factory-plugin/marketplace.json b/.factory-plugin/marketplace.json index 4282159..722dab1 100644 --- a/.factory-plugin/marketplace.json +++ b/.factory-plugin/marketplace.json @@ -53,6 +53,12 @@ "description": "Pull request lifecycle skills: create PRs with consistent conventions and follow up on them until merge-ready", "source": "./plugins/code-review", "category": "productivity" + }, + { + "name": "property-based-testing", + "description": "Property-based testing methodology: prove invariants with generated cases instead of example-based tests", + "source": "./plugins/property-based-testing", + "category": "quality" } ] } diff --git a/README.md b/README.md index 9793470..be67c28 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,14 @@ Skills for continuous learning and improvement. Autonomous experiment loop for optimization research. Try an idea, measure it, keep what works, discard what doesn't, repeat. Works standalone or as a mission worker. +### property-based-testing + +Property-based testing methodology: prove invariants with generated cases. + +**Skills:** + +- `property-based-testing` - Default to property-based tests for features and bug fixes + ## Plugin Structure Each plugin follows the Factory plugin format: diff --git a/plugins/property-based-testing/.factory-plugin/plugin.json b/plugins/property-based-testing/.factory-plugin/plugin.json new file mode 100644 index 0000000..7b001a9 --- /dev/null +++ b/plugins/property-based-testing/.factory-plugin/plugin.json @@ -0,0 +1,8 @@ +{ + "name": "property-based-testing", + "description": "Property-based testing methodology: prove invariants with generated cases instead of example-based tests", + "author": { + "name": "yifu zuo", + "email": "yzuo.2019@gmail.com" + } +} diff --git a/plugins/property-based-testing/README.md b/plugins/property-based-testing/README.md new file mode 100644 index 0000000..48a797c --- /dev/null +++ b/plugins/property-based-testing/README.md @@ -0,0 +1,9 @@ +# property-based-testing + +Property-based testing methodology: prove invariants with generated cases. + +**Skills:** + +- `property-based-testing` - Default to property-based tests for features and bug fixes; coverage lands in the same commit as the production code it proves + +Contributed from our own agent skill set, battle-tested on a hackathon repo's quality gates. diff --git a/plugins/property-based-testing/skills/property-based-testing/SKILL.md b/plugins/property-based-testing/skills/property-based-testing/SKILL.md new file mode 100644 index 0000000..1dcc06d --- /dev/null +++ b/plugins/property-based-testing/skills/property-based-testing/SKILL.md @@ -0,0 +1,46 @@ +--- +name: property-based-testing +description: "Dramatically enhance code quality and reliability with property-based testing to help build functional ambitious features quickly." +--- + +# Employ property-based tests + +While there may be occasional reasons to write conventional unit or integration tests, by default the overwhelming majority of automated tests should be property-based. Appropriate test coverage should land in the same commit as the corresponding production code. + +## Inputs + +- The change to production code or a bug report +- The high-level project goals + +## Common Pitfalls + +- Don't reimplement production code. Refactor production code so helper methods can be called directly from tests, potentially with extra parameters, as necessary. +- Good property-based tests afford greatly superior test coverage to an equivalent unit test, but only if the invariant they test is sensible. For this reason, they require more effort and contemplation to write well, and that extra effort and time can and should be expended. The time cost is paid back by drastically reducing time spent debugging down the road and enhancing the quality and reliability of the product. +- Think carefully about the best shape of the arbitraries to test the pertinent invariants. Avoid filtering arbitraries within reason as this slows down tests. Usually a filter can be replaced with a direct construction. Also think carefully about how the arbitrary will shrink; an arbitrary that shrinks better can save a lot of time in debugging down the road. +- Property-based tests are rare in your training data, so your first draft often will not be idiomatic. Look them over several times each to ensure they are idiomatic. +- Many bugs occur at third-party integration points, because our mental model doesn't match the real-world behavior of a third-party system. When interacting with APIs, find ways to prove that the system works as intended. For example, install an authoritative test bed or implementation of the API as a dev dependency, and ensure our integration is properly tested. Often consulting the API's documentation is the right place to start. +- If you discover additional pitfalls, please propose updates to this skill via chat to avoid falling into the same trap in the future, but never edit this skill directly. + +## Bugs + +### Step 1 — Reproduce Issue + +Write a new or modify an existing idiomatic property-based test to cover the reported bug. Reason carefully about the system and high-level project goals. Are there other similar or related issues in the unit under test or analogous ones in other units? Propose follow-up work to the user via chat. + +### Step 2 - Fix Production Code + +Correct the production-code in the appropriate manner that addresses the root cause. Do not weaken test coverage to produce a green build and give a false sense of confidence. If you cannot identify the root cause, explain your findings via chat, not in the repo, and ask the user to work with you to troubleshoot the issue. + +## Features + +### Step 1 - Brainstorm Invariants + +Brainstorm the most elegant invariants that illustrate the intended behavior of the new or modified feature. Reason carefully about the system and high-level project goals. What guarantees does the unit under test need to provide to achieve those? Clarify intended behavior with the user if there is ambiguity or cases that may not have been thoroughly thought out or considered based on recent prompts. + +### Step 2 - Add Test Coverage + +Enforce your best ideas as idiomatic property-based tests. The product needs to work correctly when stress tested by third parties, so adequate test coverage is essential. Think in terms of what is necessary to achieve the project's end goals, and ensure the unit under test does its part in service of those objectives. Clarify project goals and raise concerns about the unit under test's functionality or code quality if necessary. + +## Report back + +What properties you determined the pertinent production code must have and how you used property-based testing to prove that it does diff --git a/skills/property-based-testing b/skills/property-based-testing new file mode 120000 index 0000000..8573d6e --- /dev/null +++ b/skills/property-based-testing @@ -0,0 +1 @@ +../plugins/property-based-testing/skills/property-based-testing \ No newline at end of file