Is the generative-AI checkbox checked by anything, or by eye?

[augbastos]

The PR template asks contributors to tick one of:

I did not use generative AI tools when creating this PR.
I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

From outside I can't tell what happens to a PR that ticks neither. Does anything
flag it, or does it depend on a reviewer noticing?

The question I actually care about is the second one: has the box changed
anything you can observe? Fewer surprise-AI PRs, more honest disclosure, or no
measurable difference? A checkbox that changed nothing is a much more
interesting fact than a checkbox that worked.

I ask because I wrote a small Action that lints exactly that box, and I'd rather
find out the problem isn't real than keep building. github.com/augbastos/scpe
for context.

[scruss]

This post reads more like an advert/show-and-tell for your Signed Contribution Provenance Envelope.

[Josverl]

I look at more than the checkbox and canned text phrases as some authors add more context.

I seldom find it is omitted, and I can only recall a few cases,most of them when the template was circumvented.

Most authors quickly correct when reminded a by a human, less so when CI is showing ❌

[augbastos]

Thanks, this is really useful.

Two things I'm taking from it. The omissions you see are people going around
the template, not leaving the box blank, a check on the box wouldn't catch
that, so the problem I imagined isn't the one you have.

And the part I can't argue with: authors correct when a human reminds them,
less when CI shows ❌. I'd assumed a red check was stronger than a nudge. If
it's weaker, then building this as a gate is the wrong idea, not just an
unnecessary one.

If you don't mind one question: any sense of why? Does a human reminder land
better because there's a person waiting on it, or does a ❌ just read like a
build error rather than a request?

Either way this helped. I'd rather find out the idea is wrong now than later.

[Josverl]

any sense of why?

https://media.tumblr.com/ccfd0d995da79f2f284c9ad47293bf1f/tumblr_inline_mjhc3yOgMp1qz4rgp.gif

[projectgus]

From outside I can't tell what happens to a PR that ticks neither. Does anything
flag it, or does it depend on a reviewer noticing?

It depends on a reviewer noticing, but it's a very easy thing to see when first looking at a PR. In this regard I see it as something like the Van Halen brown M&Ms test. We could very easily add an automated check, but so far hasn't seemed necessary (most people fill it in).

The question I actually care about is the second one: has the box changed
anything you can observe? Fewer surprise-AI PRs, more honest disclosure, or no
measurable difference? A checkbox that changed nothing is a much more
interesting fact than a checkbox that worked.

I feel we've had less total slop PRs, although it's hard to judge exactly. Both the agentic coding models and people's attitudes to them are changing quickly so it's hard to attribute any change to one thing in particular.

There are certainly some PRs where it's been much easier and quicker to close them or not spend time on them, on the basis of seeing the author ignoring the template and by extension our generative AI policy. This also sends a signal that they're[*] probably looking to chuck some PRs over the fence and move on quickly without engaging with us.

The other benefit is that when someone says that their PR involved generative AI it frames the review: the code will superficially look like someone took care when writing it, even though it's possible nobody has taken care. So I see it as a signal to look really closely at the details.

[*] Directly, or by prompting some agent.