[Ansu (SU-AN-coder)]

Summary

_is_meta_prompt() treated any message that starts with < and ends with > as system noise, so a brief pasted into the Claude Code desktop app — delivered as <pasted_content id="…">…</pasted_content id="…"> — was dropped. The session was then mined from a short follow-up (heuristic_mine() takes the first surviving prompt), and when the brief was its only user turn the session disappeared entirely, because _is_headless_replay() reads "zero harvested user turns" as "engine replay". A user message that starts with a <system-reminder> block followed by the real request was kept, with the reminder copied verbatim into the intent.

Four changes, all in skillopt_sleep/harvest.py:

1. Pasted briefs are user content. _strip_host_wrappers() unwraps <pasted_content …> / </pasted_content …> and keeps the inner text as the user's; digest_transcript() runs it before the meta test, so the intent is what the user pasted rather than the follow-up.
2. A <system-reminder> block is dropped only while it leads the message, which is the form the host prepends; the tag match also tolerates attributes. If the user quotes one — inline code, a fenced XML sample — it stays, because that is text they wrote.
3. The "message is nothing but one host tag block" rule now matches a closed list of host tags — command-name, command-message, command-args, local-command-stdout, local-command-caveat, system-reminder, bash-input, bash-stdout, bash-stderr, ide_opened_file, ide_selection, and Copilot Chat's environment_info. A wrapper the list does not know about now keeps its content instead of the session's request.
4. A transcript with no user turns is a replay again, whatever assistant work it holds. Such a record mines nothing, so admitting it only let it take a slot under --max-sessions and displace a real session.

The closed list was picked, not guessed

The repo's own tests and fixtures pin which tags must stay out. tests/fixtures/copilot/session-v3.jsonl stores <environment_info>private machine context</environment_info> as a request's message text, and test_digest_extracts_visible_content_and_enforces_privacy_boundary requires it to stay out of the digest. harvest_copilot._sanitize_text() calls the same _is_meta_prompt(), so narrowing the rule without listing that tag fails that test (AssertionError: 2 != 1 on n_user_turns, and harvest_copilot() returns 0 sessions instead of 1) — I hit exactly that while building this. <system-reminder>, <command-name> and <command-message> are pinned by tests/test_sleep_engine.py; bash-* and ide_* are the remaining whole-message host tags of the same shape.

Review corrections

Both boundary failures the review named were introduced by this PR, not present at base, and both are now fixed:

• Quoted reminder text was being deleted. The first head removed every <system-reminder>…</system-reminder> occurrence in the message, so a prompt that quoted one in inline code or a fenced XML block lost it and the harvested request and mined intent changed. Removal is now anchored to the leading position; test_inline_code_system_reminder_example_is_user_text and test_fenced_system_reminder_example_is_user_text pin it. At base ae2f766 both pass (main preserves the text); at the reviewed head d97592c both fail; on this head both pass.
• A no-intent record could take the cap. The zero-user/assistant-present exception admitted a known host-only isMeta transcript before the cap; under --max-sessions 1 it displaced the real session and the night mined nothing. Reverted to the base behaviour, and the pasted-brief case does not need the exception: unwrapping now yields a user turn. test_host_only_transcript_does_not_take_a_session_slot builds exactly that pair (['host-note'] != ['real-task'] on the reviewed head).

Validation

Focused on the flow _is_meta_prompt() → strip → digest_transcript() → harvest() → heuristic_mine().

8 new tests in tests/test_sleep_engine.py::TestHarvest: the brief is kept and unwrapped; only known host tags are filtered (an unknown wrapper keeps its text); the pasted brief becomes the mined intent with the follow-up as context; a pasted-brief-only session survives harvest(); a leading reminder does not ride into the intent; a quoted reminder in inline code / a fenced block survives; a host-only transcript does not take a session slot.

Three commits, three regimes, same test file:

$ python -m pytest tests/test_sleep_engine.py -q     # base ae2f766 (tests copied in)
5 failed, 134 passed, 1 skipped          # the five #318 tests; the three new
                                        # regression tests pass, as the review said
$ python -m pytest tests/test_sleep_engine.py -q     # reviewed head d97592c
3 failed, 136 passed, 1 skipped          # the three boundary tests
$ python -m pytest tests/test_sleep_engine.py -q     # this head f5f1f9a
139 passed, 1 skipped, 12 subtests passed

The three failures on d97592c are the review's two points:

FAILED ...test_inline_code_system_reminder_example_is_user_text
  AssertionError: Lists differ: ['How[13 chars]ape `` inside my transcript fixture?'] != [the prompt]
FAILED ...test_fenced_system_reminder_example_is_user_text
  the reminder block was deleted from inside the fence
FAILED ...test_host_only_transcript_does_not_take_a_session_slot
  AssertionError: Lists differ: ['host-note'] != ['real-task']

The issue's own reproduction script, unchanged, base vs. branch (no model calls, no network):

# base ae2f766
_is_meta_prompt(BRIEF) = True
harvested sessions: ['s1-pasted-then-followup', 's3-reminder-prefix']
task from s3-reminder-prefix: intent = '<system-reminder>\nStatus note injected by the host app.\n</system-reminder>\n\nYes: run the third review on range r2..abc1234 and update the report.'
task from s1-pasted-then-followup: intent = 'Yes, go ahead with option A.'

# this branch
_is_meta_prompt(BRIEF) = False
harvested sessions: ['s1-pasted-then-followup', 's2-pasted-only', 's3-reminder-prefix']
task from s2-pasted-only: intent = 'Install tool X into a dedicated folder, pin the version, back up the configs first and write a short report.'
task from s3-reminder-prefix: intent = 'Yes: run the third review on range r2..abc1234 and update the report.'
task from s1-pasted-then-followup: intent = 'Install tool X into a dedicated folder, pin the version, back up the configs first and write a short report.'

s2-pasted-only (the pasted brief as its only user turn) is now harvested, and both pasted sessions are mined from the brief rather than from the follow-up. The reminder-prefix session's intent is still the request, unchanged by the narrowed stripping.

• Full suite, CI environment variables as in .github/workflows/ci.yml: 1801 passed, 11 skipped, 359 subtests passed at base ae2f766; 1809 passed, 11 skipped, 359 subtests passed on this head (the eight new tests).
• ruff check: tests/test_sleep_engine.py — "All checks passed!"; skillopt_sleep/harvest.py reports only the pre-existing I001 that is also present at base (a blank line after the import block, untouched here).
• python -m mkdocs build --strict passed; no documentation is touched.

Not in this PR

• harvest: single-turn headless sessions from other local tooling are harvested as user tasks and fill the session cap #319 (machine-written single-turn summary sessions filling the session cap) and the entrypoint question in Mining counts headless claude -p calls and sub-agent fan-out as user tasks, inflating recurring-task signal #62 are separate.
• The <cross-session-message> wrapper in tests/test_sleep_engine.py carries isMeta: true, so it never reaches this filter; I left it out of the closed list rather than list a tag no path exercises.
• bash-* double as user input in shell mode and were dropped as noise before this change too; I kept them filtered to avoid changing behaviour outside the reported shape.

I read every line of the diff myself. The branch is SU-AN-coder:fix/harvest-pasted-content-318 at f5f1f9a, and the owner has it.

Fixes #318

[Ansu (SU-AN-coder)]

@microsoft-github-policy-service agree

[Yifan Yang (Yif-Yang)]

Reviewed d97592c5c634b198d2971db390b119fd3663d266. The pasted-only and pasted-then-follow-up cases now reach the real miner with the original brief, and the normal markup, host-prefix and Copilot privacy controls pass. Two new data-loss boundaries prevent recommending this head:

• The global system-reminder substitution also removes user-authored literal examples inside inline code and fenced XML blocks, changing both the harvested request and its mined intent. Both examples are preserved on main.
• The new zero-user/assistant-present exception admits a known host-only, isMeta transcript before the session cap. A newer no-intent record can therefore displace a genuine user task and leave no mined task. Main excludes that record; independent parent verification confirms the regression.

Please retain the working pasted-brief fix while limiting host stripping so quoted user content survives, and keep known no-intent records from consuming the task cap. The passing full suite does not cover these failures. #318 stays open pending these narrow boundary corrections and retesting.

[Ansu (SU-AN-coder)]

Both boundaries were regressions this PR introduced, not gaps on main, and both are fixed in f5f1f9a.

Host stripping is anchored to the leading block. _strip_host_wrappers was deleting every <system-reminder>…</system-reminder> occurrence in the message, so a prompt that quoted one — inline code or a fenced XML example — lost it, and both the harvested request and the mined intent changed. Removal now happens only while the block leads the message, which is the form the host prepends; the match also accepts attributes (<system-reminder …>). Anything the user wrote after their own text is untouched. A message that is nothing but a reminder still ends up empty after stripping and is still filtered as meta.

Two tests pin it, and their three-commit history is the proof of direction:

• test_inline_code_system_reminder_example_is_user_text
• test_fenced_system_reminder_example_is_user_text

commit	result
base ae2f766 (tests copied in)	both pass — main preserves the text, as you said
reviewed head d97592c	both fail — ['How[13 chars]ape `` inside my transcript fixture?']
f5f1f9a	both pass

A no-intent record no longer consumes the cap. The zero-user/assistant-present exception is reverted: _is_headless_replay() calls a transcript with zero harvested user turns a replay again, regardless of assistant work. It mines nothing, so admitting it only let it displace a real session. test_host_only_transcript_does_not_take_a_session_slot builds the pair you described — a newer host-only isMeta transcript with assistant turns plus an older real task, limit=1 — and gives ['host-note'] != ['real-task'] on d97592c, ['real-task'] on this head.

The #318 case does not need that exception any more, which is why reverting it costs nothing: unwrapping a pasted brief now yields a real user turn, so the pasted-only session is admitted on its own merit (test_session_whose_only_user_turn_is_a_pasted_brief_survives_harvest, unchanged and still green).

Verification on f5f1f9a:

• pytest tests/test_sleep_engine.py -q — 139 passed, 1 skipped, 12 subtests. Same file at d97592c — 3 failed, 136 passed, 1 skipped; at base ae2f766 — 5 failed, 134 passed, 1 skipped (the five harvest: user messages wrapped in <pasted_content …> are treated as meta, so pasted briefs never become tasks #318 tests, while the three new boundary tests pass, which is what "preserved on main" means here).
• pytest -q — 1,809 passed, 11 skipped, 359 subtests (1,801 at base, +8 new tests).
• ruff check — tests/test_sleep_engine.py clean; harvest.py only the pre-existing I001 that is on base too.
• mkdocs build --strict — exit 0.

The PR body now carries the same three-commit table so the direction is visible without reading this thread. I have not rebased onto 1a44050; the branch merges cleanly and I would rather not churn the SHA you reviewed for a purely mechanical rebase — say the word if you prefer it on current main.