[fatwang2]

What changed

• s1 deepcrawl <url> starts a POST /deepcrawl task, polls GET /deepcrawl/status/{taskId} every 2 s until it finishes or --timeout runs out (default 300 s), then prints the message, a stats line, one line per page (title, URL, character count), failed URLs and the resultUrl/zipUrl. The help text and output say a task costs 20 credits.
  • Options: --type sitemap|all, -n/--max-pages (1-1000), repeatable --include <path> / --exclude <path> (up to 20 each, at most 200 characters), --format json|zip (the CLI defaults to json; the API default stays zip), --no-wait (print the task id and exit), --timeout <seconds>, -o/--output <file> (writes { pages, failed } for json, downloads the result URL when the result is too large to inline, or downloads the ZIP), --json (raw status response).
  • Options are checked against the API's limits before the task starts, so a bad option costs no credits.
  • A timeout leaves the task running, exits 1 and prints s1 deepcrawl status <taskId>. A failed task exits 1 with the API's error. A completed task that found nothing prints its message.
• s1 deepcrawl status <taskId> [-o <file>] [--json] checks a task once (free).
• SDK: deepcrawl() and deepcrawlStatus() plus types. deepcrawlStatus returns a failed task (the API sends it as an HTTP 500 that still describes the task) instead of throwing. It throws for 404 (unknown task) and other errors. api.ts gains a small send() helper so the status call can use GET. request() behaves the same as before.
• CLI parsing: the program now calls enablePositionalOptions(), so options after a subcommand belong to that subcommand (s1 deepcrawl status <id> --json reaches status). Side effect: s1 search -V now errors; s1 -V / s1 --version still work.
• Docs: README command reference, the quick-start screen, the agent skill (description, when-to-use table, crawl vs deepcrawl vs learn, a command section) and reference/examples.md. src/learn.ts doesn't list commands, so it is unchanged.
• Release: version 2.3.0 in package.json / package-lock.json, CHANGELOG entry dated 2026-11-05, matching the 2.2.0 release.

How it was tested

• npm test (build + node --test tests/*.test.mjs): 47/47 pass. No live API calls.
  • New tests/deepcrawl.test.mjs runs the built CLI against a local mock Search1API (HTTP server). It covers the request body, polling through processing, human output, --json, -o for JSON pages and ZIP download, status --json, --no-wait, failed tasks (500), timeout, an empty completed task, and 11 invalid-option cases that must not reach the API.
  • tests/sdk.test.mjs: the body sent by deepcrawl(), and how deepcrawlStatus() handles GET, task-id URL encoding, failed tasks (500), 404 and 502.
  • tests/commands.test.mjs: help output for deepcrawl and deepcrawl status.
• npx tsc --noEmit is clean. npm pack --dry-run includes dist/commands/deepcrawl.js. The repo has no lint setup.

Sample output (mocked):

Started deepcrawl task 7f3c9a1e-… (20 credits); waiting up to 300s.     # stderr
Task is processing...                                                     # stderr
Successfully crawled 4 URLs and created JSON.
212 discovered · 25 matched · 20 over max-pages · 4 crawled · 1 failed

1. Hono - Web framework built on Web Standards https://hono.dev/docs/ · 6,412 chars
2. Getting Started https://hono.dev/docs/getting-started/basic · 3,120 chars
3. Routing https://hono.dev/docs/api/routing · 11,873 chars
4. https://hono.dev/docs/api/context · 9,054 chars

Failed (1):
  https://hono.dev/docs/api/presets — Timeout after 30000ms

Result: https://…/7f3c9a1e.json
Page Markdown: s1 deepcrawl status 7f3c9a1e-… -o pages.json (free)

What to check after release

• Run a live s1 deepcrawl https://hono.dev/docs/ -n 20 --include /docs (20 credits). It should finish in about 10-20 s with 20 pages listed. Then run s1 deepcrawl status <taskId> -o pages.json and check that the file has pages[].content.
• --format zip -o site.zip downloads a valid ZIP from the live zipUrl.
• s1 deepcrawl status <unknown-id> shows the API's 404 "Task not found" error.
• Run the Release workflow on master to publish 2.3.0 (this PR doesn't tag or publish), then check that s1 update / npm deliver it.

🤖 Generated with Claude Code