Add MCP documentation discovery and help alongside Robot tools

.changeset/quiet-tools-search.md

@@ -0,0 +1,5 @@
+---
+'@transloadit/mcp-server': minor
+---
+
+Add `transloadit_list_docs` and `transloadit_get_doc_help` for discovering public documentation and reading bounded Markdown. Their discovery/help naming matches the existing Robot tools; optional search and API, SDK, FAQ, and Robot scopes cover broader integration guidance.

docs/prompts/2026-11-12-mcp-docs-review.md

@@ -0,0 +1,15 @@
+# MCP documentation tools — PR #472 review follow-up
+
+- [x] Address kvz’s September 23 review: use `transloadit_list_docs` and `transloadit_get_doc_help`, matching the existing Robot discovery/help names. Keep the existing Robot tools unchanged; use their optional `search` input convention.
+- [x] Explain the overlap: Robot tools return structured Step parameter references; documentation tools cover API authentication/endpoints, SDK setup, FAQs, integration concepts and full Robot prose. Update descriptions, README and the release changeset.
+- [x] Rebase onto current main (`80251a8`), resolving the tool registry and README conflicts without reverting authentication metadata or Builtin discovery behavior.
+- [x] Preserve the earlier scoped-index and bounded-stream review fixes. Retire the rollout fallback now that scoped indexes are deployed: missing/oversized indexes return errors instead of silently incomplete results.
+- [x] Validate today’s live indexes and fix observed duplicate URLs, missing overview links and poor multi-term ranking. Add deterministic regression coverage and MCP client-level naming, anonymous access, listing, help and sanitized-error tests.
+
+Live checks on November 12, 2026 at 14:08 UTC passed for API/authentication, FAQ/assembly expiry, SDK/Node.js, Robot/resize and all/authentication. Each expected page ranked first and was fetched as bounded Markdown. The FAQ page was returned whole (455 characters); longer pages were truncated at 1,500 characters. Unfiltered discovery returned 20 distinct pages; `/docs/` resolved to the `/docs.md` overview.
+
+The MCP workspace typecheck, all 363 MCP unit tests, root `yarn check`, and both stdio integration tests pass. A real in-memory MCP client also listed and fetched the live authentication guide without account credentials. CI is recorded in the PR’s validation section.
+
+Independent review limitation: `council-review` is not installed. A read-only Claude CLI review was attempted, but its OAuth session is expired and could not be refreshed. No independent review is claimed.
+
+GitHub prohibits force pushes to the existing PR branch. The changes were rebased locally and validated; publication retains the previous PR head as a merge parent, preserving that history without changing the tested implementation.

packages/mcp-server/CHANGELOG.md

@@ -1,5 +1,13 @@
 # @transloadit/mcp-server
 
+## 0.8.7
+
+### Patch Changes
+
+- 17ea841: Storage requests (`getStoredAsset`, `listStoredAssets`, `moveStoredAsset` and `deleteStoredAsset`) now sign their HTTP method and URL path into `params.auth` as `method` and `path`, as the Transloadit API requires for Storage requests.
+- Updated dependencies [17ea841]
+  - @transloadit/node@6.8.0
+
 ## 0.8.6
 
 ### Patch Changes

packages/mcp-server/README.md

@@ -350,6 +350,8 @@ The JSON config accepts the same keys as `createTransloaditMcpHttpHandler()`, in
 | Tool                                     | Account | OAuth scopes                         | Notes                                           |
 | ---------------------------------------- | ------- | ------------------------------------ | ----------------------------------------------- |
 | `transloadit_lint_assembly_instructions` | no      | none                                 | read-only                                       |
+| `transloadit_list_docs` | no | none | read-only; public documentation |
+| `transloadit_get_doc_help` | no | none | read-only; public documentation |
 | `transloadit_list_robots`                | no      | none                                 | read-only                                       |
 | `transloadit_get_robot_help`             | no      | none                                 | read-only                                       |
 | `transloadit_create_assembly`            | yes     | `assemblies:write`, `templates:read` | destructive, open-world (URL imports), widget   |
@@ -412,6 +414,19 @@ and null for older Builtins without a declared schema. Workspace Templates omit
 `transloadit_get_profile` returns `{ id, name?, nickname? }` for the Workspace behind the current
 credentials, derived from the Workspace's own Assemblies or Templates.
 
+### Documentation and Robot help
+
+| Discovery | Help | Coverage |
+| --- | --- | --- |
+| `transloadit_list_robots` | `transloadit_get_robot_help` | Structured Robot summaries, required/optional parameters and examples for writing Assembly Steps. |
+| `transloadit_list_docs` | `transloadit_get_doc_help` | Public Markdown documentation for API authentication and endpoints, SDK setup, FAQs, integration concepts and full Robot articles. |
+
+The existing Robot tools keep their names and behavior. Documentation tools add the broader guides and prose that a Robot parameter reference does not cover. Like `transloadit_list_robots`, `transloadit_list_docs` accepts an optional `search` filter. It also accepts `scope` (`all`, `api`, `faq`, `robots`, `sdks`) and `limit` (1–20, default 8); omitting `search` lists pages alphabetically. Search prioritizes pages matching more of the search terms and returns each Markdown URL once.
+
+`transloadit_get_doc_help` accepts `path_or_url` from a listing or a public documentation path, plus `max_chars` (1,000–100,000, default 50,000). It returns `{ title, url, markdown, truncated }`. It reads the response as a bounded stream, rejects non-documentation URLs and redirects, and reports temporary documentation failures without returning upstream diagnostics.
+
+The tools use the live `/docs/llms.txt` and scoped API/FAQ/Robot/SDK indexes. Unavailable or oversized indexes produce an explicit error instead of silently searching an incomplete root catalog. They fetch only public documentation and need no account credentials in caller-credentials mode; hosted deployments still apply the OAuth policy described above.
+
 ### Result widget
 
 `transloadit_create_assembly` and `transloadit_wait_for_assembly` link the MCP Apps resource

packages/mcp-server/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@transloadit/mcp-server",
-  "version": "0.8.6",
+  "version": "0.8.7",
   "description": "Transloadit MCP server",
   "keywords": [
     "mcp",
@@ -63,7 +63,7 @@
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.29.0",
-    "@transloadit/node": "^6.7.0",
+    "@transloadit/node": "^6.8.0",
     "@transloadit/sev-logger": "^0.1.9",
     "@transloadit/zod": "^7.5.0",
     "ajv": "^8.17.1",

packages/mcp-server/server.json

@@ -3,7 +3,7 @@
   "name": "io.github.transloadit/mcp-server",
   "title": "Transloadit Media Processing",
   "description": "Process video, audio, images and documents with Transloadit Robots. Connect by URL with OAuth.",
-  "version": "0.8.6",
+  "version": "0.8.7",
   "websiteUrl": "https://transloadit.com/docs/sdks/mcp-server/",
   "icons": [
     {
@@ -21,7 +21,7 @@
     {
       "registryType": "npm",
       "identifier": "@transloadit/mcp-server",
-      "version": "0.8.6",
+      "version": "0.8.7",
       "runtimeHint": "npx",
       "packageArguments": [
         {

packages/mcp-server/src/docs.ts

@@ -0,0 +1,203 @@
+export type DocsScope = 'all' | 'api' | 'faq' | 'robots' | 'sdks'
+
+export interface DocsSearchResult {
+  title: string
+  url: string
+  description?: string
+}
+
+export interface DocsPage {
+  title: string
+  url: string
+  markdown: string
+  truncated: boolean
+}
+
+const docsOrigin = 'https://transloadit.com'
+const docsFetchTimeoutMs = 10_000
+const docsIndexMaxChars = 500_000
+const docsIndexUrls: Record<DocsScope, readonly string[]> = {
+  all: [
+    `${docsOrigin}/docs/llms.txt`,
+    `${docsOrigin}/docs/api/llms.txt`,
+    `${docsOrigin}/docs/faq/llms.txt`,
+    `${docsOrigin}/docs/robots/llms.txt`,
+    `${docsOrigin}/docs/sdks/llms.txt`,
+  ],
+  api: [`${docsOrigin}/docs/api/llms.txt`],
+  faq: [`${docsOrigin}/docs/faq/llms.txt`],
+  robots: [`${docsOrigin}/docs/robots/llms.txt`],
+  sdks: [`${docsOrigin}/docs/sdks/llms.txt`],
+}
+const markdownLinkPattern = /^- \[([^\]]+)\]\(([^)]+)\)(?::\s*(.*))?$/u
+
+function normalizeDocsUrl(value: string): URL | undefined {
+  if (!URL.canParse(value, docsOrigin)) return
+
+  const url = new URL(value, docsOrigin)
+  if (url.origin !== docsOrigin || url.username || url.password) return
+  url.hash = ''
+  url.search = ''
+  if (url.pathname === '/llms.txt') return url
+  if (['/docs', '/docs/', '/docs.md'].includes(url.pathname)) {
+    url.pathname = '/docs.md'
+    return url
+  }
+  if (!url.pathname.startsWith('/docs/')) return
+
+  if (!url.pathname.endsWith('.md') && !url.pathname.endsWith('.txt')) {
+    url.pathname = `${url.pathname.replace(/\/$/u, '')}.md`
+  }
+  return url
+}
+
+interface DocsTextResponse {
+  text: string
+  truncated: boolean
+}
+
+async function readBoundedResponseText(
+  response: Response,
+  maxChars: number,
+): Promise<DocsTextResponse> {
+  if (response.body === null) {
+    return { text: '', truncated: false }
+  }
+
+  const decoder = new TextDecoder()
+  const reader = response.body.getReader()
+  let text = ''
+
+  while (true) {
+    const { done, value } = await reader.read()
+    if (done) {
+      text += decoder.decode()
+      return {
+        text: text.slice(0, maxChars),
+        truncated: text.length > maxChars,
+      }
+    }
+
+    text += decoder.decode(value, { stream: true })
+    if (text.length <= maxChars) continue
+
+    await reader.cancel()
+    return { text: text.slice(0, maxChars), truncated: true }
+  }
+}
+
+async function fetchDocsText(
+  url: string,
+  maxChars: number,
+  fetcher: typeof fetch,
+): Promise<DocsTextResponse> {
+  const response = await fetcher(url, {
+    headers: { Accept: 'text/markdown, text/plain;q=0.9' },
+    redirect: 'error',
+    signal: AbortSignal.timeout(docsFetchTimeoutMs),
+  })
+  if (!response.ok) {
+    throw new Error(`Documentation request failed with HTTP ${response.status}.`)
+  }
+  const contentType = response.headers.get('content-type') ?? ''
+  if (!contentType.includes('text/markdown') && !contentType.includes('text/plain')) {
+    throw new Error(`Documentation request returned unsupported content type: ${contentType}.`)
+  }
+  return readBoundedResponseText(response, maxChars)
+}
+
+function parseIndex(index: string): DocsSearchResult[] {
+  const results = new Map<string, DocsSearchResult>()
+  for (const line of index.split('\n')) {
+    const match = markdownLinkPattern.exec(line.trim())
+    if (match === null) continue
+
+    const [, title, rawUrl, description] = match
+    if (title === undefined || rawUrl === undefined) continue
+    const url = normalizeDocsUrl(rawUrl)
+    if (url === undefined || !url.pathname.endsWith('.md')) continue
+    if (!results.has(url.href)) results.set(url.href, { title, url: url.href, description })
+  }
+  return [...results.values()]
+}
+
+function matchesScope(result: DocsSearchResult, scope: DocsScope): boolean {
+  if (scope === 'all') return true
+  return new URL(result.url).pathname.startsWith(`/docs/${scope}/`)
+}
+
+function getSearchScore(
+  result: DocsSearchResult,
+  terms: string[],
+): { matches: number; weight: number } {
+  const title = result.title.toLowerCase()
+  const description = result.description?.toLowerCase() ?? ''
+  const url = result.url.toLowerCase()
+
+  let matches = 0
+  let weight = 0
+  for (const term of terms) {
+    const score = title.includes(term)
+      ? 5
+      : description.includes(term)
+        ? 2
+        : url.includes(term)
+          ? 1
+          : 0
+    if (score === 0) continue
+    matches++
+    weight += score
+  }
+  return { matches, weight }
+}
+
+/** List public documentation pages, optionally ranked by search text, without account credentials. */
+export async function listTransloaditDocs(
+  search: string | undefined,
+  scope: DocsScope,
+  limit: number,
+  fetcher: typeof fetch = fetch,
+): Promise<DocsSearchResult[]> {
+  const terms = search?.toLowerCase().split(/\s+/u).filter(Boolean) ?? []
+  // The scoped indexes are deployed. Fail explicitly if one is unavailable rather than
+  // silently replacing it with the incomplete root index and presenting missing results.
+  const indexResponses = await Promise.all(
+    docsIndexUrls[scope].map((url) => fetchDocsText(url, docsIndexMaxChars, fetcher)),
+  )
+  if (indexResponses.some(({ truncated }) => truncated)) {
+    throw new Error('A documentation index exceeded the maximum supported size.')
+  }
+  const index = indexResponses.map(({ text }) => text).join('\n')
+
+  return parseIndex(index)
+    .filter((result) => matchesScope(result, scope))
+    .map((result) => ({ result, score: getSearchScore(result, terms) }))
+    .filter(({ score }) => terms.length === 0 || score.matches > 0)
+    .sort(
+      (left, right) =>
+        right.score.matches - left.score.matches ||
+        right.score.weight - left.score.weight ||
+        left.result.title.localeCompare(right.result.title),
+    )
+    .slice(0, limit)
+    .map(({ result }) => result)
+}
+
+/** Fetch one public Transloadit documentation page as bounded Markdown. */
+export async function getTransloaditDocHelp(
+  pathOrUrl: string,
+  maxChars: number,
+  fetcher: typeof fetch = fetch,
+): Promise<DocsPage | undefined> {
+  const url = normalizeDocsUrl(pathOrUrl)
+  if (url === undefined) return
+
+  const { text: markdown, truncated } = await fetchDocsText(url.toString(), maxChars, fetcher)
+  const title = markdown.match(/^#\s+(.+)$/mu)?.[1] ?? url.pathname
+  return {
+    title,
+    url: url.toString(),
+    markdown,
+    truncated,
+  }
+}