Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/fork-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ jobs:
- name: Reused app account scenario
run: bun run e2e:self-host --test-name '^a reused hosted app account (stops serving|refuses a call)' --workers 1
- name: Operator feedback scenarios
run: bun run e2e:self-host --test-name 'self-host feedback webhook|self-host with DO_NOT_TRACK|self-host analytics pseudonymize|Hosted feedback enforces'
run: bun run e2e:self-host --test-name 'self-host feedback webhook|self-host feedback files|self-host with DO_NOT_TRACK|self-host analytics pseudonymize|Hosted feedback enforces'
# Run the large cold catalog alone, not beside the functional scenarios' servers.
- name: Cold MCP discovery
run: bun run e2e:self-host --test-name 'Cold MCP discovery serves 8549'
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/run/usage-analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ EXECUTOR_DISABLE_ANALYTICS=1
Any value other than empty, `0`, `false`, `no` or `off` turns analytics off.
Nothing is recorded or sent to analytics, including feedback: the `feedback.submit`
tool then returns `FeedbackDisabled` unless the self-host operator configured
`EXECUTOR_FEEDBACK_WEBHOOK_URL`. That independent receiver takes precedence for
`EXECUTOR_FEEDBACK_WEBHOOK_URL` or `EXECUTOR_FEEDBACK_GITHUB_REPOSITORY`. That independent receiver takes precedence for
feedback and receives the submitted message, sender identity and instance origin;
the anonymous analytics guarantees below do not apply to that explicit delivery.

Expand Down
29 changes: 27 additions & 2 deletions apps/hosted/self-host/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,13 +318,38 @@ the receiver URL or response. Delivery is not retried automatically; after an
ambiguous timeout a caller retry may duplicate feedback. `Idempotency-Key` is a
unique per-delivery ID, not a deduplication key across separate caller submissions.

The webhook takes precedence over feedback analytics and works even with
An operator receiver takes precedence over feedback analytics and works even with
`DO_NOT_TRACK=1`, `EXECUTOR_DISABLE_ANALYTICS=1`, or no analytics destination.
Other usage analytics are unchanged. With the URL unset or empty, existing
Other usage analytics are unchanged. With the URL and the GitHub repository
below unset or empty, existing
analytics feedback behavior is unchanged, including `FeedbackDisabled` when
analytics are unavailable. Submissions are not stored in the product database:
there is no feedback table, so durable retention belongs to the receiver for now.

#### GitHub issues

Set `EXECUTOR_FEEDBACK_GITHUB_REPOSITORY` to `owner/repository` and
`EXECUTOR_FEEDBACK_GITHUB_TOKEN` to a token that may create issues there (a
fine-grained token with read and write access to Issues on that repository alone)
to file each submission as an issue. Its title is the message's first non-blank
line without leading Markdown heading, quote or list markers, up to 120
characters. Its body is the whole message, which GitHub renders as Markdown,
then the sender's email (their user ID when they have none) and client, the
source, organization ID, instance origin and time. Issues identify the sender, so use a private repository.
`EXECUTOR_FEEDBACK_GITHUB_LABELS` is an optional comma-separated list of labels.
`EXECUTOR_FEEDBACK_GITHUB_API_URL` is the API root for GitHub Enterprise Server
(`https://github.example.com/api/v3`); the default is `https://api.github.com`.
A repository with a missing token, or a malformed repository or API URL, is
logged at start and every submission returns `FeedbackUnavailable`.

Issue filing has the same 10-second limit, redirect refusal and precedence over
analytics as the webhook, and does not need one. With both set, the issue is
filed first and the webhook payload gains `issue: { url, number }` and a final
`Issue: <url>` line in `body` (`Issue: not created` when GitHub refused it).
Feedback is then `accepted` when either receiver took it, so a failed
notification does not make a caller file the issue twice; the failed receiver is
logged. `FeedbackUnavailable` means neither did.

### Trace sampling

Recording every span made the telemetry collector the busiest process under load,
Expand Down
5 changes: 5 additions & 0 deletions apps/hosted/self-host/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ services:
# Optional operator-owned feedback receiver, independent of anonymous analytics.
EXECUTOR_FEEDBACK_WEBHOOK_URL:
EXECUTOR_FEEDBACK_WEBHOOK_SECRET:
# Optional: file each feedback submission as an issue in owner/repository.
EXECUTOR_FEEDBACK_GITHUB_REPOSITORY:
EXECUTOR_FEEDBACK_GITHUB_TOKEN:
EXECUTOR_FEEDBACK_GITHUB_LABELS:
EXECUTOR_FEEDBACK_GITHUB_API_URL:
# Optional override of X-Forwarded-For for client-IP assertions from trusted peers.
# Requires EXECUTOR_TRUSTED_PROXIES; see README "Rate limits behind a proxy".
EXECUTOR_TRUSTED_PROXY_HEADER:
Expand Down
117 changes: 117 additions & 0 deletions apps/hosted/self-host/src/implementation/feedback-github.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
/** Explicit feedback filed as issues in an operator-owned GitHub repository. */
import { FeedbackUnavailable } from "@executor-js/telemetry/product-analytics";
import { Config, Effect, Option, Redacted, Schema } from "effect";
import { HttpClient, HttpClientRequest } from "effect/http";

/** What an issue says about one submission; the operator's repository sees the sender. */
export interface FeedbackReport {
readonly message: string;
readonly submittedAt: string;
readonly who: string;
readonly source: string | undefined;
readonly organizationId: string;
readonly instanceOrigin: string;
}

const Created = Schema.Struct({ html_url: Schema.String, number: Schema.Int });
/** The issue GitHub created for a submission. */
export interface FeedbackIssue {
readonly url: string;
readonly number: number;
}

const Repository = /^[A-Za-z0-9][A-Za-z0-9-]*\/(?!\.{1,2}$)[\w.-]+$/u;
const titleLimit = 120;

/** The first line that says something, without Markdown heading or list markers. */
const issueTitle = (message: string) => {
const line =
message
.split("\n")
.map((text) => text.replace(/^[\s#>*-]+/u, "").trim())
.find((text) => text !== "") ?? "Executor feedback";
// By code point: a cut surrogate pair is not valid text for GitHub.
const characters = Array.from(line);
return characters.length > titleLimit ? `${characters.slice(0, titleLimit - 1).join("")}…` : line;
};

const issueBody = (report: FeedbackReport) =>
[
report.message,
"",
"---",
"",
"Submitted through Executor feedback.",
"",
`- From: ${report.who}`,
...(report.source === undefined ? [] : [`- Source: ${report.source}`]),
`- Organization: ${report.organizationId}`,
`- Instance: ${report.instanceOrigin}`,
`- Submitted: ${report.submittedAt}`,
].join("\n");

const optionalText = (name: string) =>
Config.String(name).pipe(
Config.option,
Effect.map(Option.map((text) => text.trim())),
Effect.map(Option.filter((text) => text !== "")),
);

/**
* A repository leaves issue filing off when unset. The returned effect needs an HTTP client and
* fails for any refusal; the caller bounds, reports and hides it.
*/
export const selfHostFeedbackIssues = Effect.gen(function* () {
const repository = yield* optionalText("EXECUTOR_FEEDBACK_GITHUB_REPOSITORY");
if (Option.isNone(repository)) return undefined;
const token = yield* Config.Redacted("EXECUTOR_FEEDBACK_GITHUB_TOKEN").pipe(Config.option);
const api = URL.parse(
Option.getOrElse(
yield* optionalText("EXECUTOR_FEEDBACK_GITHUB_API_URL"),
() => "https://api.github.com",
),
);
const labels = Option.getOrElse(yield* optionalText("EXECUTOR_FEEDBACK_GITHUB_LABELS"), () => "")
.split(",")
.map((label) => label.trim())
.filter((label) => label !== "");
if (
!Repository.test(repository.value) ||
Option.isNone(token) ||
Redacted.value(token.value).trim() === "" ||
api === null ||
!["https:", "http:"].includes(api.protocol) ||
api.username !== "" ||
api.password !== "" ||
api.search !== "" ||
api.hash !== ""
) {
yield* Effect.logWarning(
"EXECUTOR_FEEDBACK_GITHUB_REPOSITORY must be owner/repository with EXECUTOR_FEEDBACK_GITHUB_TOKEN set; EXECUTOR_FEEDBACK_GITHUB_API_URL must be an HTTP(S) URL without credentials or a query",
);
return () => Effect.fail(new FeedbackUnavailable());
}
const destination = `${api.href.replace(/\/+$/u, "")}/repos/${repository.value}/issues`;

return (report: FeedbackReport) =>
Effect.gen(function* () {
const response = yield* HttpClient.execute(
HttpClientRequest.post(destination).pipe(
HttpClientRequest.bodyJsonUnsafe({
title: issueTitle(report.message),
body: issueBody(report),
...(labels.length === 0 ? {} : { labels }),
}),
HttpClientRequest.setHeaders({
accept: "application/vnd.github+json",
"x-github-api-version": "2022-11-28",
"user-agent": "executor-feedback",
}),
HttpClientRequest.bearerToken(token.value),
),
);
if (response.status < 200 || response.status >= 300) return yield* new FeedbackUnavailable();
const created = yield* Schema.decodeUnknownEffect(Created)(yield* response.json);
return { url: created.html_url, number: created.number } satisfies FeedbackIssue;
});
});
148 changes: 99 additions & 49 deletions apps/hosted/self-host/src/implementation/feedback-webhook.ts
Original file line number Diff line number Diff line change
@@ -1,31 +1,68 @@
/** Explicit feedback delivery to an operator-owned receiver, independent of usage analytics. */
/** Explicit feedback delivery to operator-owned receivers, independent of usage analytics. */
import { CurrentUsage, ProductAnalytics } from "@executor-js/hosted-server";
import { FeedbackUnavailable } from "@executor-js/telemetry/product-analytics";
import { Config, DateTime, Effect, Option, Redacted, Schema } from "effect";
import { FetchHttpClient, HttpClient, HttpClientRequest } from "effect/http";
import { SqlClient } from "effect/sql";
import { type FeedbackIssue, selfHostFeedbackIssues } from "./feedback-github.ts";

const Person = Schema.Struct({ email: Schema.String });

/** An unset URL leaves the existing analytics sink (including its opt-out) untouched. */
/** One receiver's request: bounded, never traced, and reported without its URL or response. */
const delivered = <A, E>(receiver: string, request: Effect.Effect<A, E, HttpClient.HttpClient>) =>
request.pipe(
Effect.timeout("10 seconds"),
Effect.tapError(() => Effect.logWarning(`Feedback ${receiver} delivery failed`)),
// Do not trace the URL, authorization or response: all may contain receiver secrets.
Effect.provideService(HttpClient.TracerDisabledWhen, () => true),
Effect.provideService(HttpClient.TracerPropagationEnabled, false),
Effect.provideService(FetchHttpClient.RequestInit, { redirect: "manual" }),
Effect.provide(FetchHttpClient.layer),
Effect.option,
);

/** The operator's HTTP receiver; an unset URL leaves it off. */
const webhookReceiver = Effect.gen(function* () {
const configured = yield* Config.Redacted("EXECUTOR_FEEDBACK_WEBHOOK_URL").pipe(Config.option);
if (Option.isNone(configured) || Redacted.value(configured.value).trim() === "") return undefined;
const destination = URL.parse(Redacted.value(configured.value));
if (
destination === null ||
!["https:", "http:"].includes(destination.protocol) ||
destination.username !== "" ||
destination.password !== ""
) {
yield* Effect.logWarning(
"EXECUTOR_FEEDBACK_WEBHOOK_URL must be an HTTP(S) URL without embedded credentials",
);
return () => Effect.fail(new FeedbackUnavailable());
}
const secret = yield* Config.Redacted("EXECUTOR_FEEDBACK_WEBHOOK_SECRET").pipe(Config.option);
return (payload: unknown) =>
Effect.gen(function* () {
const request = HttpClientRequest.post(destination.href).pipe(
HttpClientRequest.bodyJsonUnsafe(payload),
HttpClientRequest.setHeader("Idempotency-Key", crypto.randomUUID()),
);
const response = yield* HttpClient.execute(
Option.isSome(secret) && Redacted.value(secret.value) !== ""
? HttpClientRequest.bearerToken(request, secret.value)
: request,
);
if (response.status < 200 || response.status >= 300) return yield* new FeedbackUnavailable();
});
});

/**
* With neither a GitHub repository nor a webhook URL set, the existing analytics sink (including
* its opt-out) is untouched. With both, the issue is filed first so the webhook can link it, and
* feedback is accepted when either receiver took it.
*/
export const selfHostFeedbackWebhook = (origin: string) =>
Effect.gen(function* () {
const configured = yield* Config.Redacted("EXECUTOR_FEEDBACK_WEBHOOK_URL").pipe(Config.option);
if (Option.isNone(configured) || Redacted.value(configured.value).trim() === "")
return undefined;
const destination = URL.parse(Redacted.value(configured.value));
if (
destination === null ||
!["https:", "http:"].includes(destination.protocol) ||
destination.username !== "" ||
destination.password !== ""
) {
yield* Effect.logWarning(
"EXECUTOR_FEEDBACK_WEBHOOK_URL must be an HTTP(S) URL without embedded credentials",
);
return () => Effect.fail(new FeedbackUnavailable());
}
const secret = yield* Config.Redacted("EXECUTOR_FEEDBACK_WEBHOOK_SECRET").pipe(Config.option);
const issues = yield* selfHostFeedbackIssues;
const webhook = yield* webhookReceiver;
if (issues === undefined && webhook === undefined) return undefined;
const sql = yield* SqlClient.SqlClient;

const submit: typeof ProductAnalytics.Service.submitFeedback = (feedback) =>
Expand All @@ -38,38 +75,51 @@ export const selfHostFeedbackWebhook = (origin: string) =>
...(Option.isSome(person) ? { email: person.value.email } : {}),
};
const submittedAt = DateTime.formatIso(yield* DateTime.now);
const who = [user.email ?? user.id, caller.client_name].filter(Boolean).join(" / ");
const payload = {
message: feedback.message,
submittedAt,
organization: { id: feedback.organizationId },
user,
caller,
instanceOrigin: origin,
// Common notification receivers can consume these without a translation service.
title: "Executor feedback",
body: `${feedback.message.slice(0, 1500)}\nFrom: ${who.slice(0, 300)}`,
};
const request = HttpClientRequest.post(destination.href).pipe(
HttpClientRequest.bodyJsonUnsafe(payload),
HttpClientRequest.setHeader("Idempotency-Key", crypto.randomUUID()),
const who = [user.email ?? user.id, caller.client_name]
.filter(Boolean)
.join(" / ")
// One line: a client names itself, and must not add lines to an issue's sender list.
.replace(/\s+/gu, " ")
.slice(0, 300);
const issue =
issues === undefined
? Option.none<FeedbackIssue>()
: yield* delivered(
"GitHub issue",
issues({
message: feedback.message,
submittedAt,
who,
source: caller.source,
organizationId: feedback.organizationId,
instanceOrigin: origin,
}),
);
if (webhook === undefined) {
if (Option.isNone(issue)) return yield* new FeedbackUnavailable();
return;
}
const filed =
issues === undefined
? ""
: `\nIssue: ${Option.match(issue, { onNone: () => "not created", onSome: (created) => created.url })}`;
const notified = yield* delivered(
"webhook",
webhook({
message: feedback.message,
submittedAt,
organization: { id: feedback.organizationId },
user,
caller,
instanceOrigin: origin,
...(Option.isSome(issue) ? { issue: issue.value } : {}),
// Common notification receivers can consume these without a translation service.
title: "Executor feedback",
body: `${feedback.message.slice(0, 1500)}\nFrom: ${who}${filed}`,
}),
);
const response = yield* HttpClient.execute(
Option.isSome(secret) && Redacted.value(secret.value) !== ""
? HttpClientRequest.bearerToken(request, secret.value)
: request,
);
if (response.status < 200 || response.status >= 300)
if (Option.isNone(issue) && Option.isNone(notified))
return yield* new FeedbackUnavailable();
}).pipe(
Effect.timeout("10 seconds"),
Effect.tapError(() => Effect.logWarning("Feedback webhook delivery failed")),
Effect.mapError(() => new FeedbackUnavailable()),
// Do not trace the URL, authorization or response: all may contain receiver secrets.
Effect.provideService(HttpClient.TracerDisabledWhen, () => true),
Effect.provideService(HttpClient.TracerPropagationEnabled, false),
Effect.provideService(FetchHttpClient.RequestInit, { redirect: "manual" }),
Effect.provide(FetchHttpClient.layer),
);
}).pipe(Effect.mapError(() => new FeedbackUnavailable()));
return submit;
});
4 changes: 4 additions & 0 deletions e2e/support/managed-server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ const maxClockOffset = 40 * 86_400_000;
export const OperatorSettings = Schema.Struct({
EXECUTOR_FEEDBACK_WEBHOOK_URL: Schema.optionalKey(Schema.String),
EXECUTOR_FEEDBACK_WEBHOOK_SECRET: Schema.optionalKey(Schema.String),
EXECUTOR_FEEDBACK_GITHUB_REPOSITORY: Schema.optionalKey(Schema.String),
EXECUTOR_FEEDBACK_GITHUB_TOKEN: Schema.optionalKey(Schema.String),
EXECUTOR_FEEDBACK_GITHUB_API_URL: Schema.optionalKey(Schema.String),
EXECUTOR_FEEDBACK_GITHUB_LABELS: Schema.optionalKey(Schema.String),
EXECUTOR_URL_ALLOW_LOOPBACK_HTTP: Schema.optionalKey(Schema.String),
EXECUTOR_URL_ALLOW_HTTP_ORIGINS: Schema.optionalKey(Schema.String),
NODE_EXTRA_CA_CERTS: Schema.optionalKey(Schema.NonEmptyString),
Expand Down
10 changes: 10 additions & 0 deletions e2e/test-plan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3676,6 +3676,16 @@ export const scenarios = plan({
local: na("Operator feedback webhooks are self-host only."),
},
},
feedbackGithubIssues: {
fixtures: "actors",
file: "feedback-github.spec.ts",
title: "self-host feedback files a GitHub issue and links it in the operator webhook",
targets: {
"self-host": scheduled,
cloud: na("Operator feedback receivers are self-host only."),
local: na("Operator feedback receivers are self-host only."),
},
},
selfHostAnalytics: {
fixtures: "actors",
file: "instance-analytics.spec.ts",
Expand Down
Loading
Loading