Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ curl -s -X POST "$EMULATOR_URL/_emulate/faults" \
-d '{"match":{"method":"GET","pathPattern":"/v1/*"},"response":{"status":503,"body":{"error":"temporary"}}}'
```

An arm may carry an `idempotencyKey` (1 to 255 characters, chosen by the caller). Another arm with the same key and the same fault stores nothing and returns the fault first stored for that key, with `replayed: true`, so a caller that lost an answer can send the arm again safely. The instance remembers each key for as long as its state lives, through consumption, removal by id, clearing and reset, so a late copy of an arm whose fault is already gone returns that fault with `remaining: 0` instead of arming it again. The same key with a different fault is refused with `409` and `idempotency_key_conflict`. Arms without a key are stored as before. `GET /_emulate/faults` answers `idempotencyKeys: true` so a caller can tell an emulator that honors keys.

Credential creation follows each service's real shape. For example, GitHub can mint a bearer token for a user, Spotify creates a client credentials app, Google/Microsoft/Apple/Okta/Clerk create OAuth/OIDC clients, Stripe and Resend create API-key style credentials, and AWS advertises provider-specific SDK credentials instead of pretending to be OAuth.

### MCP OAuth compliance scenarios
Expand Down
2 changes: 2 additions & 0 deletions apps/web/app/docs/ledger/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ Set `X-Correlation-Id` (or `X-Request-Id`) on a request and the emulator stores

`POST /_emulate/faults` arms a one-shot or counted response for matching provider requests. Match criteria are combined: `operationId` when the emulator can resolve one from the service manifest, `method`, and `pathPattern`. `pathPattern` is a glob where `*` matches any characters in the request path. A matched request short-circuits with the configured response, decrements the fault's remaining count, and is recorded here with `faulted: true` and `faultId`.

An arm may carry an `idempotencyKey` (1 to 255 characters, chosen by the caller). Another arm with the same key and the same fault stores nothing and returns the fault first stored for that key, with `replayed: true`, so a caller that lost an answer can send the arm again safely. The instance remembers each key for as long as its state lives, through consumption, removal by id, clearing and reset, so a late copy of an arm whose fault is already gone returns that fault with `remaining: 0` instead of arming it again. The same key with a different fault is refused with `409` and `idempotency_key_conflict`. Arms without a key are stored as before. `GET /_emulate/faults` answers `idempotencyKeys: true` so a caller can tell an emulator that honors keys.

## Persistence

On the hosted Cloudflare surface the ledger is persistent. Each instance is backed by a Durable Object that snapshots state and the ledger to storage, so records survive eviction. Locally and in process the ledger is in memory and is cleared on reset. See [Deployment](/docs/deployment) for the instance model.
Expand Down
48 changes: 48 additions & 0 deletions packages/@emulators/cloudflare/src/__tests__/worker.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -650,6 +650,54 @@ describe("cloudflare durable object control plane", () => {
expect((await pending).status).toBe(503);
});

// A caller that lost an arm's answer sends it again with the same key. The original copy can
// reach the instance at any later moment, after the instance was replaced, the fault consumed
// and the instance reset; it must never arm a second fault.
it("keeps fault idempotency keys across eviction, consumption and reset", async () => {
const { state } = makeState();
const url = "https://github.my-run.emulators.dev";
const arm = async (durableObject: EmulatorDurableObject) => {
const res = await durableObject.fetch(
new Request(`${url}/_emulate/faults`, {
method: "POST",
headers: idHeaders({ "content-type": "application/json" }),
body: JSON.stringify({
match: { method: "GET", pathPattern: "/repos/*" },
response: { status: 503, body: { message: "synthetic outage" } },
idempotencyKey: "arm-7f3a",
}),
}),
);
expect(res.status).toBe(200);
return (await res.json()) as { fault: { id: string; remaining: number }; replayed: boolean };
};
const listed = async (durableObject: EmulatorDurableObject) => {
const res = await durableObject.fetch(new Request(`${url}/_emulate/faults`, { headers: idHeaders() }));
return ((await res.json()) as { faults: Array<{ id: string }> }).faults.map((fault) => fault.id);
};
const repo = (durableObject: EmulatorDurableObject) =>
durableObject.fetch(new Request(`${url}/repos/acme/widget`, { headers: idHeaders() }));

const retry = await arm(new EmulatorDurableObject(state, {}));
expect(retry.replayed).toBe(false);
const evicted = new EmulatorDurableObject(state, {});
const original = await arm(evicted);
expect(original).toMatchObject({ replayed: true, fault: { id: retry.fault.id, remaining: 1 } });
expect(await listed(evicted)).toEqual([retry.fault.id]);
expect((await repo(evicted)).status).toBe(503);

const afterConsumed = new EmulatorDurableObject(state, {});
expect(await arm(afterConsumed)).toMatchObject({ replayed: true, fault: { id: retry.fault.id, remaining: 0 } });
const reset = await afterConsumed.fetch(
new Request(`${url}/_emulate/reset`, { method: "POST", headers: idHeaders() }),
);
expect(reset.status).toBe(200);
const afterReset = new EmulatorDurableObject(state, {});
expect((await arm(afterReset)).replayed).toBe(true);
expect(await listed(afterReset)).toEqual([]);
expect((await repo(afterReset)).status).not.toBe(503);
});

it("reports the real instance id in the manifest", async () => {
const { state } = makeState();
const durableObject = new EmulatorDurableObject(state, {});
Expand Down
156 changes: 156 additions & 0 deletions packages/@emulators/core/src/__tests__/fault-keys.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
import { describe, expect, it } from "vitest";
import { createServer } from "../server.js";
import { FaultRegistry } from "../faults.js";
import type { ServicePlugin } from "../plugin.js";

const plugin: ServicePlugin = {
name: "demo",
register(app) {
app.get("/things", (c) => c.json({ things: [] }));
},
};

const fault = {
match: { method: "get", pathPattern: "/things" },
response: { status: 503, body: { error: "planned" } },
};

type Armed = { id: string; remaining: number; idempotencyKey?: string };

function setup() {
const server = createServer(plugin, {
baseUrl: "https://demo.instance.emulators.dev",
reset() {
server.ledger.clear();
},
});
const { app } = server;
const arm = async (body: Record<string, unknown>) => {
const res = await app.request("/_emulate/faults", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
return { status: res.status, body: (await res.json()) as { fault: Armed; replayed?: boolean; error?: string } };
};
const list = async () =>
(await (await app.request("/_emulate/faults")).json()) as { faults: Armed[]; idempotencyKeys?: boolean };
const things = async () => (await app.request("/things")).status;
return { server, app, arm, list, things };
}

describe("fault idempotency keys", () => {
it("returns the fault first stored for a key to a second arm sent after it, and stores nothing", async () => {
const { arm, list, things } = setup();
const first = await arm({ ...fault, idempotencyKey: "arm-1" });
const second = await arm({ ...fault, idempotencyKey: "arm-1" });
expect(first.status).toBe(200);
expect(first.body.replayed).toBe(false);
expect(second.status).toBe(200);
expect(second.body.replayed).toBe(true);
expect(second.body.fault.id).toBe(first.body.fault.id);
expect(second.body.fault.idempotencyKey).toBe("arm-1");
expect((await list()).faults.map((f) => f.id)).toEqual([first.body.fault.id]);
expect(await things()).toBe(503);
expect(await things()).toBe(200);
});

it("stores one fault for two arms with one key sent at once", async () => {
const { arm, list } = setup();
const both = await Promise.all([
arm({ ...fault, idempotencyKey: "arm-1" }),
arm({ ...fault, idempotencyKey: "arm-1" }),
]);
expect(both.map((r) => r.status)).toEqual([200, 200]);
expect(new Set(both.map((r) => r.body.fault.id)).size).toBe(1);
expect(both.map((r) => r.body.replayed).sort()).toEqual([false, true]);
expect((await list()).faults).toHaveLength(1);
});

it("does not re-arm when a late copy arrives after its fault was consumed, removed, cleared or reset", async () => {
const { app, arm, list, things } = setup();
// Consumed.
const consumed = await arm({ ...fault, idempotencyKey: "consumed" });
expect(await things()).toBe(503);
const lateConsumed = await arm({ ...fault, idempotencyKey: "consumed" });
expect(lateConsumed.body.replayed).toBe(true);
expect(lateConsumed.body.fault).toMatchObject({ id: consumed.body.fault.id, remaining: 0 });
expect((await list()).faults).toEqual([]);
expect(await things()).toBe(200);

// Removed by id.
const removed = await arm({ ...fault, idempotencyKey: "removed" });
await app.request(`/_emulate/faults/${removed.body.fault.id}`, { method: "DELETE" });
const lateRemoved = await arm({ ...fault, idempotencyKey: "removed" });
expect(lateRemoved.body).toMatchObject({ replayed: true, fault: { id: removed.body.fault.id, remaining: 0 } });
expect((await list()).faults).toEqual([]);
expect(await things()).toBe(200);

// Every fault cleared, then the instance reset.
await arm({ ...fault, idempotencyKey: "cleared" });
await app.request("/_emulate/faults", { method: "DELETE" });
await arm({ ...fault, idempotencyKey: "reset" });
await app.request("/_emulate/reset", { method: "POST" });
expect((await arm({ ...fault, idempotencyKey: "cleared" })).body.replayed).toBe(true);
expect((await arm({ ...fault, idempotencyKey: "reset" })).body.replayed).toBe(true);
expect((await list()).faults).toEqual([]);
expect(await things()).toBe(200);
});

it("refuses a key used again for a different fault, and keeps the first", async () => {
const { arm, list } = setup();
const first = await arm({ ...fault, idempotencyKey: "arm-1" });
const other = await arm({ ...fault, times: 2, idempotencyKey: "arm-1" });
expect(other.status).toBe(409);
expect(other.body.error).toBe("idempotency_key_conflict");
const otherStatus = await arm({ ...fault, response: { status: 500 }, idempotencyKey: "arm-1" });
expect(otherStatus.status).toBe(409);
expect((await list()).faults.map((f) => f.id)).toEqual([first.body.fault.id]);
// The same fault written differently (method case, key order, `times` defaulted) is the same fault.
const same = await arm({
response: { body: { error: "planned" }, status: 503 },
match: { pathPattern: "/things", method: "GET" },
times: 1,
idempotencyKey: "arm-1",
});
expect(same.status).toBe(200);
expect(same.body.fault.id).toBe(first.body.fault.id);
expect((await arm({ ...fault, idempotencyKey: "" })).status).toBe(400);
});

it("stores every arm without a key as before, and says it honors keys", async () => {
const { arm, list } = setup();
const keyed = await arm({ ...fault, idempotencyKey: "arm-1" });
const a = await arm(fault);
const b = await arm(fault);
expect(a.body).not.toHaveProperty("replayed");
expect(a.body.fault).not.toHaveProperty("idempotencyKey");
expect(new Set([keyed.body.fault.id, a.body.fault.id, b.body.fault.id]).size).toBe(3);
const listed = await list();
expect(listed.faults).toHaveLength(3);
expect(listed.idempotencyKeys).toBe(true);
});

it("keeps the record of keys through a snapshot and restore, consumed faults included", () => {
const registry = new FaultRegistry();
const live = registry.arm({ ...fault, idempotencyKey: "live" });
const spent = registry.arm({ ...fault, idempotencyKey: "spent" });
registry.clear(spent.id);
const restored = new FaultRegistry();
restored.restore(JSON.parse(JSON.stringify(registry.serialize())));
expect(restored.armOnce({ ...fault, idempotencyKey: "live" })).toMatchObject({
replayed: true,
fault: { id: live.id, remaining: 1 },
});
expect(restored.armOnce({ ...fault, idempotencyKey: "spent" })).toMatchObject({
replayed: true,
fault: { id: spent.id, remaining: 0 },
});
expect(restored.list().map((f) => f.id)).toEqual([live.id]);
expect(() => restored.armOnce({ ...fault, times: 3, idempotencyKey: "spent" })).toThrow(/different fault/);
// A snapshot stored before keys existed restores with none.
const old = new FaultRegistry();
old.restore({ counter: 4, faults: [] });
expect(old.arm(fault).id).toBe("fault_4");
});
});
12 changes: 8 additions & 4 deletions packages/@emulators/core/src/control-plane.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import type { ConnectionVars, EmulatorInstanceInfo, ResolvedConnection, ServiceM
import { coverageReport, enrichManifest, resolveConnections } from "./manifest.js";
import type { TokenMap } from "./middleware/auth.js";
import { escapeHtml, renderCardPage } from "./ui.js";
import type { FaultArmInput, FaultRegistry } from "./faults.js";
import { FaultKeyConflictError, type FaultArmInput, type FaultRegistry } from "./faults.js";

export interface CredentialRequest {
type?: string;
Expand Down Expand Up @@ -160,13 +160,17 @@ export function registerControlPlane(app: Hono<AppEnv>, options: ControlPlaneOpt
ledger.clear();
return c.json({ ok: true });
});
app.get("/_emulate/faults", (c) => c.json({ faults: faults.list() }));
// `idempotencyKeys` tells a caller this emulator honors `idempotencyKey` on an arm.
app.get("/_emulate/faults", (c) => c.json({ faults: faults.list(), idempotencyKeys: true }));
app.post("/_emulate/faults", async (c) => {
const body = (await c.req.json().catch(() => undefined)) as FaultArmInput | undefined;
try {
const fault = faults.arm(body as FaultArmInput);
return c.json({ fault });
const { fault, replayed } = faults.armOnce(body as FaultArmInput);
return c.json(body?.idempotencyKey === undefined ? { fault } : { fault, replayed });
} catch (err) {
if (err instanceof FaultKeyConflictError) {
return c.json({ error: "idempotency_key_conflict", message: err.message }, 409);
}
return c.json({ error: "invalid_fault", message: err instanceof Error ? err.message : "Invalid fault." }, 400);
}
});
Expand Down
Loading
Loading