Skip to content

test(mockapi): validate requests and responses against the OpenAPI spec - #263

Merged
pjcdawkins merged 8 commits into
mainfrom
cli-225-validate-mock-api-requests-and-responses-against-the-openapi
Oct 22, 2026
Merged

pjcdawkins merged 8 commits into
mainfrom
cli-225-validate-mock-api-requests-and-responses-against-the-openapi

Conversation

@pjcdawkins

@pjcdawkins pjcdawkins commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The mock API (pkg/mockapi) now validates requests and responses against the API's OpenAPI spec, so mocks stay close to the real API and the CLI's requests are checked.

How it works

  • pkg/mockapi/openapi.json is a copy of https://meta.upsun.com/openapi-spec (1.6 MB, indented so refreshes diff well). Refresh it with make update-openapi-spec. Tests don't use the network.
  • A middleware on the mock handler validates each exchange with kin-openapi, including per-test handlers, and reports failures with t.Errorf.
  • Only successful (2xx) exchanges are checked. The spec documents few errors, and some success responses are only under default.
  • Requests are validated strictly. Responses are validated with the spec's required lists removed, so fixtures only need the properties a test uses. Types, enums, nullability and unknown properties are still checked.
  • Options: MultiError, IncludeResponseStatus (otherwise undocumented statuses skip validation), SkipSettingDefaults (don't modify requests), a no-op AuthenticationFunc, and DisableSchemaErrorDetails (errors would include whole schemas).
  • notInSpec lists routes that the spec lacks: billing profiles, /subscriptions/{id}, setup/options, organization trial and usage, PATCH /projects/{id}, activity logs, POST .../backups, PATCH .../deployments/next, environment settings, integration validate, observability/ (with a trailing slash) and observability/logs/query, plus test-only routes.

Spec issues

patchSpec patches these, where the API (or the API client the CLI uses) differs from the spec:

  • _links is missing from resources, and _embedded (activities, entity) from AcceptedResponse.
  • vendor is missing from Project and ProjectReference.
  • original_url is missing from routes. The route and integration oneOf variants have no type enum (or discriminator), so a value can match several variants.
  • Domains: ssl is missing (it is returned, and writing it adds a certificate), DomainPatch matches both variants, and ReplacementDomainStorageCreateInput doesn't require replacement_for.
  • Integration PATCH schemas require type and secrets such as token, but the API accepts partial updates.
  • EnvironmentPatch.http_access.basic_auth isn't nullable, but the update is a JSON merge patch, so null removes the credentials (used by httpaccess --auth 0).
  • AutoscalerSettings nests services and triggers one level too deep.
  • Deployment.tasks only has the sizing, but the API returns the task's config. App stack values are returned as configured, in several shapes.

CLI changes

Mock fixes

Validation found: missing variable fields, accepted responses without status/code, null instead of empty lists or objects, unsigned ref:* links, container profile CPU and memory as strings, app fields on services, autoscaling at /autoscaling instead of /autoscaling/settings, runtime operations off the deployment, http_access not applied as a merge patch, 200 instead of 201 on creation, and a GET environment handler that wrote a 404 after the body.

🤖 Generated with Claude Code

pjcdawkins and others added 5 commits October 22, 2026 09:50
The API treats a null options_url as if it were omitted, and its OpenAPI
spec does not allow null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mock API now checks each successful exchange against the API's
OpenAPI spec (pkg/mockapi/openapi.json, refreshed with
`make update-openapi-spec`), in a middleware that also covers per-test
handlers. Failures are reported with t.Errorf.

- Requests are validated strictly. Responses may leave out required
  properties, so fixtures stay small.
- Routes that are not in the spec are listed in notInSpec.
- patchSpec fixes differences between the spec and the API that the
  CLI depends on: HAL _links and _embedded, vendor, route types and
  original_url, integration types, domain ssl, partial PATCH bodies,
  null basic_auth (merge patch), optional activate/merge bodies, the
  autoscaling settings nesting, deployment tasks and stacks.

Mock fixes found by validation: required variable fields, accepted
responses with status and code, empty lists and objects instead of
null, signed ref links, numeric container profiles, services without
app fields, autoscaling under /autoscaling/settings, runtime operations
under the deployment, http_access as a merge patch, and a GET
environment handler that wrote a 404 after the body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cepted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Otherwise kin-openapi skips the response. Creating an organization or a
subscription now returns 201, as in the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-api-requests-and-responses-against-the-openapi

# Conflicts:
#	CLAUDE.md

@upsun-dispatch upsun-dispatch Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Changes suggested — 🟡 1 warning · 🔵 2 minor points

🔍 Full review · 46 files reviewed

Verification
  • The response router runs removeRequired on a separately loaded document, so request validation still sees the spec's required lists, including the added replacement_for.
  • Activity.MarshalJSON's outer Result *string field is shallower than the embedded field, so it overrides the embedded result and outputs null while an activity is incomplete.
  • handleGetEnvironment now returns after writing the body, so it no longer writes a 404 after a 200 body.
  • Removing options_url => null is compatible with the pinned client: SubscriptionOptions::fromArray just stores the array, and the phpstan baseline message was updated to match.
  • The validation middleware is registered after the auth middleware, so 401 responses skip validation, and it is registered before any route, as chi requires.

The new pkg/mockapi/openapi_test.go (TestValidateOpenAPI) covers valid and invalid requests and responses, unknown properties, undocumented statuses, skipped error responses, notInSpec and unknown routes. CI runs it with make test-go, and the ci.yml integration step (cd integration-tests && go test) runs the updated integration tests against the validating mock. No test covers encoded path segments in isInSpec, and unlike the schema and registry there is no CI staleness check for openapi.json.

Review details
  • Commit: 917cfe2
  • Model: claude-opus-5-5

Review 1 of 10 for this pull request · View the full run

Comment thread pkg/mockapi/openapi.go Resolved
Comment thread pkg/mockapi/openapi.go Resolved
Comment thread pkg/mockapi/environments.go Resolved

@upsun-dispatch upsun-dispatch Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new issues found · 3 still open

🔍 Full review · 46 files reviewed

Outstanding from earlier reviews:

  • 🟡 #4346432721 — pkg/mockapi/openapi.go:56: A test of a not-in-spec route on a branch with a slash in its name fails with a misleading validation error. — isInSpec still matches against req.URL.Path, the decoded path, while the gorillamux router matches the encoded path. A notInSpec route on an environment ID containing %2F is still validated and fails.
  • 🔵 #4346432737 — pkg/mockapi/openapi.go:260: A rare body-read failure shows up as a confusing empty success rather than a clear error. — When the request body cannot be read, the middleware still calls t.Errorf and returns without writing a status, so the client gets an empty 200.
  • 🔵 #4346432750 — pkg/mockapi/environments.go:73: The mock does not follow the merge-patch behaviour it claims to copy for an explicit null, so a test relying on that would pass or fail for… — handlePatchEnvironment still applies the merge only when patch.HTTPAccess != nil, so "http_access": null leaves the stored value unchanged.
Verification
  • responseRecorder keeps the first status, and counts a Write before any WriteHeader as 200. So a handler that writes its body and then calls WriteHeader(404) is still validated and sent as a 200.
  • Validation runs against the buffered reqBody through req.Clone with a fresh reader, so a handler that has already read req.Body does not leave the validator an empty body.
  • Activity.MarshalJSON wraps the converted activity in an outer struct whose *string result field is one level shallower, so encoding/json uses it and writes null for an activity with no result.
  • handlePatchEnvironment merges with mergePatch, which clones the nested maps, so the stored environment's http_access is not changed in place by a PATCH.
  • No mockapi handler streams or asserts http.Flusher (only subscriptions.go sleeps, and it does not flush), so buffering responses in the middleware does not change what any handler sends.

The new pkg/mockapi/openapi_test.go (TestValidateOpenAPI) covers valid and invalid requests and responses, undocumented statuses, notInSpec routes and unknown routes. It runs under make test-go. The integration tests in integration-tests/ exercise the middleware against the real mock routes. No test covers environment IDs with an encoded slash, a failed body read, or an explicit http_access: null.

Review details
  • Commit: 3dfc7e2
  • Model: claude-opus-5-5

Review 2 of 10 for this pull request · View the full run

pjcdawkins and others added 2 commits October 22, 2026 10:07
- Match notInSpec against the escaped path, as the router does, so
  IDs with an encoded slash still match "*".
- Respond with 400 when the request body can't be read.
- Reject "http_access": null in an environment PATCH, as the API has no
  default for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Update the API client, which now sends {} instead of no body for POST,
PUT and PATCH operations without parameters, such as activate and
merge. The OpenAPI spec requires a body for these, so the mock API no
longer needs to relax it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@upsun-dispatch

upsun-dispatch Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

📋 PR Summary

The PR makes the mock API (pkg/mockapi) check each successful (2xx) request and response against a local copy of the API's OpenAPI spec using kin-openapi, and reports any mismatch as a test failure. Routes missing from the spec are listed in notInSpec, and places where the spec differs from the real API are fixed up in patchSpec. The PR also corrects the mock fixtures and handlers that the validation flagged, drops options_url: null from project:create, and updates the PHP API client. The latest push adds a make check-openapi-spec target and a CI step that runs it to catch an outdated spec copy.

Changes
Layer / File(s) Summary
Spec tooling and CI
Makefile Adds update-openapi-spec, plus a check-openapi-spec target that refreshes the spec and fails if openapi.json changed, printing only a diff summary.
.github/workflows/ci.yml Adds a step that runs make check-openapi-spec.
pkg/mockapi/gen.go Generator that downloads the spec and saves it as indented JSON.
pkg/mockapi/openapi.json Local copy of the upstream OpenAPI spec.
Validation
pkg/mockapi/openapi.go Middleware that validates 2xx exchanges, plus the notInSpec route list and the patchSpec fixes.
pkg/mockapi/openapi_test.go Tests for the validation middleware.
Documentation
CLAUDE.md Describes the spec validation and the CI check for an up-to-date spec.
Mock fixes
pkg/mockapi/model.go Model fixes needed to match the spec: missing fields, correct types, and empty lists instead of null.
pkg/mockapi/environments.go Applies http_access as a merge patch and fixes the GET environment handler writing a 404 after the body.
pkg/mockapi/variables.go Adds missing variable fields and returns 201 on creation.
CLI
legacy/src/Command/Project/ProjectCreateCommand.php No longer sends options_url: null.
legacy/composer.lock Updates the API client so it sends {} for operations without parameters.

Add `make check-openapi-spec`, run in the non-blocking lint-upstream
job with the registry and schema checks, so upstream spec changes are
noticed without failing unrelated pull requests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pjcdawkins
pjcdawkins merged commit b6ddf7c into main Oct 22, 2026
7 checks passed
@pjcdawkins
pjcdawkins deleted the cli-225-validate-mock-api-requests-and-responses-against-the-openapi branch October 22, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant