Repository navigation
test(mockapi): validate requests and responses against the OpenAPI spec - #263
Conversation
The API treats a null options_url as if it were omitted, and its OpenAPI spec does not allow null. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mock API now checks each successful exchange against the API's OpenAPI spec (pkg/mockapi/openapi.json, refreshed with `make update-openapi-spec`), in a middleware that also covers per-test handlers. Failures are reported with t.Errorf. - Requests are validated strictly. Responses may leave out required properties, so fixtures stay small. - Routes that are not in the spec are listed in notInSpec. - patchSpec fixes differences between the spec and the API that the CLI depends on: HAL _links and _embedded, vendor, route types and original_url, integration types, domain ssl, partial PATCH bodies, null basic_auth (merge patch), optional activate/merge bodies, the autoscaling settings nesting, deployment tasks and stacks. Mock fixes found by validation: required variable fields, accepted responses with status and code, empty lists and objects instead of null, signed ref links, numeric container profiles, services without app fields, autoscaling under /autoscaling/settings, runtime operations under the deployment, http_access as a merge patch, and a GET environment handler that wrote a 404 after the body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cepted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Otherwise kin-openapi skips the response. Creating an organization or a subscription now returns 201, as in the spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-api-requests-and-responses-against-the-openapi # Conflicts: # CLAUDE.md
There was a problem hiding this comment.
Warning
Changes suggested — 🟡 1 warning · 🔵 2 minor points
🔍 Full review · 46 files reviewed
Verification
- The response router runs
removeRequiredon a separately loaded document, so request validation still sees the spec'srequiredlists, including the addedreplacement_for. Activity.MarshalJSON's outerResult *stringfield is shallower than the embedded field, so it overrides the embeddedresultand outputs null while an activity is incomplete.handleGetEnvironmentnow returns after writing the body, so it no longer writes a 404 after a 200 body.- Removing
options_url => nullis compatible with the pinned client:SubscriptionOptions::fromArrayjust stores the array, and the phpstan baseline message was updated to match. - The validation middleware is registered after the auth middleware, so 401 responses skip validation, and it is registered before any route, as chi requires.
The new pkg/mockapi/openapi_test.go (TestValidateOpenAPI) covers valid and invalid requests and responses, unknown properties, undocumented statuses, skipped error responses, notInSpec and unknown routes. CI runs it with make test-go, and the ci.yml integration step (cd integration-tests && go test) runs the updated integration tests against the validating mock. No test covers encoded path segments in isInSpec, and unlike the schema and registry there is no CI staleness check for openapi.json.
Review details
- Commit: 917cfe2
- Model: claude-opus-5-5
Review 1 of 10 for this pull request · View the full run
There was a problem hiding this comment.
Note
Reviewed — No new issues found · 3 still open
🔍 Full review · 46 files reviewed
Outstanding from earlier reviews:
- 🟡 #4346432721 —
pkg/mockapi/openapi.go:56: A test of a not-in-spec route on a branch with a slash in its name fails with a misleading validation error. —isInSpecstill matches againstreq.URL.Path, the decoded path, while the gorillamux router matches the encoded path. AnotInSpecroute on an environment ID containing%2Fis still validated and fails. - 🔵 #4346432737 —
pkg/mockapi/openapi.go:260: A rare body-read failure shows up as a confusing empty success rather than a clear error. — When the request body cannot be read, the middleware still callst.Errorfand returns without writing a status, so the client gets an empty 200. - 🔵 #4346432750 —
pkg/mockapi/environments.go:73: The mock does not follow the merge-patch behaviour it claims to copy for an explicit null, so a test relying on that would pass or fail for… —handlePatchEnvironmentstill applies the merge only whenpatch.HTTPAccess != nil, so"http_access": nullleaves the stored value unchanged.
Verification
responseRecorderkeeps the first status, and counts aWritebefore anyWriteHeaderas 200. So a handler that writes its body and then callsWriteHeader(404)is still validated and sent as a 200.- Validation runs against the buffered
reqBodythroughreq.Clonewith a fresh reader, so a handler that has already readreq.Bodydoes not leave the validator an empty body. Activity.MarshalJSONwraps the convertedactivityin an outer struct whose*stringresultfield is one level shallower, so encoding/json uses it and writes null for an activity with no result.handlePatchEnvironmentmerges withmergePatch, which clones the nested maps, so the stored environment'shttp_accessis not changed in place by a PATCH.- No mockapi handler streams or asserts
http.Flusher(onlysubscriptions.gosleeps, and it does not flush), so buffering responses in the middleware does not change what any handler sends.
The new pkg/mockapi/openapi_test.go (TestValidateOpenAPI) covers valid and invalid requests and responses, undocumented statuses, notInSpec routes and unknown routes. It runs under make test-go. The integration tests in integration-tests/ exercise the middleware against the real mock routes. No test covers environment IDs with an encoded slash, a failed body read, or an explicit http_access: null.
Review details
- Commit: 3dfc7e2
- Model: claude-opus-5-5
Review 2 of 10 for this pull request · View the full run
- Match notInSpec against the escaped path, as the router does, so IDs with an encoded slash still match "*". - Respond with 400 when the request body can't be read. - Reject "http_access": null in an environment PATCH, as the API has no default for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Update the API client, which now sends {} instead of no body for POST,
PUT and PATCH operations without parameters, such as activate and
merge. The OpenAPI spec requires a body for these, so the mock API no
longer needs to relax it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
📋 PR Summary The PR makes the mock API ( Changes
|
Add `make check-openapi-spec`, run in the non-blocking lint-upstream job with the registry and schema checks, so upstream spec changes are noticed without failing unrelated pull requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mock API (
pkg/mockapi) now validates requests and responses against the API's OpenAPI spec, so mocks stay close to the real API and the CLI's requests are checked.How it works
pkg/mockapi/openapi.jsonis a copy of https://meta.upsun.com/openapi-spec (1.6 MB, indented so refreshes diff well). Refresh it withmake update-openapi-spec. Tests don't use the network.t.Errorf.default.requiredlists removed, so fixtures only need the properties a test uses. Types, enums, nullability and unknown properties are still checked.MultiError,IncludeResponseStatus(otherwise undocumented statuses skip validation),SkipSettingDefaults(don't modify requests), a no-opAuthenticationFunc, andDisableSchemaErrorDetails(errors would include whole schemas).notInSpeclists routes that the spec lacks: billing profiles,/subscriptions/{id},setup/options, organization trial and usage,PATCH /projects/{id}, activity logs,POST .../backups,PATCH .../deployments/next, environment settings, integrationvalidate,observability/(with a trailing slash) andobservability/logs/query, plus test-only routes.Spec issues
patchSpecpatches these, where the API (or the API client the CLI uses) differs from the spec:_linksis missing from resources, and_embedded(activities, entity) fromAcceptedResponse.vendoris missing fromProjectandProjectReference.original_urlis missing from routes. The route and integrationoneOfvariants have notypeenum (or discriminator), so a value can match several variants.sslis missing (it is returned, and writing it adds a certificate),DomainPatchmatches both variants, andReplacementDomainStorageCreateInputdoesn't requirereplacement_for.typeand secrets such astoken, but the API accepts partial updates.EnvironmentPatch.http_access.basic_authisn't nullable, but the update is a JSON merge patch, sonullremoves the credentials (used byhttpaccess --auth 0).AutoscalerSettingsnests services and triggers one level too deep.Deployment.tasksonly has the sizing, but the API returns the task's config. Appstackvalues are returned as configured, in several shapes.CLI changes
project:createno longer sendsoptions_url: null. The API treats null the same as omitting it.{}for operations without parameters (e.g.activate,merge), whose request body the spec requires (fix: send an empty JSON object for operations without a body platformsh/platformsh-client-php#109).Mock fixes
Validation found: missing variable fields, accepted responses without
status/code,nullinstead of empty lists or objects, unsignedref:*links, container profile CPU and memory as strings, app fields on services, autoscaling at/autoscalinginstead of/autoscaling/settings, runtime operations off the deployment,http_accessnot applied as a merge patch, 200 instead of 201 on creation, and a GET environment handler that wrote a 404 after the body.🤖 Generated with Claude Code