diff --git a/.github/workflows/pipeline-self-test.yml b/.github/workflows/pipeline-self-test.yml index a7b2218..796898a 100644 --- a/.github/workflows/pipeline-self-test.yml +++ b/.github/workflows/pipeline-self-test.yml @@ -10,6 +10,15 @@ on: - "ci/**" - "scripts/**" - "tests/**" + pull_request: + branches: [main] + paths: + - ".github/workflows/challenge-branch-validation.yml" + - ".github/workflows/challenge-supply-chain.yml" + - ".github/workflows/pipeline-self-test.yml" + - "ci/**" + - "scripts/**" + - "tests/**" workflow_dispatch: permissions: diff --git a/docs/aws-k3s-cd-smoke.md b/docs/aws-k3s-cd-smoke.md index c14663a..dec4d8e 100644 --- a/docs/aws-k3s-cd-smoke.md +++ b/docs/aws-k3s-cd-smoke.md @@ -41,7 +41,7 @@ caller는 다음 입력을 전달합니다. ```yaml with: enable_k3s_smoke_deploy: true - runtime_target_id: aws-k3s-001 + runtime_target_id: aws-k3s-lab secrets: inherit ``` @@ -72,19 +72,94 @@ S3 파일과 Runtime API body에는 GHCR credential을 포함하지 않습니다 4. `GET /internal/v1/operations/{operation_id}`를 `SUCCEEDED`까지 조회합니다. 5. 같은 instance를 `DELETE /internal/v1/instances/{instance_id}`로 정리합니다. 6. 삭제 Operation도 `SUCCEEDED`인지 확인합니다. -7. Actions Summary에 `target_id`, `runtime_workload_id`, 생성·삭제 결과를 남깁니다. +7. Actions Summary에 `target_id`, `runtime_workload_id`, digest 이미지 목록, + `endpoints`, 생성·삭제 결과와 단계별 소요 시간을 남깁니다. 8. 성공과 실패 모두 S3 staging 파일을 삭제합니다. Smoke instance와 team UUID는 GitHub run 정보로 결정적으로 생성됩니다. 같은 run을 재시도해도 Runtime의 `request_id` 멱등 계약을 사용할 수 있습니다. +정상 실행 결과에는 다음 증거가 포함됩니다. + +```json +{ + "registry_revision": 12, + "images": [ + { + "name": "web", + "image": "ghcr.io/msg-ctf/challenges/example/web@sha256:..." + } + ], + "endpoints": [ + { + "container_name": "web", + "port": 8080, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31042" + } + ], + "create_status": "SUCCEEDED", + "create_elapsed_seconds": 12.345, + "delete_status": "SUCCEEDED", + "delete_elapsed_seconds": 1.234 +} +``` + +Smoke runner는 `artifact-v2.json`의 `revision`과 `registry_revision`이 같은지, +`isolation_profile`이 문제 category와 일치하는지, `scan_result`가 `PASS`인지 먼저 +확인합니다. `containers[]`의 digest 이미지를 같은 publish bundle에서 읽어 Runtime +요청에 전달합니다. 동일 인스턴스 내부 통신은 Runtime `STANDARD@v2` 기본 정책이며 +신규 요청에 `internal_connections`를 전달하지 않습니다. + +bundle의 컨테이너 포트는 다음처럼 Runtime PR #39 계약으로 변환합니다. + +- bundle의 전체 `ports[].port`는 Runtime 요청의 `ports`가 됩니다. +- `ports[].public: true`인 포트만 Runtime 요청의 `exposed_ports`가 됩니다. +- 모두 private인 컨테이너는 `exposed_ports: []`를 보냅니다. +- PWN은 공개 컨테이너가 정확히 1개여야 하며, 해당 컨테이너의 전체 포트도 + 정확히 1개여야 합니다. +- Runtime 요청에는 기존 `expose`를 함께 보내지 않습니다. + +따라서 한 컨테이너에 `8080 public`, `9000 private`가 함께 있어도 8080만 외부 +endpoint로 요청할 수 있습니다. Runtime 응답의 `endpoints[]`는 요청한 모든 +`(container_name, port)`와 누락, 추가, 중복 없이 정확히 일치해야 합니다. endpoint +각 항목의 `container_name`, `port`, `protocol`, `service_url`도 모두 검증합니다. + +`create_elapsed_seconds`는 생성 요청을 처음 제출한 시점부터 생성 Operation이 +`SUCCEEDED`가 될 때까지의 시간입니다. 이미지 pull뿐 아니라 Pod와 Service 준비 +시간도 포함하므로 순수 네트워크 처리량으로 해석하지 않습니다. + +## GHCR pull 검증 + +Issue #30을 닫기 전 정상 cold pull, GHCR 인증 실패와 `ImagePullBackOff`를 각각 +재현합니다. 장애 주입은 승인된 publish bundle을 변조하지 않고 Secure Provisioner +팀이 제공하는 전용 test target에서 수행합니다. + +| 시나리오 | DevSecOps 증거 | Secure Provisioner 준비와 확인 | +|---|---|---| +| cold pull | 실제 digest와 `create_elapsed_seconds` 기록 | 실행 전 해당 test digest의 node cache 제거 | +| 인증 실패 | 정상 검사된 private GHCR digest 제공 | 잘못되거나 제거된 pull credential을 가진 전용 target에서 실패 재현 | +| `ImagePullBackOff` | 운영 bundle과 분리된 테스트임을 기록 | 존재하지 않는 digest로 Pod 상태와 Operation 오류를 확인 | + +cold pull 확인은 같은 digest가 없는 상태에서 한 번 실행하고, 같은 node에서 다시 +실행한 warm-cache 결과와 비교합니다. cache 제거, pull credential 변경과 Pod 상태 +수집은 Runtime 소유 작업이며 CI가 `crictl`, `containerd`, Kubernetes Secret 또는 +`kubectl`을 직접 조작하지 않습니다. + +GHCR 인증 실패와 `ImagePullBackOff`는 정상 문제 발행의 성공 조건이 아닙니다. +따라서 일반 `k3s-smoke-deploy` job에 가짜 digest나 credential을 넣지 않고 Runtime +통합 테스트에서 실패 Operation의 `last_error_code`, Pod waiting reason과 cleanup +결과를 증거로 남깁니다. credential, service token과 raw pull 오류는 Actions Summary, +S3 staging 파일 또는 API body에 기록하지 않습니다. + ## 현재 제약 - Runtime팀 API의 현재 격리 profile에 맞춰 `pwn`은 `PWN`, 나머지는 `WEB`을 사용합니다. - `info.yaml`에 `run_as_user`가 없으면 smoke 요청은 non-root UID `10001`을 사용합니다. -- publish bundle의 `ports[].public`을 Runtime 신규 계약의 전체 `ports`와 공개 포트 - 목록인 `exposed_ports`로 변환하며 `expose`는 함께 보내지 않습니다. -- PWN은 공개 컨테이너와 해당 컨테이너의 포트를 각각 정확히 하나만 허용합니다. +- 운영 smoke 대상은 Runtime팀에서 전달한 `aws-k3s-lab`을 사용합니다. +- `aws-k3s-lab` node에서 읽을 Runtime API service token이 아직 준비되지 않아 실제 + K3s 생성·endpoint 확인·삭제는 실행하지 않았습니다. 현재 완료 범위는 요청 변환과 + cleanup 동작의 단위 테스트입니다. - 운영 참가자 instance 생성은 Backend, Scheduler, Broker, Runtime 경로가 담당합니다. 이 job은 문제 revision 발행 직후의 임시 통합 검증입니다. @@ -95,3 +170,8 @@ Smoke instance와 team UUID는 GitHub run 정보로 결정적으로 생성됩니 - `PROVISIONER_CLUSTER_REGISTRY`의 target과 K3s kubeconfig가 유효해야 합니다. - K3s/containerd가 private GHCR digest를 pull할 수 있어야 합니다. - 실패 후 Runtime Operation과 Namespace가 남지 않았는지 확인해야 합니다. +- create operation 결과가 불완전하면 runner가 `runtime-status`에서 workload ID를 + cleanup 제한 시간까지 재시도해 복구하고 삭제합니다. status 조회에서도 ID를 얻지 + 못하면 Runtime팀이 해당 instance의 잔존 리소스를 확인해야 합니다. +- cold pull 검증은 다른 workload가 없는 전용 target에서 Runtime팀이 image cache + 상태를 확인한 뒤 실행해야 합니다. diff --git a/scripts/runtime_api_smoke_runner.py b/scripts/runtime_api_smoke_runner.py index 4092868..40519b7 100644 --- a/scripts/runtime_api_smoke_runner.py +++ b/scripts/runtime_api_smoke_runner.py @@ -10,14 +10,17 @@ import time import uuid from pathlib import Path -from typing import Any +from typing import Any, Callable from urllib.error import HTTPError, URLError +from urllib.parse import urlsplit from urllib.request import Request, urlopen DIGEST_IMAGE = re.compile(r"^.+@sha256:[0-9a-f]{64}$") SERVICE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{43,128}$") FINAL_OPERATION_STATES = {"SUCCEEDED", "FAILED"} +ENDPOINT_FIELDS = {"container_name", "port", "protocol", "service_url"} +ENDPOINT_PROTOCOLS = {"HTTP", "TCP"} class RetryableRuntimeError(RuntimeError): @@ -48,6 +51,22 @@ def _isolation_profile(category: Any) -> str: return "PWN" if category.lower() == "pwn" else "WEB" +def _validate_bundle_contract(artifact: dict[str, Any]) -> tuple[int, int, str]: + revision = _positive_int(artifact.get("revision"), "artifact revision") + registry_revision = _positive_int( + artifact.get("registry_revision"), + "artifact registry_revision", + ) + if registry_revision != revision: + raise ValueError("artifact registry_revision must equal revision") + expected_profile = _isolation_profile(artifact.get("category")) + if artifact.get("isolation_profile") != expected_profile: + raise ValueError("artifact isolation_profile must match the challenge category") + if artifact.get("scan_result") != "PASS": + raise ValueError("artifact scan_result must be PASS") + return revision, registry_revision, expected_profile + + def build_create_request( artifact: dict[str, Any], *, @@ -56,6 +75,9 @@ def build_create_request( team_id: str, ) -> dict[str, Any]: """Translate a publish artifact into the Runtime team's create contract.""" + _revision, _registry_revision, isolation_profile = _validate_bundle_contract( + artifact + ) if artifact.get("runtime_type") != "KUBERNETES": raise ValueError("Runtime smoke deployment requires runtime_type KUBERNETES") if artifact.get("architecture") != "AMD64": @@ -72,6 +94,7 @@ def build_create_request( runtime_containers = [] exposed = False + exposed_container_count = 0 for container in artifact_containers: if not isinstance(container, dict): raise ValueError("artifact container must be an object") @@ -100,7 +123,11 @@ def build_create_request( if public: exposed_ports.append(port) - exposed = exposed or bool(exposed_ports) + if exposed_ports: + exposed = True + exposed_container_count += 1 + if isolation_profile == "PWN" and exposed_ports and len(ports) != 1: + raise ValueError("PWN exposed container must declare exactly one port") runtime_containers.append( { "name": name, @@ -113,14 +140,8 @@ def build_create_request( if not exposed: raise ValueError("Runtime smoke deployment requires at least one exposed container") - if _isolation_profile(artifact.get("category")) == "PWN": - public_containers = [ - container for container in runtime_containers if container["exposed_ports"] - ] - if len(public_containers) != 1: - raise ValueError("PWN Runtime requires exactly one public container") - if len(public_containers[0]["ports"]) != 1: - raise ValueError("PWN public container must declare exactly one port") + if isolation_profile == "PWN" and exposed_container_count != 1: + raise ValueError("PWN workload must declare exactly one exposed container") runtime_workload = {"containers": runtime_containers} @@ -141,7 +162,7 @@ def build_create_request( "request_id": f"ci-smoke-create-{normalized_instance_id}", "instance_id": normalized_instance_id, "team_id": normalized_team_id, - "isolation_profile": _isolation_profile(artifact.get("category")), + "isolation_profile": isolation_profile, "target": {"runtime_type": "KUBERNETES", "target_id": target_id.strip()}, "workload": {**runtime_workload, "resource_limits": resource_limits}, } @@ -172,7 +193,11 @@ def request(self, method: str, path: str, body: dict[str, Any] | None = None) -> with urlopen(request, timeout=self.timeout) as response: payload = response.read().decode("utf-8") except HTTPError as error: - detail = error.read().decode("utf-8", errors="replace")[:500] + try: + detail = error.read().decode("utf-8", errors="replace") + finally: + error.close() + detail = detail.replace(self.token, "[REDACTED]")[:500] if error.code >= 500: raise RetryableRuntimeError( f"Runtime API {method} {path} returned HTTP {error.code}: {detail}" @@ -255,6 +280,97 @@ def _poll_operation( raise TimeoutError(f"Runtime operation {operation_id} timed out") +def _runtime_endpoint_error( + create_request: dict[str, Any], + endpoints: Any, +) -> str | None: + expected = { + (container["name"], port) + for container in create_request["workload"]["containers"] + for port in container["exposed_ports"] + } + if not isinstance(endpoints, list) or not endpoints: + return "Runtime create operation did not return public endpoints" + + actual = set() + for endpoint in endpoints: + if not isinstance(endpoint, dict): + return "Runtime endpoints contain an invalid item" + if set(endpoint) != ENDPOINT_FIELDS: + return "Runtime endpoints contain missing or unknown fields" + container_name = endpoint.get("container_name") + port = endpoint.get("port") + protocol = endpoint.get("protocol") + service_url = endpoint.get("service_url") + if ( + not isinstance(container_name, str) + or not container_name + or isinstance(port, bool) + or not isinstance(port, int) + or not 1 <= port <= 65535 + or not isinstance(protocol, str) + or protocol not in ENDPOINT_PROTOCOLS + or not isinstance(service_url, str) + or not service_url + ): + return "Runtime endpoints contain missing or invalid required fields" + if any(character.isspace() for character in service_url): + return "Runtime endpoints contain an invalid service_url" + try: + parsed_service_url = urlsplit(service_url) + hostname = parsed_service_url.hostname + parsed_service_url.port + except ValueError: + return "Runtime endpoints contain an invalid service_url" + if ( + not parsed_service_url.scheme + or not parsed_service_url.netloc + or not hostname + ): + return "Runtime endpoints contain an invalid service_url" + key = (container_name, port) + if key in actual: + return "Runtime endpoints contain a duplicate container and port" + actual.add(key) + + if actual != expected: + return "Runtime endpoints do not exactly match requested public ports" + return None + + +def _recover_runtime_workload_id( + client: RuntimeClient, + create_request: dict[str, Any], + *, + poll_interval: float, + deadline: float, +) -> str: + instance_id = create_request["instance_id"] + path = f"/internal/v1/instances/{instance_id}/runtime-status" + while True: + try: + status = client.request("GET", path) + break + except RetryableRuntimeError as error: + if time.monotonic() >= deadline: + raise TimeoutError( + "Runtime status recovery timed out before cleanup" + ) from error + time.sleep(poll_interval) + if status.get("instance_id") != instance_id: + raise IncompleteOperationResult( + "Runtime status returned a different instance_id" + ) + if status.get("target_id") != create_request["target"]["target_id"]: + raise IncompleteOperationResult("Runtime status returned a different target_id") + runtime_workload_id = status.get("runtime_workload_id") + if not isinstance(runtime_workload_id, str) or not runtime_workload_id.strip(): + raise IncompleteOperationResult( + "Runtime status did not return a runtime_workload_id" + ) + return runtime_workload_id + + def run_smoke( artifact: dict[str, Any], *, @@ -266,7 +382,11 @@ def run_smoke( poll_interval: float = 2, timeout: float = 300, cleanup_timeout: float = 300, + evidence_clock: Callable[[], float] | None = None, ) -> dict[str, Any]: + if evidence_clock is None: + evidence_clock = time.monotonic + _revision, registry_revision, _profile = _validate_bundle_contract(artifact) token = token_file.read_text(encoding="utf-8").strip() client = RuntimeClient(api_url, token, timeout=min(max(timeout, cleanup_timeout, 1), 30)) create_request = build_create_request( @@ -275,6 +395,7 @@ def run_smoke( instance_id=instance_id, team_id=team_id, ) + create_started = evidence_clock() deadline = time.monotonic() + timeout recovered_after_timeout = False try: @@ -298,6 +419,7 @@ def run_smoke( poll_interval=poll_interval, deadline=create_poll_deadline, ) + created = None try: created = _poll_operation( client, @@ -306,23 +428,48 @@ def run_smoke( deadline=create_poll_deadline, require_create_result=True, ) - except (TimeoutError, IncompleteOperationResult): + except (TimeoutError, IncompleteOperationResult) as error: if recovered_after_timeout: - raise - recovered_after_timeout = True - created = _poll_operation( - client, - accepted.get("operation_id"), - poll_interval=poll_interval, - deadline=time.monotonic() + cleanup_timeout, - require_create_result=True, - ) - create_result = created.get("result") - if not isinstance(create_result, dict): - raise RuntimeError("Runtime create operation did not return a result") - runtime_workload_id = create_result.get("runtime_workload_id") - if not isinstance(runtime_workload_id, str) or not runtime_workload_id.strip(): - raise RuntimeError("Runtime create operation returned an invalid runtime_workload_id") + if isinstance(error, IncompleteOperationResult): + runtime_workload_id = _recover_runtime_workload_id( + client, + create_request, + poll_interval=poll_interval, + deadline=time.monotonic() + cleanup_timeout, + ) + else: + raise + else: + recovered_after_timeout = True + try: + created = _poll_operation( + client, + accepted.get("operation_id"), + poll_interval=poll_interval, + deadline=time.monotonic() + cleanup_timeout, + require_create_result=True, + ) + except IncompleteOperationResult: + runtime_workload_id = _recover_runtime_workload_id( + client, + create_request, + poll_interval=poll_interval, + deadline=time.monotonic() + cleanup_timeout, + ) + + if created is None: + create_result = {"runtime_workload_id": runtime_workload_id, "endpoints": []} + created = {"status": "SUCCEEDED", "result": create_result} + else: + create_result = created.get("result") + if not isinstance(create_result, dict): + raise RuntimeError("Runtime create operation did not return a result") + runtime_workload_id = create_result.get("runtime_workload_id") + if not isinstance(runtime_workload_id, str) or not runtime_workload_id.strip(): + raise RuntimeError("Runtime create operation returned an invalid runtime_workload_id") + endpoints = create_result.get("endpoints") + endpoint_error = _runtime_endpoint_error(create_request, endpoints) + create_elapsed_seconds = round(evidence_clock() - create_started, 3) delete_request = { "request_id": f"ci-smoke-delete-{create_request['instance_id']}", @@ -332,6 +479,7 @@ def run_smoke( "runtime_workload_id": runtime_workload_id, "delete_reason": "ADMIN_FORCED", } + delete_started = evidence_clock() cleanup_deadline = time.monotonic() + cleanup_timeout deleted = _submit_with_retry( client, @@ -347,15 +495,26 @@ def run_smoke( poll_interval=poll_interval, deadline=cleanup_deadline, ) + delete_elapsed_seconds = round(evidence_clock() - delete_started, 3) + if endpoint_error is not None: + raise RuntimeError(endpoint_error) return { "challenge_slug": artifact.get("challenge_slug"), "revision": artifact.get("revision"), + "registry_revision": registry_revision, "target_id": target_id, "instance_id": create_request["instance_id"], "runtime_workload_id": runtime_workload_id, + "images": [ + {"name": container["name"], "image": container["image"]} + for container in create_request["workload"]["containers"] + ], "service_url": create_result.get("service_url"), + "endpoints": endpoints, "create_status": created["status"], + "create_elapsed_seconds": create_elapsed_seconds, "delete_status": delete_snapshot["status"], + "delete_elapsed_seconds": delete_elapsed_seconds, "recovered_after_timeout": recovered_after_timeout, } diff --git a/tests/test_runtime_api_smoke_runner.py b/tests/test_runtime_api_smoke_runner.py index 445c916..a68d7b6 100644 --- a/tests/test_runtime_api_smoke_runner.py +++ b/tests/test_runtime_api_smoke_runner.py @@ -14,11 +14,15 @@ INSTANCE_ID = "018f3f1e-21b8-7a91-a30b-63b3400fd001" TEAM_ID = "00000000-0000-4000-8000-000000000018" ARTIFACT = { + "schema_version": "2.0", "challenge_slug": "koth-template", "revision": 11, + "registry_revision": 11, "category": "koth", "runtime_type": "KUBERNETES", "architecture": "AMD64", + "isolation_profile": "WEB", + "scan_result": "PASS", "workload": { "containers": [ { @@ -26,7 +30,7 @@ "image": f"ghcr.io/msg-ctf/challenges/koth-template/service@{DIGEST}", "ports": [ {"port": 8080, "public": True}, - {"port": 9090, "public": False}, + {"port": 9090, "public": True}, ], } ], @@ -47,13 +51,33 @@ class RuntimeHandler(BaseHTTPRequestHandler): create_result = { "runtime_workload_id": "aws-k3s-001/ctf-test/challenge", "service_url": "http://203.0.113.10:31042", - "endpoints": [], + "endpoints": [ + { + "container_name": "service", + "port": 8080, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31042", + }, + { + "container_name": "service", + "port": 9090, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31043", + }, + ], } drop_create_response_once = False drop_delete_response_once = False omit_create_operation_id_once = False omit_delete_operation_id_once = False omit_create_result_once = False + echo_auth_error = False + fail_runtime_status_once = False + runtime_status_result = { + "instance_id": INSTANCE_ID, + "target_id": "aws-k3s-lab", + "runtime_workload_id": "aws-k3s-lab/ctf-test/challenge", + } def log_message(self, _format, *_args): return @@ -74,6 +98,9 @@ def _respond(self, status, body): def do_POST(self): type(self).requests.append(("POST", self.path, self.headers, self._json_body())) + if type(self).echo_auth_error: + self._respond(400, {"error": self.headers.get("Authorization")}) + return if type(self).drop_create_response_once: type(self).drop_create_response_once = False self.connection.shutdown(socket.SHUT_RDWR) @@ -100,6 +127,13 @@ def do_DELETE(self): def do_GET(self): type(self).requests.append(("GET", self.path, self.headers, None)) + if self.path.endswith("/runtime-status"): + if type(self).fail_runtime_status_once: + type(self).fail_runtime_status_once = False + self._respond(503, {"error": "temporary unavailable"}) + return + self._respond(200, type(self).runtime_status_result) + return if self.path.endswith("op-create"): type(self).create_polls += 1 result = type(self).create_result @@ -137,13 +171,33 @@ def start_server(self): RuntimeHandler.create_result = { "runtime_workload_id": "aws-k3s-001/ctf-test/challenge", "service_url": "http://203.0.113.10:31042", - "endpoints": [], + "endpoints": [ + { + "container_name": "service", + "port": 8080, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31042", + }, + { + "container_name": "service", + "port": 9090, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31043", + }, + ], } RuntimeHandler.drop_create_response_once = False RuntimeHandler.drop_delete_response_once = False RuntimeHandler.omit_create_operation_id_once = False RuntimeHandler.omit_delete_operation_id_once = False RuntimeHandler.omit_create_result_once = False + RuntimeHandler.echo_auth_error = False + RuntimeHandler.fail_runtime_status_once = False + RuntimeHandler.runtime_status_result = { + "instance_id": INSTANCE_ID, + "target_id": "aws-k3s-lab", + "runtime_workload_id": "aws-k3s-lab/ctf-test/challenge", + } self.server = ThreadingHTTPServer(("127.0.0.1", 0), RuntimeHandler) self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) self.thread.start() @@ -155,6 +209,40 @@ def tearDown(self): self.server.server_close() self.thread.join(timeout=5) + def assert_endpoint_response_rejected(self, endpoint): + self.start_server() + RuntimeHandler.create_result["endpoints"][0] = endpoint + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + methods = [ + (method, path) + for method, path, _headers, _body in RuntimeHandler.requests + ] + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) + + @staticmethod + def valid_service_endpoint(): + return { + "container_name": "service", + "port": 8080, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31042", + } + def test_builds_runtime_team_multi_container_contract(self): request = build_create_request( ARTIFACT, @@ -176,7 +264,7 @@ def test_builds_runtime_team_multi_container_contract(self): "name": "service", "image": ARTIFACT["workload"]["containers"][0]["image"], "ports": [8080, 9090], - "exposed_ports": [8080], + "exposed_ports": [8080, 9090], "run_as_user": 10001, } ], @@ -184,6 +272,58 @@ def test_builds_runtime_team_multi_container_contract(self): }, ) + def test_builds_mixed_public_private_ports_with_exposed_ports(self): + artifact = copy.deepcopy(ARTIFACT) + artifact["workload"]["containers"] = [ + { + "name": "web", + "image": f"ghcr.io/msg-ctf/challenges/web-notebook/web@{DIGEST}", + "ports": [ + {"port": 8080, "public": True}, + {"port": 9000, "public": False}, + ], + }, + { + "name": "db", + "image": f"ghcr.io/msg-ctf/challenges/web-notebook/db@{DIGEST}", + "ports": [{"port": 5432, "public": False}], + }, + ] + request = build_create_request( + artifact, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + ) + + self.assertEqual(request["target"]["target_id"], "aws-k3s-lab") + self.assertEqual( + request["workload"]["containers"], + [ + { + "name": "web", + "image": artifact["workload"]["containers"][0]["image"], + "ports": [8080, 9000], + "exposed_ports": [8080], + "run_as_user": 10001, + }, + { + "name": "db", + "image": artifact["workload"]["containers"][1]["image"], + "ports": [5432], + "exposed_ports": [], + "run_as_user": 10001, + }, + ], + ) + self.assertTrue( + all( + "expose" not in container + for container in request["workload"]["containers"] + ) + ) + self.assertNotIn("internal_connections", request["workload"]) + def test_does_not_forward_legacy_internal_connections(self): artifact = copy.deepcopy(ARTIFACT) artifact["workload"]["containers"].append( @@ -210,11 +350,6 @@ def test_does_not_forward_legacy_internal_connections(self): ) self.assertNotIn("internal_connections", request["workload"]) - self.assertNotIn("expose", request["workload"]["containers"][0]) - self.assertEqual( - request["workload"]["containers"][1]["exposed_ports"], - [], - ) def test_creates_polls_and_always_deletes_runtime_workload(self): self.start_server() @@ -250,9 +385,290 @@ def test_creates_polls_and_always_deletes_runtime_workload(self): self.assertEqual(delete_body["runtime_workload_id"], "aws-k3s-001/ctf-test/challenge") self.assertEqual(delete_body["delete_reason"], "ADMIN_FORCED") + def test_reports_digest_images_and_runtime_endpoints(self): + self.start_server() + RuntimeHandler.create_result = { + "runtime_workload_id": "aws-k3s-001/ctf-test/challenge", + "service_url": "http://203.0.113.10:31042", + "endpoints": [ + { + "container_name": "service", + "port": 8080, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31042", + }, + { + "container_name": "service", + "port": 9090, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31043", + }, + ], + } + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + result = run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-001", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + self.assertEqual( + result["images"], + [ + { + "name": "service", + "image": ARTIFACT["workload"]["containers"][0]["image"], + } + ], + ) + self.assertEqual(result["endpoints"], RuntimeHandler.create_result["endpoints"]) + self.assertEqual(result["registry_revision"], 11) + self.assertNotIn("A" * 43, json.dumps(result)) + + def test_rejects_registry_revision_mismatch_before_runtime_request(self): + artifact = dict(ARTIFACT, registry_revision=12) + + with self.assertRaisesRegex(ValueError, "registry_revision"): + build_create_request( + artifact, + target_id="aws-k3s-001", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + ) + + def test_rejects_isolation_profile_mismatch_before_runtime_request(self): + artifact = dict(ARTIFACT, isolation_profile="PWN") + + with self.assertRaisesRegex(ValueError, "isolation_profile"): + build_create_request( + artifact, + target_id="aws-k3s-001", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + ) + + def test_reports_create_and_delete_elapsed_seconds(self): + self.start_server() + evidence_times = iter([10.0, 12.5, 20.0, 21.25]) + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + result = run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-001", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + evidence_clock=lambda: next(evidence_times), + ) + + self.assertEqual(result["create_elapsed_seconds"], 2.5) + self.assertEqual(result["delete_elapsed_seconds"], 1.25) + + def test_rejects_success_without_endpoints_after_cleanup(self): + self.start_server() + RuntimeHandler.create_result = { + "runtime_workload_id": "aws-k3s-001/ctf-test/challenge", + "service_url": None, + "endpoints": [], + } + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + methods = [(method, path) for method, path, _headers, _body in RuntimeHandler.requests] + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + + def test_rejects_incomplete_endpoint_set_after_cleanup(self): + self.start_server() + RuntimeHandler.create_result["endpoints"] = [ + RuntimeHandler.create_result["endpoints"][0] + ] + + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + methods = [ + (method, path) + for method, path, _headers, _body in RuntimeHandler.requests + ] + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) + + def test_rejects_unexpected_endpoint_after_cleanup(self): + self.start_server() + RuntimeHandler.create_result["endpoints"].append( + { + "container_name": "service", + "port": 9000, + "protocol": "HTTP", + "service_url": "http://203.0.113.10:31044", + } + ) + + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + methods = [ + (method, path) + for method, path, _headers, _body in RuntimeHandler.requests + ] + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) + + def test_rejects_duplicate_endpoint_after_cleanup(self): + self.start_server() + RuntimeHandler.create_result["endpoints"].append( + copy.deepcopy(RuntimeHandler.create_result["endpoints"][0]) + ) + + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + methods = [ + (method, path) + for method, path, _headers, _body in RuntimeHandler.requests + ] + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) + + def test_rejects_endpoint_missing_required_field_after_cleanup(self): + self.start_server() + del RuntimeHandler.create_result["endpoints"][0]["service_url"] + + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + methods = [ + (method, path) + for method, path, _headers, _body in RuntimeHandler.requests + ] + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) + + def test_rejects_endpoint_with_unsupported_protocol_after_cleanup(self): + endpoint = self.valid_service_endpoint() + endpoint["protocol"] = "SMTP" + + self.assert_endpoint_response_rejected(endpoint) + + def test_rejects_non_string_endpoint_protocol_after_cleanup(self): + endpoint = self.valid_service_endpoint() + endpoint["protocol"] = ["HTTP"] + + self.assert_endpoint_response_rejected(endpoint) + + def test_rejects_endpoint_with_invalid_service_url_after_cleanup(self): + endpoint = self.valid_service_endpoint() + endpoint["service_url"] = "http://bad host/path" + + self.assert_endpoint_response_rejected(endpoint) + + def test_rejects_endpoint_with_unknown_field_after_cleanup(self): + endpoint = self.valid_service_endpoint() + endpoint["unexpected"] = True + + self.assert_endpoint_response_rejected(endpoint) + + def test_redacts_service_token_from_runtime_error_details(self): + self.start_server() + RuntimeHandler.echo_auth_error = True + service_token = "A" * 43 + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text(service_token, encoding="utf-8") + with self.assertRaises(RuntimeError) as raised: + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-001", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=2, + ) + + self.assertNotIn(service_token, str(raised.exception)) + self.assertIn("[REDACTED]", str(raised.exception)) + def test_maps_pwn_category_to_pwn_isolation_profile(self): artifact = copy.deepcopy(ARTIFACT) artifact["category"] = "pwn" + artifact["isolation_profile"] = "PWN" artifact["workload"]["containers"][0]["ports"] = [ {"port": 31337, "public": True} ] @@ -266,62 +682,109 @@ def test_maps_pwn_category_to_pwn_isolation_profile(self): self.assertEqual(request["isolation_profile"], "PWN") - def test_rejects_pwn_with_multiple_public_containers(self): + def test_rejects_public_pwn_container_with_multiple_ports(self): artifact = copy.deepcopy(ARTIFACT) artifact["category"] = "pwn" + artifact["isolation_profile"] = "PWN" artifact["workload"]["containers"][0]["ports"] = [ - {"port": 31337, "public": True} + {"port": 31337, "public": True}, + {"port": 9000, "public": False}, ] - artifact["workload"]["containers"].append( - { - "name": "helper", - "image": f"ghcr.io/msg-ctf/challenges/koth-template/helper@{DIGEST}", - "ports": [{"port": 8080, "public": True}], - } - ) - with self.assertRaisesRegex(ValueError, "exactly one public container"): + with self.assertRaisesRegex(ValueError, "PWN.*one port"): build_create_request( artifact, - target_id="aws-k3s-pwn-001", + target_id="aws-k3s-lab", instance_id=INSTANCE_ID, team_id=TEAM_ID, ) - def test_rejects_pwn_public_container_with_multiple_ports(self): + def test_rejects_multiple_public_pwn_containers(self): artifact = copy.deepcopy(ARTIFACT) artifact["category"] = "pwn" + artifact["isolation_profile"] = "PWN" + artifact["workload"]["containers"] = [ + { + "name": "pwn-a", + "image": "ghcr.io/msg-ctf/pwn-a@sha256:" + "a" * 64, + "ports": [{"port": 31337, "public": True}], + "run_as_user": 10001, + }, + { + "name": "pwn-b", + "image": "ghcr.io/msg-ctf/pwn-b@sha256:" + "b" * 64, + "ports": [{"port": 31338, "public": True}], + "run_as_user": 10002, + }, + ] - with self.assertRaisesRegex(ValueError, "exactly one port"): + with self.assertRaisesRegex(ValueError, "PWN.*one exposed container"): build_create_request( artifact, - target_id="aws-k3s-pwn-001", + target_id="aws-k3s-lab", instance_id=INSTANCE_ID, team_id=TEAM_ID, ) - def test_rejects_malformed_create_result_without_invalid_delete_request(self): + def test_recovers_workload_id_and_cleans_up_incomplete_create_result(self): self.start_server() RuntimeHandler.create_result = None with tempfile.TemporaryDirectory() as directory: token_file = Path(directory) / "service-token" token_file.write_text("A" * 43, encoding="utf-8") - with self.assertRaisesRegex(RuntimeError, "did not return a result"): + with self.assertRaisesRegex(RuntimeError, "endpoints"): run_smoke( ARTIFACT, api_url=f"http://127.0.0.1:{self.server.server_port}", token_file=token_file, - target_id="aws-k3s-001", + target_id="aws-k3s-lab", instance_id=INSTANCE_ID, team_id=TEAM_ID, poll_interval=0, timeout=0.01, - cleanup_timeout=0.01, + cleanup_timeout=1, ) methods = [(method, path) for method, path, _headers, _body in RuntimeHandler.requests] - self.assertNotIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn( + ("GET", f"/internal/v1/instances/{INSTANCE_ID}/runtime-status"), + methods, + ) + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) + + def test_retries_runtime_status_recovery_before_cleanup(self): + self.start_server() + RuntimeHandler.create_result = None + RuntimeHandler.fail_runtime_status_once = True + with tempfile.TemporaryDirectory() as directory: + token_file = Path(directory) / "service-token" + token_file.write_text("A" * 43, encoding="utf-8") + + with self.assertRaisesRegex(RuntimeError, "endpoints"): + run_smoke( + ARTIFACT, + api_url=f"http://127.0.0.1:{self.server.server_port}", + token_file=token_file, + target_id="aws-k3s-lab", + instance_id=INSTANCE_ID, + team_id=TEAM_ID, + poll_interval=0, + timeout=0.01, + cleanup_timeout=0.1, + ) + + status_path = f"/internal/v1/instances/{INSTANCE_ID}/runtime-status" + methods = [ + (method, path) + for method, path, _headers, _body in RuntimeHandler.requests + ] + self.assertEqual(methods.count(("GET", status_path)), 2) + self.assertIn(("DELETE", f"/internal/v1/instances/{INSTANCE_ID}"), methods) + self.assertIn(("GET", "/internal/v1/operations/op-delete"), methods) + self.assertGreater(RuntimeHandler.delete_polls, 0) def test_recovers_timed_out_create_operation_and_deletes_workload(self): self.start_server() diff --git a/tests/test_workflow_contracts.py b/tests/test_workflow_contracts.py index c1f2b7c..90227a3 100644 --- a/tests/test_workflow_contracts.py +++ b/tests/test_workflow_contracts.py @@ -304,6 +304,7 @@ def test_pipeline_self_test_calls_reusable_supply_chain(self): text = path.read_text(encoding="utf-8") self.assertIn("push", workflow["on"]) + self.assertIn("pull_request", workflow["on"]) self.assertIn("workflow_dispatch", workflow["on"]) self.assertEqual(workflow["permissions"], {"contents": "read"}) self.assertIn(