diff --git a/docs/README.md b/docs/README.md index b998f7d..aa3d46b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,6 @@ # 문서 +- [A/B 버전 전환 사전 준비](release-transition.md): 운영 버전·후보 이미지·실행 설정·readiness 대조와 HTTP 관측, 실제 무중단 교체에 필요한 소비자 계약 구분 - [운영 버전 기준 이미지 사전 다운로드](active-release-prefetch-policy.md): 최초 active release 전체 컨테이너, 업데이트 버전 추가 준비, K3s 선언 전환과 기존 인스턴스 보존 기준 - [노드 이미지 사전 다운로드 연동](node-image-prefetch.md): 문제별·전체 이미지 목록과 변경분 취합, 읽기 전용 Actions와 수집·다운로드·적용의 책임 경계 - [관리자 대시보드 검증·발행 연동](backend-publication-dispatch.md): 정확한 main SHA 발행 요청, run 추적·상태·산출물 조회, 기존 성공본 수집과 중복·재시도 책임 diff --git a/docs/release-transition.md b/docs/release-transition.md new file mode 100644 index 0000000..ebc1352 --- /dev/null +++ b/docs/release-transition.md @@ -0,0 +1,165 @@ +# A/B 버전 전환 사전 준비 + +이 도구는 **무중단 전환의 사전검수와 HTTP 관측**을 준비합니다. 실제 이미지 pull, +active 변경, 인스턴스 생성·교체·삭제, 트래픽 전환은 실행하지 않습니다. 독립 API나 +트래픽 라우터를 새로 만들지 않으며 기존 Backend·Runtime 응답을 사용합니다. + +## 서로 다른 두 가지 전환 + +| 구분 | 의미 | 현재 확인한 코드 | +|---|---|---| +| 새 생성부터 B 사용 | 기존 참가자 A는 유지하고 이후 새 인스턴스만 B로 생성 | Backend current release 전환의 기존 동작 | +| 실행 중 A를 B로 무중단 교체 | B를 먼저 정상 기동하고 같은 접속 경로를 넘긴 뒤 A를 정리 | 소비자 계약·구현과 실제 검증이 별도로 필요 | + +Scheduler의 현재 create/replace는 A를 CLEANUP_PENDING으로 먼저 바꿉니다. 같은 +사용자의 실행 중복 제약과 기존 정리 후 재예약도 존재합니다. 이 경로를 B 사전 +기동에 쓰거나 reset을 B 업그레이드로 사용하지 않습니다. URL이 바뀐 뒤 사람이 +다시 접속하는 것과 기존 연결·상태를 유지하는 무중단 전환도 구분합니다. + +```text +A 운영 유지 + → 승인된 B 전체 이미지 준비 + → B의 현재 readiness 및 별도 QA 확인 + → [서버 측 계약 필요] B 병렬 기동·정상 판정 + → [서버 측 계약 필요] 접속 경로 전환 + → 전환·기존 연결 처리 확인 후 A 정리 + +B 실패 / 준비 만료 / 버전 변경 + → 전환 중단, A 유지 (이전 컨테이너·이미지·파일·설정 보존) +``` + +다운로드와 readiness는 Pod 기동·healthcheck·실제 공개 접속 성공이 아닙니다. +기존 [SSH 사전 다운로드](vm-image-prefetch.md)와 [readiness 대조](node-readiness-audit.md)를 +재사용하되, 그 성공만으로 운영 전환을 승인하지 않습니다. + +## 1. A/B 사전검수 + +`scripts.prepare_release_transition`은 다음 **저장된 자료만** 읽습니다. + +- A·B의 공식 성공 main `artifact-v2.json` 원본. 원본 출처는 기존 발행 검증 절차로 확인합니다. +- Backend 릴리스 목록 GET 응답의 `data` 객체. challenge_id, current_release_id, + releases, activation을 보존합니다. 서버 조회 시각은 별도로 기록합니다. +- 후보 B에 대한 Runtime readiness query의 전체 응답과 정확한 target 목록. + +Backend 파일은 60초 미만의 관측만 허용합니다. 이는 운영자가 입력한 조회 시각이며 +이 도구가 서버를 독립 조회한 증거는 아닙니다. `--as-of`는 과거 재현용입니다. +새 전환·실행 승인으로 쓰지 않고 실제 시험 직전에 최신 자료로 다시 검사합니다. + +검사하는 항목: + +- 같은 일반 Linux K3s WEB 문제·architecture, 서로 다른 A/B release ID. +- 현재 active=A, A/B 승인·배포 가능 여부, 진행 중인 다른 활성화 의도 부재. +- 전체 공개·비공개 컨테이너의 image digest·포트·공개 여부와 자원·healthcheck 보존. +- schema 2.1 env·secret 이름/등록 상태/버전. 값은 출력하지 않고 설정 fingerprint만 보존. +- B의 정확한 전체 image set, 모든 지정 target의 현재 전체 적합 노드 준비·관측·만료. +- update는 증가하는 revision만 허용합니다. 낮은 revision은 명시적 rollback 방향이 필요합니다. + +Backend 파생 릴리스, env_rules, 원본 bundle에 없는 환경값, 지원하지 않는 실행 필드는 +자동으로 추측·보완하지 않고 별도 유효 설정 검토가 필요하다고 거절합니다. KOTH·PWN· +Windows·DOCKER·VM은 이번 전환 도구의 범위가 아닙니다. Runtime의 이미지별 UID·쓰기 +경로·네트워크 정책, 전체 운영 노드의 누락 여부는 독립 확인하지 않습니다. + +```bash +python3 -B -m scripts.prepare_release_transition \ + --current-artifact /private/a/artifact-v2.json \ + --candidate-artifact /private/b/artifact-v2.json \ + --backend-releases /private/releases-data.json \ + --current-release-id CURRENT_RELEASE_UUID \ + --candidate-release-id CANDIDATE_RELEASE_UUID \ + --backend-observed-at ACTUAL_RFC3339_QUERY_TIME \ + --candidate-readiness /private/b-readiness-response.json \ + --target-id APPROVED_TARGET \ + --output-dir /private/new-transition-plan +``` + +target이 여러 곳이면 `--target-id`를 반복합니다. readiness를 생략해 준비 요청 +인계안부터 만들 수 있지만 `IMAGE_READINESS_REQUIRED`와 종료 코드 1이 나옵니다. +readiness가 전부 맞아도 판정은 `READY_FOR_REPRESENTATIVE_QA`까지만입니다. +이 경우 종료 코드 0은 계획 검수 성공이지 배포 실행·활성화·무중단 승인 코드가 아닙니다. +입력 불일치는 종료 코드 2입니다. + +`transition-plan.json`에는 A/B 원본 checksum·전체 이미지와 보존 합집합, 설정 +fingerprint, target/readiness 대조, 정확한 기존 준비·배포 테스트 API 인계안이 담깁니다. +각 POST는 사람이 승인한 뒤 기존 Backend로 접수하는 예시일 뿐 호출하지 않습니다. +준비 요청에는 고유 Idempotency-Key, 배포 테스트에는 canonical UUID request_id를 +운영자가 별도로 선택·기록해야 합니다. 생성된 문구를 그대로 요청 ID로 사용하지 않습니다. +`/activate`, reset, replace, delete 요청은 생성하지 않습니다. + +원본·Backend·Runtime 응답을 공개 artifact나 commit에 넣지 않습니다. 새 보고서 +디렉터리는0700, JSON은0600이며 기존 출력은 덮어쓰지 않습니다. env·비밀값 원문을 +복사하지 않지만 digest·release/target ID와 fingerprint도 운영 자료이므로 비공개로 +보관합니다. plan SHA는 변경 탐지용이지 서명·인증·실행 허가가 아닙니다. + +## 2. 공개 HTTP 접속 관측 + +`scripts.observe_rollout_http`는 운영자가 승인한 GET-safe 공개 WEB URL에만 접속합니다. +전환 계획과 관측 결과를 plan SHA로 연결하고 status·헤더 도착까지의 지연·고정 오류 +코드만 기록합니다. 토큰·쿠키를 보내거나 응답 본문·헤더·URL 원문을 보고서에 저장하지 않습니다. + +```bash +python3 -B -m scripts.observe_rollout_http \ + --plan /private/new-transition-plan/transition-plan.json \ + --url https://approved-public-endpoint.example/health \ + --duration-seconds 120 --interval-seconds 1 --timeout-seconds 2 \ + --output-dir /private/new-http-observation +``` + +기본 HTTPS는 인증서를 검증합니다. 공개 HTTP 문제는 승인 후 `--allow-http`를 명시할 +수 있으나 TLS 검증이 없는 사실이 결과에 남습니다. 기본 예상 상태는200이고 health +endpoint가204라면 `--expected-status 204`를 명시합니다. 로그인 화면의200 응답만으로 +문제 기능을 검증한 것으로 계산하지 않습니다. + +userinfo·query·fragment·임의 header/인증 입력, 비공개/metadata 주소, redirect와 +환경변수 proxy를 허용하지 않습니다. DNS의 모든 주소가 public인지 확인한 뒤 첫 +주소를 해당 관측 동안 고정하고 원래 hostname으로 Host/SNI·인증서를 검증합니다. +DNS 주소 교체를 검증하는 도구는 아니며, 다른 A/B URL은 별도 실행으로 관측합니다. +GET-safe 경로인지 운영자가 먼저 확인해야 합니다. + +관측은 최대900초·300회이며 느린 요청 뒤 몰아서 따라잡지 않습니다. 요청 timeout은 +소켓 작업의 제한이고 응답 헤더 전체의 엄격한 wall-clock 상한은 아닙니다. 실제 지연과 +관측 시각을 기록하며, 오류 뒤에도 정한 구간을 관측하고 중간 종료는 INTERRUPTED로 +남깁니다. 완료한 샘플과 실패도 비공개 JSON에 원자적으로 보존합니다. + +**NO_HTTP_FAILURE_OBSERVED는 표본에서 HTTP 실패를 못 봤다는 뜻입니다.** B 버전 +실행 imageID, 실제 healthcheck, 지속 TCP/WebSocket 연결, 표본 사이의 접속 보장, +DB·업로드·세션 보존이나 롤백 성공을 증명하지 않습니다. 모든 요청이200이어도 +`zero_downtime_proven=false`입니다. HTTP 관측은 기존 Pod·release·트래픽을 변경하지 않습니다. + +## 3. 실제 시험 순서와 파트 경계 + +1. 진웅/DevSecOps: 승인된 대표 WEB의 A/B 원본·현재 운영 선택을 대조하고 계획을 만듭니다. +2. 이미지 준비: 기존 경로로 B를 준비하고 현재 readiness를 재조회합니다. 과거 operation + 성공이나 캐시 존재만으로 대신하지 않습니다. A 이미지를 삭제하지 않습니다. +3. Backend/Runtime: 승인된 별도 관리자 배포 테스트로 B의 실제 imageID·healthcheck·공개 + 접속을 확인합니다. 이는 참가자 A의 접속 경로 전환이 아닙니다. QA 자원·팀·설정과 + 정리 담당을 먼저 확정하며 우리 도구는 자동 생성·삭제하지 않습니다. +4. 진웅/DevSecOps: A URL 관측을 전환 전에 시작하고 B 기동·전환·안정화 동안 유지합니다. +5. Backend·Scheduler·Runtime/접속 경로 담당: 아래 계약을 반영한 별도 전환기로 B를 + 선택합니다. 기존 replace/reset을 무중단 교체기로 대신하지 않습니다. +6. 공동: 이전 A 유지, B 실제 사용, 실패 시 A 유지, rollback, 연결·상태 보존을 확인합니다. + 성공 이후에만 합의된 시점에 A를 정리합니다. 관측 보고서 하나로 전체 통과 처리하지 않습니다. + +실제 소비자 계약에서 추가로 정할 사항: + +| 항목 | 필요한 결정 | 담당 구현 | +|---|---|---| +| 병렬 A/B | 동일 사용자·문제의 B 준비용 별도 실행과 임시 이중 자원 예약 | Backend·Scheduler·Broker | +| 접속 전환 | 안정된 URL/포트의 소유 주체, 원자적 switch·drain·확인 방법 | Backend·Runtime·접속 경로 담당 | +| 상태·연결 | DB/파일/세션 이관·호환, 기존 TCP/WebSocket 처리, TTL 보존 | 출제자·Backend·Runtime | +| 실패·rollback | B 실패 시 A를 유지하고 중복 전환·이전 generation을 막는 기준 | Backend·Scheduler·Runtime | +| 자동화 연결 | 발행·가져오기·준비·승인·전환의 식별자와 결과 연결 | 각 소비자·DevSecOps | + +완성되지 않은 switch API 이름이나 인증 범위를 임의로 만들지 않습니다. 기존 설정의 +자동 활성화 여부도 이 도구에서 바꾸지 않습니다. 자동화 연결 전까지 사전검수와 +관측 도구로 준비하고 실제 배포 권한·승인은 각 담당 시스템에서 유지합니다. + +## 확인한 기존 계약 + +- [Backend release 응답·실행 설정](https://github.com/MSG-CTF/msg-backend/blob/6ffed88cbdc743716bf8486b999d12cc9ecc6f12/apps/instances/releases.py#L470) +- [Backend current release 전환과 기존 인스턴스 유지](https://github.com/MSG-CTF/msg-backend/blob/6ffed88cbdc743716bf8486b999d12cc9ecc6f12/apps/instances/admin_views.py) +- [기존 준비 API](https://github.com/MSG-CTF/msg-backend/blob/6ffed88cbdc743716bf8486b999d12cc9ecc6f12/apps/instances/image_preparation_views.py) +- [기존 관리자 배포 테스트 API](https://github.com/MSG-CTF/msg-backend/blob/6ffed88cbdc743716bf8486b999d12cc9ecc6f12/apps/instances/deployment_test_views.py#L40) +- [Scheduler의 선행 정리·reset 버전 보존](https://github.com/MSG-CTF/instance-scheduler/blob/e94eaf17a1a3408524de90946b90f3f6cb6b9a69/src/main/kotlin/kr/msgctf/scheduler/instance/service/InstanceSchedulerService.kt) +- [Scheduler 교체 대기](https://github.com/MSG-CTF/instance-scheduler/blob/e94eaf17a1a3408524de90946b90f3f6cb6b9a69/docs/replacement-wait.md) + +이 고정 소스 대조는 해당 버전의 실제 서버 배포 확인이 아닙니다. diff --git a/scripts/README.md b/scripts/README.md index 32b7470..ef5e533 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -9,6 +9,14 @@ Provisioner API 요청으로 변환합니다. 생성 Operation을 조회한 뒤 ## 파일 +### `prepare_release_transition.py` / `observe_rollout_http.py` + +A/B 원본 bundle·Backend 운영 선택·실행 설정·B readiness를 오프라인에서 대조하고, +별도 승인된 공개 WEB URL의 HTTP 상태·지연을 관측합니다. 기존 준비/관리자 테스트 +API 인계안만 만들며 호출하지 않습니다. active·Pod·트래픽·기존 인스턴스는 바꾸지 +않고 HTTP200을 무중단 성공으로 계산하지 않습니다. +[준비 범위·실행법·소비자 계약](../docs/release-transition.md)을 참고합니다. + ### `vm_image_prefetch.py` / `vm_cache_preflight.py` 운영자가 선택한 정확한 release와 VM을 공식 발행 원본에 대조하고, 별도 수동 승인 후 diff --git a/scripts/observe_rollout_http.py b/scripts/observe_rollout_http.py new file mode 100644 index 0000000..de33787 --- /dev/null +++ b/scripts/observe_rollout_http.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Sample an explicitly approved public WEB endpoint without switching traffic.""" + +import argparse +from datetime import datetime, timezone +import hashlib +import http.client +import ipaddress +import json +import math +from pathlib import Path +import re +import socket +import ssl +import sys +import time +from urllib.parse import urlsplit + +from scripts.prepare_release_transition import plan_hash, require +from scripts.vm_image_prefetch import read_json, save_private + + +def validate_endpoint(url, *, allow_http=False, resolver=socket.getaddrinfo): + require(isinstance(url, str) and 1 <= len(url) <= 2048 + and all(33 <= ord(char) <= 126 for char in url), "INVALID_ENDPOINT") + parsed = urlsplit(url) + require(parsed.scheme == "https" or (allow_http and parsed.scheme == "http"), "HTTPS_REQUIRED") + require(parsed.hostname and parsed.username is None and parsed.password is None + and not parsed.query and not parsed.fragment and "?" not in url and "#" not in url, + "ENDPOINT_CREDENTIALS_QUERY_OR_FRAGMENT_FORBIDDEN") + path = parsed.path or "/" + require(re.fullmatch(r"/[A-Za-z0-9._~/-]*", path) and "//" not in path + and not {".", ".."}.intersection(path.split("/")), "UNSAFE_ENDPOINT_PATH") + host = parsed.hostname + require(not host.endswith((".internal", ".localhost", ".local")) and host != "localhost") + port = parsed.port if parsed.port is not None else (443 if parsed.scheme == "https" else 80) + require(1 <= port <= 65535) + answers = resolver(host, port, type=socket.SOCK_STREAM) + require(answers, "NO_PUBLIC_ENDPOINT_ADDRESS") + addresses = [] + for answer in answers: + address = ipaddress.ip_address(answer[4][0]) + require(address.is_global and not address.is_multicast, "NONPUBLIC_ENDPOINT_FORBIDDEN") + addresses.append(str(address)) + return {"host": host, "port": port, "address": addresses[0], "path": path, + "scheme": parsed.scheme} + + +class PinnedConnection(http.client.HTTPConnection): + def __init__(self, endpoint, timeout): + super().__init__(endpoint["host"], endpoint["port"], timeout=timeout) + self.address = endpoint["address"] + self.context = ssl.create_default_context() if endpoint["scheme"] == "https" else None + + def connect(self): + raw = socket.create_connection((self.address, self.port), self.timeout) + try: + # Keep the original hostname for Host/SNI and certificate verification. + self.sock = self.context.wrap_socket(raw, server_hostname=self.host) if self.context else raw + except BaseException: + raw.close() + raise + + +def probe(endpoint, timeout): + connection = PinnedConnection(endpoint, timeout) + try: + connection.request("GET", endpoint["path"], headers={"Cache-Control": "no-cache", + "Connection": "close", "User-Agent": "msgctf-rollout-observer/1"}) + response = connection.getresponse() + try: + return {"http_status": response.status, "error_code": None} + finally: + response.close() + except ssl.SSLError: + return {"http_status": None, "error_code": "TLS_FAILURE"} + except TimeoutError: + return {"http_status": None, "error_code": "REQUEST_TIMEOUT"} + except (OSError, http.client.HTTPException): + return {"http_status": None, "error_code": "NETWORK_OR_PROTOCOL_FAILURE"} + finally: + connection.close() + + +def observe(url, plan, *, duration_seconds=60, interval_seconds=1, timeout_seconds=2, + expected_status=200, allow_http=False, resolver=socket.getaddrinfo, + request=probe, monotonic=time.monotonic, sleep=time.sleep, on_result=None): + require(isinstance(plan, dict) and plan.get("scope") == "OFFLINE_WEB_RELEASE_TRANSITION_PREPARATION" + and plan.get("plan_sha256") == plan_hash(plan), "INVALID_TRANSITION_PLAN") + require(type(duration_seconds) is int and 1 <= duration_seconds <= 900) + for value, minimum, maximum in ((interval_seconds, 0.25, 30), (timeout_seconds, 0.25, 10)): + require(type(value) in (float, int) and math.isfinite(value) and minimum <= value <= maximum) + require(math.ceil(duration_seconds / interval_seconds) <= 300) + require(type(expected_status) is int and 200 <= expected_status <= 299) + endpoint = validate_endpoint(url, allow_http=allow_http, resolver=resolver) + result = {"schema_version": "1.0", "scope": "SAMPLED_PUBLIC_HTTP_AVAILABILITY_ONLY", + "plan_sha256": plan["plan_sha256"], "endpoint_sha256": hashlib.sha256(url.encode()).hexdigest(), + "status": "RUNNING", "samples": [], "expected_http_status": expected_status, + "duration_seconds": duration_seconds, "interval_seconds": interval_seconds, + "timeout_seconds": timeout_seconds, "started_at": datetime.now(timezone.utc).isoformat(), + "https_certificate_verification_enabled": endpoint["scheme"] == "https", + "http_explicitly_allowed": endpoint["scheme"] == "http" and allow_http, + "limits": {"authorization_sent": False, "response_body_saved": False, + "redirect_followed": False, "proxy_environment_used": False, + "activation_requested": False, "instance_created_or_deleted": False, + "version_b_verified": False, "existing_tcp_sessions_verified": False, + "between_sample_availability_proven": False, "zero_downtime_proven": False}} + start = monotonic() + try: + index = 0 + while monotonic() - start < duration_seconds and index < 300: + due = start + index * interval_seconds + if due > monotonic(): + sleep(min(due - monotonic(), max(0, duration_seconds - (monotonic() - start)))) + remaining = duration_seconds - (monotonic() - start) + if remaining <= 0: + break + sampled_at, began = datetime.now(timezone.utc).isoformat(), monotonic() + reply = request(endpoint, min(timeout_seconds, remaining)) + require(isinstance(reply, dict) and set(reply) == {"http_status", "error_code"}) + status, error = reply["http_status"], reply["error_code"] + require(status is None or type(status) is int and 100 <= status <= 599) + require(error in {None, "TLS_FAILURE", "REQUEST_TIMEOUT", "NETWORK_OR_PROTOCOL_FAILURE"}) + require((status is None) == (error is not None)) + result["samples"].append({"sampled_at": sampled_at, "offset_seconds": round(began - start, 6), + "elapsed_seconds": round(monotonic() - began, 6), + "http_status": status, "error_code": error, + "passed": error is None and status == expected_status}) + if on_result: + on_result(result) + # Do not issue a catch-up burst after a slow request. + index = max(index + 1, math.floor((monotonic() - start) / interval_seconds) + 1) + result["status"] = ("NO_HTTP_FAILURE_OBSERVED" if result["samples"] + and all(row["passed"] for row in result["samples"]) else "HTTP_FAILURE_OBSERVED") + except KeyboardInterrupt: + result["status"] = "INTERRUPTED" + except (ValueError, OSError, TypeError, KeyError): + result["status"] = "OBSERVER_FAILED" + raise + finally: + result["finished_at"] = datetime.now(timezone.utc).isoformat() + result["elapsed_seconds"] = round(monotonic() - start, 6) + result["sample_count"] = len(result["samples"]) + result["failure_count"] = sum(not row["passed"] for row in result["samples"]) + if on_result: + on_result(result) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--plan", required=True, type=Path) + parser.add_argument("--url", required=True) + parser.add_argument("--duration-seconds", type=int, default=60) + parser.add_argument("--interval-seconds", type=float, default=1) + parser.add_argument("--timeout-seconds", type=float, default=2) + parser.add_argument("--expected-status", type=int, default=200) + parser.add_argument("--allow-http", action="store_true") + parser.add_argument("--output-dir", required=True, type=Path) + args = parser.parse_args() + try: + plan = read_json(args.plan) + output = args.output_dir + output.mkdir(mode=0o700, parents=True, exist_ok=False) + result = observe(args.url, plan, duration_seconds=args.duration_seconds, + interval_seconds=args.interval_seconds, timeout_seconds=args.timeout_seconds, + expected_status=args.expected_status, allow_http=args.allow_http, + on_result=lambda value: save_private(output / "http-observation.json", value)) + except (ValueError, OSError, TypeError, KeyError, OverflowError, RecursionError): + print("HTTP observation failed: check approved endpoint, plan and private output path.", file=sys.stderr) + return 2 + print(json.dumps({"status": result["status"], "sample_count": result["sample_count"], + "failure_count": result["failure_count"], "zero_downtime_proven": False})) + return 0 if result["status"] == "NO_HTTP_FAILURE_OBSERVED" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/prepare_release_transition.py b/scripts/prepare_release_transition.py new file mode 100644 index 0000000..3245245 --- /dev/null +++ b/scripts/prepare_release_transition.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""Prepare an offline A/B release handoff; never activate or replace workloads.""" + +import argparse +from datetime import datetime, timezone +import hashlib +import json +from pathlib import Path +import sys +from uuid import UUID + +from scripts.audit_node_image_readiness import audit_readiness, _read_json, _time_ns, _validate_targets +from scripts.execution_settings_contract import validate_env, validate_secret_env +from scripts.generate_node_image_plan import generate_plan +from scripts.healthcheck_contract import validate_healthcheck +from scripts.vm_image_prefetch import read_json, save_private + + +MAX_SNAPSHOT_AGE_SECONDS = 60 +HANDOVER_GAPS = [ + "PARALLEL_SAME_USER_INSTANCE_CONTRACT_REQUIRED", + "STABLE_ENDPOINT_AND_TRAFFIC_SWITCH_CONTRACT_REQUIRED", + "STATE_SESSION_AND_OPEN_CONNECTION_POLICY_REQUIRED", + "FAILED_B_KEEP_A_AND_ROLLBACK_VERIFICATION_REQUIRED", +] + + +def require(condition, code="INPUT_OR_EVIDENCE_INVALID"): + if not condition: + raise ValueError(code) + + +def canonical_hash(value): + return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), + ensure_ascii=True, allow_nan=False).encode()).hexdigest() + + +def plan_hash(plan): + return canonical_hash({key: value for key, value in plan.items() if key != "plan_sha256"}) + + +def canonical_uuid(value): + require(isinstance(value, str) and str(UUID(value)) == value) + return value + + +def ports(raw): + require(isinstance(raw, list) and 1 <= len(raw) <= 32) + result = {} + for row in raw: + require(isinstance(row, dict) and set(row) == {"port", "public"}) + port = row["port"] + require(type(port) is int and 1 <= port <= 65535 and port not in result) + require(type(row["public"]) is bool) + result[port] = row["public"] + return sorted(result.items()) + + +def checked_release(bundle, checksum, row, challenge_id, *, current, as_of_ns): + generated = generate_plan(bundle, checksum) + require(generated is not None, "GENERAL_KUBERNETES_RELEASE_REQUIRED") + source, _ = generated + require(bundle.get("isolation_profile") == "WEB" and bundle["category"].lower() != "pwn", "WEB_RELEASE_REQUIRED") + require(isinstance(row, dict)) + canonical_uuid(row.get("release_id")) + require(row.get("challenge_id") == challenge_id + and row.get("challenge_slug") == source["challenge_slug"] + and type(row.get("registry_revision")) is int + and row["registry_revision"] == source["registry_revision"] + and row.get("runtime_type") == source["runtime_type"] + and row.get("architecture") == source["architecture"] + and row.get("isolation_profile") == "WEB" + and row.get("source_ref") == "refs/heads/main" + and row.get("is_deployable") is True and row.get("is_current") is current, + "BACKEND_RELEASE_OR_OPERATING_SELECTION_MISMATCH") + require("derived_from_release_id" in row and row["derived_from_release_id"] is None, + "DERIVED_RELEASE_REQUIRES_EFFECTIVE_SETTINGS_REVIEW") + require(row.get("approved_at") is not None + and _time_ns(row["approved_at"]) <= as_of_ns, "APPROVED_RELEASE_REQUIRED") + resources = bundle.get("resource_profile") + require(isinstance(resources, dict) and set(resources) == { + "cpu_millicores", "memory_mib", "ephemeral_storage_mib"}) + require(all(type(value) is int and value > 0 for value in resources.values())) + require(resources == row.get("resource_profile"), "RESOURCE_SETTINGS_MISMATCH") + supplied = row.get("containers") + require(isinstance(supplied, list) and len(supplied) == len(source["containers"])) + by_name = {} + for container in supplied: + require(isinstance(container, dict) and isinstance(container.get("name"), str) + and container["name"] not in by_name) + by_name[container["name"]] = container + require(set(by_name) == {item["name"] for item in source["containers"]}, + "FULL_CONTAINER_SET_REQUIRED") + effective, health_containers = [], [] + require(set(bundle["workload"]) <= {"containers", "healthcheck"}, "UNMODELED_WORKLOAD_SETTINGS") + for container in bundle["workload"]["containers"]: + require(set(container) <= {"name", "image", "ports", "env", "secret_env"}, "UNMODELED_CONTAINER_SETTINGS") + saved = by_name[container["name"]] + declared_ports = ports(container.get("ports")) + require(saved.get("image_ref") == container["image"] + and ports(saved.get("ports")) == declared_ports, "CONTAINER_IMAGE_OR_PORT_MISMATCH") + env = validate_env(container.get("env", {})) + aliases = validate_secret_env(container.get("secret_env", {}), env) + require(not (env or aliases) or bundle["schema_version"] == "2.1", "EXECUTION_SCHEMA_21_REQUIRED") + require(validate_env(saved.get("env", {})) == env, "ENVIRONMENT_SETTINGS_MISMATCH") + require(not saved.get("env_rules"), "ENV_RULES_REQUIRE_EFFECTIVE_SETTINGS_REVIEW") + bindings = saved.get("secret_bindings", []) + require(isinstance(bindings, list) and len(bindings) == len(aliases)) + names, versions = set(), [] + for binding in bindings: + require(isinstance(binding, dict) and isinstance(binding.get("env_name"), str)) + name = binding["env_name"] + require(name in aliases and name not in names + and binding.get("name") == aliases[name] + and binding.get("status") == "registered" + and type(binding.get("version")) is int and binding["version"] > 0, + "REGISTERED_SECRET_BINDINGS_REQUIRED") + names.add(name) + versions.append({"env_name": name, "name": binding["name"], "version": binding["version"]}) + if aliases: + canonical_uuid(saved.get("secret_ref")) + else: + require(not saved.get("secret_ref"), "UNDECLARED_SECRET_BINDING") + effective.append({"name": container["name"], "ports": declared_ports, "env": env, + "secret_bindings": sorted(versions, key=lambda item: item["env_name"])}) + health_containers.append({"name": container["name"], "ports": [port for port, _ in declared_ports]}) + require(any(public for item in effective for _, public in item["ports"]), "PUBLIC_WEB_PORT_REQUIRED") + healthcheck = validate_healthcheck(bundle["workload"].get("healthcheck"), health_containers) + require(healthcheck == row.get("healthcheck"), "HEALTHCHECK_SETTINGS_MISMATCH") + return {"release_id": row["release_id"], "challenge_slug": source["challenge_slug"], + "registry_revision": source["registry_revision"], "architecture": source["architecture"], + "artifact_sha256": checksum, "source_sha": source["source_sha"], + "containers": source["containers"], + "execution_settings_sha256": canonical_hash({"containers": sorted(effective, key=lambda item: item["name"]), + "healthcheck": healthcheck, "resource_profile": resources})} + + +def prepare_transition(current_bundle, current_checksum, candidate_bundle, candidate_checksum, + snapshot, *, current_release_id, candidate_release_id, target_ids, + backend_observed_at, as_of, candidate_readiness=None, direction="update"): + now_ns, observed_ns = _time_ns(as_of), _time_ns(backend_observed_at) + require(0 <= now_ns - observed_ns < MAX_SNAPSHOT_AGE_SECONDS * 10**9, "BACKEND_SNAPSHOT_NOT_FRESH") + _validate_targets(target_ids) + canonical_uuid(current_release_id) + canonical_uuid(candidate_release_id) + require(current_release_id != candidate_release_id, "DISTINCT_A_AND_B_REQUIRED") + require(isinstance(snapshot, dict) and {"challenge_id", "current_release_id", "activation", "releases"} <= set(snapshot)) + challenge_id = canonical_uuid(snapshot.get("challenge_id")) + require(snapshot.get("current_release_id") == current_release_id, "ACTIVE_RELEASE_CHANGED") + activation = snapshot.get("activation") + require(activation is None or (isinstance(activation, dict) + and activation.get("status") in {"ACTIVATED", "SUPERSEDED", "FAILED"}), + "EXISTING_ACTIVATION_REQUIRES_REVIEW") + rows = snapshot.get("releases") + require(isinstance(rows, list) and 2 <= len(rows) <= 1000) + by_id = {} + for row in rows: + require(isinstance(row, dict)) + key = canonical_uuid(row.get("release_id")) + require(key not in by_id) + by_id[key] = row + require(current_release_id in by_id and candidate_release_id in by_id) + a = checked_release(current_bundle, current_checksum, by_id[current_release_id], challenge_id, + current=True, as_of_ns=now_ns) + b = checked_release(candidate_bundle, candidate_checksum, by_id[candidate_release_id], challenge_id, + current=False, as_of_ns=now_ns) + require(a["challenge_slug"] == b["challenge_slug"] and a["architecture"] == b["architecture"], + "SAME_PROBLEM_AND_ARCHITECTURE_REQUIRED") + require(direction in ("update", "rollback")) + require((direction == "update" and b["registry_revision"] > a["registry_revision"]) + or (direction == "rollback" and b["registry_revision"] < a["registry_revision"]), + "EXPLICIT_UPDATE_OR_ROLLBACK_DIRECTION_REQUIRED") + readiness = None + if candidate_readiness is not None: + readiness = audit_readiness(candidate_bundle, candidate_checksum, candidate_readiness, + target_ids, as_of=as_of) + ready = readiness is not None and readiness["all_reported_eligible_nodes_ready"] + result = { + "schema_version": "1.0", "scope": "OFFLINE_WEB_RELEASE_TRANSITION_PREPARATION", + "as_of": as_of, "backend_observed_at_as_attested": backend_observed_at, + "direction": direction, "status": "READY_FOR_REPRESENTATIVE_QA" if ready else "IMAGE_READINESS_REQUIRED", + "challenge_id": challenge_id, "current": a, "candidate": b, "target_ids": target_ids, + "candidate_readiness": readiness, + "execution_settings_changed": a["execution_settings_sha256"] != b["execution_settings_sha256"], + "preserve_images": sorted({item["image"] for release in (a, b) for item in release["containers"]}), + "preparation_handoff": {"method": "POST", "path": f"/api/v1/admin/challenges/{challenge_id}/releases/{candidate_release_id}/image_preparations", + "body": {}, "idempotency_key": "OPERATOR_MUST_CHOOSE_AND_RECORD", + "call_performed": False, "requires_operator_approval": True}, + "representative_test_handoff": {"method": "POST", "path": f"/api/v1/admin/challenges/{challenge_id}/releases/{candidate_release_id}/deployment_tests", + "body": {"request_id": "OPERATOR_MUST_CHOOSE_AND_RECORD_CANONICAL_UUID"}, + "call_performed": False, "requires_operator_approval": True, + "traffic_switch": False}, + "current_platform_behavior": "EXISTING_A_REMAINS_AND_NEW_INSTANCES_USE_SELECTED_RELEASE", + "live_instance_handover": {"status": "BLOCKED_PENDING_CONSUMER_CONTRACTS", "gaps": HANDOVER_GAPS}, + "limits": {"source_provenance_live_revalidated": False, "backend_live_queried": False, + "runtime_live_queried": False, "image_pulled": False, "pod_created": False, + "runtime_uid_storage_network_policy_verified": False, + "activation_requested": False, "instance_replaced": False, + "automatic_switch_allowed": False, "zero_downtime_proven": False}, + } + result["plan_sha256"] = plan_hash(result) + return result + + +def write_plan(current_path, candidate_path, snapshot_path, output_dir, *, + current_release_id, candidate_release_id, target_ids, backend_observed_at, + as_of=None, readiness_path=None, direction="update"): + current_raw, current = _read_json(current_path) + candidate_raw, candidate = _read_json(candidate_path) + # Hash the exact immutable bytes used by the parser, not a JSON reserialization. + current_sha = hashlib.sha256(current_raw).hexdigest() + candidate_sha = hashlib.sha256(candidate_raw).hexdigest() + result = prepare_transition(current, current_sha, candidate, candidate_sha, read_json(snapshot_path), + current_release_id=current_release_id, candidate_release_id=candidate_release_id, + target_ids=target_ids, backend_observed_at=backend_observed_at, + as_of=as_of or datetime.now(timezone.utc).isoformat(), + candidate_readiness=read_json(readiness_path) if readiness_path else None, direction=direction) + result["time_basis"] = "EXPLICIT_REPLAY_TIME" if as_of else "SYSTEM_UTC_NOW" + result["plan_sha256"] = plan_hash(result) + output = Path(output_dir) + output.mkdir(mode=0o700, parents=True, exist_ok=False) + save_private(output / "transition-plan.json", result) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--current-artifact", type=Path, required=True) + parser.add_argument("--candidate-artifact", type=Path, required=True) + parser.add_argument("--backend-releases", type=Path, required=True) + parser.add_argument("--current-release-id", required=True) + parser.add_argument("--candidate-release-id", required=True) + parser.add_argument("--backend-observed-at", required=True) + parser.add_argument("--target-id", action="append", required=True) + parser.add_argument("--candidate-readiness", type=Path) + parser.add_argument("--direction", choices=("update", "rollback"), default="update") + parser.add_argument("--as-of", help="Replay only; never an execution authorization") + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + try: + result = write_plan(args.current_artifact, args.candidate_artifact, args.backend_releases, + args.output_dir, current_release_id=args.current_release_id, + candidate_release_id=args.candidate_release_id, target_ids=args.target_id, + backend_observed_at=args.backend_observed_at, as_of=args.as_of, + readiness_path=args.candidate_readiness, direction=args.direction) + except (ValueError, OSError, TypeError, KeyError, OverflowError, RecursionError): + print("Transition preparation failed: check source, release selection, settings and observation time.", file=sys.stderr) + return 2 + print(json.dumps({"status": result["status"], "plan_sha256": result["plan_sha256"], + "automatic_switch_allowed": False, "zero_downtime_proven": False})) + return 0 if result["status"] == "READY_FOR_REPRESENTATIVE_QA" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/README.md b/tests/README.md index 5edd259..3243f08 100644 --- a/tests/README.md +++ b/tests/README.md @@ -5,6 +5,14 @@ ## 테스트 구성 +### `test_prepare_release_transition.py` / `test_observe_rollout_http.py` + +A/B 전체 이미지·Backend active/승인·실행 설정·관측 만료·명시적 rollback과 API +인계안의 경계를 검증합니다. HTTP 표본 실패·중단·TLS·Host/SNI·public DNS 고정· +redirect/metadata/proxy 차단, 비공개 체크포인트와 실제 로컬 GET도 검사합니다. +운영 API·SSH·pull·Pod·active·트래픽 전환을 실행하지 않으며 HTTP200을 무중단 +성공으로 처리하지 않는지 확인합니다. + ### `test_vm_image_prefetch.py` 운영 선택과 원본 checksum·전체 이미지·architecture·노드 범위를 대조하고, 만료· diff --git a/tests/test_observe_rollout_http.py b/tests/test_observe_rollout_http.py new file mode 100644 index 0000000..7271d44 --- /dev/null +++ b/tests/test_observe_rollout_http.py @@ -0,0 +1,274 @@ +import copy +import http.client +import http.server +import io +import json +from pathlib import Path +import socket +import ssl +import tempfile +import threading +import unittest +from contextlib import redirect_stdout +from unittest.mock import patch + +from scripts import observe_rollout_http as observer +from scripts.vm_image_prefetch import save_private +from tests.test_prepare_release_transition import prepare + + +def public_resolver(host, port, **kwargs): + return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("8.8.8.8", port))] + + +class Clock: + def __init__(self): + self.value = 0 + + def monotonic(self): + return self.value + + def sleep(self, seconds): + self.value += seconds + + +def observe(*, replies=None, duration_seconds=3, interval_seconds=1, request=None, **kwargs): + clock = kwargs.pop("clock", Clock()) + replies = replies or [{"http_status": 200, "error_code": None}] + calls = [] + def call(endpoint, timeout): + calls.append((copy.deepcopy(endpoint), timeout, clock.value)) + clock.value += 0.1 + return copy.deepcopy(replies[min(len(calls) - 1, len(replies) - 1)]) + result = observer.observe("https://example.org/health", kwargs.pop("plan", prepare()), + duration_seconds=duration_seconds, interval_seconds=interval_seconds, + request=request or call, resolver=public_resolver, + monotonic=clock.monotonic, sleep=clock.sleep, **kwargs) + return result, calls + + +class RolloutHTTPTests(unittest.TestCase): + def test_passing_samples_never_claim_zero_downtime_or_b_identity(self): + result, calls = observe() + self.assertEqual(result["status"], "NO_HTTP_FAILURE_OBSERVED") + self.assertEqual(result["sample_count"], 3) + self.assertEqual([row[2] for row in calls], [0, 1, 2]) + self.assertEqual(result["failure_count"], 0) + self.assertTrue(all(value is False for value in result["limits"].values())) + self.assertNotIn("example.org", json.dumps(result)) + self.assertNotIn("/health", json.dumps(result)) + + def test_http_failures_are_recorded_and_observation_continues(self): + result, _ = observe(replies=[{"http_status": 200, "error_code": None}, + {"http_status": 503, "error_code": None}, {"http_status": 200, "error_code": None}]) + self.assertEqual(result["status"], "HTTP_FAILURE_OBSERVED") + self.assertEqual(result["sample_count"], 3) + self.assertEqual(result["failure_count"], 1) + self.assertFalse(result["samples"][1]["passed"]) + + def test_timeout_tls_and_network_errors_are_failure_not_success(self): + for error in ("REQUEST_TIMEOUT", "TLS_FAILURE", "NETWORK_OR_PROTOCOL_FAILURE"): + with self.subTest(error=error): + result, _ = observe(replies=[{"http_status": None, "error_code": error}]) + self.assertEqual(result["failure_count"], 3) + self.assertEqual(result["status"], "HTTP_FAILURE_OBSERVED") + + def test_slow_requests_do_not_cause_catch_up_bursts(self): + clock, starts = Clock(), [] + def slow(endpoint, timeout): + starts.append(clock.value) + clock.value += 2.1 + return {"http_status": 200, "error_code": None} + result, _ = observe(clock=clock, request=slow, duration_seconds=5) + self.assertEqual(starts, [0, 3]) + self.assertGreaterEqual(result["elapsed_seconds"], 5) + self.assertFalse(result["limits"]["between_sample_availability_proven"]) + + def test_interrupt_persists_samples_and_never_succeeds(self): + clock, count, saved = Clock(), 0, [] + def interrupted(endpoint, timeout): + nonlocal count + count += 1 + if count == 2: + raise KeyboardInterrupt() + clock.value += 0.1 + return {"http_status": 200, "error_code": None} + result, _ = observe(clock=clock, request=interrupted, + on_result=lambda value: saved.append(copy.deepcopy(value))) + self.assertEqual(result["status"], "INTERRUPTED") + self.assertEqual(result["sample_count"], 1) + self.assertEqual(saved[-1]["status"], "INTERRUPTED") + + def test_unexpected_reply_is_not_copied_into_report(self): + saved = [] + with self.assertRaises(ValueError): + observe(replies=[{"http_status": 200, "error_code": None, "body": "private-response"}], + on_result=lambda value: saved.append(copy.deepcopy(value))) + self.assertEqual(saved[-1]["status"], "OBSERVER_FAILED") + self.assertNotIn("private-response", json.dumps(saved)) + + def test_malformed_status_or_error_is_rejected(self): + for reply in ({"http_status": True, "error_code": None}, + {"http_status": None, "error_code": None}, + {"http_status": 200, "error_code": "TLS_FAILURE"}, + {"http_status": None, "error_code": "private-error-text"}): + with self.subTest(reply=reply), self.assertRaises(ValueError): + observe(replies=[reply]) + + def test_tampered_plan_does_not_send_a_request(self): + plan = prepare() + plan["candidate"]["registry_revision"] += 1 + with self.assertRaises(ValueError): + observe(plan=plan, request=lambda *_: self.fail("must not request")) + + def test_request_budget_and_invalid_numeric_limits_are_rejected(self): + for kwargs in ({"duration_seconds": 901}, {"duration_seconds": True}, + {"duration_seconds": 100, "interval_seconds": 0.25}, + {"interval_seconds": float("nan")}, {"timeout_seconds": 0}, + {"timeout_seconds": float("inf")}, {"expected_status": 302}): + with self.subTest(kwargs=kwargs), self.assertRaises(ValueError): + observe(**kwargs) + + def test_an_explicit_204_health_endpoint_is_supported(self): + result, _ = observe(expected_status=204, replies=[{"http_status": 204, "error_code": None}]) + self.assertEqual(result["status"], "NO_HTTP_FAILURE_OBSERVED") + + def test_https_default_and_explicit_http_only(self): + with self.assertRaises(ValueError): + observer.validate_endpoint("http://example.org/", resolver=public_resolver) + target = observer.validate_endpoint("http://example.org:8080/", allow_http=True, resolver=public_resolver) + self.assertEqual(target["scheme"], "http") + self.assertEqual(target["port"], 8080) + + def test_credentials_queries_fragments_and_unsafe_paths_are_rejected(self): + for url in ("https://user:password@example.org/", "https://example.org/?token=private", + "https://example.org/#fragment", "https://example.org/?", "https://example.org/#", + "https://example.org/../admin", "https://example.org//admin", + "https://example.org/%2e%2e/admin", "https://example.org/a\nheader", + "https://example.org:0/", "ftp://example.org/", "https://example.org/a b"): + with self.subTest(url=url), self.assertRaises(ValueError): + observer.validate_endpoint(url, resolver=public_resolver) + + def test_all_dns_answers_must_be_public_and_metadata_is_forbidden(self): + for address in ("127.0.0.1", "10.0.0.1", "169.254.169.254", "::1", "224.0.0.1"): + def resolver(host, port, **kwargs): + return public_resolver(host, port) + [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (address, port))] + with self.subTest(address=address), self.assertRaises(ValueError): + observer.validate_endpoint("https://example.org/", resolver=resolver) + for url in ("https://localhost/", "https://metadata.google.internal/", "https://service.local/"): + with self.subTest(url=url), self.assertRaises(ValueError): + observer.validate_endpoint(url, resolver=public_resolver) + + def test_socket_is_pinned_and_tls_checks_original_hostname(self): + endpoint = observer.validate_endpoint("https://example.org/", resolver=public_resolver) + connection = observer.PinnedConnection(endpoint, 2) + self.assertTrue(connection.context.check_hostname) + self.assertEqual(connection.context.verify_mode, ssl.CERT_REQUIRED) + with patch("socket.create_connection") as connect, patch.object(connection.context, "wrap_socket") as wrap: + connection.connect() + connect.assert_called_once_with(("8.8.8.8", 443), 2) + wrap.assert_called_once_with(connect.return_value, server_hostname="example.org") + connection.sock = None + + def test_tls_wrapping_failure_closes_raw_socket(self): + endpoint = observer.validate_endpoint("https://example.org/", resolver=public_resolver) + connection = observer.PinnedConnection(endpoint, 2) + with patch("socket.create_connection") as connect, \ + patch.object(connection.context, "wrap_socket", side_effect=ssl.SSLError("private-detail")): + with self.assertRaises(ssl.SSLError): + connection.connect() + connect.return_value.close.assert_called_once() + + def test_probe_discards_body_headers_and_does_not_follow_redirect(self): + endpoint = observer.validate_endpoint("https://example.org/", resolver=public_resolver) + with patch.object(observer, "PinnedConnection") as factory: + connection = factory.return_value + connection.getresponse.return_value.status = 302 + result = observer.probe(endpoint, 2) + self.assertEqual(result, {"http_status": 302, "error_code": None}) + connection.request.assert_called_once() + args, kwargs = connection.request.call_args + self.assertEqual(args, ("GET", "/")) + self.assertFalse({"Authorization", "Cookie", "Proxy-Authorization"}.intersection(kwargs["headers"])) + connection.getresponse.return_value.read.assert_not_called() + connection.getresponse.return_value.getheader.assert_not_called() + connection.close.assert_called_once() + + def test_probe_redacts_transport_errors_and_closes_connection(self): + endpoint = observer.validate_endpoint("https://example.org/", resolver=public_resolver) + for error, code in ((ssl.SSLError("private-token"), "TLS_FAILURE"), + (TimeoutError("private-token"), "REQUEST_TIMEOUT"), + (http.client.BadStatusLine("private-token"), "NETWORK_OR_PROTOCOL_FAILURE")): + with self.subTest(code=code), patch.object(observer, "PinnedConnection") as factory: + factory.return_value.request.side_effect = error + result = observer.probe(endpoint, 2) + self.assertEqual(result["error_code"], code) + self.assertNotIn("private-token", json.dumps(result)) + factory.return_value.close.assert_called_once() + + def test_private_checkpoints_do_not_contain_endpoint_or_body(self): + with tempfile.TemporaryDirectory() as temporary: + target = Path(temporary) / "http-observation.json" + result, _ = observe(on_result=lambda value: save_private(target, value)) + saved = json.loads(target.read_text()) + self.assertEqual(saved, result) + self.assertEqual(target.stat().st_mode & 0o777, 0o600) + self.assertNotIn("example.org", target.read_text()) + + def test_cli_writes_private_result_without_network_or_url_output(self): + real_observe = observer.observe + clock = Clock() + def fake_observe(url, plan, **kwargs): + def request(endpoint, timeout): + clock.value += 0.1 + return {"http_status": 200, "error_code": None} + return real_observe(url, plan, resolver=public_resolver, request=request, + monotonic=clock.monotonic, sleep=clock.sleep, **kwargs) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + plan = root / "plan.json" + plan.write_text(json.dumps(prepare())) + output, stdout = root / "observation", io.StringIO() + args = ["observe", "--plan", str(plan), "--url", "https://example.org/health", + "--duration-seconds", "2", "--output-dir", str(output)] + with patch("sys.argv", args), patch.object(observer, "observe", side_effect=fake_observe), redirect_stdout(stdout): + code = observer.main() + self.assertEqual(code, 0) + self.assertEqual(output.stat().st_mode & 0o777, 0o700) + self.assertEqual((output / "http-observation.json").stat().st_mode & 0o777, 0o600) + self.assertNotIn("example.org", stdout.getvalue()) + self.assertFalse(json.loads(stdout.getvalue())["zero_downtime_proven"]) + + def test_local_transport_sends_only_get_and_does_not_use_proxy_or_redirect(self): + seen = [] + class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + seen.append((self.path, dict(self.headers))) + self.send_response(302 if self.path == "/redirect" else 200) + self.send_header("Location", "http://127.0.0.1/should-not-follow") + self.end_headers() + + def log_message(self, *args): + pass + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + # Transport only: the production URL validator rejects this private address. + endpoint = {"host": "127.0.0.1", "address": "127.0.0.1", "port": server.server_port, + "scheme": "http", "path": "/redirect"} + with patch.dict("os.environ", {"HTTP_PROXY": "http://private-proxy.invalid:1"}): + result = observer.probe(endpoint, 2) + self.assertEqual(result["http_status"], 302) + self.assertEqual(len(seen), 1) + self.assertEqual(seen[0][0], "/redirect") + self.assertNotIn("Authorization", seen[0][1]) + self.assertNotIn("Cookie", seen[0][1]) + finally: + server.shutdown() + server.server_close() + thread.join(timeout=2) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_prepare_release_transition.py b/tests/test_prepare_release_transition.py new file mode 100644 index 0000000..b43a853 --- /dev/null +++ b/tests/test_prepare_release_transition.py @@ -0,0 +1,346 @@ +import copy +import hashlib +import json +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +from scripts import prepare_release_transition as transition +from scripts.audit_node_image_readiness import calculate_image_set_id +from tests.test_generate_node_image_plan import sample_artifact + + +ROOT = Path(__file__).resolve().parents[1] +AS_OF = "2026-11-04T00:00:30Z" +OBSERVED = "2026-11-04T00:00:00Z" +A_ID = "00000000-0000-4000-8000-000000000001" +B_ID = "00000000-0000-4000-8000-000000000002" +CHALLENGE_ID = "00000000-0000-4000-8000-000000000003" + + +def inputs(): + a = sample_artifact() + a.update(isolation_profile="WEB", resource_profile={"cpu_millicores": 500, + "memory_mib": 512, "ephemeral_storage_mib": 1024}) + a["workload"]["healthcheck"] = {"container": "web", "port": 8080, "path": "/health"} + b = copy.deepcopy(a) + b.update(registry_revision=6, revision=6, source_sha="2" * 40) + b["workload"]["containers"][0]["image"] = b["workload"]["containers"][0]["image"].replace("a" * 64, "d" * 64) + b["evidence"]["containers"][0]["published_image"] = b["workload"]["containers"][0]["image"] + def saved(bundle, release_id, current): + return {"release_id": release_id, "challenge_id": CHALLENGE_ID, + "challenge_slug": bundle["challenge_slug"], "registry_revision": bundle["registry_revision"], + "runtime_type": "KUBERNETES", "architecture": "AMD64", "isolation_profile": "WEB", + "source_ref": "refs/heads/main", "is_deployable": True, "is_current": current, + "derived_from_release_id": None, "approved_at": OBSERVED, + "resource_profile": copy.deepcopy(bundle["resource_profile"]), + "healthcheck": copy.deepcopy(bundle["workload"]["healthcheck"]), + "containers": [{"name": row["name"], "image_ref": row["image"], + "ports": copy.deepcopy(row["ports"]), "secret_bindings": [], "env_rules": {}} + for row in bundle["workload"]["containers"]]} + snapshot = {"challenge_id": CHALLENGE_ID, "current_release_id": A_ID, "activation": None, + "releases": [saved(a, A_ID, True), saved(b, B_ID, False)]} + readiness = {"architecture": "AMD64", "image_set_id": calculate_image_set_id("AMD64", + [row["image"] for row in b["workload"]["containers"]]), + "targets": [{"target_id": "target-a", "ready": True, + "ready_node_count": 1, "eligible_node_count": 1, + "observed_at": OBSERVED, "valid_until": "2026-11-04T00:02:00Z"}]} + return a, b, snapshot, readiness + + +def prepare(values=None, **kwargs): + a, b, snapshot, readiness = values or inputs() + return transition.prepare_transition(a, "a" * 64, b, "b" * 64, snapshot, + current_release_id=kwargs.pop("current_release_id", A_ID), + candidate_release_id=kwargs.pop("candidate_release_id", B_ID), + target_ids=kwargs.pop("target_ids", ["target-a"]), + backend_observed_at=kwargs.pop("backend_observed_at", OBSERVED), + as_of=kwargs.pop("as_of", AS_OF), candidate_readiness=readiness, **kwargs) + + +class ReleaseTransitionTests(unittest.TestCase): + def test_full_sets_and_current_are_preserved_without_mutation_or_network(self): + values = inputs() + original = copy.deepcopy(values) + with patch("socket.create_connection", side_effect=AssertionError("network forbidden")), \ + patch("subprocess.Popen", side_effect=AssertionError("process forbidden")): + result = prepare(values) + self.assertEqual(values, original) + self.assertEqual(result["status"], "READY_FOR_REPRESENTATIVE_QA") + self.assertEqual(len(result["preserve_images"]), 3) + self.assertEqual(len(result["current"]["containers"]), 2) + self.assertEqual(result["current"]["release_id"], A_ID) + self.assertEqual(result["plan_sha256"], transition.plan_hash(result)) + self.assertTrue(all(value is False for value in result["limits"].values())) + self.assertEqual(result["live_instance_handover"]["status"], "BLOCKED_PENDING_CONSUMER_CONTRACTS") + + def test_exact_existing_preparation_path_never_generates_activate_or_replace(self): + handoff = prepare()["preparation_handoff"] + self.assertEqual(handoff["method"], "POST") + self.assertEqual(handoff["path"], f"/api/v1/admin/challenges/{CHALLENGE_ID}/releases/{B_ID}/image_preparations") + self.assertEqual(handoff["body"], {}) + self.assertFalse(handoff["call_performed"]) + self.assertTrue(handoff["requires_operator_approval"]) + self.assertNotIn("/activate", json.dumps(prepare())) + self.assertNotIn("replace_instance_id", json.dumps(prepare())) + test_handoff = prepare()["representative_test_handoff"] + self.assertEqual(test_handoff["path"], f"/api/v1/admin/challenges/{CHALLENGE_ID}/releases/{B_ID}/deployment_tests") + self.assertEqual(set(test_handoff["body"]), {"request_id"}) + self.assertFalse(test_handoff["call_performed"]) + self.assertFalse(test_handoff["traffic_switch"]) + + def test_missing_readiness_is_a_plan_not_permission_to_switch(self): + a, b, snapshot, _ = inputs() + result = prepare((a, b, snapshot, None)) + self.assertEqual(result["status"], "IMAGE_READINESS_REQUIRED") + self.assertIsNone(result["candidate_readiness"]) + self.assertFalse(result["limits"]["automatic_switch_allowed"]) + + def test_partial_missing_and_expired_readiness_stay_blocked(self): + for change in ({"eligible_node_count": 2}, {"valid_until": AS_OF}, + {"observed_at": None}, {"target_id": "unexpected"}): + values = inputs() + values[3]["targets"][0].update(change) + with self.subTest(change=change): + if "target_id" in change: + with self.assertRaises(ValueError): + prepare(values) + else: + self.assertEqual(prepare(values)["status"], "IMAGE_READINESS_REQUIRED") + + def test_a_readiness_cannot_be_used_for_b(self): + values = inputs() + values[3]["image_set_id"] = calculate_image_set_id("AMD64", [row["image"] for row in values[0]["workload"]["containers"]]) + with self.assertRaises(ValueError): + prepare(values) + + def test_changed_active_and_same_release_are_rejected(self): + values = inputs() + values[2]["current_release_id"] = B_ID + with self.assertRaises(ValueError): + prepare(values) + with self.assertRaises(ValueError): + prepare(candidate_release_id=A_ID) + + def test_old_future_and_boundary_snapshot_are_rejected(self): + for observed in ("2026-11-03T23:59:30Z", "2026-11-04T00:00:31Z", "2026-11-03T00:00:00Z"): + with self.subTest(observed=observed), self.assertRaises(ValueError): + prepare(backend_observed_at=observed) + + def test_pending_and_unknown_activation_are_rejected_but_history_is_allowed(self): + for status in ("PENDING", "UNKNOWN", "ACTIVATED", "SUPERSEDED", "FAILED"): + values = inputs() + values[2]["activation"] = {"status": status} + with self.subTest(status=status): + if status in ("PENDING", "UNKNOWN"): + with self.assertRaises(ValueError): + prepare(values) + else: + self.assertEqual(prepare(values)["status"], "READY_FOR_REPRESENTATIVE_QA") + + def test_duplicate_release_ids_and_container_names_are_rejected(self): + for duplicate in ("release", "container"): + values = inputs() + if duplicate == "release": + values[2]["releases"].append(copy.deepcopy(values[2]["releases"][0])) + else: + values[2]["releases"][1]["containers"][1] = copy.deepcopy(values[2]["releases"][1]["containers"][0]) + with self.subTest(duplicate=duplicate), self.assertRaises(ValueError): + prepare(values) + + def test_candidate_approval_and_deployability_are_required(self): + for change in ({"approved_at": None}, {"approved_at": "2026-11-05T00:00:00Z"}, + {"is_deployable": False}, {"is_current": True}, {"registry_revision": True}): + values = inputs() + values[2]["releases"][1].update(change) + with self.subTest(change=change), self.assertRaises(ValueError): + prepare(values) + + def test_wrong_problem_architecture_or_backend_revision_are_rejected(self): + for key, value in (("challenge_id", A_ID), ("challenge_slug", "other-problem"), + ("architecture", "ARM64"), ("registry_revision", 7), + ("source_ref", "refs/heads/feature"), ("runtime_type", "DOCKER")): + values = inputs() + values[2]["releases"][1][key] = value + with self.subTest(key=key), self.assertRaises(ValueError): + prepare(values) + + def test_unscanned_tagged_pwn_vm_and_koth_are_rejected(self): + for change in ({"scan_result": "FAIL"}, {"isolation_profile": "PWN"}, + {"runtime_type": "VM"}, {"category": "koth"}): + values = inputs() + values[1].update(change) + with self.subTest(change=change), self.assertRaises(ValueError): + prepare(values) + values = inputs() + values[1]["workload"]["containers"][0]["image"] = "ghcr.io/msg-ctf/challenges/web-notebook/web:latest" + with self.assertRaises(ValueError): + prepare(values) + + def test_private_container_omission_and_wrong_digest_are_rejected(self): + for mode in ("missing", "digest"): + values = inputs() + if mode == "missing": + values[2]["releases"][1]["containers"].pop() + else: + values[2]["releases"][1]["containers"][1]["image_ref"] = "incorrect" + with self.subTest(mode=mode), self.assertRaises(ValueError): + prepare(values) + + def test_port_visibility_boolean_duplicates_and_health_loss_are_rejected(self): + for mode in ("public", "bool", "duplicate", "health"): + values = inputs() + saved = values[2]["releases"][1] + if mode == "health": + saved["healthcheck"] = None + elif mode == "duplicate": + saved["containers"][0]["ports"] *= 2 + else: + saved["containers"][0]["ports"][0]["public" if mode == "public" else "port"] = False + with self.subTest(mode=mode), self.assertRaises(ValueError): + prepare(values) + + def test_resource_and_environment_drift_are_rejected(self): + for mode in ("resource", "env"): + values = inputs() + saved = values[2]["releases"][1] + if mode == "resource": + saved["resource_profile"]["memory_mib"] = 1024 + else: + saved["containers"][0]["env"] = {"APP_MODE": "unexpected"} + with self.subTest(mode=mode), self.assertRaises(ValueError): + prepare(values) + + def test_schema_21_settings_are_fingerprinted_without_exporting_values(self): + values = inputs() + values[1]["schema_version"] = "2.1" + values[1]["workload"]["containers"][0].update(env={"APP_MODE": "private-app-value"}, secret_env={"FLAG": "flag"}) + saved = values[2]["releases"][1]["containers"][0] + saved.update(env={"APP_MODE": "private-app-value"}, secret_ref=CHALLENGE_ID, + secret_bindings=[{"env_name": "FLAG", "name": "flag", "version": 2, "status": "registered"}]) + result = prepare(values) + self.assertTrue(result["execution_settings_changed"]) + for value in ("private-app-value", "secret_bindings", "secret_ref"): + self.assertNotIn(value, json.dumps(result)) + saved["secret_bindings"][0]["status"] = "missing" + with self.assertRaises(ValueError): + prepare(values) + + def test_unmodeled_derived_release_and_environment_rules_require_review(self): + for mode in ("derived", "rules"): + values = inputs() + saved = values[2]["releases"][1] + if mode == "derived": + saved["derived_from_release_id"] = A_ID + else: + saved["containers"][0]["env_rules"] = {"FLAG": {"kind": "unknown"}} + with self.subTest(mode=mode), self.assertRaises(ValueError): + prepare(values) + + def test_unmodeled_workload_and_container_fields_cannot_be_ignored(self): + for key in ("storage", "network_policy", "internal_connections", "command", "run_as_user"): + values = inputs() + target = values[1]["workload"] if key in ("storage", "network_policy", "internal_connections") else values[1]["workload"]["containers"][0] + target[key] = "unmodeled" + with self.subTest(key=key), self.assertRaises(ValueError): + prepare(values) + + def test_execution_settings_require_schema_21(self): + values = inputs() + values[1]["workload"]["containers"][0]["env"] = {"APP_MODE": "private-value"} + values[2]["releases"][1]["containers"][0]["env"] = {"APP_MODE": "private-value"} + with self.assertRaises(ValueError): + prepare(values) + + def test_missing_backend_contract_fields_cannot_be_assumed_safe(self): + for field in ("activation", "derived_from_release_id"): + values = inputs() + if field == "activation": + del values[2][field] + else: + del values[2]["releases"][1][field] + with self.subTest(field=field), self.assertRaises(ValueError): + prepare(values) + + def test_secrets_cannot_appear_as_plain_env_or_undeclared_binding(self): + for change in ({"env": {"FLAG": "private-flag"}}, {"secret_ref": A_ID}): + values = inputs() + values[2]["releases"][1]["containers"][0].update(change) + with self.subTest(change=change), self.assertRaises(ValueError): + prepare(values) + + def test_explicit_rollback_and_no_implicit_downgrade(self): + a, b, snapshot, _ = inputs() + snapshot["current_release_id"] = B_ID + snapshot["releases"][0]["is_current"] = False + snapshot["releases"][1]["is_current"] = True + with self.assertRaises(ValueError): + prepare((b, a, snapshot, None), current_release_id=B_ID, candidate_release_id=A_ID) + result = prepare((b, a, snapshot, None), current_release_id=B_ID, candidate_release_id=A_ID, direction="rollback") + self.assertEqual(result["direction"], "rollback") + self.assertEqual(result["candidate"]["release_id"], A_ID) + + def test_targets_and_reordering_are_handled_without_losing_private_images(self): + values = inputs() + expected = prepare(values) + values[2]["releases"].reverse() + for row in values[2]["releases"]: + row["containers"].reverse() + self.assertEqual(prepare(values), expected) + with self.assertRaises(ValueError): + prepare(target_ids=["target-a", "target-a"]) + + def test_private_files_exact_raw_checksum_and_no_overwrite(self): + a, b, snapshot, readiness = inputs() + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + paths = [root / name for name in ("a.json", "b.json", "backend.json", "readiness.json")] + for path, value in zip(paths, (a, b, snapshot, readiness)): + path.write_text(json.dumps(value, indent=4) + "\n") + output = root / "private" + kwargs = dict(current_release_id=A_ID, candidate_release_id=B_ID, target_ids=["target-a"], + backend_observed_at=OBSERVED, as_of=AS_OF, readiness_path=paths[3]) + report = transition.write_plan(*paths[:3], output, **kwargs) + self.assertEqual(report["current"]["artifact_sha256"], hashlib.sha256(paths[0].read_bytes()).hexdigest()) + self.assertEqual(output.stat().st_mode & 0o777, 0o700) + self.assertEqual((output / "transition-plan.json").stat().st_mode & 0o777, 0o600) + self.assertEqual(report["time_basis"], "EXPLICIT_REPLAY_TIME") + with self.assertRaises(FileExistsError): + transition.write_plan(*paths[:3], output, **kwargs) + + def test_cli_error_does_not_print_path_or_payload(self): + command = [sys.executable, "-B", "-m", "scripts.prepare_release_transition", + "--current-artifact", "/private-operator-source/missing", "--candidate-artifact", "/missing", + "--backend-releases", "/missing", "--current-release-id", A_ID, "--candidate-release-id", B_ID, + "--backend-observed-at", OBSERVED, "--target-id", "target-a", "--output-dir", "/unused"] + result = subprocess.run(command, cwd=ROOT, text=True, capture_output=True) + self.assertEqual(result.returncode, 2) + self.assertNotIn("private-operator-source", result.stderr) + + def test_cli_ready_and_missing_readiness_have_distinct_exit_codes(self): + values = inputs() + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + paths = [root / name for name in ("a.json", "b.json", "backend.json", "ready.json")] + for path, value in zip(paths, values): + path.write_text(json.dumps(value)) + command = [sys.executable, "-B", "-m", "scripts.prepare_release_transition", + "--current-artifact", str(paths[0]), "--candidate-artifact", str(paths[1]), + "--backend-releases", str(paths[2]), "--current-release-id", A_ID, + "--candidate-release-id", B_ID, "--backend-observed-at", OBSERVED, + "--as-of", AS_OF, "--target-id", "target-a"] + waiting = subprocess.run(command + ["--output-dir", str(root / "waiting")], + cwd=ROOT, text=True, capture_output=True) + self.assertEqual(waiting.returncode, 1) + self.assertEqual(json.loads(waiting.stdout)["status"], "IMAGE_READINESS_REQUIRED") + ready = subprocess.run(command + ["--candidate-readiness", str(paths[3]), + "--output-dir", str(root / "ready")], cwd=ROOT, text=True, capture_output=True) + self.assertEqual(ready.returncode, 0, ready.stderr) + self.assertFalse(json.loads(ready.stdout)["automatic_switch_allowed"]) + + +if __name__ == "__main__": + unittest.main()