Phase 4: Integrate HashiCorp Vault for application secrets and dynamic PostgreSQL credentials

Type: Security / Infrastructure
Priority: Critical
Parent: PostgreSQL & Vault Migration
Depends on: Phase 1 and PostgreSQL role/schema design from Phase 2

Objective

Replace insecure or manually managed runtime secrets with a service-scoped Vault integration that supports safe credential acquisition, renewal, rotation, revocation, and failure recovery.

Tasks

Vault Infrastructure

• Select and pin a supported self-hosted Vault edition.
• Configure a persistent non-development Vault environment.
• Enable TLS and certificate verification.
• Configure protected persistent storage.
• Define initialization, unseal, backup, and recovery procedures.
• Enable suitable Vault audit logging.
• Provide isolated disposable development/test configuration where useful.

Authentication and Authorization

• Define a separate Vault identity per service and environment.
• Configure least-privilege service policies.
• Implement the selected service authentication method.
• Define secure bootstrap credential delivery.
• Prevent application use of root tokens.
• Test cross-service access denial.
• Define token renewal and re-authentication behavior.

PostgreSQL Dynamic Credentials

• Configure the PostgreSQL database secrets engine.
• Create independently scoped database roles per service.
• Separate Vault credential-management privileges from runtime privileges.
• Define credential TTL, maximum TTL, and revocation policy.
• Integrate credentials into the application DataSource.
• Handle lease renewal and replacement.
• Coordinate credential replacement with HikariCP.
• Retire outdated pooled connections safely.
• Handle in-flight transactions and revocation limitations.
• Test credential rotation without depending solely on process restart.

Application Secrets

• Configure Vault KV v2 for appropriate application secrets.
• Securely provision the OTP HMAC pepper.
• Remove insecure production-like fallback secrets.
• Define HMAC pepper versioning and rotation compatibility.
• Preserve verification of still-valid challenges during key transitions.
• Review any remaining authenticator-secret port only if retained by the approved architecture.
• Prevent sensitive data from entering logs or ordinary diagnostics.

Failure Handling

• Handle Vault unavailability.
• Handle missing or denied secrets.
• Configure bounded retry and recovery.
• Define fail-closed startup/readiness behavior.
• Prevent fallback to hardcoded credentials.
• Test token expiration and database credential revocation.

Acceptance Criteria

• Applications obtain database credentials from Vault.
• Service-specific Vault isolation is enforced.
• Secrets are not committed or exposed in logs.
• Credential lifecycle works with the real connection pool.
• Startup, renewal, revocation, and outage behavior are documented and tested.
• No paid-only Vault feature is required by the chosen implementation.

Definition of Done

Vault integration works end to end and satisfies the required credential lifecycle and security controls.