Skip to main content

Documentation Changelog

Recent changes to Prism, listed in reverse chronological order.

November 2025​

2025-11-19​

Pattern Proxy Integration Tests with Proper Layer Isolation ✅​

Branch: feature/vault-integration-phase2

Summary: Implemented proper integration tests that validate complete data flow through prism-proxy using pattern-level gRPC APIs. Tests connect to proxy (NOT backends directly), ensuring proper layer isolation and security. All backend services (Kafka, NATS, Redis, PostgreSQL) have no external port mappings - access is exclusively through prism-proxy.

Key Architectural Fix: Previous approach bypassed the proxy and connected tests directly to backends. This violated production architecture where backends are internal-only. New tests properly validate the entire stack: Test Client → Proxy (gRPC) → Pattern Runner → Pattern → Driver → Backend.

Security Model:

  • Backend Isolation: No external ports on Kafka (9092), NATS (4222), Redis (6379), PostgreSQL (5432)
  • Single Entry Point: Only prism-proxy exposed via localhost:50090 (gRPC), localhost:50091 (metrics)
  • Internal Networks: Backends accessible only within Docker network
  • Debugging: Use Prometheus metrics and container exec, not direct backend access

Test Architecture:

Test gRPC Client
    ↓
prism-proxy:50090 (ONLY exposed port)
    ↓ routes request
Pattern Runner (internal)
    ↓ executes pattern
Driver (internal)
    ↓ connects to backend
Backend (internal, no external ports)

Test Scenarios:

1. KeyValue Round-Trip with Random Data

  • Generate random test data (UUID, timestamp, random int)
  • Store via KeyValue.Store() gRPC call to proxy
  • Retrieve via KeyValue.Retrieve() gRPC call to proxy
  • Validate exact field-by-field match
  • Proves: Data integrity through Client → Proxy → Pattern → Redis → Pattern → Proxy → Client

2. Producer/Consumer Message Flow

  • Generate unique topic and random message
  • Publish via Producer.Publish() with metadata
  • Consume via Consumer.Consume() with timeout
  • Validate message payload and envelope metadata
  • Proves: Message flow through Client → Proxy → Producer → Kafka → Consumer → Proxy → Client

3. Session Context Propagation

  • Publish message with session context (user_id, tenant_id, roles)
  • Session credentials flow through envelope headers
  • Pattern runners can access session for authorization

Test Execution:

# Start complete stack (proxy waits for backends)
docker-compose -f docker-compose.test.yml up -d

# Verify ONLY proxy is accessible
nc -zv localhost 50090  # ✓ Should succeed
nc -zv localhost 4222   # ✗ Should FAIL (NATS internal)
nc -zv localhost 6379   # ✗ Should FAIL (Redis internal)

# Run integration tests
go test -v ./tests/integration/patterns -run TestPatternProxyIntegration

Infrastructure Changes:

  • docker-compose.test.yml: Removed all ports: mappings from backends
  • Only prism-proxy and prism-admin have external ports
  • Healthchecks run inside containers, not from host

New Files:

  • tests/integration/patterns/pattern_proxy_integration_test.go (238 lines) - Proxy integration test
  • pkg/patterns/common/envelope.go (185 lines) - Universal envelope format
  • pkg/patterns/common/audit.go (175 lines) - Audit logging interfaces

Updated Files:

  • docker-compose.test.yml - Removed external ports from kafka, nats, redis, postgres, localstack
  • tests/integration/patterns/README.md - Updated with proper architecture diagrams and troubleshooting
  • tests/integration/patterns/namespace_integration_test.go - Marked as DEPRECATED (bypassed proxy)

Benefits:

  • Production-Like Testing: Tests mirror actual deployment architecture
  • Security Validation: Proves backends are not externally accessible
  • Complete Stack Coverage: Validates proxy routing, pattern execution, driver operation
  • No False Positives: Random test data ensures proper validation
  • Observability Focus: Debugging via metrics/logs, not direct backend access

Debugging Approach:

# Check proxy health
docker logs prism-proxy

# Verify backend connectivity (from inside container)
docker exec prism-proxy sh -c "nc -zv prism-kafka 9092"

# View metrics endpoint
curl localhost:50091/metrics

# Execute commands in containers
docker exec prism-nats nats server info
docker exec prism-redis redis-cli info

Next: E2E testing with local Vault + Dex for authentication through proxy

2025-11-18​

Phase 3: Pattern Runner Authentication Integration ✅​

Branch: feature/vault-integration-phase2

Summary: Extended authentication support to all three pattern runners (KeyValue, Consumer, Producer). Each pattern runner now supports optional Vault-based dynamic credential management with dual-mode operation (auth enabled/disabled).

Implementation Details:

Consumer Pattern Runner:

  • Added SessionManager and service identity authentication
  • Implemented initializeAuth() method with full Vault integration
  • Updated lifecycle methods (Initialize/Stop) for session management
  • Built successfully: 23MB binary
  • Example configs: consumer-auth-disabled.yaml, consumer-auth-enabled.yaml

Producer Pattern Runner:

  • Refactored to ProducerRunner struct with auth support
  • Created RunnerConfig wrapper for auth configuration
  • Implemented complete auth lifecycle (Initialize/Start/Stop)
  • Built successfully: 28MB binary
  • Example configs: producer-auth-disabled.json, producer-auth-enabled.json

KeyValue Pattern Runner (completed earlier):

  • Per-request authentication with SessionMiddleware
  • gRPC interceptor for JWT validation
  • Session-based credential injection
  • Example configs: keyvalue-auth-disabled.yaml, keyvalue-auth-enabled.yaml

New Files:

  • examples/configs/consumer-auth-disabled.yaml - Consumer without auth
  • examples/configs/consumer-auth-enabled.yaml - Consumer with Vault integration
  • examples/configs/producer-auth-disabled.json - Producer without auth
  • examples/configs/producer-auth-enabled.json - Producer with Vault integration
  • docs-cms/memos/memo-091-local-vault-dex-setup.md - Local development setup guide

Modified Files:

  • patterns/consumer/cmd/consumer-runner/main.go - Added auth support (320+ lines)
  • patterns/consumer/cmd/consumer-runner/go.mod - Added authz dependency
  • patterns/producer/cmd/producer-runner/main.go - Complete refactor with auth (330+ lines)
  • patterns/producer/go.mod - Added authz dependency
  • go.work - Added producer pattern to workspace

Architecture Models:

  • KeyValue: Per-request authentication (users send JWT with each request)
  • Consumer/Producer: Service identity authentication (single session for service lifetime)

Authentication Flow:

Service Startup
    ↓
initializeAuth() → Create TokenValidator + VaultClient + SessionManager
    ↓
Service obtains JWT (from K8s SA or environment)
    ↓
JWT validated → Exchange for Vault token
    ↓
Fetch dynamic credentials (database/creds/role)
    ↓
Backend connections use dynamic credentials
    ↓
Background renewal every lease_duration/2
    ↓
On shutdown: Revoke credentials via Vault

Features:

  • ✅ All 3 patterns support Vault authentication
  • ✅ Dual-mode operation (auth enabled/disabled)
  • ✅ Dynamic credential fetching and renewal
  • ✅ Automatic credential revocation on shutdown
  • ✅ Session caching to avoid repeated Vault calls
  • ✅ TLS support for Vault connections
  • ✅ Configurable session TTL and idle timeout
  • ✅ JWT validation with OIDC provider integration
  • ✅ Support for K8s and AWS service identities

Local Development Setup:

  • MEMO-091 provides complete Docker Compose setup
  • Includes Dex (OIDC), Vault (dev mode), Redis, NATS
  • Step-by-step configuration and testing guide
  • Full end-to-end authentication flow examples

Next: E2E testing with local infrastructure, production operator guide (MEMO-092), troubleshooting guide (MEMO-093)

Phase 2: Vault Integration and Session Management ✅​

Branch: feature/vault-integration-phase2

Summary: Implemented complete Vault integration for dynamic credential management. The pkg/authz package now provides session lifecycle management with automatic credential renewal, JWT token validation with OIDC providers, and Vault client operations.

Core Components:

Session Management (pkg/authz/session_manager.go):

  • CreateSession: JWT → Vault token → Backend credentials
  • GetSession: Retrieve active session by ID
  • CloseSession: Revoke credentials and clean up
  • CleanupExpiredSessions: Background cleanup task
  • Session caching with TTL and idle timeout support

Vault Client (pkg/authz/vault_client.go, vault_credentials.go, vault_renewal.go):

  • AuthenticateWithJWT: Exchange JWT for Vault token
  • GetBackendCredentials: Fetch dynamic credentials from secret paths
  • RenewCredentials: Background renewal worker
  • RevokeCredentials: Explicit lease revocation
  • TLS support with CA cert validation

Token Validation (pkg/authz/token_validator.go):

  • OIDC provider integration with JWKS fetching
  • JWT signature verification
  • Issuer and audience claim validation
  • Expiry checking with configurable skip flags

Service Identity (pkg/authz/k8s_auth.go, aws_auth.go):

  • Kubernetes service account token authentication
  • AWS IAM role authentication
  • Automatic service identity detection

Session Middleware (pkg/plugin/session_middleware.go):

  • gRPC interceptors (unary and streaming)
  • Automatic JWT extraction from metadata
  • Session creation and caching
  • Credential injection into context
  • 32.8% test coverage

New Files:

  • pkg/authz/session_manager.go (350+ lines) - Core session lifecycle
  • pkg/authz/vault_client.go (280+ lines) - Vault SDK wrapper
  • pkg/authz/vault_credentials.go (120+ lines) - Credential operations
  • pkg/authz/vault_renewal.go (150+ lines) - Background renewal
  • pkg/authz/token_validator.go (180+ lines) - JWT validation
  • pkg/authz/k8s_auth.go (90+ lines) - K8s service identity
  • pkg/authz/aws_auth.go (85+ lines) - AWS service identity
  • pkg/plugin/session_middleware.go (200+ lines) - gRPC interceptors
  • docs-cms/memos/memo-083-phase2-vault-integration-plan.md - Implementation plan
  • docs-cms/memos/memo-089-session-middleware-integration-guide.md - Integration guide

Security Features:

  • ✅ JWT signature verification with JWKS
  • ✅ Dynamic credentials from Vault (1h TTL default)
  • ✅ Automatic credential renewal (every 30min default)
  • ✅ Credential revocation on session end
  • ✅ TLS for Vault connections with CA validation
  • ✅ Per-session credential isolation
  • ✅ Token expiry validation
  • ✅ Issuer and audience claim checks

Integration Tests:

  • pkg/authz/integration_test.go - Full authentication flow
  • Uses Vault testcontainer for realistic testing
  • JWT → Vault token → Dynamic credentials
  • Currently blocked by Go module dependencies

Next: Pattern runner integration (Phase 3), E2E testing with local Vault + Dex

Control Plane RPC Handler (Namespace Management Complete) ✅​

Branch: feature/vault-integration-phase2

Summary: Implemented complete namespace creation RPC handler with pattern selection (Layer 3) and backend assignment (Layer 4). Control plane now fully supports unified namespace model.

New Files:

  • cmd/prism-admin/pattern_selection.go (126 lines):

    • Layer 3: Pattern selection logic
    • Validates pattern registry (SessionManager, KeyValue, Producer, etc.)
    • Assigns pattern versions and metadata
    • Marks session-aware patterns

  • cmd/prism-admin/backend_assignment.go (177 lines):

    • Layer 4: Backend assignment logic
    • Maps slots to backend instances (Redis, Postgres, Kafka, NATS, etc.)
    • Generates connection info and Vault credential paths
    • Default local dev endpoints

  • cmd/prism-admin/namespace_handler.go (195 lines):

    • Complete namespace lifecycle (Create, Get, Update, Delete)
    • Unified creation flow: Validate → Pattern Selection → Backend Assignment → Persist
    • Returns NamespaceResponse with assigned patterns and slot bindings
    • Ready for launcher integration

Modified:

  • cmd/prism-admin/control_plane.go:
    • CreateNamespace RPC now uses configv1.NamespaceRequest (not old CreateNamespaceRequest)
    • Returns configv1.NamespaceResponse with full assignments
    • Integrated with NamespaceHandler

Complete Pipeline:

Client: NamespaceRequest (protobuf)
    ↓
Control Plane: Validate (business rules)
    ↓
Layer 3: Pattern Selection (choose implementations)
    ↓
Layer 4: Backend Assignment (assign backend instances to slots)
    ↓
Storage: Persist namespace configuration
    ↓
Response: NamespaceResponse (assigned patterns, slot bindings, partition)

Supported Backends:

  • Redis (localhost:6379)
  • PostgreSQL (localhost:5432)
  • Kafka (localhost:9092)
  • NATS (localhost:4222)
  • SQLite (/tmp/prism.db)
  • MemStore (memory://)
  • S3/MinIO (localhost:9000)
  • ClickHouse (localhost:9000)

Features:

  • ✅ Multi-pattern composition
  • ✅ Slot-based backend dependencies
  • ✅ Vault dynamic credential paths (vault://secret/data/prism/...)
  • ✅ Session-aware pattern handling
  • ✅ Partition assignment (consistent hashing)
  • ✅ Pattern version management
  • ✅ Metadata tracking

Next: Integration tests and prism-launcher deployment

Protobuf-First YAML Loader (Phase 2 Complete) ✅​

Branch: feature/vault-integration-phase2

Summary: Implemented protobuf-first configuration loading where YAML is an import format only, not the source of truth. Protobuf types are canonical.

New Files:

  • pkg/config/protobuf_loader.go (240 lines):

    • LoadNamespaceProto: YAML file → protobuf NamespaceRequest
    • ParseNamespaceYAML: YAML bytes → protobuf with validation
    • FormatNamespaceYAML: protobuf → YAML (export/preview)
    • Multi-namespace configuration support

  • pkg/config/validation.go (260 lines):

    • Business logic validation (cross-field rules)
    • SessionManager requirements enforcement
    • Session-aware pattern constraints
    • Slot naming conventions ({purpose}-{type})
    • Credential detection (vault:// allowed, static credentials flagged)

Test Coverage:

  • pkg/config/protobuf_loader_test.go: 14 test cases, YAML round-trip testing
  • pkg/config/validation_test.go: 24 test cases, validation rules
  • Total: 38 tests, all passing (0.571s)

Key Features:

  • ✅ YAML → JSON → protobuf conversion pipeline
  • ✅ Validates all business rules (SessionManager, session-aware, slots)
  • ✅ Round-trip support (YAML ↔ protobuf)
  • ✅ Multi-namespace config files
  • ✅ Static credential detection in session-aware patterns
  • ✅ Vault dynamic credential support (vault:// paths)

Benefits:

  • Single source of truth: Protobuf (ADR-003)
  • YAML as human-friendly import/export format only
  • Type-safe construction via protobuf
  • Multi-language support (Go, Python, Rust, Java)

Next: Phase 3 - Implement CreateNamespace RPC handler in control plane

Protobuf Import Cycle Resolution 🔧​

Branch: feature/vault-integration-phase2

Summary: Fixed import cycle between prism and prism.config.v1 packages by moving shared field options to new prism.common package.

Problem: Import cycle prevented compilation:

  • prism package (control_plane.proto) imported prism.config.v1 package
  • prism.config.v1 imported prism package for field options (prism.validation, prism.required, etc.)
  • Go compiler: "import cycle not allowed"

Solution: Created prism.common package for shared annotations:

  • Moved proto/prism/options.proto → proto/prism/common/options.proto
  • Changed package from prism to prism.common
  • Updated all field option references: prism.* → prism.common.*
  • Fixed go_package paths: proto/gen → pkg/plugin/gen

Files Modified:

  • proto/prism/common/options.proto (new location)
  • All proto files updated with correct imports and option references
  • examples/client/namespace_client.go (import paths)

Result:

  • ✅ All protobuf code compiles without errors
  • ✅ No import cycles
  • ✅ Example client compiles successfully
  • ✅ Generated code in pkg/plugin/gen/prism/ and pkg/plugin/gen/prism/config/v1/

Unified Namespace Model with Protobuf 🎯​

Branch: feature/vault-integration-phase2

Summary: Established the UNIFIED namespace model where ALL namespaces natively support multi-pattern composition, slot-based backends, and authentication. No "simple" vs "composable" split - just ONE namespace type that does it right.

New Documents:

  • MEMO-092: Unified Namespace Model with Protobuf:
    • Protobuf as canonical format (ADR-003, ADR-002)
    • ONE NamespaceRequest type (no backward compat needed - Prism is new)
    • Patterns, slots, auth built into core namespace model
    • YAML/JSON as export/preview format only
    • Client SDK examples (Go, Python, Rust)

Enhanced Proto Files:

  • proto/prism/config/v1/namespace_request.proto - Enhanced with patterns, slots, auth (550 lines)
    • Pattern message for multi-pattern composition
    • Slot message for backend dependencies
    • AuthConfig, VaultConfig, JWTConfig for authentication
    • DeploymentPreferences for topology hints
    • NamespaceResponse with assigned patterns and slot bindings
  • proto/prism/control_plane.proto - Simplified CreateNamespace RPC

Client Example:

  • examples/client/namespace_client.go - Unified namespace construction

Key Improvements:

  • Removed unnecessary complexity (no "composable" prefix)
  • ONE namespace type that supports everything
  • Type-safe construction with protobuf
  • Multi-language client SDKs (Go, Python, Rust, Java)
  • No schema drift (protobuf is canonical)

Updated Documents:

  • MEMO-083 - Added Phase 2 completion status (85% complete, code done)
  • MEMO-089 - Fixed broken ADR link

2025-11-17​

Phase 2 Vault Integration Plan (MEMO-083) 🔐​

Branch: feature/vault-integration-phase2

Summary: Detailed 6-week implementation plan for Phase 2 of Prism's authentication system. Builds on Phase 1 (JWT auth, completed) to add HashiCorp Vault integration for per-session, dynamic backend credentials.

New Document:

  • MEMO-083: Phase 2 Vault Integration Implementation Plan:
    • Week-by-week implementation breakdown (6 weeks total)
    • Core Vault infrastructure (JWT → Vault token exchange)
    • Credential lifecycle (fetch, renew, revoke)
    • Session management with per-user isolation
    • Service identity authentication (K8s SA, AWS IAM, Azure MI, GCP SA)
    • Integration testing strategy with Vault testcontainer
    • Vault operator setup guide (auth methods, secrets engines, policies)

Key Benefits:

  • Per-session credential isolation (each user gets unique backend username)
  • Automatic credential rotation (every 1 hour)
  • Zero shared credentials (breach of one session doesn't compromise others)
  • Per-user audit trails in backend logs (e.g., v-jwt-alice-abc123)

Current State:

  • Phase 1 Complete ✅: JWT authentication, namespace authorization, auth context pass-through
  • Phase 2 Not Implemented ❌: Vault integration is design-only (MEMO-008), zero code exists

Netflix Video Reference: Database Migrations at Scale​

Summary: Added AWS re:Invent 2023 video reference featuring Netflix engineers discussing strategies for safely migrating databases that handle millions of requests per second.

New Document:

Updates:

  • Netflix index page now includes direct links to all three video references

2025-11-16​

  • Namespace Configuration - Updated Foundations page with namespace references (RFC-056, RFC-047)
  • Pattern Proxy Integration Tests - Proper layer isolation with backend security
  • Phase 3 Authentication - All pattern runners support Vault authentication

Week of Nov 16-18​

  • Phase 2 Vault Integration - Session management, JWT validation, dynamic credentials
  • Unified Namespace Model - Protobuf-first configuration with multi-pattern support (MEMO-092)
  • Control Plane RPC - Namespace creation with pattern selection and backend assignment

Week of Nov 15-17​

  • Vault Integration Plan - 6-week implementation roadmap (MEMO-083)
  • Documentation Feedback - Added user engagement mechanism
  • Netflix Video References - Database migrations at scale

Full November 2025 changelog

October 2025​

Week of Oct 13-15​

  • Admin UI MVP - Real-time KPI monitoring with templ + htmx + Gin (MEMO-035)
  • Mailbox Pattern - Searchable event store with SQLite indexing (RFC-037)
  • Control Plane Architecture - ADRs 054-057 defining admin, proxy, and launcher protocols
  • Pattern Launcher Complete - Full RFC-035 implementation with developer ergonomics

Week of Oct 11-12​

  • Claim Check Pattern - Large payload handling with object storage (RFC-033)
  • Message Envelope Protocol - Pub/Sub envelope with encryption support (RFC-031)
  • CI Migration - Pattern-based acceptance testing with GitHub Actions summary reporter
  • Schema Registry - Minimal local testing registry (RFC-032)

Week of Oct 7-9​

  • Parallel Linting - 54-90x faster linting with comprehensive Python tooling
  • OIDC Integration Tests - prismctl authentication infrastructure (Phases 1-3)
  • Pattern SDK - Core SDK architecture and build system (RFC-022, RFC-025)
  • Authorization Layer - Token validation with Vault integration (RFC-019)
  • Topaz Integration - Policy-based authorization (ADR-050)

Full October 2025 changelog

Archive​

Monthly archives with detailed entries:

  • November 2025 - Vault Integration, Unified Namespaces, Pattern Authentication
  • October 2025 - Admin UI, Mailbox Pattern, Control Plane, Pattern SDK