Configuration Overview

Prism uses a 6-layer configuration model that separates concerns between users, platform policies, and runtime implementation. This architecture ensures that application teams can declaratively specify their needs while the platform handles the operational complexity.

Key Principle

Users declare WHAT they need, not HOW to implement it. The platform automatically handles pattern selection, backend binding, and resource provisioning.

Six-Layer Model

The configuration flows through six distinct layers, each with a specific purpose and owner:

┌─────────────────────────────────────────────────────────────┐
│ Layer 1: User Request (Application Owner)                   │
│ - Namespace configuration                                   │
│ - Pattern type (queue, pubsub, keyvalue, etc.)            │
│ - Functional requirements (durability, throughput, etc.)   │
│ - Access control and compliance policies                   │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│ Layer 2: Platform Policy (Authorization & Quotas)           │
│ - Team permission levels (Guided, Advanced, Expert)         │
│ - Capacity quotas (max RPS, data size, retention)          │
│ - Resource limits and cost budgets                          │
│ - Compliance requirements                                   │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│ Layer 3: Pattern Selection (Platform Intelligence)          │
│ - Automatic pattern selection based on needs               │
│ - Pattern composition (WAL, claim-check, etc.)            │
│ - Backend slot requirements                                │
│ - Resource calculations                                    │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│ Layer 4: Backend Binding (Operator-Managed)                 │
│ - Global backend registry (Kafka, Postgres, Redis, etc.)   │
│ - Backend capabilities and interfaces                       │
│ - Connection configurations                                │
│ - Slot → Backend mappings                                   │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│ Layer 5: Frontend Exposure (Operator-Managed)               │
│ - API protocol bindings (REST, GraphQL, gRPC)             │
│ - Route mappings and translations                          │
│ - API compatibility layers                                 │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│ Layer 6: Runtime Execution (Pattern Runners & Proxies)      │
│ - Pattern runner process lifecycle                          │
│ - Backend connections and health monitoring                │
│ - Request routing and load balancing                        │
│ - Observability (metrics, traces, logs)                     │
└─────────────────────────────────────────────────────────────┘

Layer Details

Layer 1: User Request

Owner: Application developers, service owners

What it controls:

• Namespace identity and ownership
• High-level pattern type (queue, pubsub, keyvalue, etc.)
• Functional requirements (durability, throughput, consistency)
• Access control (who can read/write)
• Compliance policies (PII handling, audit logging)

What it cannot control:

• Which specific patterns are used
• Which backends store the data
• How resources are allocated
• Network and infrastructure configuration

Configuration Format: YAML namespace request

Documentation: Namespace Configuration

Schema: schemas/config/namespace-request.schema.json

Example:

namespace: order-processing
team: payments-team
pattern: queue
needs:
  durability: strong
  write_rps: 5000
  consistency: strong

---

Layer 2: Platform Policy

Owner: Platform operators, SREs

What it controls:

• Team permission levels (Guided, Advanced, Expert)
• Resource quotas per team
• Cost budgets and alerting
• Compliance requirements
• Policy enforcement rules

Configuration Format: Team policy definitions

Schema: schemas/config/platform-policy.schema.json (TODO)

Key Concepts:

Permission Levels

Level	Access	Use Case
Guided	Standard patterns, moderate limits	Most application teams (default)
Advanced	Advanced patterns, higher limits	Performance-critical services
Expert	All patterns, unlimited	Platform team only

Team Quotas

team: payments-team
permission_level: advanced
quotas:
  max_namespaces: 50
  max_total_write_rps: 500000
  max_total_data_size: 10TB
  max_monthly_cost: 50000  # USD

References:

• ADR-002: Client-Originated Configuration - Permission levels
• ADR-050: Topaz Policy Authorization - Policy enforcement

---

Layer 3: Pattern Selection

Owner: Prism platform (automated)

What it does:

• Analyzes user requirements from Layer 1
• Validates against team policies from Layer 2
• Automatically selects implementation patterns
• Calculates required resources
• Determines backend slot requirements

Configuration Format: Pattern selection output (generated)

Schema: schemas/config/pattern-selection.schema.json (TODO)

Selection Rules (examples):

Requirement	Triggers Pattern
durability: strong	Write-Ahead Log (WAL)
max_message_size > 1MB	Claim-Check pattern
consistency: strong + pattern: queue	Outbox pattern
replay: enabled	Replay-Store pattern
data_size > 1TB	Tiered Storage pattern

Example Output:

selected_patterns:
  - pattern_id: wal-001
    pattern_type: wal
    reason: "Strong durability requirement"
    config:
      fsync: true
      batch_size: 100

  - pattern_id: claim-check-001
    pattern_type: claim-check
    reason: "Messages up to 10MB"
    config:
      threshold: 1MB
      storage: s3

slot_bindings:
  - pattern_id: wal-001
    slot_name: storage
    backend_id: postgres-primary
    reason: "Strong consistency required"

References:

• RFC-056: Unified Configuration Model - Pattern selection algorithm
• RFC-014: Layered Data Access Patterns - Pattern architecture

---

Layer 4: Backend Registry

Owner: Platform operators

What it controls:

• Global registry of available backends
• Backend capabilities and interfaces
• Connection configurations (hosts, ports, auth)
• Capacity limits and current usage
• Cost information

Configuration Format: Backend registration

Schema: schemas/config/backend-registry.schema.json (TODO)

Example:

backend_id: kafka-prod-us-east
backend_type: kafka
display_name: Production Kafka (US East)

connection:
  endpoint: kafka-prod.prism.local:9092
  auth:
    type: mtls
    certificate: /path/to/cert.pem

capabilities:
  - name: publish
    max_write_rps: 1000000
  - name: subscribe
    max_read_rps: 5000000

interfaces:
  - name: PubSubInterface
    version: v1
    operations: [Publish, Subscribe, CreateTopic]

capacity:
  max_write_rps: 1000000
  max_storage: 10TB
  current:
    write_rps: 250000
    storage_used: 2.5TB

cost:
  cost_per_1k_writes: 0.001  # USD
  cost_per_gb_month: 0.10    # USD

References:

• RFC-039: Backend Configuration Registry
• MEMO-006: Backend Interface Decomposition

---

Layer 5: Frontend Registry

Owner: Platform operators

What it controls:

• API protocol bindings (REST, GraphQL, gRPC)
• HTTP route mappings to gRPC services
• Protocol translation configurations
• API versioning and compatibility

Configuration Format: Frontend binding definitions

Schema: schemas/config/frontend-registry.schema.json (TODO)

Example:

frontend_id: confluent-schema-registry-rest
frontend_type: rest
pattern_mapping: multicast-registry

routes:
  - path: /subjects
    method: GET
    grpc_service: prism.Registry
    grpc_method: ListSchemas

  - path: /subjects/{subject}/versions
    method: POST
    grpc_service: prism.Registry
    grpc_method: RegisterSchema
    request_transform: confluent_to_prism
    response_transform: prism_to_confluent

References:

• RFC-039: Backend Configuration Registry - Frontend registry section

---

Layer 6: Runtime Configuration

Owner: Prism admin cluster (automated)

What it controls:

• Pattern runner process configuration
• Proxy routing configuration
• Health check intervals
• Observability settings
• Resource limits (CPU, memory)

Configuration Format: Runtime process configuration

Schema: schemas/config/runtime-config.schema.json (TODO)

Example:

process_type: pattern
pattern_type: multicast-registry
namespace: service-discovery

slots:
  registry:
    backend_id: postgres-primary
    config:
      connection_pool_size: 20
      query_timeout: 5s

  messaging:
    backend_id: nats-prod
    config:
      max_subscriptions: 1000
      ack_wait: 30s

resources:
  cpu_limit: 2000m
  memory_limit: 4Gi

observability:
  metrics_port: 9090
  tracing_enabled: true
  log_level: info

References:

• ADR-055: Proxy-Admin Control Plane
• ADR-056: Launcher-Admin Control Plane
• ADR-057: Prism-Launcher Refactoring

---

Configuration Hierarchy

Configuration precedence (highest to lowest):

1. Platform Policy (Layer 2) - Hard limits, compliance requirements
2. User Request (Layer 1) - Explicit namespace configuration
3. Pattern Selection (Layer 3) - Automatic pattern defaults
4. Backend Capabilities (Layer 4) - Physical backend limitations
5. Runtime Defaults (Layer 6) - System-wide default settings

Example Precedence

User requests: write_rps: 100000
Team policy: max_write_rps: 50000
Result: Request DENIED (exceeds team quota)

User requests: consistency: strong
Pattern selected: Postgres backend
Backend available: Kafka only (eventual consistency)
Result: Request QUEUED (wait for Postgres availability)

---

Validation Flow

1. User submits namespace request (Layer 1)
   ↓
2. Validate against team policy (Layer 2)
   - Check permission level allows pattern
   - Check quotas not exceeded
   - Enforce compliance requirements
   ↓
3. Platform selects patterns (Layer 3)
   - Match requirements to patterns
   - Calculate resource needs
   - Determine backend slots
   ↓
4. Platform binds backends (Layer 4)
   - Match capabilities to requirements
   - Check backend availability
   - Validate interface compatibility
   ↓
5. Admin creates namespace via Raft
   ↓
6. Launcher receives configuration (Layer 6)
   - Start pattern runner process
   - Establish backend connections
   - Register with proxy

---

Configuration Schema System

All configuration layers are defined using JSON Schema as the source of truth. This prevents drift between documentation, validation, and implementation.

Available Tools

# Validate configuration against schema
uv run tooling/config_schema_tools.py validate <config.yaml> --schema <schema.json>

# Generate documentation from schema
uv run tooling/config_schema_tools.py docs <schema.json> --output <output.md>

# Generate example configuration
uv run tooling/config_schema_tools.py example <schema.json> --output <output.yaml>

# Check for configuration drift
uv run tooling/config_schema_tools.py drift <config-dir> --schema <schema.json>

Schema Locations

Layer	Schema File	Status
Layer 1	schemas/config/namespace-request.schema.json	✅ Complete
Layer 2	schemas/config/platform-policy.schema.json	📝 TODO
Layer 3	schemas/config/pattern-selection.schema.json	📝 TODO
Layer 4	schemas/config/backend-registry.schema.json	📝 TODO
Layer 5	schemas/config/frontend-registry.schema.json	📝 TODO
Layer 6	schemas/config/runtime-config.schema.json	📝 TODO

---

Terminology

Configuration Terms

Term	Definition	Example
pattern	High-level data access pattern	queue, pubsub, keyvalue
needs	Functional requirements	durability: strong
slot	Capability requirement of a pattern	registry slot in multicast-registry
backend	Physical data store	kafka-prod, postgres-primary
capability	High-level operation	publish, subscribe, scan
interface	Specific API contract	KeyValueBasicInterface, PubSubInterface
frontend	API protocol exposure	confluent-schema-registry-rest

Authorization Terms

Term	Definition	Example
team	Group of users who own resources	payments-team
namespace	Isolated data resource	order-processing
permission level	Coarse-grained capability grant	Guided, Advanced, Expert
policy	Fine-grained authorization rule	"payments-team can admin order-*"
quota	Resource limit	max_write_rps: 50000
compliance	Regulatory requirement	PCI, SOC2, GDPR

---

See Also

Documentation

• Namespace Configuration - Layer 1 configuration guide
• RFC-056: Unified Configuration Model - Complete specification
• MEMO-050: Configuration Schema System - Implementation guide

Related ADRs

• ADR-002: Client-Originated Configuration - Permission levels
• ADR-022: Dynamic Client Configuration - Runtime updates
• ADR-050: Topaz Policy Authorization - Policy enforcement

Schema System

• schemas/README.md - Schema system documentation
• tooling/config_schema_tools.py - Validation and codegen tools