Backend Registry

The Backend Registry is Layer 4 of the configuration model and defines all available backends that Prism can use for data storage and messaging. Platform operators manage this registry to control where data is physically stored.

Configuration Layer

Layer 4: Backend Binding - Operator-managed registry of available backends and their capabilities.

Overview

The backend registry serves as the single source of truth for:

• Available backends (Kafka, Postgres, Redis, NATS, etc.)
• Backend capabilities (what operations each backend supports)
• Backend interfaces (specific API contracts implemented)
• Connection configurations (endpoints, authentication, pooling)
• Capacity limits and current usage
• Cost information for resource planning

Backend Types

Prism supports multiple backend types, each optimized for different data access patterns:

Backend Type	Primary Use Cases	Capabilities
Kafka	High-throughput messaging, event streaming	Publish, Subscribe, Replay, Partitioning
NATS	Low-latency messaging, microservices communication	Publish, Subscribe, Request-Reply
Postgres	Strong consistency, ACID transactions, complex queries	CRUD, Transactions, Indexes, Foreign Keys
Redis	Fast caching, session storage, simple key-value	Get, Set, TTL, Pub/Sub, Sorted Sets
SQLite	Local storage, embedded databases, testing	CRUD, Transactions, Indexes
Neptune	Graph databases, relationship queries	Graph traversal, Gremlin queries
SQS	Queue messaging, AWS integration	Enqueue, Dequeue, Dead-letter queues
S3	Object storage, large payloads, archival	Put, Get, List, Versioning
MemStore	In-memory storage, testing, ephemeral data	Get, Set, volatile storage

References:

• RFC-039: Backend Configuration Registry
• ADR-041: Graph Database Backend - Neptune integration
• ADR-042: SQS Queue Backend

---

Backend Registration

Registration Structure

Each backend must be registered with complete metadata:

backend_id: kafka-prod-us-east-1
backend_type: kafka
display_name: Production Kafka (US East)
description: High-throughput Kafka cluster for production workloads

# Connection configuration
connection:
  endpoint: kafka-prod.prism.local:9092
  auth:
    type: mtls
    certificate_path: /etc/prism/certs/kafka-prod.pem
    key_path: /etc/prism/certs/kafka-prod-key.pem
    ca_cert_path: /etc/prism/certs/ca.pem

  tls:
    enabled: true
    verify_server: true
    server_name: kafka-prod.prism.local

  pool:
    min_connections: 10
    max_connections: 100
    max_idle_time: 300s
    max_lifetime: 3600s

  timeouts:
    connect_timeout: 10s
    read_timeout: 30s
    write_timeout: 30s
    idle_timeout: 60s

  retry:
    max_attempts: 3
    initial_backoff: 100ms
    max_backoff: 5s
    backoff_multiplier: 2.0

# Capabilities this backend provides
capabilities:
  - name: publish
    version: v1
    performance:
      max_write_rps: 1000000
      latency:
        p50_ms: 5
        p95_ms: 15
        p99_ms: 25
        p999_ms: 50
      supports_batch: true
      max_batch_size: 1000

  - name: subscribe
    version: v1
    performance:
      max_read_rps: 5000000
      supports_batch: true
      max_concurrent_operations: 10000

  - name: replay
    version: v1
    limitations:
      - "Replay limited to 7 days retention"
      - "Requires consumer group reset"

# Interfaces this backend implements
interfaces:
  - name: PubSubInterface
    version: v1
    proto_service: prism.PubSub
    operations:
      - name: Publish
        idempotent: true
      - name: Subscribe
        idempotent: false
      - name: CreateTopic
        idempotent: true
        transactional: false

  - name: StreamingInterface
    version: v1
    proto_service: prism.Streaming
    operations:
      - name: ReadStream
        idempotent: true
      - name: SeekToOffset
        idempotent: true

# Capacity and limits
capacity:
  max_write_rps: 1000000
  max_read_rps: 5000000
  max_storage: 10TB
  max_connections: 10000
  max_namespaces: 500

  # Current usage
  current:
    write_rps: 250000
    read_rps: 1200000
    storage_used: 2.5TB
    active_connections: 2500
    namespace_count: 125

# Cost information
cost:
  cost_model: pay_per_use
  cost_per_1k_writes: 0.001  # USD
  cost_per_1k_reads: 0.0005  # USD
  cost_per_gb_month: 0.10    # USD per GB per month
  cost_per_connection_hour: 0.01  # USD per connection per hour

  transfer_costs:
    cost_per_gb_intra_region: 0.01   # Within same region
    cost_per_gb_inter_region: 0.02   # Between regions
    cost_per_gb_internet: 0.09       # To internet

# Health and status
health:
  status: healthy  # healthy | degraded | unhealthy | maintenance
  last_check_at: 2025-11-16T10:30:00Z
  uptime_percent_24h: 99.95

  recent_incidents:
    - started_at: 2025-11-15T14:22:00Z
      resolved_at: 2025-11-15T14:45:00Z
      severity: warning
      description: "Elevated latency in partition 12"
      impact: "P99 latency increased from 25ms to 150ms"

# Deployment regions
regions:
  - us-east-1
  - us-west-2

# Compliance certifications
compliance:
  - pci
  - soc2
  - hipaa

# Metadata
metadata:
  owner_team: platform-infrastructure
  oncall_slack: "#kafka-oncall"
  runbook: "https://wiki.example.com/kafka-prod-runbook"
  monitoring_dashboard: "https://grafana.example.com/d/kafka-prod"

---

Capabilities

Capabilities define what operations a backend supports at a high level.

Core Capabilities

Read/Write Capabilities

Capability	Description	Required For Patterns
get	Retrieve single record by key	KeyValue, Reader
set	Store single record	KeyValue, Writer
delete	Remove single record	KeyValue
scan	Iterate over range of keys	Reader, Query
batch_get	Retrieve multiple records	KeyValue (performance)
batch_set	Store multiple records	KeyValue (performance)

Messaging Capabilities

Capability	Description	Required For Patterns
publish	Send message to topic	PubSub, Queue
subscribe	Receive messages from topic	PubSub, Consumer
enqueue	Add message to queue	Queue
dequeue	Remove message from queue	Queue
ack	Acknowledge message receipt	Queue, Consumer
nack	Reject message	Queue, Consumer

Advanced Capabilities

Capability	Description	Required For Patterns
transaction	Multi-operation transactions	Transact
replay	Read from beginning of stream	Event Sourcing, Replay
ttl	Automatic expiration	KeyValue (with TTL)
watch	Notification on changes	Registry, Live Updates
query	Complex queries	Mailbox, Search
graph_traversal	Graph queries	Graph

Capability Performance

Each capability includes performance characteristics:

capability:
  name: publish
  performance:
    max_write_rps: 1000000
    latency:
      p50_ms: 5
      p95_ms: 15
      p99_ms: 25
      p999_ms: 50
    supports_batch: true
    max_batch_size: 1000
    max_concurrent_operations: 10000

References:

• ADR-043: Plugin Capability Discovery

---

Interfaces

Interfaces define specific API contracts that backends implement. They are more detailed than capabilities and map directly to gRPC service definitions.

Interface Hierarchy

Capability (high-level)
    ↓
Interface (specific API contract)
    ↓
Operations (individual RPC methods)

Example Interfaces

KeyValueBasicInterface

interface:
  name: KeyValueBasicInterface
  version: v1
  proto_service: prism.KeyValue

  operations:
    - name: Get
      idempotent: true
      parameters:
        key: string (required)
        namespace: string (required)
      returns: value, version, ttl

    - name: Set
      idempotent: false
      parameters:
        key: string (required)
        value: bytes (required)
        namespace: string (required)
        ttl: duration (optional)
      returns: version

    - name: Delete
      idempotent: true
      parameters:
        key: string (required)
        namespace: string (required)
      returns: success

    - name: Exists
      idempotent: true
      parameters:
        key: string (required)
        namespace: string (required)
      returns: exists

PubSubInterface

interface:
  name: PubSubInterface
  version: v1
  proto_service: prism.PubSub

  operations:
    - name: Publish
      idempotent: true  # With deduplication key
      parameters:
        topic: string (required)
        message: bytes (required)
        namespace: string (required)
        deduplication_key: string (optional)
      returns: message_id, offset

    - name: Subscribe
      idempotent: false
      parameters:
        topic: string (required)
        consumer_group: string (required)
        namespace: string (required)
      returns: stream of messages

    - name: CreateTopic
      idempotent: true
      parameters:
        topic: string (required)
        namespace: string (required)
        config: TopicConfig
      returns: success

TransactionalInterface

interface:
  name: TransactionalInterface
  version: v1
  proto_service: prism.Transaction

  operations:
    - name: BeginTransaction
      idempotent: false
      returns: transaction_id

    - name: CommitTransaction
      idempotent: false
      parameters:
        transaction_id: string (required)
      returns: success

    - name: RollbackTransaction
      idempotent: true
      parameters:
        transaction_id: string (required)
      returns: success

    - name: TransactionalGet
      idempotent: true
      transactional: true
      parameters:
        transaction_id: string (required)
        key: string (required)
      returns: value, version

    - name: TransactionalSet
      idempotent: false
      transactional: true
      parameters:
        transaction_id: string (required)
        key: string (required)
        value: bytes (required)
      returns: version

Interface Compatibility

Patterns declare which interfaces they require via slots. The platform validates that backends implement the required interfaces:

# Pattern declares requirements
pattern: multicast-registry
slots:
  registry:
    required_interfaces:
      - KeyValueBasicInterface
      - WatchInterface

  messaging:
    required_interfaces:
      - PubSubInterface

# Backend must implement these interfaces
backend: postgres-primary
interfaces:
  - KeyValueBasicInterface
  - WatchInterface
  # ✅ Compatible with registry slot

backend: kafka-prod
interfaces:
  - PubSubInterface
  # ✅ Compatible with messaging slot

References:

• MEMO-006: Backend Interface Decomposition
• RFC-014: Layered Data Access Patterns - 45 thin interfaces

---

Slot Matching

Slots are capability requirements that patterns have. The platform matches slots to backends based on interface compatibility and performance requirements.

Slot Definition

slot_requirement:
  slot_name: storage

  # Required capabilities
  capabilities:
    - get
    - set
    - delete
    - transaction

  # Required interfaces
  interfaces:
    - KeyValueBasicInterface
    - TransactionalInterface

  # Optional (can fail gracefully)
  optional: false

  # Performance requirements
  performance:
    min_write_rps: 10000
    min_read_rps: 50000
    max_latency_p99: 25ms
    requires_transactions: true
    requires_atomic_writes: true

Matching Algorithm

The platform uses this algorithm to match slots to backends:

def match_slot_to_backend(slot_requirement, available_backends):
    """
    Match a slot requirement to the best available backend.

    Returns: (backend_id, match_score, reasons)
    """
    candidates = []

    for backend in available_backends:
        # 1. Check interface compatibility (required)
        if not all(iface in backend.interfaces for iface in slot_requirement.interfaces):
            continue  # Skip backend, missing required interface

        # 2. Check capability compatibility (required)
        if not all(cap in backend.capabilities for cap in slot_requirement.capabilities):
            continue  # Skip backend, missing required capability

        # 3. Check performance requirements (required)
        if backend.capacity.max_write_rps < slot_requirement.performance.min_write_rps:
            continue  # Skip backend, insufficient write capacity

        if backend.capacity.max_read_rps < slot_requirement.performance.min_read_rps:
            continue  # Skip backend, insufficient read capacity

        # 4. Calculate match score
        score = calculate_score(backend, slot_requirement)

        candidates.append({
            'backend_id': backend.backend_id,
            'score': score,
            'reasons': generate_reasons(backend, slot_requirement)
        })

    # 5. Sort by score (highest first)
    candidates.sort(key=lambda x: x['score'], reverse=True)

    return candidates[0] if candidates else None

def calculate_score(backend, requirement):
    """Calculate how well a backend matches requirements."""
    score = 0.0

    # Performance match (40% weight)
    write_ratio = backend.capacity.available_write_rps / requirement.performance.min_write_rps
    read_ratio = backend.capacity.available_read_rps / requirement.performance.min_read_rps
    score += 0.4 * min(write_ratio, read_ratio, 2.0)  # Cap at 2x

    # Cost efficiency (30% weight)
    cost_score = 1.0 / (backend.cost.cost_per_1k_writes + 1.0)
    score += 0.3 * cost_score

    # Availability (20% weight)
    availability_score = backend.health.uptime_percent_24h / 100.0
    score += 0.2 * availability_score

    # Region affinity (10% weight)
    if backend.region == requirement.preferred_region:
        score += 0.1

    return score

Match Reasons

When a backend is selected, the platform generates reasons explaining the choice:

slot_binding:
  pattern_id: wal-001
  slot_name: storage
  backend_id: postgres-primary

  reasons:
    - type: capability_match
      explanation: "Backend implements TransactionalInterface required for WAL pattern"

    - type: performance
      explanation: "Backend provides 100k write RPS (required: 10k)"

    - type: consistency
      explanation: "Backend offers strong consistency required for durability: strong"

    - type: region_affinity
      explanation: "Backend in same region (us-east-1) as requester"

References:

• RFC-056: Unified Configuration Model - Slot binding flow

---

Backend Querying

The platform provides APIs to query available backends:

Query by Capabilities

query:
  required_capabilities:
    - transaction
    - get
    - set

  required_interfaces:
    - KeyValueBasicInterface
    - TransactionalInterface

  min_capacity:
    min_write_rps: 10000
    min_read_rps: 50000
    min_storage: 1TB

  regions:
    - us-east-1
    - us-west-2

  health_statuses:
    - healthy

response:
  backends:
    - backend_id: postgres-primary
      score: 0.92
      reasons: [capability_match, performance, region_affinity]

    - backend_id: postgres-secondary
      score: 0.78
      reasons: [capability_match, region_affinity]

  total_count: 2

Query by Type

query:
  backend_types:
    - kafka
    - nats

  regions:
    - us-east-1

response:
  backends:
    - kafka-prod-us-east-1
    - nats-prod-us-east-1

---

Connection Configuration

Authentication Methods

Backends support multiple authentication methods:

mTLS (Mutual TLS)

auth:
  type: mtls
  certificate_path: /etc/prism/certs/client.pem
  key_path: /etc/prism/certs/client-key.pem
  ca_cert_path: /etc/prism/certs/ca.pem

Username/Password

auth:
  type: username_password
  username: prism-service
  password: ${SECRET:prism/backend/kafka-prod/password}  # From secret store

Token-Based

auth:
  type: token
  token: ${SECRET:prism/backend/redis-cache/token}

IAM (AWS)

auth:
  type: iam
  role_arn: arn:aws:iam::123456789012:role/PrismBackendAccess
  region: us-east-1

Connection Pooling

pool:
  min_connections: 10        # Minimum pool size
  max_connections: 100       # Maximum pool size
  max_idle_time: 300s        # Max time connection can be idle
  max_lifetime: 3600s        # Max connection lifetime (force reconnect)

Timeouts

timeouts:
  connect_timeout: 10s       # How long to wait for connection
  read_timeout: 30s          # How long to wait for read
  write_timeout: 30s         # How long to wait for write
  idle_timeout: 60s          # How long before idle connection closed

Retry Policies

retry:
  max_attempts: 3                 # Maximum retry attempts
  initial_backoff: 100ms          # Initial backoff duration
  max_backoff: 5s                 # Maximum backoff duration
  backoff_multiplier: 2.0         # Exponential backoff multiplier
  retryable_errors:               # Which errors to retry
    - connection_timeout
    - temporary_failure
    - rate_limit_exceeded

References:

• ADR-035: Connection Pooling

---

Health Monitoring

Health Status

Backends report health status:

Status	Description	Action
healthy	Operating normally	Route traffic normally
degraded	Experiencing issues but operational	Reduce traffic, alert operators
unhealthy	Not operational	Stop routing traffic, failover
maintenance	Scheduled maintenance	Route to alternative backends

Health Checks

health_check:
  endpoint: /health
  interval: 30s
  timeout: 5s
  healthy_threshold: 2    # Consecutive successes to mark healthy
  unhealthy_threshold: 3  # Consecutive failures to mark unhealthy

  checks:
    - name: connectivity
      weight: 1.0
      command: "ping backend"

    - name: latency
      weight: 0.8
      threshold: 100ms
      command: "measure p99 latency"

    - name: error_rate
      weight: 1.0
      threshold: 1%  # More than 1% errors = degraded
      command: "check error rate"

Incident Tracking

incident:
  started_at: 2025-11-15T14:22:00Z
  resolved_at: 2025-11-15T14:45:00Z
  severity: warning  # info | warning | critical
  description: "Elevated latency in partition 12"
  impact: "P99 latency increased from 25ms to 150ms"
  root_cause: "Network congestion in AZ-1a"
  remediation: "Traffic shifted to AZ-1b"

---

Cost Tracking

Cost Models

Model	Description	Use Case
pay_per_use	Pay per operation	Variable workloads
provisioned	Pre-allocated capacity	Predictable workloads
reserved	Long-term commitment discount	Production stable workloads
spot	Cheaper interruptible capacity	Batch processing, non-critical

Cost Calculation

cost_calculation:
  # Operation costs
  writes: 1000000 ops * $0.001 per 1k = $1.00
  reads: 5000000 ops * $0.0005 per 1k = $2.50

  # Storage costs
  storage: 2500 GB * $0.10 per GB/month = $250.00

  # Connection costs
  connections: 2500 conn * 720 hours * $0.01 per conn/hour = $18,000.00

  # Data transfer costs
  intra_region: 100 GB * $0.01 per GB = $1.00
  inter_region: 50 GB * $0.02 per GB = $1.00
  internet: 10 GB * $0.09 per GB = $0.90

  # Total monthly cost
  total: $18,255.40

---

Example Backend Configurations

Kafka Production

backend_id: kafka-prod-us-east-1
backend_type: kafka
display_name: Production Kafka (US East)

connection:
  endpoint: kafka-prod.prism.local:9092
  auth:
    type: mtls
  tls:
    enabled: true
    verify_server: true

capabilities:
  - publish
  - subscribe
  - replay

interfaces:
  - PubSubInterface
  - StreamingInterface

capacity:
  max_write_rps: 1000000
  max_read_rps: 5000000
  max_storage: 10TB

regions: [us-east-1, us-west-2]
compliance: [pci, soc2]

Postgres Primary

backend_id: postgres-primary
backend_type: postgres
display_name: Primary Postgres Database

connection:
  endpoint: postgres-primary.prism.local:5432
  auth:
    type: username_password
    username: prism
  pool:
    min_connections: 20
    max_connections: 100

capabilities:
  - get
  - set
  - delete
  - scan
  - transaction
  - query

interfaces:
  - KeyValueBasicInterface
  - TransactionalInterface
  - QueryInterface

capacity:
  max_write_rps: 50000
  max_read_rps: 200000
  max_storage: 5TB

regions: [us-east-1]
compliance: [pci, soc2, hipaa]

Redis Cache

backend_id: redis-cache-us-east-1
backend_type: redis
display_name: Redis Cache (US East)

connection:
  endpoint: redis-cache.prism.local:6379
  auth:
    type: token
  pool:
    min_connections: 50
    max_connections: 500

capabilities:
  - get
  - set
  - delete
  - ttl
  - publish
  - subscribe

interfaces:
  - KeyValueBasicInterface
  - TTLInterface
  - PubSubInterface

capacity:
  max_write_rps: 500000
  max_read_rps: 2000000
  max_storage: 100GB

regions: [us-east-1]

---

See Also

• Configuration Overview - 6-layer model
• Slot Matching - Detailed slot matching algorithm
• Cluster Administration - Admin cluster configuration
• RFC-039: Backend Configuration Registry
• MEMO-006: Backend Interface Decomposition
• ADR-043: Plugin Capability Discovery