Skip to main content

Core Concepts

Read time: 10 minutes

The Three-Layer Architecture​

Prism separates what you want from how it works and where data lives.

Layer 1: Client API (What)​

The stable interface your application uses. Five core APIs:

KeyValue  → Store and retrieve by key
PubSub    → Publish events, subscribe to topics
Queue     → Send and receive ordered messages
Reader    → Read-only access to data streams
Transact  → Multi-operation transactions

Key point: These APIs never change, even when backends or patterns change.

Layer 2: Patterns (How)​

Reliability patterns Prism applies automatically:

WAL (Write-Ahead Log) → Durability guarantees
Outbox                → Transactional messaging
Claim Check           → Handle large messages
CDC (Change Data)     → Database change capture
Tiered Storage        → Hot/cold data management

Key point: You request guarantees (durability: strong), Prism selects patterns.

Layer 3: Backends (Where)​

Data storage backends:

Kafka      → High-throughput event streaming
PostgreSQL → Transactional data
Redis      → Fast key-value and pub/sub
NATS       → Lightweight messaging

Key point: Platform team controls backend selection and provisioning.

Example: How Layers Work Together​

Your request:

client_api: queue
needs:
  durability: strong
  retention: 30days
  write_rps: 5000

What Prism does:

Layer 1 (What): Expose Queue API
            ↓
Layer 2 (How): Apply WAL pattern for durability
            ↓
Layer 3 (Where): Use Kafka + PostgreSQL

Result:

  • You call queue.send(message) (Layer 1)
  • Prism writes to WAL on disk, then Kafka (Layer 2)
  • Data stored in Kafka topic + PostgreSQL WAL table (Layer 3)
  • You never see Layers 2-3—handled transparently

Namespaces​

A namespace is an isolated data access scope.

Namespace Properties​

namespace: order-processing  # Unique identifier
team: payments              # Owning team
client_api: queue           # API type

Namespace Isolation​

Each namespace provides:

  • Separate data: No cross-namespace visibility
  • Independent config: Different durability, retention per namespace
  • Access control: Per-namespace authorization
  • Resource allocation: Dedicated backend resources

Example:

orders-service namespace:
  - client_api: queue
  - backend: Kafka topic "orders-service-queue"
  - access: payments team only

analytics namespace:
  - client_api: pubsub
  - backend: NATS subject "analytics.events"
  - access: analytics team + data-science team

Data Models​

Prism provides 10 data models composed of 45 thin interfaces.

KeyValue Model​

Store and retrieve data by key.

Interfaces:

  • keyvalue_basic: Get, Set, Delete
  • keyvalue_scan: Scan keys by pattern
  • keyvalue_ttl: Automatic expiration
  • keyvalue_transactional: Multi-key operations
  • keyvalue_batch: Bulk operations
  • keyvalue_cas: Compare-and-swap

Backends: Redis, PostgreSQL, DynamoDB, MemStore

Use cases:

  • Session storage
  • Configuration management
  • Caching
  • Feature flags

PubSub Model​

Publish events to topics, subscribe to receive events.

Interfaces:

  • pubsub_basic: Publish, Subscribe
  • pubsub_wildcards: Pattern subscriptions (e.g., user.*)
  • pubsub_persistent: Durable subscriptions
  • pubsub_filtering: Server-side event filtering
  • pubsub_ordering: Ordered delivery guarantees

Backends: NATS, Redis, Kafka

Use cases:

  • Event notifications
  • Real-time updates
  • Microservice communication
  • Activity feeds

Queue Model​

Ordered message delivery with acknowledgment.

Interfaces:

  • queue_basic: Send, Receive, Acknowledge
  • queue_visibility: Visibility timeout for retries
  • queue_dead_letter: Failed message handling
  • queue_priority: Priority-based delivery
  • queue_delayed: Scheduled message delivery

Backends: PostgreSQL, SQS, RabbitMQ

Use cases:

  • Background job processing
  • Task queues
  • Order processing
  • Workflow orchestration

Stream Model​

Append-only log of events with replay capability.

Interfaces:

  • stream_basic: Append, Read
  • stream_consumer_groups: Parallel consumption
  • stream_replay: Read from any point
  • stream_retention: Automatic old data cleanup
  • stream_partitioning: Ordered parallelism

Backends: Kafka, Redis Streams, NATS JetStream

Use cases:

  • Event sourcing
  • Audit logs
  • Change data capture
  • Metrics collection

TimeSeries Model​

Time-ordered data with aggregation.

Interfaces:

  • timeseries_basic: Append, Query by time range
  • timeseries_aggregation: Downsampling, rollups
  • timeseries_retention: Automatic data expiration
  • timeseries_interpolation: Fill missing data points

Backends: ClickHouse, TimescaleDB, InfluxDB

Use cases:

  • Metrics and monitoring
  • Sensor data
  • Financial data
  • Application performance monitoring

Backend Capabilities​

Backends implement thin interfaces instead of exposing all features.

Example: Redis Capabilities​

Redis implements 16 interfaces across 5 models:

KeyValue Model:
  ✓ keyvalue_basic (Get, Set, Delete)
  ✓ keyvalue_scan (Scan, Count)
  ✓ keyvalue_ttl (Expire, GetTTL)
  ✓ keyvalue_batch (MGet, MSet)

PubSub Model:
  ✓ pubsub_basic (Publish, Subscribe)
  ✓ pubsub_wildcards (Pattern matching)

Stream Model:
  ✓ stream_basic (XADD, XREAD)
  ✓ stream_consumer_groups (XGROUP)

Example: Kafka Capabilities​

Kafka implements 8 interfaces focused on streaming:

Stream Model:
  ✓ stream_basic (Produce, Consume)
  ✓ stream_consumer_groups (Consumer groups)
  ✓ stream_replay (Seek to offset)
  ✓ stream_retention (Topic retention)
  ✓ stream_partitioning (Topic partitions)

PubSub Model:
  ✓ pubsub_basic (Topics as pub/sub)
  ✓ pubsub_ordering (Partition ordering)
  ✓ pubsub_persistent (Durable storage)

Why interfaces matter:

  • Type-safe: Compiler enforces contracts
  • Composable: Multiple backends can fulfill pattern needs
  • Explicit: No hidden capabilities or surprises

Configuration Ownership​

Client-Controlled Configuration​

Application teams control what they need:

# You specify:
client_api: pubsub
needs:
  write_rps: 1000          # Capacity estimate
  retention: 7days         # Data lifetime
  durability: strong       # Reliability level
  max_message_size: 100KB  # Message size limit

Platform-Controlled Configuration​

Platform team controls how and where:

# Platform configures (you never see this):
patterns:
  - wal: {fsync: true}
  - claim_check: {threshold: 50KB}
backend:
  type: kafka
  partitions: 20
  replication: 3

Why this separation?

  • Prevents misconfiguration: Clients can't select incompatible patterns
  • Enables evolution: Platform changes implementation without breaking apps
  • Enforces best practices: Platform ensures correct pattern composition

Authorization Boundaries​

Platform team defines policy boundaries to guide client choices.

Guided Mode (Default)​

Pre-approved backends for all teams:

allowed_backends:
  - postgres  # For queues, transactions
  - kafka     # For streams, pub/sub
  - redis     # For caching, key-value
  - nats      # For lightweight messaging

Result: Teams self-service within safe boundaries.

Advanced Mode (Approval Required)​

Backend-specific tuning needs platform approval:

# Requires platform team review:
needs:
  kafka_partitions: 50       # Custom partition count
  postgres_pool_size: 100    # Custom connection pool

Expert Mode (Platform Team Only)​

Unrestricted access for platform team:

# Platform engineers only:
backend:
  type: custom-redis-cluster
  connection: "redis://custom-endpoint"
  replication_factor: 5

PII Handling​

Prism automatically handles Personally Identifiable Information (PII) based on protobuf annotations.

Annotating PII​

message UserProfile {
  string user_id = 1;

  string email = 2 [
    (prism.pii) = "email",
    (prism.encrypt_at_rest) = true,
    (prism.mask_in_logs) = true
  ];

  string ssn = 3 [
    (prism.pii) = "ssn",
    (prism.encrypt_at_rest) = true,
    (prism.mask_in_logs) = true,
    (prism.audit_access) = true
  ];
}

Automatic PII Handling​

Prism generates code that:

  • Encrypts at rest: Data encrypted before storage
  • Masks in logs: PII redacted from observability
  • Audits access: All PII reads logged for compliance
  • Validates schemas: Ensures PII annotations are enforced

Example log output:

INFO: Processing UserProfile user_id=123 email=***@***.*** ssn=***-**-****

Observability​

Every namespace includes automatic observability.

Built-In Metrics​

prism_requests_total{namespace="orders", api="queue", status="success"}
prism_latency_seconds{namespace="orders", api="queue", percentile="p99"}
prism_backend_errors_total{namespace="orders", backend="kafka"}

Built-In Tracing​

Distributed traces span all layers:

Trace: order-processing-request
  ├─ queue.send() [1.2ms]
  │  ├─ wal.write() [0.3ms]
  │  └─ kafka.produce() [0.9ms]
  └─ response [1.2ms total]

Built-In Logs​

Structured JSON logs:

{
  "level": "info",
  "namespace": "orders",
  "api": "queue",
  "operation": "send",
  "latency_ms": 1.2,
  "backend": "kafka",
  "trace_id": "abc123"
}

Key Takeaways​

  1. Three layers separate concerns: Client API (what), Patterns (how), Backends (where)
  2. Namespaces provide isolation: Each namespace has independent config and access control
  3. Interfaces compose backends: Thin interfaces enable type-safe backend composition
  4. Clients control needs, platform controls implementation: Clear separation prevents misconfiguration
  5. PII and observability are automatic: No manual implementation required

Next Steps​