Data Patterns

Read time: 10 minutes

Prism automatically applies reliability patterns based on your requirements. You don't implement these patterns—Prism does.

Pattern Selection

You declare needs:

needs:
  durability: strong
  max_message_size: 5MB
  consistency: strong

Prism selects patterns:

durability: strong        → WAL (Write-Ahead Log)
max_message_size: 5MB     → Claim Check
consistency: strong       → Transactional Outbox

Core Patterns

WAL (Write-Ahead Log)

Purpose: Guarantee zero data loss

When Prism applies:

• durability: strong is set
• Financial transactions
• Audit logs
• Order processing

What it does:

1. Write to disk (with fsync)
2. Acknowledge to client
3. Asynchronously flush to backend

Guarantees:

• ✓ Zero data loss (survives crashes)
• ✓ Ordered writes
• ✓ Replay capability

Performance:

• Latency: 2-5ms (disk write overhead)
• Throughput: Limited by disk I/O

Example use case:

# E-commerce order processing
namespace: orders
needs:
  durability: strong  # → Prism adds WAL pattern

What happens:

App sends order
  ↓
Prism writes to WAL on disk (fsync)
  ↓
Prism acknowledges (order is durable)
  ↓
Prism asynchronously flushes to Kafka

Result: Order survives proxy crashes, zero loss.

Claim Check Pattern

Purpose: Handle large messages efficiently

When Prism applies:

• Message size > 100KB
• Backend has message size limits
• Large file attachments

What it does:

1. Store large payload in object storage (S3)
2. Send small reference (claim check) via queue
3. Consumer retrieves payload using claim check

Guarantees:

• ✓ Bypass backend message size limits
• ✓ Efficient queue processing
• ✓ Automatic garbage collection

Example use case:

# Image processing pipeline
namespace: image-uploads
needs:
  max_message_size: 10MB  # → Prism adds Claim Check

What happens:

App uploads 10MB image
  ↓
Prism stores image in S3
  ↓
Prism sends reference to queue: {s3_key: "img_123"}
  ↓
Consumer receives reference
  ↓
Consumer fetches image from S3
  ↓
Consumer processes image
  ↓
Prism deletes S3 object (after TTL)

Configuration:

needs:
  max_message_size: 5MB
  retention: 7days  # Claim check TTL

Performance:

• Latency: +10-50ms (S3 upload)
• Storage: Separate from queue backend

Outbox Pattern

Purpose: Transactional messaging (database + queue atomically)

When Prism applies:

• client_api: transact with messaging
• Need atomic database write + event publish
• Saga pattern implementations

What it does:

1. Write to database and outbox table (single transaction)
2. Background process reads outbox
3. Publish to message queue
4. Mark outbox entry as sent

Guarantees:

• ✓ Atomic database + messaging
• ✓ At-least-once delivery
• ✓ No lost messages

Example use case:

# Order creation with notification
namespace: order-service
client_api: transact
needs:
  consistency: strong  # → Prism adds Outbox pattern

What happens:

BEGIN TRANSACTION
  INSERT INTO orders (id, amount) VALUES (123, 99.99)
  INSERT INTO outbox (event_type, payload) VALUES ('order_created', {...})
COMMIT

Background process:
  SELECT * FROM outbox WHERE sent = false
  PUBLISH to message queue
  UPDATE outbox SET sent = true

Result: Order write and notification are atomic—both happen or neither happens.

CDC (Change Data Capture)

Purpose: Replicate database changes to event streams

When Prism applies:

• Read-only event streaming from databases
• Data synchronization
• Audit trails from database changes

What it does:

1. Monitor database transaction log
2. Capture INSERT/UPDATE/DELETE operations
3. Publish changes as events
4. Maintain ordering guarantees

Guarantees:

• ✓ Every database change becomes an event
• ✓ Ordered event stream
• ✓ No application code changes

Example use case:

# Replicate user profile changes
namespace: user-profile-events
client_api: reader
needs:
  ordered: true  # → Prism adds CDC pattern

What happens:

App writes to users table:
  UPDATE users SET email='new@example.com' WHERE id=123

CDC process detects change:
  {
    "operation": "UPDATE",
    "table": "users",
    "before": {"email": "old@example.com"},
    "after": {"email": "new@example.com"},
    "timestamp": "2025-10-19T10:30:00Z"
  }

Prism publishes event to stream

Result: Every database write automatically becomes an event.

Tiered Storage

Purpose: Optimize storage costs (hot/cold data separation)

When Prism applies:

• Long retention periods (>30 days)
• Large data volumes
• Infrequent access to old data

What it does:

1. Store recent data in fast backend (Redis, Kafka)
2. Archive old data to cold storage (S3, ClickHouse)
3. Automatically move data based on age
4. Transparent retrieval from both tiers

Guarantees:

• ✓ Lower storage costs
• ✓ Fast access to recent data
• ✓ Transparent to application

Example use case:

# Long-term event storage
namespace: audit-logs
needs:
  retention: 365days  # → Prism adds Tiered Storage

What happens:

Day 0-7: Data in Kafka (fast access)
Day 8-365: Data in S3 (cold storage)

App requests data from day 5: Read from Kafka (fast)
App requests data from day 90: Read from S3 (slower, but cheaper)

Cost comparison:

365 days in Kafka: $1000/month
7 days Kafka + 358 days S3: $200/month

Multicast Registry

Purpose: Multi-backend write with consensus

When Prism applies:

• High availability requirements
• Cross-region replication
• Multiple backend redundancy

What it does:

1. Write to multiple backends in parallel
2. Wait for quorum acknowledgment (e.g., 2 of 3)
3. Return success to client
4. Handle backend failures gracefully

Guarantees:

• ✓ Survives single backend failure
• ✓ Data replicated across backends
• ✓ Configurable consistency (quorum)

Example use case:

# Critical configuration storage
namespace: feature-flags
needs:
  durability: strong
  consistency: strong  # → Prism adds Multicast Registry

What happens:

Write feature flag to 3 backends:
  Redis (primary)
  PostgreSQL (secondary)
  DynamoDB (tertiary)

Wait for 2 acknowledgments (quorum)
Return success

Read: Try Redis first, fallback to PostgreSQL/DynamoDB

Result: System survives Redis failure, data still available.

Pattern Composition

Prism combines multiple patterns to meet complex requirements.

Example: Durable Queue with Large Messages

Requirements:

needs:
  durability: strong          # Zero data loss
  max_message_size: 10MB      # Large payloads
  retention: 30days           # Long retention

Prism applies:

1. WAL pattern (durability)
2. Claim Check pattern (large messages)
3. Tiered Storage pattern (long retention)

Data flow:

Client sends 10MB message
  ↓
WAL: Write to disk (durability)
  ↓
Claim Check: Store payload in S3
  ↓
Queue: Send reference {s3_key: "..."}
  ↓
Tiered Storage: Move old references to cold storage after 7 days

Example: Transactional Outbox with CDC

Requirements:

client_api: transact
needs:
  consistency: strong     # Atomic operations
  ordered: true          # Event ordering

Prism applies:

1. Outbox pattern (transactional messaging)
2. CDC pattern (capture database changes)

Data flow:

BEGIN TRANSACTION
  INSERT INTO orders (...)
  INSERT INTO outbox (...)
COMMIT
  ↓
Outbox publisher: Read outbox table
  ↓
Publish to Kafka
  ↓
CDC: Capture order table changes
  ↓
Publish change events to separate stream

Result: Atomic order creation + immediate event + change log.

Pattern Performance Characteristics

Pattern	Latency Impact	Throughput Impact	Storage Impact
WAL	+1-3ms	-20% (disk I/O)	+10% (WAL files)
Claim Check	+10-50ms	-5% (S3 overhead)	+100% (object storage)
Outbox	+0ms (async)	-0%	+5% (outbox table)
CDC	+0ms (async)	-0%	+0% (log-based)
Tiered Storage	+50-200ms (cold)	-0%	-70% (cold storage)
Multicast	+5-20ms	-30% (parallel writes)	+200% (redundancy)

Choosing Patterns

Decision Matrix

Requirement	Pattern Applied
Zero data loss	WAL
Messages >100KB	Claim Check
Atomic database + messaging	Outbox
Database change streaming	CDC
Retention >30 days	Tiered Storage
Multi-backend redundancy	Multicast Registry

Example Decisions

Use case: E-commerce order processing

Requirements:

• Zero data loss (financial)
• Atomic order creation + notification
• Long-term audit trail

Patterns:

• WAL (durability)
• Outbox (atomic messaging)
• Tiered Storage (long retention)

---

Use case: Image processing pipeline

Requirements:

• Large file uploads (10MB+)
• High throughput
• No durability requirements

Patterns:

• Claim Check (large files)
• Best-effort durability (no WAL)

---

Use case: Real-time analytics

Requirements:

• High throughput (50K RPS)
• Low latency
• Short retention (24 hours)

Patterns:

• No WAL (best-effort durability)
• No Claim Check (small messages)
• No Tiered Storage (short retention)
• Direct to backend (NATS)

Pattern Monitoring

View Applied Patterns

prism namespace show user-events

# Output:
# Namespace: user-events
# Client API: pubsub
# Patterns:
#   - wal (durability: strong)
#   - claim_check (threshold: 100KB)
# Backend: Kafka + S3

Pattern Performance

prism namespace metrics user-events --patterns

# Output:
# WAL:
#   Write latency: 2.3ms (P99: 4.5ms)
#   Flush rate: 1000 ops/sec
#   WAL size: 1.2GB
#
# Claim Check:
#   S3 uploads: 50/sec
#   Upload latency: 25ms (P99: 80ms)
#   Objects stored: 50K

Key Takeaways

1. Patterns are automatic: Declare needs, Prism selects patterns
2. Patterns compose: Multiple patterns work together
3. Patterns have tradeoffs: Durability vs. latency, cost vs. redundancy
4. Patterns are transparent: Application code doesn't change
5. Monitor pattern impact: Check metrics to understand overhead

Next Steps

• Operations - Day-to-day operational tasks
• RFC-014: Layered Data Access Patterns - Deep dive into pattern composition