Using Prism

Read time: 15 minutes

Requesting a Namespace

Step 1: Choose Your Client API

Select the API that matches your use case:

Client API	Use Case	Example
keyvalue	Store/retrieve by key	Session storage, caching, config
pubsub	Event publish/subscribe	Notifications, real-time updates
queue	Ordered message processing	Background jobs, task queues
reader	Read-only data access	Analytics, reporting
transact	Multi-operation transactions	Financial operations, ACID needs

Step 2: Define Your Needs

Create a namespace configuration file:

# my-namespace.yaml
namespace: user-events
team: analytics-team

# What API do you need?
client_api: pubsub

# What are your requirements?
needs:
  # Capacity estimates
  write_rps: 1000           # Peak writes per second
  read_rps: 5000            # Peak reads per second
  data_size: 10GB           # Estimated total data size
  retention: 7days          # How long to keep data

  # Reliability requirements
  durability: strong        # strong | eventual | best-effort
  ordered: true             # Preserve message order
  consistency: eventual     # strong | eventual | bounded_staleness

  # Message characteristics
  max_message_size: 100KB   # Largest expected message

# Who can access this namespace?
access:
  owners:
    - team: analytics-team
  consumers:
    - service: analytics-api      # Read-write
    - service: data-pipeline      # Read-only

# Compliance requirements
policies:
  pii_handling: enabled     # Enable automatic PII handling
  audit_logging: enabled    # Log all access

Step 3: Submit the Request

# Submit namespace request
prism namespace create my-namespace.yaml

# Output:
# ✓ Namespace user-events created
# ✓ Backend provisioned: NATS subject "user-events"
# ✓ Access policies configured
# ✓ Ready to use

Step 4: Use the Namespace

# Python client example
from prism import Client

client = Client(namespace="user-events")

# Publish event
client.publish({
  "user_id": 123,
  "action": "login",
  "timestamp": "2025-10-19T10:30:00Z"
})

# Subscribe to events
for event in client.subscribe():
    print(f"Event: {event}")

Choosing Durability Levels

Durability determines how Prism handles data persistence.

Best-Effort Durability

When to use: Non-critical data, metrics, real-time dashboards

Guarantees:

• ✓ Fast (no disk writes)
• ✓ Low latency (<1ms)
• ✗ May lose data on crash

Configuration:

needs:
  durability: best-effort

What Prism does:

• Writes directly to backend (e.g., NATS)
• No WAL (Write-Ahead Log)
• Acknowledges immediately

Use cases:

• Live metrics (ok to lose a few data points)
• Real-time dashboards
• Debug logs

Eventual Durability

When to use: Important data, can tolerate small data loss

Guarantees:

• ✓ Fast (asynchronous disk writes)
• ✓ Low latency (1-2ms)
• ✓ Survives most crashes
• ✗ May lose seconds of data on crash

Configuration:

needs:
  durability: eventual

What Prism does:

• Writes to backend immediately
• Writes to WAL asynchronously (no fsync)
• Acknowledges after backend write

Use cases:

• User activity logs
• Application events
• Non-financial transactions

Strong Durability

When to use: Critical data, zero loss tolerance

Guarantees:

• ✓ Zero data loss
• ✓ Survives all crashes
• ✓ Guaranteed persistence
• ✗ Higher latency (2-5ms)

Configuration:

needs:
  durability: strong

What Prism does:

• Writes to WAL with fsync (disk flush)
• Acknowledges only after disk persistence
• Asynchronously flushes to backend

Use cases:

• Financial transactions
• Order processing
• Audit logs
• Compliance data

Durability Comparison

Level	Latency	Data Loss Risk	Use Case
best-effort	<1ms	Seconds to minutes	Metrics, logs
eventual	1-2ms	Seconds	Activity tracking
strong	2-5ms	Zero	Financial data

Estimating Capacity

Prism uses capacity estimates to provision backends.

Write RPS (Requests Per Second)

needs:
  write_rps: 5000  # Peak writes per second

How to estimate:

1. Measure current peak write rate
2. Add 50% buffer for growth
3. Round up to nearest thousand

Example:

Current peak: 3,200 writes/sec
With 50% buffer: 4,800 writes/sec
Rounded up: 5,000 writes/sec

Read RPS

needs:
  read_rps: 10000  # Peak reads per second

Note: Reads are often higher than writes (especially for caching, pub/sub).

Data Size

needs:
  data_size: 100GB  # Total data size

How to estimate:

Estimated messages: 10 million
Average message size: 10 KB
Total: 10M * 10KB = 100GB

Retention

needs:
  retention: 30days  # How long to keep data

Retention determines:

• Storage requirements
• Replay capability window
• Compliance adherence

Common values:

• 1day - Temporary data, metrics
• 7days - Application events, logs
• 30days - Business events, compliance
• 90days - Financial data, audit logs
• 365days - Long-term compliance

Access Control

Team Ownership

Every namespace has an owning team:

access:
  owners:
    - team: analytics-team

Owners can:

• Read and write data
• Modify namespace configuration
• Grant access to other teams
• Delete the namespace

Consumer Access

Grant access to other services:

access:
  consumers:
    - service: analytics-api       # Full read-write access
      permissions: [read, write]

    - service: reporting-dashboard # Read-only access
      permissions: [read]

    - service: data-pipeline       # Write-only access
      permissions: [write]

Permission Types

Permission	Operations Allowed
read	Subscribe, consume, query
write	Publish, send, append
admin	All operations + config changes

Compliance Policies

PII Handling

Enable automatic PII detection and handling:

policies:
  pii_handling: enabled

What this enables:

• Automatic encryption of annotated PII fields
• Log masking for PII
• Audit trails for PII access
• Schema validation for PII annotations

Example protobuf:

message User {
  string user_id = 1;
  string email = 2 [(prism.pii) = "email"];  // Auto-encrypted
}

Audit Logging

Track all data access:

policies:
  audit_logging: enabled

Logs include:

• Who accessed the data (service identity)
• When (timestamp)
• What operation (read/write)
• Which records (key/offset)

Example audit log:

{
  "timestamp": "2025-10-19T10:30:00Z",
  "namespace": "user-events",
  "service": "analytics-api",
  "operation": "read",
  "record_key": "user:123",
  "compliance": "pci"
}

Compliance Standards

Declare compliance requirements:

policies:
  compliance: pci  # pci | hipaa | gdpr | sox

Compliance controls:

Standard	Controls Applied
pci	Encryption, audit logs, access restrictions
hipaa	PHI encryption, audit trails, data retention
gdpr	Right to erasure, data portability, consent tracking
sox	Immutable audit logs, access controls, retention

Common Configuration Patterns

High-Throughput Event Streaming

namespace: click-events
client_api: pubsub
needs:
  write_rps: 50000      # High write volume
  read_rps: 100000      # High read volume
  durability: eventual  # Tolerate small loss
  retention: 7days
  max_message_size: 10KB

Prism provisions: Kafka with 50 partitions, consumer groups.

Durable Task Queue

namespace: order-processing
client_api: queue
needs:
  write_rps: 1000
  durability: strong     # Zero data loss
  ordered: true          # Process in order
  retention: 30days

Prism provisions: PostgreSQL + Kafka with WAL pattern.

Session Storage

namespace: user-sessions
client_api: keyvalue
needs:
  read_rps: 10000       # High read volume
  write_rps: 2000
  durability: eventual  # Sessions can be recreated
  retention: 1day       # Short-lived

Prism provisions: Redis with automatic expiration.

Financial Transactions

namespace: payments
client_api: transact
needs:
  write_rps: 500
  durability: strong     # Zero loss tolerance
  consistency: strong    # ACID guarantees
  retention: 365days
policies:
  pii_handling: enabled
  audit_logging: enabled
  compliance: pci

Prism provisions: PostgreSQL with transactional patterns, full audit trail.

Analytics Data Collection

namespace: metrics
client_api: reader
needs:
  write_rps: 5000
  durability: best-effort  # Ok to lose some metrics
  retention: 30days
  max_message_size: 1KB    # Small metrics

Prism provisions: ClickHouse for fast analytics queries.

Modifying Namespaces

Scaling Capacity

Update capacity estimates:

# Update namespace config
prism namespace update user-events \
  --write-rps 10000 \
  --read-rps 50000

# Prism automatically:
# - Increases backend partitions
# - Adjusts connection pools
# - Updates resource allocation

Changing Retention

prism namespace update user-events \
  --retention 30days

# Prism automatically:
# - Updates backend retention policies
# - Migrates existing data (if needed)
# - Adjusts storage allocation

Adding Consumers

prism namespace grant user-events \
  --service new-dashboard \
  --permissions read

# Prism automatically:
# - Creates access policies
# - Issues credentials
# - Updates audit logs

Monitoring Your Namespace

View Namespace Status

prism namespace status user-events

# Output:
# Namespace: user-events
# Status: healthy
# Backend: NATS subject "user-events"
# Consumers: 3 active
# Write RPS: 850 (capacity: 1000)
# Read RPS: 4200 (capacity: 5000)
# Storage: 2.3GB / 10GB

View Metrics

prism namespace metrics user-events

# Output:
# Latency (P50/P99): 0.8ms / 3.2ms
# Throughput: 850 writes/sec, 4200 reads/sec
# Errors: 0.01% (5 errors in last hour)
# Backend health: healthy

View Audit Logs

prism namespace audit user-events --last 1h

# Output:
# 2025-10-19 10:30:00 analytics-api READ user:123
# 2025-10-19 10:30:01 analytics-api WRITE event:456
# 2025-10-19 10:30:02 data-pipeline READ stream:789

Troubleshooting

Namespace Creation Failed

Symptom: prism namespace create returns error

Common causes:

1. Invalid configuration:

   # Check config syntax
   prism namespace validate my-namespace.yaml

2. Capacity too high:

   Error: Requested write_rps (100000) exceeds limit (50000)
   Solution: Contact platform team for capacity increase

3. Namespace already exists:

   Error: Namespace "user-events" already exists
   Solution: Choose different name or use existing namespace

High Latency

Symptom: Slow read/write operations

Diagnosis:

prism namespace metrics user-events

# Check:
# - P99 latency > 50ms? (Backend overloaded)
# - Error rate > 1%? (Backend connectivity issues)
# - Write RPS near capacity? (Need to scale)

Solutions:

1. Scale capacity:

   prism namespace update user-events --write-rps 10000

2. Check backend health:

   prism namespace status user-events
   # Look for "Backend health: degraded"

Data Loss

Symptom: Messages/events missing

Diagnosis:

1. Check durability setting:

   prism namespace show user-events
   # Look for "durability: best-effort" (may lose data)

2. Check audit logs:

   prism namespace audit user-events --last 24h
   # Look for gaps in timestamps

Solutions:

1. Increase durability:

   needs:
     durability: strong  # Zero data loss

2. Enable replay:

   needs:
     retention: 30days  # Keep data for replay

Key Takeaways

1. Define clear requirements: Be specific about capacity, durability, and retention
2. Choose appropriate durability: Balance latency vs. data loss tolerance
3. Estimate capacity generously: Add 50% buffer for growth
4. Use compliance policies: Enable PII handling and audit logging for sensitive data
5. Monitor namespace health: Use prism namespace status and metrics regularly

Next Steps

• Data Patterns - Learn about Prism's reliability patterns
• Operations - Day-to-day operational tasks