MEMO-092: Unified Namespace Model with Protobuf

Executive Summary

This memo establishes the unified namespace model where ALL namespaces natively support multi-pattern composition, slot-based backends, and authentication. Protobuf is the canonical format (ADR-003), clients construct NamespaceRequest messages via gRPC, and YAML/JSON is for export/preview only.

Key Principle: There is ONE namespace type, not "simple" vs "composable". Every namespace can compose patterns because that's how namespaces work.

Problem Statement

Current Misalignment

We've been building YAML parsers and example YAML files for namespace configuration, but this violates our core architectural principles:

1. ADR-003: Protobuf Single Source of Truth - Proto files should define all data models
2. ADR-002: Client-Originated Configuration - Clients declare needs to control plane
3. RFC-056: Unified Configuration Model - Layer 1 (NamespaceRequest) is canonical

Issues with YAML-First Approach

# Problem: YAML is parsed into Go structs, not protobuf
namespace: team-alpha
patterns:
  - name: session-manager
    type: session-manager
    # ... 50 more lines of YAML

Problems:

• YAML schema separate from protobuf schema (duplication)
• Validation logic in Go code (pkg/config/composable.go) not in proto
• Clients write YAML files instead of constructing protobuf messages
• Control plane receives parsed Go structs, not wire-format protobuf
• No versioning, no backward compatibility story
• Harder to generate client SDKs (Python, Rust, Java)

Solution: Protobuf-First with YAML Export

Canonical Data Flow

┌─────────────────────────────────────────────────────────────┐
│                    Protobuf-First Flow                      │
│                                                             │
│  Client SDK                                                 │
│  ├─ Build protobuf message                                 │
│  │  req := &configv1.ComposableNamespaceRequest{...}       │
│  │                                                          │
│  └─ Send via gRPC                                           │
│     └─> ControlPlane.CreateComposableNamespace(req)        │
│                                                             │
│  Control Plane                                              │
│  ├─ Receive protobuf on wire                               │
│  ├─ Validate using proto validation rules                  │
│  ├─ Process and assign backends                            │
│  └─ Return ComposableNamespaceResponse                     │
│                                                             │
│  Optional: Export to YAML (for documentation)              │
│  └─ protojson.Marshal(req) → JSON → yaml.Marshal()        │
│     (This is for HUMANS, not for config storage)           │
│                                                             │
└─────────────────────────────────────────────────────────────┘

YAML Role: Export/Preview Only

YAML/JSON serves documentation and preview purposes:

# Client constructs protobuf in code
prismctl namespace create \
  --from-code team-alpha.go

# Export to YAML for review
prismctl namespace export team-alpha --format yaml > team-alpha.yaml

# View in browser (web UI renders YAML for humans)
prismctl namespace view team-alpha

YAML is generated FROM protobuf, not the other way around.

Unified Namespace Design

Core Principle: No "Simple" vs "Composable"

We don't have two namespace types. There is ONE NamespaceRequest that supports:

• Single pattern or multiple patterns (both supported natively)
• Slot-based backend dependencies
• Authentication (optional)
• Session-aware patterns (optional)

We don't need backward compatibility - Prism is new, so we do it right from the start.

Files Updated

1. proto/prism/config/v1/namespace_request.proto (ENHANCED)

   • NamespaceRequest now includes patterns array (multi-pattern native)
   • Pattern - Individual pattern within namespace
   • Slot - Backend slot dependencies (requires/produces)
   • AuthConfig - Vault + JWT authentication
   • NamespaceResponse - Response with pattern assignments and slot bindings

2. proto/prism/control_plane.proto (SIMPLIFIED)

   • CreateNamespace RPC uses unified NamespaceRequest
   • No separate "CreateComposableNamespace" - unnecessary complexity

Example: Protobuf-Based Client Code

// examples/client/namespace_client.go

func BuildTeamAlphaNamespace() *configv1.NamespaceRequest {
    return &configv1.NamespaceRequest{
        Namespace: "team-alpha",
        Team:      "platform-team",
        Description: "Multi-pattern namespace with authentication",
        Needs: &configv1.NamespaceNeeds{
            Durability:  configv1.DurabilityLevel_DURABILITY_LEVEL_STRONG,
            WriteRps:    1000,
            ReadRps:     5000,
            DataSize:    "100GB",
            Consistency: configv1.ConsistencyLevel_CONSISTENCY_LEVEL_STRONG,
        },
        Patterns: []*configv1.Pattern{
            // SessionManager with Redis + NATS
            {
                Name: "session-manager",
                Type: configv1.PatternType_PATTERN_TYPE_SESSION_MANAGER,
                Requires: []*configv1.Slot{
                    {
                        Name:        "session-store",
                        PatternType: configv1.PatternType_PATTERN_TYPE_KEYVALUE,
                        BackendType: configv1.BackendType_BACKEND_TYPE_REDIS,
                    },
                    {
                        Name:        "session-events",
                        PatternType: configv1.PatternType_PATTERN_TYPE_PUBSUB,
                        BackendType: configv1.BackendType_BACKEND_TYPE_NATS,
                    },
                },
            },
            // Session-aware KeyValue with Postgres
            {
                Name:         "user-data",
                Type:         configv1.PatternType_PATTERN_TYPE_KEYVALUE,
                SessionAware: true,
                Requires: []*configv1.Slot{
                    {
                        Name:        "data-store",
                        PatternType: configv1.PatternType_PATTERN_TYPE_KEYVALUE,
                        BackendType: configv1.BackendType_BACKEND_TYPE_POSTGRES,
                    },
                },
            },
        },
        Auth: &configv1.AuthConfig{
            Enabled: true,
            Vault: &configv1.VaultAuthConfig{
                Address:    "https://vault.internal:8200",
                Role:       "prism-patterns",
                SecretPath: "database/creds/team-alpha",
            },
            Jwt: &configv1.JWTAuthConfig{
                Issuer:   "https://dex.prism.local:5556/dex",
                Audience: "prism-patterns",
            },
        },
    }
}

// Send to control plane
client := prism.NewControlPlaneClient(conn)
resp, err := client.CreateNamespace(ctx, req)

Key Benefits:

• Type safety from protobuf generated code
• Compiler catches missing required fields
• IDE autocomplete for all options
• Easy to version (add fields with field numbers)
• Works across languages (Python, Rust, Java, Go)

Migration Strategy

Phase 1: Unified Namespace Model ✅

Status: COMPLETE

• Enhanced namespace_request.proto with patterns, slots, auth built-in
• Updated CreateNamespace RPC to use unified model
• Created client example (namespace_client.go)
• No separate "composable" type - ONE namespace model

Phase 2: Refactor pkg/config (Pending)

Current State:

• ✅ Protobuf types defined and generating correctly
• ✅ Import cycle resolved
• ✅ Example client demonstrates protobuf usage
• ❌ pkg/config still uses custom Go structs

Goal: Make pkg/config consume protobuf, not define its own structs

Current pkg/config Structure (572 lines in composable.go):

• Custom structs: ComposableNamespace, ComposedPattern, PatternSlot, ComposableConfig
• Validation logic in Go code
• YAML loading via gopkg.in/yaml.v3
• Legacy format conversion for backward compatibility

Refactoring Plan:

1. Create protobuf loader (new file: pkg/config/protobuf_loader.go):

func LoadNamespaceProto(path string) (*configv1.NamespaceRequest, error) {
    // Read YAML file
    yamlBytes, err := os.ReadFile(path)

    // Convert YAML → JSON (intermediate)
    var yamlData map[string]interface{}
    yaml.Unmarshal(yamlBytes, &yamlData)
    jsonBytes, _ := json.Marshal(yamlData)

    // Unmarshal JSON → protobuf
    req := &configv1.NamespaceRequest{}
    if err := protojson.Unmarshal(jsonBytes, req); err != nil {
        return nil, err
    }

    // Validate using business logic
    if err := ValidateNamespace(req); err != nil {
        return nil, err
    }

    return req, nil
}

2. Add validation adapter (new file: pkg/config/validation.go):

func ValidateNamespace(req *configv1.NamespaceRequest) error {
    // Business logic validation (cross-field rules)
    // Rule: Max 1 SessionManager per namespace
    // Rule: SessionManager must have session-store slot
    // Rule: Session-aware patterns require SessionManager
    // Rule: Session-aware patterns cannot have static credentials
}

3. Deprecate custom structs:

• Mark composable.go as deprecated
• Keep for 1-2 releases for backward compatibility
• Add conversion functions: ComposableNamespace → NamespaceRequest

4. Update tests:

• Refactor composable_test.go to use protobuf types
• Keep existing test cases (validation rules unchanged)
• Add YAML → protobuf conversion tests

Benefits:

• Single source of validation (protobuf + business logic)
• YAML is just an import format
• Type-safe construction with protobuf
• Multi-language support (Python, Rust clients can use same proto)

Phase 3: Update Control Plane (Week 3)

Implement RPC handler:

// cmd/prism-admin/namespace_handler.go

func (s *ControlPlaneServer) CreateNamespace(
    ctx context.Context,
    req *configv1.NamespaceRequest,
) (*configv1.NamespaceResponse, error) {
    // Validate request (protobuf validation)
    if err := s.validator.Validate(req); err != nil {
        return nil, status.Errorf(codes.InvalidArgument, "validation failed: %v", err)
    }

    // Pattern selection (Layer 3)
    patterns, err := s.patternSelector.SelectPatterns(req)
    if err != nil {
        return nil, err
    }

    // Backend assignment (Layer 4)
    bindings, err := s.backendRegistry.AssignBackends(patterns)
    if err != nil {
        return nil, err
    }

    // Deploy patterns via launcher
    deployment, err := s.launcher.DeployPatterns(patterns, bindings)
    if err != nil {
        return nil, err
    }

    // Return response
    return &configv1.NamespaceResponse{
        Success:          true,
        NamespaceId:      req.Base.Namespace,
        AssignedPatterns: patterns,
        SlotBindings:     bindings,
        Deployment:       deployment,
    }, nil
}

Phase 4: Generate Client SDKs (Week 4)

Go:

buf generate
# Generates Go code in proto/gen/prism/config/v1

Python:

buf generate --template buf.gen.python.yaml
# Generates Python code with type stubs

Example Python client:

from prism.config.v1 import namespace_request_pb2
from prism import control_plane_pb2_grpc

req = namespace_request_pb2.NamespaceRequest(
    namespace="team-alpha",
    team="platform-team",
    description="Multi-pattern namespace",
    patterns=[
        namespace_request_pb2.Pattern(
            name="session-manager",
            type=namespace_request_pb2.PATTERN_TYPE_SESSION_MANAGER,
            requires=[
                namespace_request_pb2.Slot(
                    name="session-store",
                    pattern_type=namespace_request_pb2.PATTERN_TYPE_KEYVALUE,
                    backend_type=namespace_request_pb2.BACKEND_TYPE_REDIS,
                ),
            ],
        ),
    ],
)

stub = control_plane_pb2_grpc.ControlPlaneStub(channel)
resp = stub.CreateNamespace(req)

Validation Strategy

Protobuf Validation with buf validate

Add validation rules in proto:

import "buf/validate/validate.proto";

message NamespaceRequest {
  string namespace = 1 [(buf.validate.field).required = true];
  string team = 2 [(buf.validate.field).required = true];

  repeated Pattern patterns = 3 [
    (buf.validate.field).required = true,
    (buf.validate.field).repeated.min_items = 1
  ];
}

message Pattern {
  string name = 1 [
    (buf.validate.field).required = true,
    (buf.validate.field).string.pattern = "^[a-z][a-z0-9-]{2,62}$"
  ];

  PatternType type = 2 [
    (buf.validate.field).required = true,
    (buf.validate.field).enum.defined_only = true
  ];
}

Validation at runtime:

import "github.com/bufbuild/protovalidate-go"

validator, _ := protovalidate.New()
if err := validator.Validate(req); err != nil {
    return nil, fmt.Errorf("invalid request: %w", err)
}

Business Logic Validation

Some rules require cross-field validation:

func ValidateNamespace(req *configv1.NamespaceRequest) error {
    // Rule: Max 1 SessionManager per namespace
    sessionManagers := countPatternsByType(req.Patterns,
        configv1.ComposablePatternType_COMPOSABLE_PATTERN_TYPE_SESSION_MANAGER)
    if sessionManagers > 1 {
        return fmt.Errorf("namespace can have at most 1 SessionManager, found %d", sessionManagers)
    }

    // Rule: SessionManager must have session-store slot
    if sessionManagers == 1 {
        sm := findSessionManager(req.Patterns)
        if !hasSlot(sm.Requires, "session-store") {
            return fmt.Errorf("SessionManager must have session-store slot")
        }
    }

    // Rule: Session-aware patterns require SessionManager
    for _, p := range req.Patterns {
        if p.SessionAware && sessionManagers == 0 {
            return fmt.Errorf("pattern %s is session-aware but no SessionManager found", p.Name)
        }
    }

    // Rule: Session-aware patterns cannot have credentials
    // (Credentials come from SessionManager via Vault)
    for _, p := range req.Patterns {
        if p.SessionAware && hasCredentials(p) {
            return fmt.Errorf("session-aware pattern %s cannot have static credentials", p.Name)
        }
    }

    return nil
}

Benefits of Protobuf-First Approach

1. Single Source of Truth

• Schema defined once in .proto files
• Generated code for all languages
• No drift between Go structs, YAML schema, validation logic

2. Versioning and Compatibility

// Add new field with field number
message ComposableNamespaceRequest {
  NamespaceRequest base = 1;
  repeated ComposedPattern patterns = 2;
  NamespaceAuthConfig auth = 3;

  // NEW in v2: Add deployment preferences
  DeploymentPreferences deployment = 4; // Clients without this field still work
}

Protobuf ensures backward/forward compatibility with field numbers.

3. Language Portability

Same proto generates code for:

• Go (buf generate)
• Python (buf generate --template buf.gen.python.yaml)
• Rust (buf generate --template buf.gen.rust.yaml)
• Java, TypeScript, Ruby, etc.

4. Type Safety

// Compiler catches this at build time
req := &configv1.ComposableNamespaceRequest{
    Patterns: []*configv1.ComposedPattern{
        {
            // ERROR: PatternType is required field
            Name: "session-manager",
        },
    },
}

YAML doesn't catch missing required fields until runtime.

5. Tooling Support

• buf lint - Lint proto files for best practices
• buf breaking - Detect breaking changes
• buf format - Auto-format proto files
• protovalidate - Runtime validation from proto annotations

6. Documentation Generation

buf generate --template buf.gen.docs.yaml
# Generates markdown docs from proto comments

Proto comments become API documentation automatically.

Comparison: YAML vs Protobuf

Aspect	YAML-First	Protobuf-First
Schema	Separate schema doc	Proto defines schema
Validation	Go code	Proto + buf validate
Versioning	No built-in support	Field numbers
Type Safety	Runtime errors	Compile-time errors
Multi-language	Manual porting	Auto-generated
Wire Format	Text (inefficient)	Binary (efficient)
Compatibility	Manual checks	Protobuf guarantees
Tooling	Limited	buf, protovalidate
Documentation	Separate docs	Generated from proto

YAML Usage: Export and Preview

YAML still has value for human readability:

Export Namespace to YAML

func ExportNamespaceToYAML(req *configv1.NamespaceRequest) (string, error) {
    // Convert protobuf → JSON
    jsonBytes, err := protojson.Marshal(req)
    if err != nil {
        return "", err
    }

    // Convert JSON → YAML
    var data interface{}
    json.Unmarshal(jsonBytes, &data)
    yamlBytes, err := yaml.Marshal(data)
    if err != nil {
        return "", err
    }

    return string(yamlBytes), nil
}

CLI Tool

# Create namespace from Go code (protobuf)
prismctl namespace create --from-code team-alpha.go

# Export to YAML for documentation
prismctl namespace export team-alpha > team-alpha.yaml

# View in web UI (renders YAML for humans)
prismctl namespace view team-alpha

# Import from YAML (convenience feature)
# Internally: YAML → JSON → protobuf → gRPC
prismctl namespace create --from-yaml team-alpha.yaml

Key Point: YAML import is a convenience feature that converts to protobuf internally.

Implementation Roadmap

Week 1: Unified Namespace Model ✅

• ✅ Enhanced namespace_request.proto with patterns, slots, auth
• ✅ Updated CreateNamespace RPC to use unified model
• ✅ Created Go client example (namespace_client.go)
• ✅ Removed unnecessary "composable" complexity
• ✅ Documented unified approach

Week 2: Refactor pkg/config

• Update pkg/config/loader.go to consume protobuf
• Remove custom Go structs (pkg/config/composable.go)
• Add YAML → protobuf conversion utility
• Update tests to use protobuf messages

Week 3: Control Plane Implementation

• Implement CreateComposableNamespace RPC handler
• Add protobuf validation
• Add business logic validation
• Integration test with protobuf client

Week 4: Client SDKs and Tooling

• Generate Python client SDK
• Generate Rust client SDK
• Create prismctl namespace commands
• Add YAML export utility
• Documentation generation from proto

Related Documents

• ADR-003: Protobuf Single Source of Truth
• ADR-002: Client-Originated Configuration
• MEMO-086: Composable Pattern Architecture with SessionManager
• MEMO-087: Composable Patterns Implementation Progress

Appendix: Proto File Structure

proto/prism/config/v1/
├── namespace_request.proto       # UNIFIED: Patterns, slots, auth built-in
├── platform_policy.proto         # Layer 2: Team permissions
├── pattern_selection.proto       # Layer 3: Pattern selection (HOW)
└── backend_registry.proto        # Layer 4: Operator backends

proto/prism/
└── control_plane.proto           # SIMPLIFIED: CreateNamespace uses unified model

Revision History

• 2025-11-18: Initial protobuf-first architecture design
• 2025-11-18: Refactored to unified namespace model (removed "composable" complexity)