MEMO-084: Vault Operator Guide for Prism

Executive Summary

This guide provides step-by-step instructions for operators to configure HashiCorp Vault for use with Prism's authentication system. It covers JWT auth method setup, database secrets engine configuration, policy creation, and operational best practices.

Prerequisites

Required Components

1. HashiCorp Vault 1.12+ (1.15+ recommended)
2. Dex OIDC Provider (for JWT token issuance)
3. Backend Databases:
   • Redis 7.0+
   • PostgreSQL 14+
   • Kafka (optional, for dynamic ACLs)
4. Network Connectivity:
   • Pattern plugins → Vault (port 8200)
   • Vault → Backend databases
   • Vault → Dex OIDC provider

Operator Permissions

• Vault root token or admin policy
• Database admin credentials
• Dex admin access for OIDC configuration

Architecture Overview

┌─────────────┐
│   Client    │
└──────┬──────┘
       │ JWT token
       ↓
┌─────────────────┐
│ Pattern Plugin  │
└────────┬────────┘
         │ 1. Validate JWT (JWKS)
         ↓
┌────────────────────────────┐
│        Vault               │
│  ┌──────────────────────┐  │
│  │  JWT Auth Method     │  │ 2. Exchange JWT → Vault token
│  └──────────────────────┘  │
│  ┌──────────────────────┐  │
│  │ Database Secrets     │  │ 3. Generate backend credentials
│  │ Engine               │  │
│  └──────────────────────┘  │
└────────┬───────────────────┘
         │ Username: v-jwt-alice-abc123
         │ Password: <random>
         ↓
┌────────────────────────────┐
│   Backend (Redis/PG)       │
└────────────────────────────┘

Step 1: Initialize Vault

Development Setup (Single Node)

# Start Vault in dev mode (NOT for production)
vault server -dev -dev-root-token-id="root-token"

# Export environment variables
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root-token'

Production Setup (HA Cluster)

# vault-config.hcl
storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/vault/tls/vault.crt"
  tls_key_file  = "/vault/tls/vault.key"
}

api_addr = "https://vault-1.prism.internal:8200"
cluster_addr = "https://vault-1.prism.internal:8201"

ui = true

# Start Vault server
vault server -config=/vault/config/vault-config.hcl

# Initialize Vault (first node only)
vault operator init -key-shares=5 -key-threshold=3

# Unseal Vault (on each node, 3 of 5 keys required)
vault operator unseal <key-1>
vault operator unseal <key-2>
vault operator unseal <key-3>

# Log in with root token
vault login <root-token>

CRITICAL: Store unseal keys and root token securely (HashiCorp Vault, AWS Secrets Manager, etc.).

Step 2: Configure JWT Auth Method

Enable JWT Auth

# Enable JWT auth method
vault auth enable jwt

# Verify enabled
vault auth list

Configure OIDC Provider (Dex)

# Configure JWT auth with Dex OIDC discovery
vault write auth/jwt/config \
    oidc_discovery_url="https://dex.prism.local:5556/dex" \
    oidc_discovery_ca_pem=@/etc/vault/dex-ca.pem \
    default_role="prism-patterns"

Create JWT Role for Pattern Plugins

# Create role for pattern plugins
vault write auth/jwt/role/prism-patterns \
    role_type="jwt" \
    bound_audiences="prism-patterns" \
    user_claim="sub" \
    groups_claim="groups" \
    token_ttl="1h" \
    token_max_ttl="2h" \
    token_policies="prism-patterns-policy"

Role Parameters:

• bound_audiences: JWT aud claim must match "prism-patterns"
• user_claim: User ID extracted from sub claim
• groups_claim: User groups extracted from groups claim
• token_ttl: Initial Vault token TTL (1 hour)
• token_max_ttl: Maximum renewal time (2 hours)
• token_policies: Vault policy attached to generated tokens

Test JWT Authentication

# Get a JWT token from Dex (use Dex OAuth2 flow)
JWT_TOKEN="<jwt-token-from-dex>"

# Test authentication
vault write auth/jwt/login \
    role="prism-patterns" \
    jwt="$JWT_TOKEN"

# Expected output:
# Key                  Value
# ---                  -----
# token                hvs.CAESIAbc123...
# token_accessor       abc123...
# token_duration       1h
# token_policies       [default prism-patterns-policy]

Step 3: Configure Database Secrets Engine

Enable Database Secrets Engine

# Enable database secrets engine
vault secrets enable database

# Verify enabled
vault secrets list

Configure Redis Backend

# Configure Redis connection
vault write database/config/redis \
    plugin_name="redis-database-plugin" \
    host="redis.prism.internal" \
    port=6379 \
    username="vault-admin" \
    password="admin-password" \
    allowed_roles="redis-role"

# Create role for dynamic Redis credentials
vault write database/roles/redis-role \
    db_name="redis" \
    creation_statements='["ACL SETUSER {{username}} on >{{password}} ~* +@all"]' \
    revocation_statements='["ACL DELUSER {{username}}"]' \
    default_ttl="1h" \
    max_ttl="2h"

Redis ACL Template:

• {{username}}: Vault-generated username (format: v-jwt-alice-abc123)
• {{password}}: Vault-generated password (random 32 characters)
• ~*: Allow access to all keys
• +@all: Allow all Redis commands

Configure PostgreSQL Backend

# Configure PostgreSQL connection
vault write database/config/postgres \
    plugin_name="postgresql-database-plugin" \
    connection_url="postgresql://{{username}}:{{password}}@postgres.prism.internal:5432/prism?sslmode=require" \
    username="vault-admin" \
    password="admin-password" \
    allowed_roles="postgres-role"

# Create role for dynamic PostgreSQL credentials
vault write database/roles/postgres-role \
    db_name="postgres" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="2h"

Test Database Credentials

# Generate Redis credentials
vault read database/creds/redis-role

# Expected output:
# Key                Value
# ---                -----
# lease_id           database/creds/redis-role/abc123
# lease_duration     1h
# lease_renewable    true
# password           A1a-random-password-xyz789
# username           v-jwt-alice-abc123

# Test Redis login
redis-cli -h redis.prism.internal
AUTH v-jwt-alice-abc123 A1a-random-password-xyz789
PING
# PONG

Step 4: Create Vault Policies

Pattern Plugin Policy

Create prism-patterns-policy.hcl:

# Allow token renewal
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Allow lease renewal
path "sys/leases/renew" {
  capabilities = ["update"]
}

# Allow lease revocation
path "sys/leases/revoke" {
  capabilities = ["update"]
}

# Allow reading Redis credentials
path "database/creds/redis-role" {
  capabilities = ["read"]
}

# Allow reading PostgreSQL credentials
path "database/creds/postgres-role" {
  capabilities = ["read"]
}

# Allow reading Kafka credentials (optional)
path "database/creds/kafka-role" {
  capabilities = ["read"]
}

Apply policy:

vault policy write prism-patterns-policy prism-patterns-policy.hcl

Service Account Policy (Kubernetes)

Create prism-service-policy.hcl:

# Same permissions as pattern plugin policy
# Plus additional service-specific paths if needed

path "auth/token/renew-self" {
  capabilities = ["update"]
}

path "sys/leases/renew" {
  capabilities = ["update"]
}

path "sys/leases/revoke" {
  capabilities = ["update"]
}

path "database/creds/*" {
  capabilities = ["read"]
}

Apply policy:

vault policy write prism-service-policy prism-service-policy.hcl

Step 5: Configure Kubernetes Auth (Optional)

If deploying pattern plugins in Kubernetes:

# Enable Kubernetes auth method
vault auth enable kubernetes

# Configure Kubernetes auth with K8s API server
vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# Create role for pattern plugin ServiceAccounts
vault write auth/kubernetes/role/prism-patterns \
    bound_service_account_names="keyvalue-sa,consumer-sa,producer-sa" \
    bound_service_account_namespaces="prism-prod" \
    policies="prism-service-policy" \
    token_ttl="24h" \
    token_max_ttl="48h"

Step 6: Configure AWS Auth (Optional)

If deploying pattern plugins in AWS:

# Enable AWS auth method
vault auth enable aws

# Configure AWS auth
vault write auth/aws/config/client \
    access_key="<aws-access-key>" \
    secret_key="<aws-secret-key>"

# Create role for EC2 instances
vault write auth/aws/role/prism-patterns-ec2 \
    auth_type="ec2" \
    bound_ami_id="ami-abc123" \
    bound_account_id="123456789012" \
    policies="prism-service-policy" \
    token_ttl="24h" \
    token_max_ttl="48h"

# Create role for IAM roles
vault write auth/aws/role/prism-patterns-iam \
    auth_type="iam" \
    bound_iam_principal_arn="arn:aws:iam::123456789012:role/prism-patterns" \
    policies="prism-service-policy" \
    token_ttl="24h" \
    token_max_ttl="48h"

Step 7: Enable Audit Logging

# Enable file audit logging
vault audit enable file file_path=/vault/logs/audit.log

# Enable syslog audit logging
vault audit enable syslog tag="vault" facility="AUTH"

# Verify audit backends
vault audit list

Audit Log Contents:

• All authentication attempts
• Credential generation events
• Lease renewals and revocations
• Policy changes

Step 8: Pattern Plugin Configuration

Update pattern plugin configuration to use Vault:

# patterns/keyvalue/config.yaml
auth:
  vault:
    enabled: true
    address: https://vault.prism.internal:8200

    # JWT authentication
    jwt_auth:
      role: prism-patterns
      auth_path: auth/jwt

    # Backend credentials
    credentials:
      secret_path: database/creds/redis-role
      renew_interval: 1800s  # 30 minutes

    # TLS configuration
    tls:
      enabled: true
      ca_cert: /etc/prism/vault-ca.pem
      skip_verify: false

  # Token validation
  token:
    enabled: true
    issuer: https://dex.prism.local:5556/dex
    audience: prism-patterns
    cache_ttl: 1h

Operational Tasks

Rotate Vault Root Token

# Generate new root token
vault operator generate-root -init

# Provide unseal keys when prompted
vault operator generate-root -nonce=<nonce> <unseal-key-1>
vault operator generate-root -nonce=<nonce> <unseal-key-2>
vault operator generate-root -nonce=<nonce> <unseal-key-3>

# Decode new root token
vault operator generate-root -decode=<encoded-token> -otp=<otp>

# Revoke old root token
vault token revoke <old-root-token>

Rotate Database Admin Credentials

# Update Redis admin password
vault write database/rotate-root/redis

# Update PostgreSQL admin password
vault write database/rotate-root/postgres

Monitor Lease Usage

# List active leases
vault list sys/leases/lookup/database/creds/redis-role

# Count active leases
vault list -format=json sys/leases/lookup/database/creds/redis-role | jq '. | length'

# Revoke specific lease
vault lease revoke database/creds/redis-role/abc123

# Revoke all leases for a role
vault lease revoke -prefix database/creds/redis-role

Backup Vault Data

# Take Raft snapshot (HA cluster)
vault operator raft snapshot save vault-snapshot.snap

# Restore from snapshot
vault operator raft snapshot restore vault-snapshot.snap

Upgrade Vault

# 1. Take snapshot
vault operator raft snapshot save pre-upgrade-snapshot.snap

# 2. Upgrade follower nodes one at a time
systemctl stop vault
# Update Vault binary
systemctl start vault
vault operator unseal <key-1>
vault operator unseal <key-2>
vault operator unseal <key-3>

# 3. Step down leader and upgrade
vault operator step-down
# Upgrade leader node same as followers

# 4. Verify cluster health
vault operator raft list-peers

Security Hardening

TLS Configuration

# Generate TLS certificates (use cert-manager in K8s or internal CA)
# vault.crt, vault.key, ca.crt

# Update Vault config
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/vault/tls/vault.crt"
  tls_key_file  = "/vault/tls/vault.key"
  tls_min_version = "tls12"
}

Seal Vault with Cloud KMS

AWS KMS:

seal "awskms" {
  region     = "us-west-2"
  kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/abc123"
}

GCP KMS:

seal "gcpckms" {
  project    = "prism-prod"
  region     = "us-west1"
  key_ring   = "vault-keyring"
  crypto_key = "vault-key"
}

Azure Key Vault:

seal "azurekeyvault" {
  tenant_id     = "<tenant-id>"
  vault_name    = "prism-vault"
  key_name      = "vault-key"
}

Network Policies

Kubernetes NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-access
  namespace: prism-prod
spec:
  podSelector:
    matchLabels:
      app: vault
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: keyvalue-runner
    - podSelector:
        matchLabels:
          app: consumer-runner
    - podSelector:
        matchLabels:
          app: producer-runner
    ports:
    - protocol: TCP
      port: 8200

Monitoring and Alerting

Prometheus Metrics

# prometheus.yml
scrape_configs:
  - job_name: 'vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    bearer_token: '<vault-token-with-metrics-policy>'
    static_configs:
      - targets: ['vault.prism.internal:8200']

Key Metrics

• vault_core_unsealed: Vault seal status (0 = sealed, 1 = unsealed)
• vault_token_creation_count: Token creation rate
• vault_token_count: Active token count
• vault_runtime_alloc_bytes: Memory usage
• vault_database_secrets_creation_count: Credential generation rate

Grafana Dashboard

Import Vault dashboard: https://grafana.com/grafana/dashboards/12904

Alerts

# alerts.yml
groups:
- name: vault
  rules:
  - alert: VaultSealed
    expr: vault_core_unsealed == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Vault is sealed"

  - alert: VaultDown
    expr: up{job="vault"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Vault is down"

  - alert: HighLeaseCount
    expr: vault_token_count > 10000
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High active lease count"

Troubleshooting

See MEMO-085: Vault Troubleshooting Guide for detailed troubleshooting procedures.

Related Documents

• RFC-062: Unified Authentication and Session Management
• ADR-007: Authentication and Authorization
• MEMO-008: Vault Token Exchange Flow
• MEMO-083: Phase 2 Vault Integration Implementation Plan
• MEMO-085: Vault Troubleshooting Guide

Revision History

• 2025-11-17: Initial operator guide for Vault setup