MEMO-037: Launcher Callback Protocol with Dynamic Port Allocation

Overview

Implemented a launcher callback protocol that enables launched processes to use dynamic port allocation (port 0) and report their actual ports back to the launcher. This eliminates hard-coded ports and enables automatic topology discovery.

Problem Statement

Previously, all processes required hard-coded port assignments:

• Admin instances: 8981, 8982, 8983
• Proxy instances: 9090, 9091, 9092
• Pattern runners: 9095+

This approach had several limitations:

1. Port conflicts in development
2. Complex port management for multiple instances
3. No way for processes to discover each other dynamically
4. Difficult to scale beyond pre-allocated port ranges

Solution: Launcher Callback Protocol

Architecture

┌─────────────────┐
│ Pattern Runner  │
│                 │
│  1. Get port 0  │──┐
│  2. Handshake   │  │
│  3. Discover    │  │ LauncherControl gRPC
│  4. Heartbeat   │  │
└─────────────────┘  │
                     ▼
┌─────────────────┐
│ Prism Launcher  │
│                 │
│  Track ports    │──┐
│  Forward to     │  │
│    admin        │  │ Forward handshake
│  Relay topology │  │
└─────────────────┘  │
                     ▼
┌─────────────────┐
│ Admin Cluster   │
│                 │
│  Register       │
│    process      │
│  Return topo    │
└─────────────────┘

Protocol Flow

1. Process Launch: Launcher spawns process with --admin=launcher://localhost:7070 and --process-id=unique-id
2. Dynamic Port: Process binds to port 0, OS assigns actual port
3. Handshake: Process calls ProcessHandshake RPC with actual port
4. Topology Discovery: Launcher returns all admin/proxy endpoints
5. Heartbeat: Process sends periodic heartbeats (30s interval)
6. Shutdown: Process notifies launcher on graceful exit

Protobuf Service Definition

service LauncherControl {
  rpc ProcessHandshake(ProcessHandshakeRequest) returns (ProcessHandshakeResponse);
  rpc ProcessHeartbeat(ProcessHeartbeatRequest) returns (ProcessHeartbeatResponse);
  rpc ProcessShutdown(ProcessShutdownRequest) returns (ProcessShutdownResponse);
}

message ProcessHandshakeRequest {
  string process_id = 1;        // Unique ID assigned at launch
  string process_type = 2;       // "proxy", "pattern-runner", "admin"
  int32 actual_port = 3;         // Port assigned by OS
  string hostname = 4;           // Hostname or IP
  int64 timestamp = 5;           // Unix timestamp
  map<string, string> metadata = 6;
}

message ProcessHandshakeResponse {
  bool success = 1;
  string message = 2;
  string launcher_id = 3;
  repeated string admin_endpoints = 4;   // All admin endpoints
  repeated string proxy_endpoints = 5;   // All proxy endpoints
  string namespace = 6;
  map<string, string> config = 7;
}

Implementation

1. Launcher Client Library (pkg/launcherclient/)

Core client library for launched processes:

// Parse admin endpoint to detect launcher
launcherAddr, isLauncher, err := launcherclient.ParseAdminEndpoint("launcher://localhost:7070")

// Get dynamic port from OS
listener, actualPort, err := launcherclient.GetActualPort()

// Create client
client, err := launcherclient.New(&launcherclient.Config{
    LauncherAddr: launcherAddr,
    ProcessID:    processID,
    ProcessType:  "pattern-runner",
    ActualPort:   actualPort,
})

// Handshake and discover topology
topology, err := client.Handshake(ctx)
// topology.AdminEndpoints = ["localhost:8981", "localhost:8982", ...]
// topology.ProxyEndpoints = ["localhost:9090", "localhost:9091", ...]

// Send periodic heartbeats
client.SendHeartbeat(ctx)

Key Functions:

• ParseAdminEndpoint(): Detects launcher:// scheme
• GetActualPort(): Binds to port 0, returns listener and actual port
• Handshake(): Sends actual port, receives topology
• SendHeartbeat(): Periodic health check

2. LauncherControl Service (pkg/launcher/control_service.go)

Server-side implementation in prism-launcher:

type ControlService struct {
    launcherID      string
    processes       map[string]*CallbackProcessInfo
    adminEndpoints  []string
    proxyEndpoints  []string
}

func (s *ControlService) ProcessHandshake(ctx, req) (*ProcessHandshakeResponse, error) {
    // Store process info with actual port
    s.processes[req.ProcessId] = &CallbackProcessInfo{
        ProcessID:   req.ProcessId,
        ProcessType: req.ProcessType,
        ActualPort:  int(req.ActualPort),
        Hostname:    req.Hostname,
    }

    // Return current topology
    return &ProcessHandshakeResponse{
        Success:        true,
        AdminEndpoints: s.adminEndpoints,
        ProxyEndpoints: s.proxyEndpoints,
    }, nil
}

3. Integration in Components

prism-admin (cmd/prism-admin/serve.go):

• Added --admin and --process-id flags
• Launcher callback support with dynamic port override
• Falls back to configured port in standalone mode

keyvalue-runner (patterns/keyvalue/cmd/keyvalue-runner/main.go):

• Integrated launcher client
• Dynamic port allocation
• Topology discovery for admin connections

mailbox-runner (patterns/mailbox/cmd/mailbox-runner/main.go):

• Integrated launcher client
• Heartbeat support
• Graceful shutdown notification

4. Local Stack Updates (pkg/launcher/local_stack.go)

Updated to pass launcher endpoints to runners:

runnerSpec := &ComponentSpec{
    Name:   "keyvalue-runner",
    Binary: "keyvalue-runner",
    Args: []string{
        "--admin=launcher://localhost:7070",
        "--process-id=keyvalue-runner-1",
        "--grpc-port=0",  // Will be dynamically assigned
    },
}

Testing Results

Successfully tested with local stack startup:

keyvalue-runner → launcher:
  ✓ Connected to launcher://localhost:7070
  ✓ Got dynamic port: 61716
  ✓ Handshake successful
  ✓ Started on dynamic port

mailbox-runner → launcher:
  ✓ Connected to launcher://localhost:7070
  ✓ Got dynamic port: 61715
  ✓ Handshake successful
  ✓ Started on dynamic port

prism-launcher logs:
  [LAUNCHER-CONTROL] ProcessHandshake from keyvalue-runner-1 (port=61716)
  [LAUNCHER-CONTROL] ProcessHandshake from mailbox-runner-admin (port=61715)

Benefits Achieved

1. No Hard-Coded Ports: Processes use OS-assigned ephemeral ports
2. Automatic Discovery: Processes learn topology via handshake response
3. Process Tracking: Launcher tracks all process ports centrally
4. Health Monitoring: Periodic heartbeats detect failures
5. Graceful Shutdown: Processes notify before exit
6. Backward Compatible: Standalone mode still works without launcher

Usage Examples

Pattern Runner Integration

func main() {
    adminEndpoint := flag.String("admin", "", "Admin/launcher endpoint")
    processID := flag.String("process-id", "", "Process ID")
    flag.Parse()

    // Detect launcher mode
    launcherAddr, isLauncher, _ := launcherclient.ParseAdminEndpoint(*adminEndpoint)

    if isLauncher {
        // Get dynamic port
        listener, actualPort, _ := launcherclient.GetActualPort()
        defer listener.Close()

        // Create launcher client
        client, _ := launcherclient.New(&launcherclient.Config{
            LauncherAddr: launcherAddr,
            ProcessID:    *processID,
            ProcessType:  "pattern-runner",
            ActualPort:   actualPort,
        })
        defer client.Close()

        // Connect and handshake
        client.Connect(ctx)
        topology, _ := client.Handshake(ctx)

        // Use topology to connect to admin
        connectToAdmin(topology.AdminEndpoints[0])

        // Start heartbeats
        go startHeartbeats(client)

        // Use listener for your server
        grpcServer.Serve(listener)
    } else {
        // Standalone mode with fixed port
        runStandalone()
    }
}

Local Stack Startup

# Start stack with launcher callbacks
prismctl local start

# All processes use dynamic ports:
# - Admin: 8981, 8982, 8983 (still use configured ports)
# - Launcher: 7070 (fixed)
# - KeyValue: 61716 (dynamic)
# - Mailbox: 61715 (dynamic)

# Check logs to see callback flow
prismctl local logs launcher
prismctl local logs keyvalue-runner

Files Modified

New Files:

• pkg/launcherclient/client.go - Client library (200 lines)
• pkg/launcherclient/example_integration.go - Complete working example showing launcher client integration with dynamic ports, handshake, heartbeats, and graceful shutdown. See this file for a full reference implementation of the pattern shown above.
• pkg/launcherclient/README.md - API documentation
• pkg/launcher/control_service.go - Server implementation (196 lines)

Modified Files:

• cmd/prism-admin/serve.go - Launcher client integration
• patterns/keyvalue/cmd/keyvalue-runner/main.go - Launcher support
• patterns/mailbox/cmd/mailbox-runner/main.go - Launcher support
• cmd/prism-launcher/main.go - Registered LauncherControl service
• pkg/launcher/local_stack.go - Pass launcher:// endpoints
• proto/prism/launcher/launcher.proto - Added LauncherControl service

Go Module Updates:

• cmd/prism-admin/go.mod - Added launcherclient dependency
• patterns/keyvalue/cmd/keyvalue-runner/go.mod - Added launcherclient
• patterns/mailbox/go.mod - Added launcherclient
• pkg/launcherclient/go.mod - New module

Future Enhancements

1. TLS Support: Secure launcher-process communication
2. Service Discovery: Dynamic proxy endpoint updates via heartbeat
3. Health Metrics: Rich health reporting in heartbeat messages
4. Configuration Hot-Reload: Update config via heartbeat response
5. Multi-Launcher Support: Failover between multiple launchers
6. Admin Callback: Admin instances could also use launcher callbacks for full dynamic allocation

References

• RFC-035: Pattern Process Launcher
• ADR-056: Launcher Admin Control Plane
• ADR-057: Prism Launcher Refactoring
• MEMO-034: Pattern Launcher Quickstart

Conclusion

The launcher callback protocol successfully eliminates hard-coded port requirements and enables automatic topology discovery. All pattern runners now use dynamic ports and can discover the full cluster topology through a simple handshake with the launcher.

The implementation is backward compatible (standalone mode still works) and provides a foundation for future enhancements like dynamic service discovery and multi-launcher failover.