# FiberLB Specification > Version: 1.0 | Status: Draft | Last Updated: 2025-12-08 ## 1. Overview ### 1.1 Purpose FiberLB is a multi-tenant load balancer service providing L4 (TCP/UDP) and L7 (HTTP/HTTPS) traffic distribution for the cloud platform. It enables organizations and projects to create and manage load balancers that distribute traffic across backend pools of PlasmaVMC virtual machines with configurable algorithms, health checking, and TLS termination. The name "FiberLB" reflects high-speed, reliable traffic distribution with the "Fiber" prefix denoting throughput and the cloud platform family branding. ### 1.2 Scope - **In scope**: L4 load balancing (TCP, UDP), L7 load balancing (HTTP, HTTPS), TLS termination and passthrough, backend pool management, multiple load balancing algorithms (RoundRobin, LeastConnections, IpHash, Random, WeightedRoundRobin), active health checks (TCP, HTTP, HTTPS, DNS), circuit breaker patterns, multi-tenant LBs (org/project scoped), gRPC management API, aegis integration for access control, ChainFire storage backend, FlashDNS integration for DNS-based health checks - **Out of scope**: Global load balancing (GeoDNS-based), API gateway features (rate limiting, request transformation), Web Application Firewall (WAF), DDoS mitigation (handled at network layer), Service mesh integration (planned), automatic TLS certificate provisioning (planned via LightningStor integration) ### 1.3 Design Goals - **Dual-mode operation**: Both L4 and L7 load balancing in a single service - **Multi-tenant from day one**: Full org/project LB isolation with aegis integration - **High-performance data plane**: Low-latency traffic forwarding with minimal overhead - **Flexible health checking**: Multiple health check types with circuit breaker support - **Algorithm diversity**: Support common load balancing algorithms for different use cases - **Cloud-native management**: gRPC API for LB/pool/backend management, Prometheus metrics - **Consistent storage**: ChainFire for LB configuration persistence with strong consistency ## 2. Architecture ### 2.1 Crate Structure ``` fiberlb/ ├── crates/ │ ├── fiberlb-api/ # gRPC service implementations │ ├── fiberlb-client/ # Rust client library │ ├── fiberlb-server/ # Server binary (control + data plane) │ ├── fiberlb-proxy/ # L4/L7 proxy engine │ └── fiberlb-types/ # Shared types (LoadBalancer, Pool, Backend, etc.) └── proto/ └── fiberlb.proto # gRPC API definitions ``` ### 2.2 Component Topology ``` ┌─────────────────────────────────────────────────────────────────────┐ │ FiberLB Server │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ │ │ fiberlb-proxy │ │ fiberlb-api │ │ fiberlb-types │ │ │ │ (L4/L7 data │ │ (gRPC) │ │ (core types) │ │ │ │ plane) │ │ │ │ │ │ │ └────────┬────────┘ └────────┬────────┘ └──────────┬──────────┘ │ │ │ │ │ │ │ └────────────────────┼──────────────────────┘ │ │ │ │ │ ┌──────▼──────┐ │ │ │ Core │ │ │ │ (config, │ │ │ │ health, │ │ │ │ routing) │ │ │ └──────┬──────┘ │ └────────────────────────────────┼────────────────────────────────────┘ │ ┌────────────┼────────────┬────────────┐ ▼ ▼ ▼ ▼ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ ChainFire │ │ Aegis │ │ FlashDNS │ │ PlasmaVMC │ │ (storage) │ │ (IAM) │ │ (DNS HC) │ │ (backends)│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ ``` ### 2.3 Data Flow **L4 Traffic Flow (TCP/UDP)**: ``` [Client] → [Listener :port] → [Connection Accept] │ [Backend Selection] (algorithm + health) │ [Connection Forward] │ [Backend Server] │ [Response Relay] │ [Client] ``` **L7 Traffic Flow (HTTP/HTTPS)**: ``` [Client] → [Listener :port] → [TLS Termination (if HTTPS)] │ [HTTP Parse] │ [Route Matching] (host, path, headers) │ [Backend Selection] (pool + algorithm) │ [Request Forward] (+ headers: X-Forwarded-*) │ [Backend Server] │ [Response Relay] │ [Client] ``` **Management API Flow**: ``` [gRPC Client] → [fiberlb-api] → [Aegis AuthZ] → [Core Service] │ [ChainFire Store] │ [Config Reload] │ [Data Plane Update] ``` ### 2.4 Dependencies | Crate | Version | Purpose | |-------|---------|---------| | tokio | 1.x | Async runtime | | tonic | 0.12 | gRPC framework | | prost | 0.13 | Protocol buffers | | hyper | 1.x | HTTP/1.1 and HTTP/2 | | tokio-rustls | 0.26 | TLS for HTTPS | | dashmap | 6.x | Concurrent backend/pool state | | uuid | 1.x | LB/pool/backend identifiers | ## 3. Core Concepts ### 3.1 LoadBalancer A load balancer instance scoped to an organization and optionally a project. ```rust pub struct LoadBalancer { pub id: String, // UUID pub name: String, // Display name (unique within scope) pub org_id: String, // Owner organization pub project_id: Option, // Optional project scope pub description: Option, pub listeners: Vec, // Associated listeners pub default_pool_id: Option, // Default backend pool pub status: LbStatus, pub created_at: u64, // Creation timestamp (Unix ms) pub updated_at: u64, // Last modification pub created_by: String, // Principal ID pub metadata: HashMap, pub tags: HashMap, } pub enum LbStatus { Creating, // Being provisioned Active, // Operational Updating, // Configuration change in progress Error, // Provisioning/config error Deleting, // Being removed Disabled, // Manually disabled } pub struct LbStatusInfo { pub status: LbStatus, pub message: Option, // Error details if applicable pub last_transition: u64, // When status changed } ``` **Naming Rules**: - 1-63 characters - Lowercase alphanumeric and hyphens - Must start with letter - Must end with alphanumeric - Unique within org (or project if project-scoped) ### 3.2 Listener A network endpoint that accepts incoming traffic for a load balancer. ```rust pub struct Listener { pub id: String, // UUID pub lb_id: String, // Parent load balancer pub name: String, // Display name pub protocol: ListenerProtocol, pub port: u16, // Listen port (1-65535) pub tls_config: Option, // For HTTPS/TLS listeners pub default_pool_id: Option, // Pool for unmatched requests pub rules: Vec, // L7 routing rules pub connection_limit: Option, // Max concurrent connections pub timeout_client: u32, // Client timeout (ms) pub timeout_backend: u32, // Backend timeout (ms) pub enabled: bool, pub created_at: u64, pub updated_at: u64, } pub enum ListenerProtocol { Tcp, // L4 TCP Udp, // L4 UDP Http, // L7 HTTP Https, // L7 HTTPS (TLS termination) TcpTls, // L4 TCP with TLS passthrough } pub struct TlsConfig { pub certificate_id: String, // Reference to cert in LightningStor pub min_version: TlsVersion, // Minimum TLS version pub cipher_suites: Vec, // Allowed cipher suites (or default) pub client_auth: ClientAuthMode, // mTLS settings pub sni_certificates: HashMap, // SNI -> cert_id mapping } pub enum TlsVersion { Tls12, Tls13, } pub enum ClientAuthMode { None, // No client cert required Optional, // Request but don't require Required, // Require valid client cert } ``` **Port Restrictions**: - Privileged ports (1-1023) require elevated permissions - Port conflicts prevented within same LB - Well-known ports: 80 (HTTP), 443 (HTTPS), custom for TCP/UDP ### 3.3 RoutingRule L7 routing rules for HTTP/HTTPS listeners. ```rust pub struct RoutingRule { pub id: String, // UUID pub name: String, pub priority: u32, // Lower = higher priority pub conditions: Vec, // AND logic pub action: RuleAction, pub enabled: bool, } pub enum RuleCondition { HostHeader { values: Vec, // Exact match or wildcard (*.example.com) }, PathPrefix { value: String, // e.g., "/api/" }, PathExact { value: String, // e.g., "/health" }, PathRegex { pattern: String, // Regex pattern }, Header { name: String, values: Vec, // Match any }, Method { methods: Vec, }, QueryParam { name: String, value: String, }, SourceIp { cidrs: Vec, // Client IP CIDR match }, } pub enum RuleAction { ForwardToPool { pool_id: String, }, Redirect { url: String, status_code: u16, // 301, 302, 307, 308 }, FixedResponse { status_code: u16, content_type: String, body: String, }, } pub enum HttpMethod { Get, Post, Put, Delete, Patch, Head, Options, } ``` ### 3.4 Pool A backend pool containing a group of servers with a load balancing algorithm. ```rust pub struct Pool { pub id: String, // UUID pub lb_id: String, // Parent load balancer pub name: String, // Display name pub algorithm: Algorithm, pub backends: Vec, // Member backends pub health_check: Option, pub session_persistence: Option, pub circuit_breaker: Option, pub enabled: bool, pub created_at: u64, pub updated_at: u64, } pub enum Algorithm { RoundRobin, // Sequential distribution LeastConnections, // Fewest active connections IpHash, // Consistent hashing by client IP Random, // Random selection WeightedRoundRobin, // Round robin with weights LeastResponseTime, // Fastest backend (requires active monitoring) } pub struct SessionPersistence { pub mode: PersistenceMode, pub cookie_name: Option, // For cookie-based pub ttl_seconds: u32, // Session timeout } pub enum PersistenceMode { None, SourceIp, // Stick by client IP Cookie, // Insert/track LB cookie AppCookie, // Track application cookie } ``` **Algorithm Selection Guide**: | Algorithm | Best For | Considerations | |-----------|----------|----------------| | RoundRobin | Equal capacity backends | Simple, even distribution | | LeastConnections | Variable request duration | Tracks connection count | | IpHash | Session affinity without cookies | May unbalance with NAT | | Random | Simple, stateless | Good entropy needed | | WeightedRoundRobin | Mixed capacity backends | Manual weight tuning | | LeastResponseTime | Performance-sensitive | Requires active probing | ### 3.5 Backend An individual backend server (typically a PlasmaVMC VM) in a pool. ```rust pub struct Backend { pub id: String, // UUID pub pool_id: String, // Parent pool pub name: String, // Display name pub address: BackendAddress, pub port: u16, // Backend port pub weight: u32, // For weighted algorithms (1-100, default: 1) pub status: BackendStatus, pub health_status: HealthStatus, pub enabled: bool, // Admin enable/disable pub metadata: HashMap, pub created_at: u64, pub updated_at: u64, } pub enum BackendAddress { Ip(IpAddr), // Direct IP address Hostname(String), // DNS hostname (resolved) VmId(String), // PlasmaVMC VM ID (resolved via API) } pub enum BackendStatus { Creating, // Being added Active, // Ready to receive traffic Draining, // Graceful removal (no new connections) Removed, // Removed from pool } pub struct HealthStatus { pub healthy: bool, pub last_check: u64, // Timestamp of last check pub last_healthy: Option, // Last time marked healthy pub consecutive_failures: u32, pub consecutive_successes: u32, pub last_error: Option, // Most recent failure reason } ``` ### 3.6 HealthCheck Configuration for backend health monitoring. ```rust pub struct HealthCheck { pub id: String, // UUID pub check_type: HealthCheckType, pub interval: u32, // Check interval (seconds) pub timeout: u32, // Check timeout (seconds) pub healthy_threshold: u32, // Consecutive successes to mark healthy pub unhealthy_threshold: u32, // Consecutive failures to mark unhealthy pub enabled: bool, } pub enum HealthCheckType { Tcp { // Just TCP connection success }, Http { path: String, // e.g., "/health" method: HttpMethod, // GET, HEAD expected_codes: Vec, // e.g., [200, 204] host_header: Option, headers: HashMap, }, Https { path: String, method: HttpMethod, expected_codes: Vec, host_header: Option, headers: HashMap, verify_tls: bool, // Verify backend cert }, Dns { hostname: String, // Query via FlashDNS record_type: DnsRecordType, // A, AAAA expected_address: Option, // Expected response }, Grpc { service: Option, // gRPC health check service }, } pub enum DnsRecordType { A, Aaaa, } impl Default for HealthCheck { fn default() -> Self { Self { id: uuid::Uuid::new_v4().to_string(), check_type: HealthCheckType::Tcp {}, interval: 30, // 30 seconds timeout: 10, // 10 seconds healthy_threshold: 2, // 2 consecutive successes unhealthy_threshold: 3, // 3 consecutive failures enabled: true, } } } ``` ### 3.7 CircuitBreaker Circuit breaker configuration for backend fault isolation. ```rust pub struct CircuitBreakerConfig { pub enabled: bool, pub failure_threshold: u32, // Failures to open circuit pub success_threshold: u32, // Successes to close circuit pub timeout_seconds: u32, // Time in open state before half-open pub failure_rate_threshold: f32, // 0.0-1.0, alternative to count pub min_request_volume: u32, // Min requests before rate calculation pub slow_call_threshold_ms: u32, // Response time to count as slow pub slow_call_rate_threshold: f32, // Slow call rate to open circuit } pub enum CircuitState { Closed, // Normal operation Open, // Failing, reject requests HalfOpen, // Testing recovery } impl Default for CircuitBreakerConfig { fn default() -> Self { Self { enabled: false, failure_threshold: 5, success_threshold: 3, timeout_seconds: 60, failure_rate_threshold: 0.5, min_request_volume: 10, slow_call_threshold_ms: 5000, slow_call_rate_threshold: 0.8, } } } ``` **Circuit Breaker States**: ``` [Closed] ──(failures >= threshold)──► [Open] ▲ │ │ (timeout expires) │ ▼ └───(successes >= threshold)─── [Half-Open] │ (failure) ▼ [Open] ``` ## 4. API ### 4.1 gRPC Services #### LbService (`fiberlb.v1.LbService`) ```protobuf service LbService { // Load Balancer CRUD rpc CreateLoadBalancer(CreateLoadBalancerRequest) returns (LoadBalancer); rpc GetLoadBalancer(GetLoadBalancerRequest) returns (LoadBalancer); rpc UpdateLoadBalancer(UpdateLoadBalancerRequest) returns (LoadBalancer); rpc DeleteLoadBalancer(DeleteLoadBalancerRequest) returns (DeleteLoadBalancerResponse); rpc ListLoadBalancers(ListLoadBalancersRequest) returns (ListLoadBalancersResponse); // Status operations rpc GetLoadBalancerStatus(GetLoadBalancerStatusRequest) returns (LoadBalancerStatus); rpc EnableLoadBalancer(EnableLoadBalancerRequest) returns (LoadBalancer); rpc DisableLoadBalancer(DisableLoadBalancerRequest) returns (LoadBalancer); } message CreateLoadBalancerRequest { string name = 1; string org_id = 2; optional string project_id = 3; optional string description = 4; map tags = 5; map metadata = 6; } message GetLoadBalancerRequest { string lb_id = 1; } message UpdateLoadBalancerRequest { string lb_id = 1; optional string name = 2; optional string description = 3; optional string default_pool_id = 4; map tags = 5; } message DeleteLoadBalancerRequest { string lb_id = 1; bool force = 2; // Delete even with active listeners } message ListLoadBalancersRequest { string org_id = 1; optional string project_id = 2; optional string name_filter = 3; // Prefix match optional LbStatus status_filter = 4; uint32 limit = 5; // Max results (default: 100) string page_token = 6; } message ListLoadBalancersResponse { repeated LoadBalancer load_balancers = 1; string next_page_token = 2; uint32 total_count = 3; } ``` #### ListenerService (`fiberlb.v1.ListenerService`) ```protobuf service ListenerService { rpc CreateListener(CreateListenerRequest) returns (Listener); rpc GetListener(GetListenerRequest) returns (Listener); rpc UpdateListener(UpdateListenerRequest) returns (Listener); rpc DeleteListener(DeleteListenerRequest) returns (DeleteListenerResponse); rpc ListListeners(ListListenersRequest) returns (ListListenersResponse); // Routing rules rpc AddRoutingRule(AddRoutingRuleRequest) returns (Listener); rpc UpdateRoutingRule(UpdateRoutingRuleRequest) returns (Listener); rpc RemoveRoutingRule(RemoveRoutingRuleRequest) returns (Listener); } message CreateListenerRequest { string lb_id = 1; string name = 2; ListenerProtocol protocol = 3; uint32 port = 4; optional TlsConfig tls_config = 5; optional string default_pool_id = 6; optional uint32 connection_limit = 7; optional uint32 timeout_client = 8; optional uint32 timeout_backend = 9; } message UpdateListenerRequest { string listener_id = 1; optional string name = 2; optional TlsConfig tls_config = 3; optional string default_pool_id = 4; optional uint32 connection_limit = 5; optional uint32 timeout_client = 6; optional uint32 timeout_backend = 7; optional bool enabled = 8; } message ListListenersRequest { string lb_id = 1; optional ListenerProtocol protocol_filter = 2; uint32 limit = 3; string page_token = 4; } ``` #### PoolService (`fiberlb.v1.PoolService`) ```protobuf service PoolService { rpc CreatePool(CreatePoolRequest) returns (Pool); rpc GetPool(GetPoolRequest) returns (Pool); rpc UpdatePool(UpdatePoolRequest) returns (Pool); rpc DeletePool(DeletePoolRequest) returns (DeletePoolResponse); rpc ListPools(ListPoolsRequest) returns (ListPoolsResponse); // Health check management rpc SetHealthCheck(SetHealthCheckRequest) returns (Pool); rpc RemoveHealthCheck(RemoveHealthCheckRequest) returns (Pool); // Pool status rpc GetPoolStatus(GetPoolStatusRequest) returns (PoolStatus); } message CreatePoolRequest { string lb_id = 1; string name = 2; Algorithm algorithm = 3; optional HealthCheck health_check = 4; optional SessionPersistence session_persistence = 5; optional CircuitBreakerConfig circuit_breaker = 6; } message UpdatePoolRequest { string pool_id = 1; optional string name = 2; optional Algorithm algorithm = 3; optional SessionPersistence session_persistence = 4; optional CircuitBreakerConfig circuit_breaker = 5; optional bool enabled = 6; } message ListPoolsRequest { string lb_id = 1; uint32 limit = 2; string page_token = 3; } message PoolStatus { string pool_id = 1; uint32 total_backends = 2; uint32 healthy_backends = 3; uint32 unhealthy_backends = 4; uint32 draining_backends = 5; CircuitState circuit_state = 6; } ``` #### BackendService (`fiberlb.v1.BackendService`) ```protobuf service BackendService { rpc AddBackend(AddBackendRequest) returns (Backend); rpc GetBackend(GetBackendRequest) returns (Backend); rpc UpdateBackend(UpdateBackendRequest) returns (Backend); rpc RemoveBackend(RemoveBackendRequest) returns (RemoveBackendResponse); rpc ListBackends(ListBackendsRequest) returns (ListBackendsResponse); // Drain operations rpc DrainBackend(DrainBackendRequest) returns (Backend); rpc UndainBackend(UndrainBackendRequest) returns (Backend); // Health status rpc GetBackendHealth(GetBackendHealthRequest) returns (HealthStatus); rpc ListBackendHealth(ListBackendHealthRequest) returns (ListBackendHealthResponse); } message AddBackendRequest { string pool_id = 1; string name = 2; BackendAddress address = 3; uint32 port = 4; optional uint32 weight = 5; // Default: 1 map metadata = 6; } message UpdateBackendRequest { string backend_id = 1; optional string name = 2; optional uint32 port = 3; optional uint32 weight = 4; optional bool enabled = 5; } message RemoveBackendRequest { string backend_id = 1; bool drain_first = 2; // Graceful removal optional uint32 drain_timeout_seconds = 3; } message ListBackendsRequest { string pool_id = 1; optional bool healthy_only = 2; uint32 limit = 3; string page_token = 4; } message DrainBackendRequest { string backend_id = 1; optional uint32 timeout_seconds = 2; // Max drain time } message ListBackendHealthResponse { repeated BackendHealthEntry entries = 1; } message BackendHealthEntry { string backend_id = 1; string backend_name = 2; HealthStatus health_status = 3; } ``` ### 4.2 Authentication **gRPC API**: - aegis bearer tokens in `authorization` metadata - mTLS for service-to-service communication - API key header (`x-api-key`) for programmatic access **Data Plane**: - No authentication for load balanced traffic (passthrough) - mTLS between LB and backends (optional) - Client certificate validation (optional, for HTTPS listeners) ### 4.3 Client Library ```rust use fiberlb_client::FiberLbClient; let client = FiberLbClient::connect("http://127.0.0.1:6300").await?; // Create load balancer let lb = client.create_load_balancer(CreateLoadBalancerRequest { name: "web-lb".into(), org_id: "acme".into(), project_id: Some("web-prod".into()), description: Some("Production web load balancer".into()), ..Default::default() }).await?; // Create HTTP listener let listener = client.create_listener(CreateListenerRequest { lb_id: lb.id.clone(), name: "http".into(), protocol: ListenerProtocol::Http, port: 80, ..Default::default() }).await?; // Create HTTPS listener with TLS let https_listener = client.create_listener(CreateListenerRequest { lb_id: lb.id.clone(), name: "https".into(), protocol: ListenerProtocol::Https, port: 443, tls_config: Some(TlsConfig { certificate_id: "cert-123".into(), min_version: TlsVersion::Tls12, ..Default::default() }), ..Default::default() }).await?; // Create backend pool let pool = client.create_pool(CreatePoolRequest { lb_id: lb.id.clone(), name: "web-backends".into(), algorithm: Algorithm::LeastConnections, health_check: Some(HealthCheck { check_type: HealthCheckType::Http { path: "/health".into(), method: HttpMethod::Get, expected_codes: vec![200], ..Default::default() }, interval: 10, timeout: 5, healthy_threshold: 2, unhealthy_threshold: 3, ..Default::default() }), session_persistence: Some(SessionPersistence { mode: PersistenceMode::Cookie, cookie_name: Some("SERVERID".into()), ttl_seconds: 3600, }), ..Default::default() }).await?; // Add backends (PlasmaVMC VMs) for (i, vm_id) in ["vm-001", "vm-002", "vm-003"].iter().enumerate() { client.add_backend(AddBackendRequest { pool_id: pool.id.clone(), name: format!("web-{}", i + 1), address: BackendAddress::VmId(vm_id.to_string()), port: 8080, weight: Some(1), ..Default::default() }).await?; } // Update listener to use pool client.update_listener(UpdateListenerRequest { listener_id: listener.id.clone(), default_pool_id: Some(pool.id.clone()), ..Default::default() }).await?; // Add L7 routing rules client.add_routing_rule(AddRoutingRuleRequest { listener_id: https_listener.id.clone(), rule: RoutingRule { name: "api-route".into(), priority: 10, conditions: vec![ RuleCondition::PathPrefix { value: "/api/".into() }, ], action: RuleAction::ForwardToPool { pool_id: pool.id.clone() }, enabled: true, ..Default::default() }, }).await?; // Check pool health status let status = client.get_pool_status(GetPoolStatusRequest { pool_id: pool.id.clone(), }).await?; println!("Healthy backends: {}/{}", status.healthy_backends, status.total_backends); // Drain backend for maintenance client.drain_backend(DrainBackendRequest { backend_id: "backend-001".into(), timeout_seconds: Some(30), }).await?; ``` ## 5. Multi-Tenancy ### 5.1 Scope Hierarchy ``` System (platform operators) └─ Organization (tenant boundary) ├─ Org-level load balancers (shared across projects) └─ Project (workload isolation) └─ Project-level load balancers ``` ### 5.2 LoadBalancer Scoping ```rust pub enum LbScope { /// LB accessible to all projects in org Organization { org_id: String }, /// LB scoped to specific project Project { org_id: String, project_id: String }, } impl LoadBalancer { pub fn scope(&self) -> LbScope { match &self.project_id { Some(pid) => LbScope::Project { org_id: self.org_id.clone(), project_id: pid.clone() }, None => LbScope::Organization { org_id: self.org_id.clone() }, } } } ``` ### 5.3 Access Control Integration ```rust // aegis action patterns for fiberlb const ACTIONS: &[&str] = &[ "fiberlb:loadbalancers:create", "fiberlb:loadbalancers:get", "fiberlb:loadbalancers:list", "fiberlb:loadbalancers:update", "fiberlb:loadbalancers:delete", "fiberlb:listeners:create", "fiberlb:listeners:get", "fiberlb:listeners:list", "fiberlb:listeners:update", "fiberlb:listeners:delete", "fiberlb:pools:create", "fiberlb:pools:get", "fiberlb:pools:list", "fiberlb:pools:update", "fiberlb:pools:delete", "fiberlb:backends:add", "fiberlb:backends:get", "fiberlb:backends:list", "fiberlb:backends:update", "fiberlb:backends:remove", "fiberlb:backends:drain", ]; // Resource path format // org/{org_id}/project/{project_id}/lb/{lb_id} // org/{org_id}/project/{project_id}/lb/{lb_id}/listener/{listener_id} // org/{org_id}/project/{project_id}/lb/{lb_id}/pool/{pool_id} // org/{org_id}/project/{project_id}/lb/{lb_id}/pool/{pool_id}/backend/{backend_id} async fn authorize_lb_access( iam: &IamClient, principal: &PrincipalRef, action: &str, lb: &LoadBalancer, ) -> Result<()> { let resource = ResourceRef { kind: "loadbalancer".into(), id: lb.id.clone(), org_id: lb.org_id.clone(), project_id: lb.project_id.clone().unwrap_or_default(), ..Default::default() }; let allowed = iam.authorize(principal, action, &resource).await?; if !allowed { return Err(Error::AccessDenied); } Ok(()) } ``` ### 5.4 Resource Isolation - Load balancers with same name can exist in different orgs/projects - Backends can only reference VMs within same org/project scope - Cross-project backend references require explicit binding - Listener ports unique within a load balancer ## 6. Health Checking ### 6.1 Health Check Engine ```rust pub struct HealthChecker { pools: Arc>, check_interval: Duration, http_client: reqwest::Client, dns_client: Arc, } pub struct PoolState { pub pool: Pool, pub backends: DashMap, } pub struct BackendState { pub backend: Backend, pub health: HealthStatus, pub circuit: CircuitState, pub last_response_time: Option, } impl HealthChecker { /// Start background health checking for all pools pub async fn start(&self); /// Run single health check for a backend pub async fn check_backend( &self, backend: &Backend, check: &HealthCheck, ) -> HealthCheckResult; /// Update backend health status based on check result pub fn update_health( &self, backend_id: &str, result: HealthCheckResult, ); } pub struct HealthCheckResult { pub success: bool, pub response_time: Duration, pub error: Option, pub details: HealthCheckDetails, } pub enum HealthCheckDetails { Tcp { connected: bool }, Http { status_code: u16, body_preview: String }, Dns { resolved: bool, addresses: Vec }, } ``` ### 6.2 Health Check Types **TCP Health Check**: ```rust async fn check_tcp(addr: SocketAddr, timeout: Duration) -> HealthCheckResult { match tokio::time::timeout(timeout, TcpStream::connect(addr)).await { Ok(Ok(_)) => HealthCheckResult::success(), Ok(Err(e)) => HealthCheckResult::failure(format!("Connection failed: {}", e)), Err(_) => HealthCheckResult::failure("Connection timeout"), } } ``` **HTTP Health Check**: ```rust async fn check_http( url: &str, method: HttpMethod, headers: &HashMap, expected_codes: &[u16], timeout: Duration, ) -> HealthCheckResult { let response = http_client .request(method, url) .headers(headers) .timeout(timeout) .send() .await?; let status = response.status().as_u16(); if expected_codes.contains(&status) { HealthCheckResult::success_with_details(HealthCheckDetails::Http { status_code: status, body_preview: response.text().await?.chars().take(100).collect(), }) } else { HealthCheckResult::failure(format!("Unexpected status: {}", status)) } } ``` **DNS Health Check (via FlashDNS)**: ```rust async fn check_dns( hostname: &str, record_type: DnsRecordType, expected: Option<&str>, dns_client: &FlashDnsClient, ) -> HealthCheckResult { let records = dns_client.resolve(hostname, record_type).await?; if records.is_empty() { return HealthCheckResult::failure("No DNS records found"); } if let Some(expected_addr) = expected { if records.iter().any(|r| r.address() == expected_addr) { HealthCheckResult::success() } else { HealthCheckResult::failure(format!("Expected {} not found", expected_addr)) } } else { HealthCheckResult::success() } } ``` ### 6.3 Circuit Breaker Implementation ```rust impl CircuitBreaker { pub fn record_success(&mut self) { self.consecutive_successes += 1; self.consecutive_failures = 0; match self.state { CircuitState::HalfOpen => { if self.consecutive_successes >= self.config.success_threshold { self.transition_to(CircuitState::Closed); } } CircuitState::Closed => { // Reset failure rate window } CircuitState::Open => { // Shouldn't happen, but handle gracefully } } } pub fn record_failure(&mut self) { self.consecutive_failures += 1; self.consecutive_successes = 0; match self.state { CircuitState::Closed => { if self.should_open() { self.transition_to(CircuitState::Open); } } CircuitState::HalfOpen => { self.transition_to(CircuitState::Open); } CircuitState::Open => { // Already open } } } pub fn allow_request(&mut self) -> bool { match self.state { CircuitState::Closed => true, CircuitState::Open => { if self.timeout_elapsed() { self.transition_to(CircuitState::HalfOpen); true // Allow probe request } else { false } } CircuitState::HalfOpen => { // Limited requests allowed self.half_open_requests < self.config.half_open_max_requests } } } } ``` ### 6.4 PlasmaVMC Integration ```rust /// Resolve backend address from PlasmaVMC VM ID async fn resolve_vm_address( plasmavmc: &PlasmaVmcClient, vm_id: &str, ) -> Result { let vm = plasmavmc.get_vm(vm_id).await?; // Prefer private IP for internal LB, public for external vm.network_interfaces .iter() .find_map(|nic| nic.private_ipv4) .ok_or(Error::NoBackendAddress) } /// Watch VM status changes for backend health async fn watch_vm_health( plasmavmc: &PlasmaVmcClient, backend: &Backend, ) -> Result<()> { if let BackendAddress::VmId(vm_id) = &backend.address { let vm = plasmavmc.get_vm(vm_id).await?; match vm.status { VmStatus::Running => Ok(()), VmStatus::Stopped | VmStatus::Terminated => { Err(Error::BackendUnavailable) } _ => Err(Error::BackendUnknownState), } } else { Ok(()) } } ``` ## 7. Storage ### 7.1 ChainFire Key Schema **Load Balancers**: ``` fiberlb/lbs/{lb_id} # LB record (by ID) fiberlb/lbs/by-name/{org_id}/{name} # Name lookup (org-level) fiberlb/lbs/by-name/{org_id}/{project_id}/{name} # Name lookup (project-level) fiberlb/lbs/by-org/{org_id}/{lb_id} # Org index fiberlb/lbs/by-project/{project_id}/{lb_id} # Project index ``` **Listeners**: ``` fiberlb/listeners/{listener_id} # Listener by ID fiberlb/listeners/by-lb/{lb_id}/{listener_id} # LB index fiberlb/listeners/by-port/{lb_id}/{port} # Port lookup ``` **Pools**: ``` fiberlb/pools/{pool_id} # Pool by ID fiberlb/pools/by-lb/{lb_id}/{pool_id} # LB index fiberlb/pools/by-name/{lb_id}/{name} # Name lookup ``` **Backends**: ``` fiberlb/backends/{backend_id} # Backend by ID fiberlb/backends/by-pool/{pool_id}/{backend_id} # Pool index fiberlb/backends/by-address/{address_hash} # Address lookup ``` **Health State (ephemeral)**: ``` fiberlb/health/{backend_id} # Current health status fiberlb/circuit/{pool_id} # Circuit breaker state ``` ### 7.2 Storage Operations ```rust #[async_trait] pub trait LbStore: Send + Sync { async fn create_lb(&self, lb: &LoadBalancer) -> Result<()>; async fn get_lb(&self, lb_id: &str) -> Result>; async fn get_lb_by_name( &self, org_id: &str, project_id: Option<&str>, name: &str, ) -> Result>; async fn update_lb(&self, lb: &LoadBalancer) -> Result<()>; async fn delete_lb(&self, lb_id: &str) -> Result; async fn list_lbs( &self, org_id: &str, project_id: Option<&str>, limit: usize, page_token: Option<&str>, ) -> Result<(Vec, Option)>; } #[async_trait] pub trait PoolStore: Send + Sync { async fn create_pool(&self, pool: &Pool) -> Result<()>; async fn get_pool(&self, pool_id: &str) -> Result>; async fn update_pool(&self, pool: &Pool) -> Result<()>; async fn delete_pool(&self, pool_id: &str) -> Result; async fn list_pools_by_lb( &self, lb_id: &str, limit: usize, page_token: Option<&str>, ) -> Result<(Vec, Option)>; } #[async_trait] pub trait BackendStore: Send + Sync { async fn add_backend(&self, backend: &Backend) -> Result<()>; async fn get_backend(&self, backend_id: &str) -> Result>; async fn update_backend(&self, backend: &Backend) -> Result<()>; async fn remove_backend(&self, backend_id: &str) -> Result; async fn list_backends_by_pool( &self, pool_id: &str, limit: usize, page_token: Option<&str>, ) -> Result<(Vec, Option)>; async fn get_healthy_backends(&self, pool_id: &str) -> Result>; } ``` ### 7.3 Configuration Cache ```rust pub struct ConfigCache { load_balancers: DashMap, listeners: DashMap, pools: DashMap, backends: DashMap>, // pool_id -> backends config: CacheConfig, } impl ConfigCache { /// Load all config for an LB into cache pub async fn load_lb(&self, store: &dyn LbStore, lb_id: &str) -> Result<()>; /// Invalidate and reload on config change pub fn invalidate_lb(&self, lb_id: &str); /// Get routing config for data plane pub fn get_routing_config(&self, lb_id: &str) -> Option; } ``` ## 8. Configuration ### 8.1 Config File Format (TOML) ```toml [server] grpc_addr = "0.0.0.0:6300" # gRPC management API metrics_addr = "0.0.0.0:9090" # Prometheus metrics [server.tls] cert_file = "/etc/fiberlb/tls/server.crt" key_file = "/etc/fiberlb/tls/server.key" ca_file = "/etc/fiberlb/tls/ca.crt" [storage] backend = "chainfire" # "chainfire" | "memory" chainfire_endpoints = ["http://chainfire-1:2379", "http://chainfire-2:2379"] [proxy] # L4 settings tcp_keepalive_seconds = 60 tcp_nodelay = true connection_timeout_ms = 5000 # L7 settings http_idle_timeout_seconds = 60 max_header_size_bytes = 8192 max_body_size_bytes = 10485760 # 10MB # Buffer sizes recv_buffer_size = 65536 send_buffer_size = 65536 [proxy.tls] default_min_version = "tls12" session_cache_size = 10000 session_timeout_seconds = 3600 [health_check] default_interval_seconds = 30 default_timeout_seconds = 10 default_healthy_threshold = 2 default_unhealthy_threshold = 3 max_concurrent_checks = 100 check_jitter_percent = 10 # Spread checks over interval [circuit_breaker] default_failure_threshold = 5 default_success_threshold = 3 default_timeout_seconds = 60 [iam] endpoint = "http://aegis:9090" service_account = "fiberlb" token_path = "/var/run/secrets/iam/token" [plasmavmc] endpoint = "http://plasmavmc:8080" cache_ttl_seconds = 30 [flashdns] endpoint = "http://flashdns:5300" dns_addr = "127.0.0.1:53" # For DNS health checks [logging] level = "info" format = "json" ``` ### 8.2 Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `FIBERLB_CONFIG` | - | Config file path | | `FIBERLB_GRPC_ADDR` | `0.0.0.0:6300` | gRPC listen address | | `FIBERLB_METRICS_ADDR` | `0.0.0.0:9090` | Metrics listen address | | `FIBERLB_LOG_LEVEL` | `info` | Log level | | `FIBERLB_STORE_BACKEND` | `memory` | Storage backend | ### 8.3 CLI Arguments ``` fiberlb-server [OPTIONS] -c, --config Config file path --grpc-addr gRPC listen address --metrics-addr Metrics listen address -l, --log-level Log level -h, --help Print help -V, --version Print version ``` ## 9. Security ### 9.1 Authentication **gRPC API**: - aegis bearer tokens for user/service authentication - mTLS for service-to-service communication - API key header for programmatic access **Data Plane**: - TLS termination at listener (for HTTPS) - mTLS to backends (optional) - Client certificate validation (optional) ### 9.2 Authorization - All management operations authorized via aegis - LB-level, pool-level, and backend-level permissions - Scope enforcement (org/project boundaries) - Owner-based access patterns supported ### 9.3 Data Security - TLS 1.2/1.3 for HTTPS listeners - Certificate storage in LightningStor (reference by ID) - Private keys never exposed via API - Backend traffic encryption (optional mTLS) ### 9.4 Network Security ```rust pub struct SecurityConfig { /// Restrict listener binding to specific IPs pub allowed_listener_ips: Vec, /// Restrict backend addresses to trusted ranges pub allowed_backend_cidrs: Vec, /// Maximum connections per source IP pub max_connections_per_ip: Option, /// Rate limiting for new connections pub connection_rate_limit: Option, } ``` ### 9.5 Audit - All management API calls logged with principal, action, resource - Connection logs for traffic analytics - Health check results logged - Integration with platform audit system ## 10. Operations ### 10.1 Deployment **Single Node (Development)**: ```bash fiberlb-server --config config.toml ``` **Production Cluster**: ```bash # Multiple FiberLB instances # - Stateless control plane (shared ChainFire) # - Data plane with health state sync fiberlb-server --config config.toml # Behind external load balancer for HA # Or with BGP anycast for direct traffic distribution ``` ### 10.2 Monitoring **Metrics (Prometheus)**: | Metric | Type | Description | |--------|------|-------------| | `fiberlb_connections_total` | Counter | Total connections accepted | | `fiberlb_connections_active` | Gauge | Current active connections | | `fiberlb_requests_total` | Counter | Total L7 requests | | `fiberlb_request_duration_seconds` | Histogram | Request latency | | `fiberlb_bytes_in_total` | Counter | Total bytes received | | `fiberlb_bytes_out_total` | Counter | Total bytes sent | | `fiberlb_backend_health{status}` | Gauge | Backends by health status | | `fiberlb_health_checks_total` | Counter | Total health checks | | `fiberlb_health_check_duration_seconds` | Histogram | Health check latency | | `fiberlb_circuit_breaker_state` | Gauge | Circuit breaker states | | `fiberlb_pools_total` | Gauge | Total pools | | `fiberlb_backends_total` | Gauge | Total backends | | `fiberlb_grpc_requests_total` | Counter | gRPC API requests | **Health Endpoints**: - `GET /health` - Liveness check - `GET /ready` - Readiness check (storage connected, data plane ready) ### 10.3 Backup & Recovery - **LB configuration**: ChainFire snapshots - **Export**: Configuration export via gRPC API - **Import**: Configuration import for disaster recovery ### 10.4 Graceful Operations ```rust /// Graceful backend removal pub async fn drain_backend(backend_id: &str, timeout: Duration) -> Result<()> { // 1. Mark backend as draining backend.status = BackendStatus::Draining; // 2. Stop sending new connections routing.exclude_backend(backend_id); // 3. Wait for existing connections to complete let deadline = Instant::now() + timeout; while backend.active_connections() > 0 && Instant::now() < deadline { tokio::time::sleep(Duration::from_secs(1)).await; } // 4. Force-close remaining connections if backend.active_connections() > 0 { backend.force_close_connections(); } // 5. Remove backend backend.status = BackendStatus::Removed; Ok(()) } /// Graceful LB shutdown pub async fn graceful_shutdown(&self, timeout: Duration) -> Result<()> { // 1. Stop accepting new connections self.listeners.stop_accepting(); // 2. Drain all backends for pool in self.pools.values() { for backend in pool.backends.values() { self.drain_backend(&backend.id, timeout / 2).await?; } } // 3. Close management API self.grpc_server.shutdown().await; Ok(()) } ``` ## 11. Compatibility ### 11.1 API Versioning - gRPC package: `fiberlb.v1` - Semantic versioning for breaking changes - Backward compatible additions within major version ### 11.2 Protocol Support | Protocol | Version | Status | |----------|---------|--------| | HTTP | 1.0, 1.1 | Supported | | HTTP | 2 | Supported | | HTTP | 3 (QUIC) | Planned | | TLS | 1.2, 1.3 | Supported | | TCP | - | Supported | | UDP | - | Supported | | WebSocket | - | Supported (via HTTP upgrade) | ### 11.3 Backend Compatibility - Direct IP addresses - DNS hostnames (with periodic re-resolution) - PlasmaVMC VM IDs (resolved via API) ## Appendix ### A. Error Codes **gRPC Errors**: | Error | Description | |-------|-------------| | LB_NOT_FOUND | Load balancer does not exist | | LISTENER_NOT_FOUND | Listener does not exist | | POOL_NOT_FOUND | Pool does not exist | | BACKEND_NOT_FOUND | Backend does not exist | | LB_ALREADY_EXISTS | LB name already in use | | LISTENER_PORT_CONFLICT | Port already in use on LB | | INVALID_LB_NAME | LB name format invalid | | INVALID_BACKEND_ADDRESS | Backend address invalid | | ACCESS_DENIED | Permission denied | | POOL_NOT_EMPTY | Cannot delete pool with backends | | BACKEND_UNHEALTHY | Backend failed health check | | CIRCUIT_OPEN | Circuit breaker is open | | QUOTA_EXCEEDED | LB/pool/backend quota exceeded | ### B. Port Assignments | Port | Protocol | Purpose | |------|----------|---------| | 6300 | gRPC | Management API | | 9090 | HTTP | Prometheus metrics | | 80 | HTTP | Default HTTP listener (configurable) | | 443 | HTTPS | Default HTTPS listener (configurable) | ### C. Glossary - **Load Balancer**: A logical grouping of listeners and pools for traffic distribution - **Listener**: A network endpoint that accepts incoming traffic - **Pool**: A group of backend servers with a load balancing algorithm - **Backend**: An individual server that receives traffic from a pool - **Health Check**: Periodic probe to verify backend availability - **Circuit Breaker**: Pattern to prevent cascading failures by failing fast - **Drain**: Graceful removal of a backend by stopping new connections - **L4**: Layer 4 (transport layer) - TCP/UDP load balancing - **L7**: Layer 7 (application layer) - HTTP/HTTPS load balancing with routing - **TLS Termination**: Decrypting TLS at the load balancer - **TLS Passthrough**: Forwarding encrypted traffic directly to backends ### D. Example Configurations **Basic HTTP Load Balancer**: ```rust // Create LB with HTTP listener and round-robin pool let lb = client.create_load_balancer(CreateLoadBalancerRequest { name: "simple-http".into(), org_id: "acme".into(), project_id: Some("web".into()), ..Default::default() }).await?; let pool = client.create_pool(CreatePoolRequest { lb_id: lb.id.clone(), name: "backends".into(), algorithm: Algorithm::RoundRobin, health_check: Some(HealthCheck::http("/health", vec![200])), ..Default::default() }).await?; // Add backends for addr in ["10.0.1.10", "10.0.1.11", "10.0.1.12"] { client.add_backend(AddBackendRequest { pool_id: pool.id.clone(), name: format!("backend-{}", addr), address: BackendAddress::Ip(addr.parse()?), port: 8080, ..Default::default() }).await?; } let listener = client.create_listener(CreateListenerRequest { lb_id: lb.id.clone(), name: "http".into(), protocol: ListenerProtocol::Http, port: 80, default_pool_id: Some(pool.id.clone()), ..Default::default() }).await?; ``` **HTTPS with Path-Based Routing**: ```rust // Create pools for different services let api_pool = create_pool("api-pool", Algorithm::LeastConnections); let static_pool = create_pool("static-pool", Algorithm::RoundRobin); // HTTPS listener with routing rules let https = client.create_listener(CreateListenerRequest { lb_id: lb.id.clone(), name: "https".into(), protocol: ListenerProtocol::Https, port: 443, tls_config: Some(TlsConfig { certificate_id: "cert-web-prod".into(), min_version: TlsVersion::Tls12, ..Default::default() }), default_pool_id: Some(static_pool.id.clone()), ..Default::default() }).await?; // Route /api/* to API pool client.add_routing_rule(AddRoutingRuleRequest { listener_id: https.id.clone(), rule: RoutingRule { name: "api-route".into(), priority: 10, conditions: vec![ RuleCondition::PathPrefix { value: "/api/".into() }, ], action: RuleAction::ForwardToPool { pool_id: api_pool.id.clone() }, enabled: true, ..Default::default() }, }).await?; ``` **TCP Load Balancer (Database)**: ```rust let lb = client.create_load_balancer(CreateLoadBalancerRequest { name: "postgres-lb".into(), org_id: "acme".into(), project_id: Some("data".into()), ..Default::default() }).await?; let pool = client.create_pool(CreatePoolRequest { lb_id: lb.id.clone(), name: "postgres-replicas".into(), algorithm: Algorithm::LeastConnections, health_check: Some(HealthCheck { check_type: HealthCheckType::Tcp {}, interval: 10, timeout: 5, ..Default::default() }), ..Default::default() }).await?; let listener = client.create_listener(CreateListenerRequest { lb_id: lb.id.clone(), name: "postgres".into(), protocol: ListenerProtocol::Tcp, port: 5432, default_pool_id: Some(pool.id.clone()), ..Default::default() }).await?; ``` ### E. Performance Considerations - **Connection pooling**: Reuse backend connections for L7 - **Zero-copy forwarding**: Use splice/sendfile where possible for L4 - **Health check spreading**: Jitter to avoid thundering herd - **Circuit breaker**: Fail fast to prevent cascade failures - **Backend caching**: Cache VM IP resolution from PlasmaVMC - **Hot config reload**: Update routing without connection drops