11 KiB
11 KiB
T055.S3: BGP Integration Strategy Specification
Author: PeerA Date: 2025-12-12 Status: DRAFT
1. Executive Summary
This document specifies the BGP Anycast integration strategy for FiberLB to enable VIP (Virtual IP) advertisement to upstream routers. The recommended approach is a sidecar pattern using GoBGP with gRPC API integration.
2. Background
2.1 Current State
- FiberLB binds listeners to
0.0.0.0:{port}on each node - LoadBalancer resources have
vip_addressfield (currently unused for routing) - No mechanism exists to advertise VIPs to physical network infrastructure
2.2 Requirements (from PROJECT.md Item 7)
- "BGP AnycastによるL2ロードバランシング" (BGP Anycast L2 LB)
- VIPs must be reachable from external networks
- Support for ECMP (Equal-Cost Multi-Path) across multiple FiberLB nodes
- Graceful withdrawal when load balancer is unhealthy/deleted
3. BGP Library Options Analysis
3.1 Option A: GoBGP Sidecar (RECOMMENDED)
Description: Run GoBGP as a sidecar container/process, control via gRPC API
| Aspect | Details |
|---|---|
| Language | Go |
| Maturity | Production-grade, widely deployed |
| API | gRPC with well-documented protobuf |
| Integration | FiberLB calls GoBGP gRPC to add/withdraw routes |
| Deployment | Separate process, co-located with FiberLB |
Pros:
- Battle-tested in production (Google, LINE, Yahoo Japan)
- Extensive BGP feature support (ECMP, BFD, RPKI)
- Clear separation of concerns
- Minimal code changes to FiberLB
Cons:
- External dependency (Go binary)
- Additional process management
- Network overhead for gRPC calls (minimal)
3.2 Option B: RustyBGP Sidecar
Description: Same sidecar pattern but using RustyBGP daemon
| Aspect | Details |
|---|---|
| Language | Rust |
| Maturity | Active development, less production deployment |
| API | GoBGP-compatible gRPC |
| Performance | Higher than GoBGP (multicore optimized) |
Pros:
- Rust ecosystem alignment
- Drop-in replacement for GoBGP (same API)
- Better performance in benchmarks
Cons:
- Less production history
- Smaller community
3.3 Option C: Embedded zettabgp
Description: Build custom BGP speaker using zettabgp library
| Aspect | Details |
|---|---|
| Language | Rust |
| Type | Parsing/composing library only |
| Integration | Embedded directly in FiberLB |
Pros:
- No external dependencies
- Full control over BGP behavior
- Single binary deployment
Cons:
- Significant implementation effort (FSM, timers, peer state)
- Risk of BGP protocol bugs
- Months of additional development
3.4 Option D: OVN Gateway Integration
Description: Leverage OVN's built-in BGP capabilities via OVN gateway router
| Aspect | Details |
|---|---|
| Dependency | Requires OVN deployment |
| Integration | FiberLB configures OVN via OVSDB |
Pros:
- No additional BGP daemon
- Integrated with SDN layer
Cons:
- Tightly couples to OVN
- Limited BGP feature set
- May not be deployed in all environments
4. Recommended Architecture
┌─────────────────────────────────────────────────────────────┐
│ FiberLB Node │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ │ gRPC │ │ │
│ │ FiberLB │───────>│ GoBGP │──── BGP ──│──> ToR Router
│ │ Server │ │ Daemon │ │
│ │ │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ VIP Traffic │ │
│ │ (Data Plane) │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
4.1 Components
- FiberLB Server - Existing service, adds BGP client module
- GoBGP Daemon - BGP speaker process, controlled via gRPC
- BGP Client Module - New Rust module using
gobgp-clientcrate or raw gRPC
4.2 Communication Flow
- LoadBalancer created with VIP address
- FiberLB checks backend health
- When healthy backends exist →
AddPath(VIP/32) - When all backends fail →
DeletePath(VIP/32) - LoadBalancer deleted →
DeletePath(VIP/32)
5. Implementation Design
5.1 New Module: fiberlb-bgp
// fiberlb/crates/fiberlb-bgp/src/lib.rs
pub struct BgpManager {
client: GobgpClient,
config: BgpConfig,
advertised_vips: HashSet<IpAddr>,
}
impl BgpManager {
/// Advertise a VIP to BGP peers
pub async fn advertise_vip(&mut self, vip: IpAddr) -> Result<()>;
/// Withdraw a VIP from BGP peers
pub async fn withdraw_vip(&mut self, vip: IpAddr) -> Result<()>;
/// Check if VIP is currently advertised
pub fn is_advertised(&self, vip: &IpAddr) -> bool;
}
5.2 Configuration Schema
# fiberlb-server config
bgp:
enabled: true
gobgp_address: "127.0.0.1:50051" # GoBGP gRPC address
local_as: 65001
router_id: "10.0.0.1"
neighbors:
- address: "10.0.0.254"
remote_as: 65000
description: "ToR Router"
5.3 GoBGP Configuration (sidecar)
# /etc/gobgp/gobgp.yaml
global:
config:
as: 65001
router-id: 10.0.0.1
port: 179
neighbors:
- config:
neighbor-address: 10.0.0.254
peer-as: 65000
afi-safis:
- config:
afi-safi-name: ipv4-unicast
add-paths:
config:
send-max: 8
5.4 Integration Points in FiberLB
// In loadbalancer_service.rs
impl LoadBalancerService {
async fn on_loadbalancer_active(&self, lb: &LoadBalancer) {
if let Some(vip) = &lb.vip_address {
if let Some(bgp) = &self.bgp_manager {
bgp.advertise_vip(vip.parse()?).await?;
}
}
}
async fn on_loadbalancer_deleted(&self, lb: &LoadBalancer) {
if let Some(vip) = &lb.vip_address {
if let Some(bgp) = &self.bgp_manager {
bgp.withdraw_vip(vip.parse()?).await?;
}
}
}
}
6. Deployment Patterns
6.1 NixOS Module
# modules/fiberlb-bgp.nix
{ config, lib, pkgs, ... }:
{
services.fiberlb = {
bgp = {
enable = true;
localAs = 65001;
routerId = "10.0.0.1";
neighbors = [
{ address = "10.0.0.254"; remoteAs = 65000; }
];
};
};
# GoBGP sidecar
services.gobgpd = {
enable = true;
config = fiberlb-bgp-config;
};
}
6.2 Container/Pod Deployment
# kubernetes deployment with sidecar
spec:
containers:
- name: fiberlb
image: plasmacloud/fiberlb:latest
env:
- name: BGP_GOBGP_ADDRESS
value: "localhost:50051"
- name: gobgp
image: osrg/gobgp:latest
args: ["-f", "/etc/gobgp/config.yaml"]
ports:
- containerPort: 179 # BGP
- containerPort: 50051 # gRPC
7. Health-Based VIP Withdrawal
7.1 Logic
┌─────────────────────────────────────────┐
│ Health Check Loop │
│ │
│ FOR each LoadBalancer WITH vip_address │
│ healthy_backends = count_healthy() │
│ │
│ IF healthy_backends > 0 │
│ AND NOT advertised(vip) │
│ THEN │
│ advertise(vip) │
│ │
│ IF healthy_backends == 0 │
│ AND advertised(vip) │
│ THEN │
│ withdraw(vip) │
│ │
└─────────────────────────────────────────┘
7.2 Graceful Shutdown
- SIGTERM received
- Withdraw all VIPs (allow BGP convergence)
- Wait for configurable grace period (default: 5s)
- Shutdown data plane
8. ECMP Support
With multiple FiberLB nodes advertising the same VIP:
┌─────────────┐
│ ToR Router │
│ (AS 65000) │
└──────┬──────┘
│ ECMP
┌──────────┼──────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│FiberLB-1│ │FiberLB-2│ │FiberLB-3│
│ VIP: X │ │ VIP: X │ │ VIP: X │
│AS 65001 │ │AS 65001 │ │AS 65001 │
└─────────┘ └─────────┘ └─────────┘
- All nodes advertise same VIP with same attributes
- Router distributes traffic via ECMP hashing
- Node failure = route withdrawal = automatic failover
9. Future Enhancements
- BFD (Bidirectional Forwarding Detection) - Faster failure detection
- BGP Communities - Traffic engineering support
- Route Filtering - Export policies per neighbor
- RustyBGP Migration - Switch from GoBGP for performance
- Embedded Speaker - Long-term: native Rust BGP using zettabgp
10. Implementation Phases
Phase 1: Basic Integration
- GoBGP sidecar deployment
- Simple VIP advertise/withdraw API
- Manual configuration
Phase 2: Health-Based Control
- Automatic VIP withdrawal on backend failure
- Graceful shutdown handling
Phase 3: Production Hardening
- BFD support
- Metrics and observability
- Operator documentation
11. References
- GoBGP - Official documentation
- RustyBGP - Rust BGP daemon
- zettabgp - Rust BGP library
- kube-vip BGP Mode - Similar pattern
- MetalLB BGP - Kubernetes LB BGP
12. Decision Summary
| Decision | Choice | Rationale |
|---|---|---|
| Integration Pattern | Sidecar | Clear separation, proven pattern |
| BGP Daemon | GoBGP | Production maturity, extensive features |
| API | gRPC | Native GoBGP interface, language-agnostic |
| Future Path | RustyBGP | Same API, better performance when stable |