# Overlay Network Specification > Version: 1.0 | Status: Draft | Last Updated: 2025-12-08 ## 1. Overview ### 1.1 Purpose Overlay Network provides multi-tenant network isolation for PlasmaVMC virtual machines. It enables secure, isolated network environments per organization and project using OVN (Open Virtual Network) as the underlying network virtualization platform. The overlay network abstracts physical network infrastructure and provides logical networking constructs (VPCs, subnets, security groups) that ensure complete tenant isolation while maintaining flexibility for inter-tenant communication when explicitly configured. ### 1.2 Scope - **In scope**: Multi-tenant network isolation, VPC/subnet management, IP address allocation (DHCP/static), security groups, NAT (SNAT/DNAT), OVN integration, network policies, tenant-scoped networking - **Out of scope**: Physical network infrastructure management, BGP routing configuration, hardware load balancer integration (handled by FiberLB), DNS resolution (handled by FlashDNS), network monitoring/analytics (future) ### 1.3 Design Goals - **Strong tenant isolation**: Complete network separation between organizations and projects by default - **OVN-based**: Leverage mature OVN platform for proven multi-tenant networking - **PlasmaVMC integration**: Seamless integration with VM lifecycle management - **Automatic IPAM**: DHCP-based IP allocation with optional static assignment - **Security-first**: Security groups and network policies enforced at the network layer - **Scalable**: Support thousands of VMs across multiple tenants ## 2. Architecture ### 2.1 Crate Structure ``` overlay-network/ ├── crates/ │ ├── overlay-network-api/ # gRPC service definitions │ ├── overlay-network-client/ # Client library │ ├── overlay-network-core/ # Core network logic │ ├── overlay-network-server/ # Server binary │ ├── overlay-network-ovn/ # OVN integration layer │ ├── overlay-network-storage/ # Persistence layer (ChainFire) │ └── overlay-network-types/ # Shared types └── proto/ └── overlay-network.proto # Protocol definitions ``` ### 2.2 Component Topology ``` ┌─────────────────────────────────────────────────────────────┐ │ Overlay Network Service │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Network API │──│ Network Core │──│ OVN Adapter │ │ │ │ (gRPC) │ │ (VPC/Subnet)│ │ (OVN Client) │ │ │ └──────────────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │ │ ┌──────▼──────┐ ┌──────▼──────┐ │ │ │ ChainFire │ │ OVN North │ │ │ │ (state) │ │ (logical) │ │ │ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────┘ │ ┌──────────────────┼──────────────────┐ │ │ │ ┌───────▼──────┐ ┌───────▼──────┐ ┌───────▼──────┐ │ Node 1 │ │ Node 2 │ │ Node N │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ │ OVN │ │ │ │ OVN │ │ │ │ OVN │ │ │ │ Controller│ │ │ │Controller│ │ │ │Controller│ │ │ └────┬─────┘ │ │ └────┬─────┘ │ │ └────┬─────┘ │ │ │ │ │ │ │ │ │ │ │ ┌────▼─────┐ │ │ ┌────▼─────┐ │ │ ┌────▼─────┐ │ │ │ OVS │ │ │ │ OVS │ │ │ │ OVS │ │ │ │ Bridge │ │ │ │ Bridge │ │ │ │ Bridge │ │ │ └────┬─────┘ │ │ └────┬─────┘ │ │ └────┬─────┘ │ │ │ │ │ │ │ │ │ │ │ ┌────▼─────┐ │ │ ┌────▼─────┐ │ │ ┌────▼─────┐ │ │ │ VM Ports │ │ │ │ VM Ports │ │ │ │ VM Ports │ │ │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │ └──────────────┘ └──────────────┘ └──────────────┘ ``` ### 2.3 Data Flow ``` [PlasmaVMC VmService] → [NetworkService.create_port()] │ ▼ [Network Core Logic] │ ┌─────────────────────┼─────────────────────┐ │ │ │ ▼ ▼ ▼ [ChainFire] [OVN Northbound] [IPAM Logic] (state storage) (logical network) (IP allocation) │ │ │ └─────────────────────┼─────────────────────┘ │ ▼ [OVN Southbound] │ ▼ [OVN Controller] │ ▼ [OVS Bridge] │ ▼ [VM TAP Interface] ``` ### 2.4 Dependencies | Crate | Purpose | |-------|---------| | tokio | Async runtime | | tonic | gRPC framework | | prost | Protocol buffers | | chainfire-client | State persistence | | ovsdb-client | OVN Northbound API client | | ipnet | IP address/CIDR handling | | uuid | Resource identifiers | ## 3. Core Concepts ### 3.1 Tenant Hierarchy ``` Organization (org_id) └── Project (project_id) └── VPC (Virtual Private Cloud) └── Subnet(s) └── VM Port(s) ``` ### 3.2 VPC (Virtual Private Cloud) Each project has exactly one VPC (1:1 relationship). **VPC Identifier:** ``` vpc_id = "{org_id}/{project_id}" ``` **OVN Mapping:** - OVN Logical Router: Project VPC router - OVN Logical Switches: Subnets within VPC **CIDR Allocation:** - Default pool: `10.0.0.0/8` divided into `/16` subnets - Each project: `/16` subnet (65,536 IPs) - Example: Project 1 → `10.1.0.0/16`, Project 2 → `10.2.0.0/16` ### 3.3 Subnet Each VPC contains one or more subnets. **Subnet Identifier:** ``` subnet_id = "{org_id}/{project_id}/{subnet_name}" ``` **Default Subnet:** - Created automatically with project - Name: `default` - CIDR: `/24` within VPC CIDR (256 IPs) - Example: VPC `10.1.0.0/16` → Subnet `10.1.0.0/24` **OVN Mapping:** - OVN Logical Switch: Each subnet ### 3.4 Port (VM Network Interface) A port represents a VM's network interface attached to a subnet. **Port Identifier:** ``` port_id = UUID ``` **Attributes:** - MAC address (auto-generated or user-specified) - IP address (DHCP or static) - Security groups - Subnet attachment **OVN Mapping:** - OVN Logical Port: VM's network interface ### 3.5 Security Group A security group defines firewall rules for network traffic. **Security Group Identifier:** ``` sg_id = "{org_id}/{project_id}/{sg_name}" ``` **Default Security Group:** - Created automatically with project - Name: `default` - Rules: - Ingress: Allow all from same security group - Egress: Allow all **OVN Mapping:** - OVN ACL (Access Control List): Applied to logical ports ## 4. API ### 4.1 gRPC Services ```protobuf service NetworkService { // VPC operations rpc CreateVpc(CreateVpcRequest) returns (Vpc); rpc GetVpc(GetVpcRequest) returns (Vpc); rpc ListVpcs(ListVpcsRequest) returns (ListVpcsResponse); rpc DeleteVpc(DeleteVpcRequest) returns (Empty); // Subnet operations rpc CreateSubnet(CreateSubnetRequest) returns (Subnet); rpc GetSubnet(GetSubnetRequest) returns (Subnet); rpc ListSubnets(ListSubnetsRequest) returns (ListSubnetsResponse); rpc DeleteSubnet(DeleteSubnetRequest) returns (Empty); // Port operations (VM NIC attachment) rpc CreatePort(CreatePortRequest) returns (Port); rpc GetPort(GetPortRequest) returns (Port); rpc ListPorts(ListPortsRequest) returns (ListPortsResponse); rpc DeletePort(DeletePortRequest) returns (Empty); rpc AttachPort(AttachPortRequest) returns (Port); rpc DetachPort(DetachPortRequest) returns (Empty); // Security Group operations rpc CreateSecurityGroup(CreateSecurityGroupRequest) returns (SecurityGroup); rpc GetSecurityGroup(GetSecurityGroupRequest) returns (SecurityGroup); rpc ListSecurityGroups(ListSecurityGroupsRequest) returns (ListSecurityGroupsResponse); rpc UpdateSecurityGroup(UpdateSecurityGroupRequest) returns (SecurityGroup); rpc DeleteSecurityGroup(DeleteSecurityGroupRequest) returns (Empty); // NAT operations rpc CreateSnat(CreateSnatRequest) returns (SnatConfig); rpc DeleteSnat(DeleteSnatRequest) returns (Empty); rpc CreateDnat(CreateDnatRequest) returns (DnatConfig); rpc DeleteDnat(DeleteDnatRequest) returns (Empty); } ``` ### 4.2 Key Request/Response Types ```protobuf message CreateVpcRequest { string org_id = 1; string project_id = 2; string name = 3; string cidr = 4; // Optional, auto-allocated if not specified } message CreateSubnetRequest { string org_id = 1; string project_id = 2; string vpc_id = 3; string name = 4; string cidr = 5; // Must be within VPC CIDR bool dhcp_enabled = 6; repeated string dns_servers = 7; } message CreatePortRequest { string org_id = 1; string project_id = 2; string subnet_id = 3; string vm_id = 4; string mac_address = 5; // Optional, auto-generated if not specified string ip_address = 6; // Optional, DHCP if not specified repeated string security_group_ids = 7; } message CreateSecurityGroupRequest { string org_id = 1; string project_id = 2; string name = 3; string description = 4; repeated SecurityRule ingress_rules = 5; repeated SecurityRule egress_rules = 6; } message SecurityRule { Protocol protocol = 1; PortRange port_range = 2; // Optional, all ports if not specified SourceType source_type = 3; string source = 4; // CIDR or security_group_id } enum Protocol { PROTOCOL_UNSPECIFIED = 0; PROTOCOL_TCP = 1; PROTOCOL_UDP = 2; PROTOCOL_ICMP = 3; PROTOCOL_ALL = 4; } message PortRange { uint32 min = 1; uint32 max = 2; } enum SourceType { SOURCE_TYPE_UNSPECIFIED = 0; SOURCE_TYPE_CIDR = 1; SOURCE_TYPE_SECURITY_GROUP = 2; } ``` ### 4.3 Public Traits ```rust pub trait NetworkService: Send + Sync { async fn create_vpc(&self, req: CreateVpcRequest) -> Result; async fn get_vpc(&self, org_id: &str, project_id: &str) -> Result>; async fn create_subnet(&self, req: CreateSubnetRequest) -> Result; async fn create_port(&self, req: CreatePortRequest) -> Result; async fn attach_port_to_vm(&self, port_id: &str, vm_id: &str) -> Result<()>; async fn create_security_group(&self, req: CreateSecurityGroupRequest) -> Result; } ``` ### 4.4 Client Library ```rust let client = NetworkClient::connect("http://localhost:8081").await?; // Create VPC let vpc = client.create_vpc(CreateVpcRequest { org_id: "org1".to_string(), project_id: "proj1".to_string(), name: "my-vpc".to_string(), cidr: None, // Auto-allocate }).await?; // Create subnet let subnet = client.create_subnet(CreateSubnetRequest { org_id: "org1".to_string(), project_id: "proj1".to_string(), vpc_id: vpc.id.clone(), name: "default".to_string(), cidr: "10.1.0.0/24".to_string(), dhcp_enabled: true, dns_servers: vec!["8.8.8.8".to_string()], }).await?; // Create port for VM let port = client.create_port(CreatePortRequest { org_id: "org1".to_string(), project_id: "proj1".to_string(), subnet_id: subnet.id.clone(), vm_id: "vm-123".to_string(), mac_address: None, // Auto-generate ip_address: None, // DHCP security_group_ids: vec!["default".to_string()], }).await?; ``` ## 5. Data Models ### 5.1 Core Types ```rust pub struct Vpc { pub id: String, // "{org_id}/{project_id}" pub org_id: String, pub project_id: String, pub name: String, pub cidr: String, // "10.1.0.0/16" pub created_at: u64, pub updated_at: u64, } pub struct Subnet { pub id: String, // "{org_id}/{project_id}/{subnet_name}" pub org_id: String, pub project_id: String, pub vpc_id: String, pub name: String, pub cidr: String, // "10.1.0.0/24" pub gateway_ip: String, // "10.1.0.1" pub dns_servers: Vec, pub dhcp_enabled: bool, pub created_at: u64, } pub struct Port { pub id: String, // UUID pub org_id: String, pub project_id: String, pub subnet_id: String, pub vm_id: Option, // None if detached pub mac_address: String, pub ip_address: Option, // None if DHCP pending pub security_group_ids: Vec, pub ovn_port_uuid: String, // OVN logical port UUID pub created_at: u64, } pub struct SecurityGroup { pub id: String, // "{org_id}/{project_id}/{sg_name}" pub org_id: String, pub project_id: String, pub name: String, pub description: String, pub ingress_rules: Vec, pub egress_rules: Vec, pub created_at: u64, } pub struct SecurityRule { pub protocol: Protocol, pub port_range: Option<(u16, u16)>, pub source_type: SourceType, pub source: String, } pub enum Protocol { Tcp, Udp, Icmp, All, } pub enum SourceType { Cidr, // "10.1.0.0/24" SecurityGroup, // "{org_id}/{project_id}/{sg_name}" } ``` ### 5.2 Storage Format - **Engine**: ChainFire (distributed KVS) - **Serialization**: JSON (via serde_json) - **Key format**: Hierarchical keys for tenant scoping **ChainFire Keys:** ``` # VPC /networks/vpcs/{org_id}/{project_id} = Vpc (JSON) # Subnet /networks/subnets/{org_id}/{project_id}/{subnet_name} = Subnet (JSON) # Port /networks/ports/{org_id}/{project_id}/{port_id} = Port (JSON) # Security Group /networks/security_groups/{org_id}/{project_id}/{sg_name} = SecurityGroup (JSON) # IPAM /networks/ipam/{org_id}/{project_id}/{subnet_name}/allocated = ["10.1.0.10", ...] (JSON) # CIDR Allocation /networks/cidr/allocations/{org_id}/{project_id} = "10.1.0.0/16" (string) /networks/cidr/pool/used = ["10.1.0.0/16", "10.2.0.0/16", ...] (JSON) ``` ## 6. Network Isolation ### 6.1 Inter-Tenant Isolation **Organization Level:** - Default: Complete isolation (no communication) - Exception: Explicit peering configuration required **Project Level (Same Organization):** - Default: Isolated (no communication) - Exception: VPC peering or shared network for connectivity **OVN Implementation:** - Logical Switches: L2 isolation per subnet - Logical Routers: L3 routing control - ACLs: Security group enforcement ### 6.2 Intra-Tenant Communication **Same Subnet:** - L2 forwarding (MAC address-based) - Direct communication via OVN Logical Switch **Different Subnets (Same VPC):** - L3 routing via OVN Logical Router - Router forwards packets between Logical Switches **Packet Flow Example:** ``` VM1 (10.1.0.10) → VM2 (10.1.1.10) 1. VM1 sends packet to 10.1.1.10 2. TAP interface → OVS bridge 3. OVS → OVN Logical Switch (L2, no match) 4. OVN → Logical Router (L3 forwarding) 5. Logical Router → Destination Logical Switch 6. OVN ACL check (security groups) 7. Packet forwarded to VM2's TAP interface ``` ## 7. IP Address Management (IPAM) ### 7.1 IP Allocation Strategy **Automatic (DHCP):** - Default allocation method - OVN DHCP server assigns IPs from subnet pool - IPs tracked in ChainFire for conflict prevention **Static Assignment:** - User-specified IP address - Must be within subnet CIDR - Duplicate check required **IP Allocation Tracking:** ``` /networks/ipam/{org_id}/{project_id}/{subnet_name}/allocated = ["10.1.0.10", "10.1.0.11", ...] /networks/ipam/{org_id}/{project_id}/{subnet_name}/reserved = ["10.1.0.1", "10.1.0.254"] // gateway, broadcast ``` ### 7.2 DHCP Configuration **OVN DHCP Options:** ```rust pub struct DhcpOptions { pub subnet_id: String, pub gateway_ip: String, pub dns_servers: Vec, pub domain_name: Option, pub ntp_servers: Vec, pub lease_time: u32, // seconds } ``` **OVN Implementation:** - DHCP options configured on OVN Logical Switch - OVN acts as DHCP server - VMs receive IP, gateway, DNS via DHCP ## 8. Security ### 8.1 Security Groups **Default Security Group:** - Created automatically with project - Ingress: Allow from same security group - Egress: Allow all **OVN ACL Implementation:** - ACLs applied to Logical Ports - Direction: `from-lport` (egress), `to-lport` (ingress) - Action: `allow`, `drop`, `reject` **ACL Example:** ``` # Ingress: Allow TCP port 80 from security group "web" from-lport 1000 "tcp && tcp.dst == 80 && ip4.src == $sg_web" allow-related # Egress: Allow all to-lport 1000 "1" allow ``` ### 8.2 Network Policies **Policy Types:** 1. **Ingress Policy**: Inbound traffic control 2. **Egress Policy**: Outbound traffic control 3. **Isolation Policy**: Inter-network isolation settings **Implementation:** - OVN ACLs for enforcement - Combined with security groups ### 8.3 IP Spoofing Prevention - OVN validates source IP addresses - Blocks traffic from IPs not assigned to port ### 8.4 ARP Spoofing Prevention - OVN manages ARP tables - Blocks invalid ARP responses ## 9. NAT (Network Address Translation) ### 9.1 SNAT (Source NAT) **Purpose:** Private IP to external (Internet) communication **Configuration:** ```rust pub struct SnatConfig { pub vpc_id: String, pub external_ip: String, pub enabled: bool, } ``` **OVN Implementation:** - SNAT rule added to Logical Router - `ovn-nbctl lr-nat-add snat ` ### 9.2 DNAT (Destination NAT) **Purpose:** External to specific VM communication (port forwarding) **Configuration:** ```rust pub struct DnatConfig { pub vpc_id: String, pub external_ip: String, pub external_port: u16, pub internal_ip: String, pub internal_port: u16, pub protocol: Protocol, } ``` **OVN Implementation:** - `ovn-nbctl lr-nat-add dnat ` ## 10. Configuration ### 10.1 Config File Format (TOML) ```toml [network] ovn_northbound_endpoint = "tcp:127.0.0.1:6641" ovn_southbound_endpoint = "tcp:127.0.0.1:6642" cidr_pool = "10.0.0.0/8" default_subnet_size = 24 [storage] chainfire_endpoint = "http://127.0.0.1:50051" [server] listen_addr = "0.0.0.0:8081" ``` ### 10.2 Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `OVERLAY_NETWORK_OVN_NB_ENDPOINT` | `tcp:127.0.0.1:6641` | OVN Northbound DB endpoint | | `OVERLAY_NETWORK_OVN_SB_ENDPOINT` | `tcp:127.0.0.1:6642` | OVN Southbound DB endpoint | | `OVERLAY_NETWORK_CHAINFIRE_ENDPOINT` | `http://127.0.0.1:50051` | ChainFire endpoint | | `OVERLAY_NETWORK_CIDR_POOL` | `10.0.0.0/8` | CIDR pool for VPC allocation | | `OVERLAY_NETWORK_LISTEN_ADDR` | `0.0.0.0:8081` | gRPC server listen address | ### 10.3 CLI Arguments ``` overlay-network-server [OPTIONS] --config Config file path --ovn-nb-endpoint OVN Northbound endpoint --ovn-sb-endpoint OVN Southbound endpoint --chainfire-endpoint ChainFire endpoint --listen-addr gRPC listen address ``` ## 11. Operations ### 11.1 Deployment **Single Node:** - Network service runs alongside PlasmaVMC control plane - OVN Northbound/Southbound DBs on same node - Suitable for development/testing **Cluster Mode:** - Network service can be distributed - OVN databases replicated (OVSDB clustering) - ChainFire provides distributed state ### 11.2 Monitoring **Metrics (Prometheus format):** - `overlay_network_vpcs_total`: Total number of VPCs - `overlay_network_subnets_total`: Total number of subnets - `overlay_network_ports_total`: Total number of ports - `overlay_network_ip_allocations_total`: Total IP allocations - `overlay_network_ovn_operations_duration_seconds`: OVN operation latency **Health Endpoints:** - `/health`: Service health check - `/ready`: Readiness check (OVN connectivity) ### 11.3 Integration with PlasmaVMC **VM Creation Flow:** ``` 1. VmService.create_vm() called with NetworkSpec 2. NetworkService.create_port() creates OVN Logical Port 3. OVN assigns IP address (DHCP or static) 4. Security groups applied to port (OVN ACLs) 5. VM NIC attached to port (TAP interface) ``` **NetworkSpec Extension:** ```rust pub struct NetworkSpec { pub id: String, pub network_id: String, // subnet_id: "{org_id}/{project_id}/{subnet_name}" pub mac_address: Option, pub ip_address: Option, // None = DHCP pub model: NicModel, pub security_groups: Vec, // security_group_ids } ``` ## 12. Compatibility ### 12.1 API Versioning - Version scheme: Semantic versioning (v1.0, v1.1, etc.) - Deprecation policy: 2 major versions support - Breaking changes: New major version ### 12.2 Wire Protocol - Protocol buffer version: proto3 - Backward compatibility: Maintained within major version ## Appendix ### A. Error Codes | Code | Meaning | |------|---------| | `INVALID_CIDR` | Invalid CIDR format | | `CIDR_OVERLAP` | CIDR overlaps with existing allocation | | `SUBNET_OUTSIDE_VPC` | Subnet CIDR not within VPC CIDR | | `IP_OUTSIDE_SUBNET` | IP address not within subnet CIDR | | `IP_ALREADY_ALLOCATED` | IP address already in use | | `OVN_CONNECTION_FAILED` | Failed to connect to OVN | | `SECURITY_GROUP_NOT_FOUND` | Security group does not exist | | `PORT_ALREADY_ATTACHED` | Port already attached to VM | ### B. Glossary - **VPC (Virtual Private Cloud)**: Isolated network environment per project - **Subnet**: L2 network segment within VPC - **Port**: VM network interface attached to subnet - **Security Group**: Firewall rules for network traffic - **OVN (Open Virtual Network)**: Network virtualization platform - **Logical Switch**: OVN L2 network construct - **Logical Router**: OVN L3 routing construct - **Logical Port**: OVN port attached to Logical Switch - **ACL (Access Control List)**: OVN firewall rules - **IPAM (IP Address Management)**: IP allocation and tracking - **SNAT (Source NAT)**: Outbound NAT for external connectivity - **DNAT (Destination NAT)**: Inbound NAT for port forwarding ### C. OVN Integration Details **OVN Northbound Operations:** - Create Logical Switch: `ovn-nbctl ls-add ` - Create Logical Router: `ovn-nbctl lr-add ` - Create Logical Port: `ovn-nbctl lsp-add ` - Set DHCP Options: `ovn-nbctl dhcp-options-create` - Add ACL: `ovn-nbctl acl-add ` - Add NAT: `ovn-nbctl lr-nat-add ` **OVN Southbound State:** - Physical port bindings - Flow table entries - Chassis mappings