# lightscale lab (NixOS VMs)

This folder contains NixOS tests that boot QEMU VMs, register them against the
lightscale control plane, bring up WireGuard interfaces, and verify ping across overlay IPs.

## Run the lab test

Fast (2 nodes) test to avoid long builds:

```sh
cd /home/centra/dev/lightscale
nix build .#nixosTests.lightscale-lab
./result/bin/nixos-test-driver
```

Or use the helper script (prebuilds client/server first):

```sh
cd /home/centra/dev/lightscale
./lab/run.sh
```

To watch logs interactively (recommended when debugging hangs):

```sh
cd /home/centra/dev/lightscale
./lab/run.sh --interactive --keep
```

In the Python prompt, run:

```python
node1.execute("tail -f /tmp/lightscale-server.log")
```

Full (5 nodes) test:

```sh
cd /home/centra/dev/lightscale
nix build .#nixosTests.lightscale-lab-5
./result/bin/nixos-test-driver
```

Helper script:

```sh
cd /home/centra/dev/lightscale
./lab/run.sh full
```

Firewall (UDP blocked, stream relay fallback):

```sh
cd /home/centra/dev/lightscale
./lab/run.sh firewall
```

Control plane restart/outage resilience:

```sh
cd /home/centra/dev/lightscale
./lab/run.sh controlplane
```

Control plane HA (shared DB + client failover across control URLs):

```sh
cd /home/centra/dev/lightscale
./lab/run.sh controlplane-ha
```

Stream relay failover (multi relay servers, first down):

```sh
cd /home/centra/dev/lightscale
./lab/run.sh relay-failover
```

The tests will:
- start node1 as the control server
- register clients
- run `agent` on each node (endpoint rotation + stream relay fallback phases)
- ping full-mesh between overlay IPv4 addresses in both phases
- for the firewall test, block UDP/51820 between nodes and verify stream relay fallback