photoncloud-monorepo/docs/por/T036-vm-cluster-deployment/task.yaml

id: T036
name: VM Cluster Deployment (T032 Validation)
goal: Deploy and validate a 3-node PlasmaCloud cluster using T032 bare-metal provisioning tools in a VM environment to validate end-to-end provisioning flow before physical deployment.
status: active
priority: P0
owner: peerA
created: 2025-12-11
depends_on: [T032, T035]
blocks: []

context: |
  PROJECT.md Principal: "Peer Aへ：**自分で戦略を**決めて良い！好きにやれ！"

  Strategic Decision: Pursue VM-based testing cluster (Option A from deployment readiness assessment)
  to validate T032 tools end-to-end before committing to physical infrastructure.

  T032 delivered: PXE boot infra, NixOS image builder, first-boot automation, documentation (17,201L)
  T035 validated: Single-VM build integration (10/10 services, dev builds)

  This task validates: Multi-node cluster deployment, PXE boot flow, nixos-anywhere,
  Raft cluster formation, first-boot automation, and operational procedures.

acceptance:
  - 3 VMs deployed with libvirt/KVM
  - Virtual network configured for PXE boot
  - PXE server running and serving netboot images
  - All 3 nodes provisioned via nixos-anywhere
  - Chainfire + FlareDB Raft clusters formed (3-node quorum)
  - IAM service operational on all control-plane nodes
  - Health checks passing on all services
  - T032 RUNBOOK validated end-to-end

steps:
  - step: S1
    name: VM Infrastructure Setup
    done: 3 VMs created with QEMU, multicast socket network configured, launch scripts ready
    status: complete
    owner: peerA
    priority: P0
    progress: |
      **COMPLETED** — VM infrastructure operational, pivoted to ISO boot approach

      Completed:
      - ✅ Created VM working directory: /home/centra/cloud/baremetal/vm-cluster
      - ✅ Created disk images: node01/02/03.qcow2 (100GB each)
      - ✅ Wrote launch scripts: launch-node{01,02,03}.sh
      - ✅ Configured QEMU multicast socket networking (230.0.0.1:1234)
      - ✅ VM specs: 8 vCPU, 16GB RAM per node
      - ✅ MACs assigned: 52:54:00:00:01:{01,02,03} (nodes)
      - ✅ Netboot artifacts built successfully (bzImage 14MB, initrd 484MB, ZFS disabled)
      - ✅ **PIVOT DECISION**: ISO boot approach (QEMU 10.1.2 initrd compatibility bug)
      - ✅ Downloaded NixOS 25.11 minimal ISO (1.6GB)
      - ✅ Node01 booting from ISO, multicast network configured

    notes: |
      **Topology Change:** Abandoned libvirt bridges (required root). Using QEMU directly with:
      - Multicast socket networking (no root required): `-netdev socket,mcast=230.0.0.1:1234`
      - 3 node VMs (pxe-server dropped due to ISO pivot)
      - All VMs share L2 segment via multicast

      **PIVOT JUSTIFICATION (MID: cccc-1765406017-b04a6e):**
      - Netboot artifacts validated ✓ (build process, kernel-6.18 ZFS fix)
      - QEMU 10.1.2 initrd bug blocks PXE testing (environmental, not T032 issue)
      - ISO + nixos-anywhere validates core T032 provisioning capability
      - PXE boot protocol deferred for bare-metal validation

  - step: S2
    name: Network Access Configuration
    done: Node VMs configured with SSH access for nixos-anywhere (netboot key auth)
    status: complete
    owner: peerB
    priority: P0
    progress: |
      **COMPLETED** — Custom netboot with SSH key auth bypasses VNC/telnet entirely

      Completed (2025-12-11):
      - ✅ Updated nix/images/netboot-base.nix with real SSH key (centra@cn-nixos-think)
      - ✅ Added netboot-base to flake.nix nixosConfigurations
      - ✅ Built netboot artifacts (kernel 14MB, initrd 484MB)
      - ✅ Created launch-node01-netboot.sh (QEMU -kernel/-initrd direct boot)
      - ✅ Fixed init path in kernel append parameter
      - ✅ SSH access verified (port 2201, key auth, zero manual interaction)

      Evidence:
      ```
      ssh -p 2201 root@localhost -> SUCCESS: nixos at Thu Dec 11 12:48:13 AM UTC 2025
      ```

      **PIVOT DECISION (2025-12-11, MID: cccc-1765413547-285e0f):**
      - PeerA directive: Build custom netboot with SSH key baked in
      - Eliminates VNC/telnet/password setup entirely
      - Netboot approach superior to ISO for automated provisioning
    notes: |
      **Solution Evolution:**
      - Initial: VNC (Option C) - requires user
      - Investigation: Alpine/telnet (Options A/B) - tooling gap/fragile
      - Final: Custom netboot with SSH key (PeerA strategy) - ZERO manual steps

      Files created:
      - baremetal/vm-cluster/launch-node01-netboot.sh (direct kernel/initrd boot)
      - baremetal/vm-cluster/netboot-{kernel,initrd}/ (nix build outputs)

  - step: S3
    name: TLS Certificate Generation
    done: CA and per-node certificates generated, ready for deployment
    status: complete
    owner: peerA
    priority: P0
    progress: |
      **COMPLETED** — TLS certificates generated and deployed to node config directories

      Completed:
      - ✅ Generated CA certificate and key
      - ✅ Generated node01.crt/.key (192.168.100.11)
      - ✅ Generated node02.crt/.key (192.168.100.12)
      - ✅ Generated node03.crt/.key (192.168.100.13)
      - ✅ Copied to docs/por/T036-vm-cluster-deployment/node*/secrets/
      - ✅ Permissions set (ca.crt/node*.crt: 644, node*.key: 400)
      - ✅ **CRITICAL FIX (2025-12-11):** Renamed certs to match cluster-config.json expectations
        - ca-cert.pem → ca.crt, cert.pem → node0X.crt, key.pem → node0X.key (all 3 nodes)
        - Prevented first-boot automation failure (services couldn't load TLS certs)

    notes: |
      Certificates ready for nixos-anywhere deployment (will be placed at /etc/nixos/secrets/)
      **Critical naming fix applied:** Certs renamed to match cluster-config.json paths

  - step: S4
    name: Node Configuration Preparation
    done: configuration.nix, disko.nix, cluster-config.json ready for all 3 nodes
    status: complete
    owner: peerB
    priority: P0
    progress: |
      **COMPLETED** — All node configurations created and validated

      Deliverables (13 files, ~600 LOC):
      - ✅ node01/configuration.nix (112L) - NixOS system config, control-plane services
      - ✅ node01/disko.nix (62L) - Disk partitioning (EFI + LVM)
      - ✅ node01/secrets/cluster-config.json (28L) - Raft bootstrap config
      - ✅ node01/secrets/README.md - TLS documentation
      - ✅ node02/* (same structure, IP: 192.168.100.12)
      - ✅ node03/* (same structure, IP: 192.168.100.13)
      - ✅ DEPLOYMENT.md (335L) - Comprehensive deployment guide

      Configuration highlights:
      - All 9 control-plane services enabled per node
      - Bootstrap mode: `bootstrap: true` on all 3 nodes (simultaneous initialization)
      - Network: Static IPs 192.168.100.11/12/13
      - Disk: Single-disk LVM (512MB EFI + 80GB root + 19.5GB data)
      - First-boot automation: Enabled with cluster-config.json
      - **CRITICAL FIX (2025-12-11):** Added networking.hosts to all 3 nodes (configuration.nix:14-19)
        - Maps node01/02/03 hostnames to 192.168.100.11/12/13
        - Prevented Raft bootstrap failure (cluster-config.json uses hostnames, DNS unavailable)

    notes: |
      Node configurations ready for nixos-anywhere provisioning (S5)
      TLS certificates from S3 already in secrets/ directories
      **Critical fixes applied:** TLS cert naming (S3), hostname resolution (/etc/hosts)

  - step: S5
    name: Cluster Provisioning
    done: All 3 nodes provisioned via nixos-anywhere, first-boot automation completed
    status: in_progress
    owner: peerB
    priority: P0
    progress: |
      **BLOCKED** — nixos-anywhere flake path resolution errors (nix store vs git working tree)

      Completed:
      - ✅ All 3 VMs launched with custom netboot (SSH ports 2201/2202/2203, key auth)
      - ✅ SSH access verified on all nodes (zero manual interaction)
      - ✅ Node configurations staged in git (node0{1,2,3}/configuration.nix + disko.nix + secrets/)
      - ✅ nix/modules staged (first-boot-automation, k8shost, metricstor, observability)
      - ✅ Launch scripts created: launch-node0{1,2,3}-netboot.sh

      Blocked:
      - ❌ nixos-anywhere failing with path resolution errors
      - ❌ Error: `/nix/store/.../docs/nix/modules/default.nix does not exist`
      - ❌ Root cause: Git tree dirty + files not in nix store
      - ❌ 3 attempts made, each failing on different missing path

      Next (awaiting PeerA decision):
      - Option A: Continue debug (may need git commit or --impure flag)
      - Option B: Alternative provisioning (direct configuration.nix)
      - Option C: Hand off to PeerA
        - Analyzed telnet serial console automation viability
        - Presented 3 options: Alpine automation (A), NixOS+telnet (B), VNC (C)

      Blocked:
      - ❌ SSH access unavailable (connection refused to 192.168.100.11)
      - ❌ S2 dependency: VNC network configuration or telnet console bypass required

      Next steps (when unblocked):
      - [ ] Choose unblock strategy: VNC (C), NixOS+telnet (B), or Alpine (A)
      - [ ] Run nixos-anywhere for node01/02/03
      - [ ] Monitor first-boot automation logs
      - [ ] Verify cluster formation (Chainfire, FlareDB Raft)

    notes: |
      **Unblock Options (peerB investigation 2025-12-11):**
      - Option A: Alpine virt ISO + telnet automation (viable but fragile)
      - Option B: NixOS + manual telnet console (recommended: simple, reliable)
      - Option C: Original VNC approach (lowest risk, requires user)

      ISO boot approach (not PXE):
      - Boot VMs from NixOS/Alpine ISO
      - Configure SSH via VNC or telnet serial console
      - Execute nixos-anywhere with node configurations from S4
      - First-boot automation will handle cluster initialization

  - step: S6
    name: Cluster Validation
    done: All acceptance criteria met, cluster operational, RUNBOOK validated
    status: pending
    owner: peerA
    priority: P0
    notes: |
      Validate cluster per T032 QUICKSTART:
      - Chainfire cluster: 3 members, leader elected, health OK
      - FlareDB cluster: 3 members, quorum formed, health OK
      - IAM service: all nodes responding
      - CRUD operations: write/read/delete working
      - Data persistence: verify across restarts
      - Metrics: Prometheus endpoints responding

evidence: []
notes: |
  **Strategic Rationale:**
  - VM deployment validates T032 tools without hardware dependency
  - Fastest feedback loop (~3-4 hours total)
  - After success, physical bare-metal deployment has validated blueprint
  - Failure discovery in VMs is cheaper than on physical hardware

  **Timeline Estimate:**
  - S1 VM Infrastructure: 30 min
  - S2 PXE Server: 30 min
  - S3 TLS Certs: 15 min
  - S4 Node Configs: 30 min
  - S5 Provisioning: 60 min
  - S6 Validation: 30 min
  - Total: ~3.5 hours

  **Success Criteria:**
  - All 6 steps complete
  - 3-node Raft cluster operational
  - T032 RUNBOOK procedures validated
  - Ready for physical bare-metal deployment