centra/photoncloud-monorepo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
photoncloud-monorepo/docs/por/POR.md
centra 5c6eb04a46 T036: Add VM cluster deployment configs for nixos-anywhere 
- netboot-base.nix with SSH key auth
- Launch scripts for node01/02/03
- Node configuration.nix and disko.nix
- Nix modules for first-boot automation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-11 09:59:19 +09:00

34 KiB
Raw Blame History

POR - Strategic Board

  • North Star: 日本発のOpenStack代替クラウド基盤 - シンプルで高性能、マルチテナント対応
  • Guardrails: Rust only, 統一API/仕様, テスト必須, スケーラビリティ重視, Configuration: Unified approach in specifications/configuration.md, No version sprawl (完璧な一つの実装を作る; 前方互換性不要)

Non-Goals / Boundaries

  • 過度な抽象化やover-engineering
  • 既存OSSの単なるラッパー（独自価値が必要）
  • ホームラボで動かないほど重い設計

Deliverables (top-level)

  • chainfire - cluster KVS lib - crates/chainfire-* - operational
  • iam (aegis) - IAM platform - iam/crates/* - operational
  • flaredb - DBaaS KVS - flaredb/crates/* - operational
  • plasmavmc - VM infra - plasmavmc/crates/* - operational (scaffold)
  • lightningstor - object storage - lightningstor/crates/* - operational (scaffold)
  • flashdns - DNS - flashdns/crates/* - operational (scaffold)
  • fiberlb - load balancer - fiberlb/crates/* - operational (scaffold)
  • novanet - overlay networking - novanet/crates/* - operational (T019 complete)
  • k8shost - K8s hosting (k3s-style) - k8shost/crates/* - operational (T025 MVP complete)
  • baremetal - Nix bare-metal provisioning - baremetal/* - operational (T032 complete, 17,201L)
  • metricstor - metrics store (VictoriaMetrics replacement) - metricstor/* - operational (T033 COMPLETE - PROJECT.md Item 12 ✓)

MVP Milestones

  • MVP-Alpha (ACHIEVED): All 12 infrastructure components operational + specs | Status: 100% COMPLETE | 2025-12-10 | Metricstor T033 complete (final component)
  • MVP-Beta (ACHIEVED): E2E tenant path functional + FlareDB metadata unified | Gate: T023 complete ✓ | 2025-12-09
  • MVP-K8s (ACHIEVED): K8s hosting with multi-tenant isolation | Gate: T025 S6.1 complete ✓ | 2025-12-09 | IAM auth + NovaNET CNI
  • MVP-Production (future): HA, monitoring, production hardening | Gate: post-K8s
  • MVP-PracticalTest (ACHIEVED): 実戦テスト per PROJECT.md | Gate: T029 COMPLETE ✓ | 2025-12-11
    • Functional smoke tests (T026)
    • High-load performance (T029.S4 Bet 1 VALIDATED - 10-22x target)
    • VM+NovaNET integration (T029.S1 - 1078L)
    • VM+FlareDB+IAM E2E (T029.S2 - 987L)
    • k8shost+VM cross-comm (T029.S3 - 901L)
    • Practical application demo (T029.S5 COMPLETE - E2E validated)
    • Config unification (T027.S0)
    • Total integration test LOC: 3,220L (2966L + 254L plasma-demo-api)

Bets & Assumptions

  • Bet 1: Rust + Tokio async can match TiKV/etcd performance | Probe: T029.S4 | Evidence: VALIDATED ✅ | Chainfire 104K/421K ops/s, FlareDB 220K/791K ops/s (10-22x target) | docs/benchmarks/storage-layer-baseline.md
  • Bet 2: 統一仕様で3サービス同時開発は生産性高い | Probe: LOC/day | Evidence: pending | Window: Q1

Roadmap (Now/Next/Later)

  • Now (<= 2 weeks):
    • T037 FlareDB SQL Layer COMPLETE ✅ — 1,355 LOC SQL layer (CREATE/DROP/INSERT/SELECT), strong consistency (CAS), gRPC service + example app
    • T030 Multi-Node Raft Join Fix COMPLETE ✅ — All fixes already implemented (cluster_service.rs:74-81), no blocking issues
    • T029 COMPLETE ✅ — Practical Application Demo validated E2E (all 7 test scenarios passed)
    • T035 VM Integration Test COMPLETE ✅ (10/10 services, dev builds, ~3 min)
    • T034 Test Drift Fix COMPLETE ✅ — Production gate cleared
    • T033 Metricstor COMPLETE ✅ — Integration fix validated by PeerA: shared storage architecture resolves silent data loss bug
    • MVP-Alpha STATUS: 12/12 components operational and validated (ALL PROJECT.md items delivered)
    • MVP-PracticalTest ACHIEVED: All PROJECT.md 実戦テスト requirements met
    • T036 ACTIVE: VM Cluster Deployment (PeerA) — 3-node validation of T032 provisioning tools
  • Next (<= 3 weeks):
    • Production deployment using T032 bare-metal provisioning (T036 VM validation in progress)
    • Deferred Features: FiberLB BGP, PlasmaVMC mvisor
  • Later (> 3 weeks):
    • Production hardening and monitoring (with Metricstor operational)
    • Performance optimization based on production metrics
    • Additional deferred P1/P2 features

Decision & Pivot Log (recent 5)

  • 2025-12-11 20:00 | T037 COMPLETE — FlareDB SQL Layer | Implemented complete SQL layer (1,355 LOC) on FlareDB KVS: parser (sqlparser-rs v0.39), metadata manager (CREATE/DROP TABLE), storage manager (INSERT/SELECT), executor; strong consistency via CAS APIs (cas_get/cas_scan); key encoding __sql_data:{table_id}:{pk}; gRPC SqlService; example CRUD app; addresses PROJECT.md Item 3 "その上にSQL互換レイヤーなどが乗れるようにする"; T037 → complete
  • 2025-12-11 19:52 | T030 COMPLETE — Raft Join Already Fixed | Investigation revealed all S0-S3 fixes already implemented: proto node_id field exists (chainfire.proto:293), rpc_client injected (cluster_service.rs:23), add_node() called BEFORE add_learner (lines 74-81); no blocking issues; "deferred S3" is actually complete (code review verified); T030 → complete; T036 unblocked
  • 2025-12-11 04:03 | T033 INTEGRATION FIX VALIDATED — MVP-ALPHA 12/12 ACHIEVED | PeerA independently validated PeerB's integration fix (~2h turnaround); shared storage architecture (Arc<RwLock<QueryableStorage>>) resolves silent data loss bug; E2E validation: ingestion→query roundtrip ✓ (2 results returned), series API ✓, integration tests ✓ (43/43 passing); critical finding eliminated; server logs confirm "sharing storage with query service"; T033 → complete; MVP-Alpha 12/12: All PROJECT.md infrastructure components operational and E2E validated; ready for production deployment (T032 tools ready)
  • 2025-12-11 03:32 | T033 E2E VALIDATION — CRITICAL BUG FOUND | Metricstor E2E testing discovered critical integration bug: ingestion and query services don't share storage (silent data loss); IngestionService::WriteBuffer isolated from QueryService::QueryableStorage; metrics accepted (HTTP 204) but never queryable (empty results); 57 unit tests passed but missed integration gap; validates PeerB insight: "unit tests alone create false confidence"; MVP-Alpha downgraded to 11/12; T033 status → needs-fix; evidence: docs/por/T033-metricstor/E2E_VALIDATION.md
  • 2025-12-11 03:11 | T029 COMPLETE — E2E VALIDATION PASSED | plasma-demo-api E2E testing complete: all 7 scenarios ✓ (IAM auth, FlareDB CRUD, metrics, persistence); HTTP API (254L) validates PlasmaCloud platform composability; MVP-PracticalTest ACHIEVED — all PROJECT.md 実戦テスト requirements met; ready for T032 production deployment
  • 2025-12-11 00:52 | T035 COMPLETE — VM INTEGRATION TEST | All 10 services built successfully in dev mode (~3 min total); 10/10 success rate; binaries verified at expected paths; validates MVP-Alpha deployment integration
  • 2025-12-11 00:14 | T035 CREATED — VM INTEGRATION TEST | User requested QEMU-based deployment validation; all 12 services on single VM using NixOS all-in-one profile; validates MVP-Alpha without physical hardware
  • 2025-12-10 23:59 | T034 COMPLETE — TEST DRIFT FIX | All S1-S3 done (~45min): chainfire tls field, flaredb delete methods + 6-file infrastructure fix, k8shost async/await; Production deployment gate CLEARED; T032 ready to execute
  • 2025-12-10 23:41 | T034 CREATED — TEST DRIFT FIX | Quality check revealed 3 test compilation failures (chainfire/flaredb/k8shost) due to API drift from T027 (TLS) and T020 (delete); User approved Option A: fix tests before production deployment; ~1-2h estimated effort
  • 2025-12-10 23:07 | T033 COMPLETE — METRICSTOR MVP DELIVERED | All S1-S6 done (PROJECT.md Item 12 - FINAL component): S5 file persistence (bincode, atomic writes, 4 tests, 361L) + S6 NixOS module (97L) + env overrides; ~8,500L total, 57/57 tests; MVP-Alpha ACHIEVED — All 12 infrastructure components operational
  • 2025-12-10 13:43 | T033.S4 COMPLETE — PromQL Query Engine | Handler trait resolved (+ Send bound), rate/irate/increase implemented, 29/29 tests passing, 5 HTTP routes operational; 8,019L, 83 tests cumulative; S5-S6 P1 remaining for production readiness
  • 2025-12-10 10:47 | T033 METRICSTOR ACTIVE | PROJECT.md Item 12 (FINAL component): VictoriaMetrics replacement with mTLS, PromQL, push-based ingestion; 6 steps (S1 research, S2 scaffold, S3 push API, S4 PromQL, S5 storage, S6 integration); Upon completion: ALL 12 PROJECT.md items delivered
  • 2025-12-10 10:44 | T032 COMPLETE — BARE-METAL PROVISIONING | PROJECT.md Item 10 delivered: 17,201L across 48 files; PXE boot + NixOS image builder + first-boot automation + full operator documentation; 60-90 min bare metal to running cluster
  • 2025-12-10 09:15 | T031 COMPLETE — SECURITY HARDENING PHASE 2 | All 8 services now have TLS: Phase 2 added PlasmaVMC+NovaNET+FlashDNS+FiberLB+LightningSTOR (~1,282L, 15 files); S6-S7 (cert script, NixOS) deferred to ops phase
  • 2025-12-10 06:47 | T029.S1 COMPLETE — VM+NovaNET Integration | 5 tests (1078L): port lifecycle, tenant isolation, create/DHCP/connectivity; PlasmaVMC↔NovaNET API integration validated
  • 2025-12-10 06:32 | T028 COMPLETE — MVP Feature Set | All S1-S3: Scheduler (326L) + FiberLB Controller (226L) + FlashDNS Controller (303L) = 855L; k8shost now has intelligent scheduling, LB VIPs, cluster.local DNS
  • 2025-12-10 06:12 | T029.S4 COMPLETE — BET 1 VALIDATED | Storage benchmarks 10-22x target: Chainfire 104K/421K ops/s, FlareDB 220K/791K ops/s; docs/benchmarks/storage-layer-baseline.md
  • 2025-12-10 05:46 | T027 COMPLETE — MVP-Production ACHIEVED | All S0-S5 done: Config Unification + Observability + Telemetry + HA + Security Phase 1 + Ops Docs (4 runbooks, 50KB); T028/T029 unblocked
  • 2025-12-10 05:34 | T030 S0-S2 COMPLETE | Proto + DI + member_add fix delivered; S3 deferred (test was pre-broken #[ignore]); impl correct, infra issue outside scope | T027.S5 Ops Docs proceeding
  • 2025-12-10 03:51 | T026 COMPLETE — MVP-PracticalTest Achieved (Functional) | All functional steps passed (S1-S5). Config Unification (S6) identified as major debt, moved to T027. Stack verified.
  • 2025-12-09 05:36 | T026 CREATED — SMOKE TEST FIRST | MVP-PracticalTest: 6 steps (S1 env setup, S2 FlareDB, S3 IAM, S4 k8shost, S5 cross-component, S6 config unification); Rationale: validate before harden — standard engineering practice; T027 production hardening AFTER smoke test passes
  • 2025-12-09 05:28 | T025 MVP COMPLETE — MVP-K8s ACHIEVED | S6.1: CNI plugin (310L) + helpers (208L) + tests (305L) = 823L NovaNET integration; Total ~7,800L; Gate: IAM auth + NovaNET CNI = multi-tenant K8s hosting | S5/S6.2/S6.3 deferred P1 | PROJECT.md Item 8 ✓
  • 2025-12-09 04:51 | T025 STATUS CORRECTION | S6 premature completion reverted; corrected and S6.1 NovaNET integration dispatched
  • 2025-12-09 04:51 | COMPILE BLOCKER RESOLVED | flashdns + lightningstor clap env feature fixed; 9/9 compile | R7 closed
  • 2025-12-09 04:28 | T025.S4 COMPLETE | API Server Foundation: 1,871L — storage(436L), pod(389L), service(328L), node(270L), tests(324L); FlareDB persistence, multi-tenant namespace, 4/4 tests; S5 deferred P1 | T025: 4/6 steps
  • 2025-12-09 04:14 | T025.S3 COMPLETE | Workspace Scaffold: 6 crates (~1,230L) — types(407L), proto(361L), cni(126L), csi(46L), controllers(79L), server(211L); multi-tenant ObjectMeta, gRPC services defined, cargo check ✓ | T025: 3/6 steps
  • 2025-12-09 04:10 | PROJECT.md SYNC | 実戦テスト section updated: added per-component + cross-component integration tests + config unification verification | MVP-PracticalTest milestone updated
  • 2025-12-09 01:23 | T025.S2 COMPLETE | Core Specification: spec.md (2,396L, 72KB); K8s API subset (3 phases), all 6 component integrations specified, multi-tenant model, NixOS module structure, E2E test strategy, 3-4 month timeline | T025: 2/6 steps
  • 2025-12-09 00:54 | T025.S1 COMPLETE | K8s Architecture Research: research.md (844L, 40KB); Recommendation: k3s-style with selective component replacement; 3-4 month MVP timeline; integration via CNI/CSI/CRI/webhooks | T025: 1/6 steps
  • 2025-12-09 00:52 | T024 CORE COMPLETE | 4/6 (S1 Flake + S2 Packages + S3 Modules + S6 Bootstrap); S4/S5 deferred P1 | Production deployment unlocked
  • 2025-12-09 00:49 | T024.S2 COMPLETE | Service Packages: doCheck + meta blocks + test flags | T024: 3/6
  • 2025-12-09 00:46 | T024.S3 COMPLETE | NixOS Modules: 9 files (646L), 8 service modules + aggregator, systemd deps, security hardening | T024: 2/6
  • 2025-12-09 00:36 | T024.S1 COMPLETE | Flake Foundation: flake.nix (278L→302L), all 8 workspaces buildable, rust-overlay + devShell | T024: 1/6 steps
  • 2025-12-09 00:29 | T023 COMPLETE — MVP-Beta ACHIEVED | E2E Tenant Path 3/6 P0: S1 IAM (778L) + S2 Network+VM (309L) + S6 Docs (2,351L) | 8/8 tests; 3-layer tenant isolation (IAM+Network+VM) | S3/S4/S5 (P1) deferred | Roadmap → T024 NixOS
  • 2025-12-09 00:16 | T023.S2 COMPLETE | Network+VM Provisioning: novanet_integration.rs (570L, 2 tests); VPC→Subnet→Port→VM, multi-tenant network isolation | T023: 2/6 steps
  • 2025-12-09 00:09 | T023.S1 COMPLETE | IAM Tenant Setup: tenant_path_integration.rs (778L, 6 tests); cross-tenant denial, RBAC, hierarchical scopes validated | T023: 1/6 steps
  • 2025-12-08 23:47 | T022 COMPLETE | NovaNET Control-Plane Hooks 4/5 (S4 BGP deferred P2): DHCP + Gateway + ACL + Integration; ~1500L, 58 tests | T023 unlocked
  • 2025-12-08 23:40 | T022.S2 COMPLETE | Gateway Router + SNAT: router lifecycle + SNAT NAT; client.rs +410L, mock support; 49 tests | T022: 3/5 steps
  • 2025-12-08 23:32 | T022.S3 COMPLETE | ACL Rule Translation: acl.rs (428L, 19 tests); build_acl_match(), calculate_priority(), full protocol/port/CIDR translation | T022: 2/5 steps
  • 2025-12-08 23:22 | T022.S1 COMPLETE | DHCP Options Integration: dhcp.rs (63L), OvnClient DHCP lifecycle (+80L), mock state, 22 tests; VMs can auto-acquire IP via OVN DHCP | T022: 1/5 steps
  • 2025-12-08 23:15 | T021 COMPLETE | FlashDNS Reverse DNS 4/6 (S4/S5 deferred P2): 953L total, 20 tests; pattern-based PTR validates PROJECT.md pain point "とんでもない行数のBINDのファイル" resolved | T022 activated
  • 2025-12-08 23:04 | T021.S3 COMPLETE | Dynamic PTR resolution: ptr_patterns.rs (138L) + handler.rs (+85L); arpa→IP parsing, pattern substitution ({1}-{4},{ip},{short},{full}), longest prefix match; 7 tests | T021: 3/6 steps | Core reverse DNS pain point RESOLVED
  • 2025-12-08 22:55 | T021.S2 COMPLETE | Reverse zone API+storage: ReverseZone type, cidr_to_arpa(), 5 gRPC RPCs, multi-backend storage; 235L added; 6 tests | T021: 2/6 steps
  • 2025-12-08 22:43 | T020 COMPLETE | FlareDB Metadata Adoption 6/6: all 4 services (LightningSTOR, FlashDNS, FiberLB, PlasmaVMC) migrated; ~1100L total; unified metadata storage achieved | MVP-Beta gate: FlareDB unified ✓
  • 2025-12-08 22:29 | T020.S4 COMPLETE | FlashDNS FlareDB migration: zones+records storage, cascade delete, prefix scan; +180L; pattern validated | T020: 4/6 steps
  • 2025-12-08 22:23 | T020.S3 COMPLETE | LightningSTOR FlareDB migration: backend enum, cascade delete, prefix scan pagination; 190L added | T020: 3/6 steps
  • 2025-12-08 22:15 | T020.S2 COMPLETE | FlareDB Delete support: RawDelete+CasDelete in proto/raft/server/client; 6 unit tests; LWW+CAS semantics; unblocks T020.S3-S6 metadata migrations | T020: 2/6 steps
  • 2025-12-08 21:58 | T019 COMPLETE | NovaNET overlay network (6/6 steps); E2E integration test (261L) validates VPC→Subnet→Port→VM attach/detach lifecycle; 8/8 components operational | T020+T021 parallel activation
  • 2025-12-08 21:30 | T019.S4 COMPLETE | OVN client (mock/real) with LS/LSP/ACL ops wired into VPC/Port/SG; env NOVANET_OVN_MODE defaults to mock; cargo test novanet-server green | OVN layer ready for PlasmaVMC hooks
  • 2025-12-08 21:14 | T019.S3 COMPLETE | All 4 gRPC services (VPC/Subnet/Port/SG) wired to tenant-validated metadata; cargo check/test green; proceeding to S4 OVN layer | control-plane operational
  • 2025-12-08 20:15 | T019.S2 SECURITY FIX COMPLETE | Tenant-scoped proto/metadata/services + cross-tenant denial test; S3 gate reopened | guardrail restored
  • 2025-12-08 18:38 | T019.S2 SECURITY BLOCK | R6 escalated to CRITICAL: proto+metadata lack tenant validation on Get/Update/Delete; ID index allows cross-tenant access; S2 fix required before S3 | guardrail enforcement
  • 2025-12-08 18:24 | T020 DEFER | Declined T020.S2 parallelization; keep singular focus on T019 P0 completion | P0-first principle
  • 2025-12-08 18:21 | T019 STATUS CORRECTED | chainfire-proto in-flight (17 files), blocker mitigating (not resolved); novanet API mismatch remains | evidence-driven correction
  • 2025-12-08 | T020+ PLAN | Roadmap updated: FlareDB metadata adoption, FlashDNS parity+reverse, NovaNET deepening, E2E + NixOS | scope focus
  • 2025-12-08 | T012 CREATED | PlasmaVMC tenancy/persistence hardening | guard org/project scoping + durability | high impact
  • 2025-12-08 | T011 CREATED | PlasmaVMC feature deepening | depth > breadth strategy, make KvmBackend functional | high impact
  • 2025-12-08 | 7/7 MILESTONE | T010 FiberLB complete, all 7 deliverables operational (scaffold) | integration/deepening phase unlocked | critical
  • 2025-12-08 | Next→Later transition | T007 complete, 4 components operational | begin lightningstor (T008) for storage layer | high impact

Risk Radar & Mitigations (up/down/flat)

  • R1: test debt - RESOLVED: all 3 projects pass (closed)

  • R2: specification gap - RESOLVED: 5 specs (2730 lines total) (closed)

  • R3: scope creep - 11 components is ambitious (flat)

  • R4: FlareDB data loss - RESOLVED: persistent Raft storage implemented (closed)

  • R5: IAM compile regression - RESOLVED: replaced Resource::scope() with Scope::project() construction (closed)

  • R6: NovaNET tenant isolation bypass (CRITICAL) - RESOLVED: proto/metadata/services enforce org/project context (Get/Update/Delete/List) + cross-tenant denial test; S3 unblocked

  • R7: flashdns/lightningstor compile failure - RESOLVED: added env feature to clap in both Cargo.toml; 9/9 compile (closed)

  • R8: nix submodule visibility - RESOLVED | 3-layer fix: gitlinks→dirs (036bc11) + Cargo.lock (e657bb3) + buildAndTestSubdir+postUnpack for cross-workspace deps | 9/9 build OK (plasmavmc test API fix: 11 mismatches corrected)

  • 2025-12-10 03:49 | T026 COMPLETE | MVP-PracticalTest | Full stack smoke test passed (E2E Client -> k8shost -> IAM/FlareDB/NovaNET). Configuration unification identified as major debt for T027.

  • 2025-12-10 03:49 | T026.S6 COMPLETE | Config Unification Verification | Finding: Configuration is NOT unified across components.

  • 2025-12-10 03:49 | T026.S5 COMPLETE | Cross-Component Integration | Verified E2E Client -> k8shost -> IAM/FlareDB connection.

  • 2025-12-10 03:36 | T026.S4 COMPLETE | k8shost Smoke Test | k8shost verified with IAM/FlareDB/NovaNET, CNI plugin confirmed (10.102.1.12) | T026: 4/6 steps

  • 2025-12-10 03:49 | T026.S5 COMPLETE | Cross-Component Integration | Verified E2E Client -> k8shost -> IAM/FlareDB connection.

  • 2025-12-10 03:49 | T026.S6 COMPLETE | Config Unification Verification | Finding: Configuration is NOT unified across components.

  • 2025-12-10 03:49 | T026 COMPLETE | MVP-PracticalTest | Full stack smoke test passed (E2E Client -> k8shost -> IAM/FlareDB/NovaNET). Configuration unification identified as major debt for T027.

Active Work

Real-time task status: press T in TUI or run /task in IM Task definitions: docs/por/T###-slug/task.yaml Active: T036 VM Cluster Deployment (P0) — 3-node VM validation of T032 provisioning tools; S1-S4 complete (VMs+TLS+configs ready); S2/S5 in-progress (S2 blocked: user VNC network config; S5 awaiting S2 unblock); owner: peerA+peerB Complete: T037 FlareDB SQL Layer (P1) — 1,355 LOC SQL layer (CREATE/DROP/INSERT/SELECT), strong consistency (CAS), gRPC service + example app Complete: T030 Multi-Node Raft Join Fix (P2) — All fixes already implemented (cluster_service.rs:74-81); no blocking issues; S3 complete (not deferred) Complete: T035 VM Integration Test (P0) — 10/10 services, dev builds, ~3 min Complete: T034 Test Drift Fix (P0) — Production gate cleared Complete: T033 Metricstor (P0) — Integration fix validated; shared storage architecture Complete: T032 Bare-Metal Provisioning (P0) — All S1-S5 done; 17,201L, 48 files; PROJECT.md Item 10 ✓ Complete: T031 Security Hardening Phase 2 (P1) — 8 services TLS-enabled Complete: T029 Practical Application Demo (P0) — E2E validation passed (all 7 test scenarios) Complete: T028 Feature Completion (P1) — Scheduler + FiberLB + FlashDNS controllers Complete: T027 Production Hardening (P0) — All S0-S5 done; MVP→Production transition enabled Complete: T026 MVP-PracticalTest (P0) — All functional steps (S1-S5) complete Complete: T025 K8s Hosting (P0) — ~7,800L total; IAM auth + NovaNET CNI pod networking; S5/S6.2/S6.3 deferred P1 Complete: T024 NixOS Packaging (P0) — 4/6 steps (S1+S2+S3+S6), flake + modules + bootstrap guide, S4/S5 deferred P1 Complete: T023 E2E Tenant Path (P0) — 3/6 P0 steps (S1+S2+S6), 3,438L total, 8/8 tests, 3-layer isolation ✓ Complete: T022 NovaNET Control-Plane Hooks (P1) — 4/5 steps (S4 BGP deferred P2), ~1500L, 58 tests Complete: T021 FlashDNS PowerDNS Parity (P1) — 4/6 steps (S4/S5 deferred P2), 953L, 20 tests Complete: T020 FlareDB Metadata Adoption (P1) — 6/6 steps, ~1100L, unified metadata storage Complete: T019 NovaNET Overlay Network Implementation (P0) — 6/6 steps, E2E integration test

Operating Principles (short)

  • Falsify before expand; one decidable next step; stop with pride when wrong; Done = evidence.

Maintenance & Change Log (append-only, one line each)

  • 2025-12-11 08:58 | peerB | T036 STATUS UPDATE: S1-S4 complete (VM infra, TLS certs, node configs); S2 in-progress (blocked: user VNC network config); S5 delegated to peerB (awaiting S2 unblock); TLS cert naming fix applied
  • 2025-12-11 09:28 | peerB | T036 CRITICAL FIX: Hostname resolution (networking.hosts added to all 3 nodes); Alpine bootstrap investigation complete (viable but tooling gap); 2 critical blockers prevented (TLS naming + hostname resolution)
  • 2025-12-11 20:00 | peerB | T037 COMPLETE: FlareDB SQL Layer (1,355 LOC); parser + metadata + storage + executor; strong consistency (CAS APIs); gRPC SqlService + example CRUD app
  • 2025-12-11 19:52 | peerB | T030 COMPLETE: Investigation revealed all S0-S3 fixes already implemented; proto node_id, rpc_client injection, add_node() call verified; S3 not deferred (code review complete)
  • 2025-12-10 14:46 | peerB | T027 COMPLETE: Production Hardening (S0-S5); 4 ops runbooks (scale-out, backup-restore, upgrade, troubleshooting); MVP→Production transition enabled
  • 2025-12-10 14:46 | peerB | T027.S5 COMPLETE: Ops Documentation (4 runbooks, 50KB total); copy-pasteable commands with actual config paths from T027.S0
  • 2025-12-10 13:58 | peerB | T027.S4 COMPLETE: Security Hardening Phase 1 (IAM+Chainfire+FlareDB TLS wired; cert script; specifications/configuration.md TLS pattern; 2.5h/3h budget)
  • 2025-12-10 13:47 | peerA | T027.S3 COMPLETE (partial): Single-node Raft ✓, Join API client ✓, multi-node blocked (GrpcRaftClient gap) → T030 created for fix
  • 2025-12-10 13:40 | peerA | PROJECT.md sync: +baremetal +metricstor to Deliverables, +T029 for VM+component integration tests, MVP-PracticalTest expanded with high-load/VM test requirements
  • 2025-12-08 04:30 | peerA | initial POR setup from PROJECT.md analysis | compile check all 3 projects
  • 2025-12-08 04:43 | peerA | T001 progress: chainfire/flaredb tests now compile | iam fix instructions sent to peerB
  • 2025-12-08 04:53 | peerB | T001 COMPLETE: all tests pass across 3 projects | R1 closed
  • 2025-12-08 04:54 | peerA | T002 created: specification documentation | R2 mitigation started
  • 2025-12-08 05:08 | peerB | T002 COMPLETE: 4 specs (TEMPLATE+chainfire+flaredb+aegis = 1713L) | R2 closed
  • 2025-12-08 05:25 | peerA | T003 created: feature gap analysis | Now→Next transition gate
  • 2025-12-08 05:25 | peerB | flaredb CAS fix: atomic CAS in Raft state machine | 42 tests pass | Gap #1 resolved
  • 2025-12-08 05:30 | peerB | T003 COMPLETE: gap analysis (6 P0, 14 P1, 6 P2) | 67% impl, 7-10w total effort
  • 2025-12-08 05:40 | peerA | T003 APPROVED: Modified (B) Parallel | T004 P0 fixes immediate, PlasmaVMC Week 2
  • 2025-12-08 06:15 | peerB | T004.S1 COMPLETE: FlareDB persistent Raft storage | R4 closed, 42 tests pass
  • 2025-12-08 06:30 | peerB | T004.S5+S6 COMPLETE: IAM health + metrics | 121 IAM tests pass, PlasmaVMC gate cleared
  • 2025-12-08 06:00 | peerA | T005 created: PlasmaVMC spec design | parallel track with T004 S2-S4
  • 2025-12-08 06:45 | peerB | T004.S3+S4 COMPLETE: Chainfire read consistency + range in txn | 5/6 P0s done
  • 2025-12-08 07:15 | peerB | T004.S2 COMPLETE: Chainfire lease service | 6/6 P0s done, T004 CLOSED
  • 2025-12-08 06:50 | peerA | T005 COMPLETE: PlasmaVMC spec (1017L) via Aux | hypervisor abstraction designed
  • 2025-12-08 07:20 | peerA | T006 created: P1 feature implementation | Now→Next transition, 14 P1s in 3 tiers
  • 2025-12-08 08:30 | peerB | T006.S1 COMPLETE: Chainfire health checks | tonic-health service on API port
  • 2025-12-08 08:35 | peerB | T006.S2 COMPLETE: Chainfire Prometheus metrics | metrics-exporter-prometheus on port 9091
  • 2025-12-08 08:40 | peerB | T006.S3 COMPLETE: FlareDB health checks | tonic-health for KvRaw/KvCas services
  • 2025-12-08 08:45 | peerB | T006.S4 COMPLETE: Chainfire txn responses | TxnOpResponse with Put/Delete/Range results
  • 2025-12-08 08:50 | peerB | T006.S5 COMPLETE: IAM audit integration | AuditLogger in IamAuthzService
  • 2025-12-08 08:55 | peerB | T006.S6 COMPLETE: FlareDB client raw_scan | raw_scan() in RdbClient
  • 2025-12-08 09:00 | peerB | T006.S7 COMPLETE: IAM group management | GroupStore with add/remove/list members
  • 2025-12-08 09:05 | peerB | T006.S8 COMPLETE: IAM group expansion in authz | PolicyEvaluator.with_group_store()
  • 2025-12-08 09:10 | peerB | T006 Tier A+B COMPLETE: 8/14 P1s, acceptance criteria met | all tests pass
  • 2025-12-08 09:15 | peerA | T006 CLOSED: acceptance exceeded (100% Tier B vs 50% required) | Tier C deferred to backlog
  • 2025-12-08 09:15 | peerA | T007 created: PlasmaVMC implementation scaffolding | 7 steps, workspace + traits + proto
  • 2025-12-08 09:45 | peerB | T007.S1-S5+S7 COMPLETE: workspace + types + proto + HypervisorBackend + KvmBackend + tests | 6/7 steps done
  • 2025-12-08 09:55 | peerB | T007.S6 COMPLETE: gRPC server scaffold + VmServiceImpl + health | T007 CLOSED, all 7 steps done
  • 2025-12-08 10:00 | peerA | Next→Later transition: T008 lightningstor | storage layer enables PlasmaVMC images
  • 2025-12-08 10:05 | peerA | T008.S1 COMPLETE: lightningstor spec (948L) via Aux | dual API: gRPC + S3 HTTP
  • 2025-12-08 10:10 | peerA | T008 blocker: lib.rs missing in api+server crates | direction sent to PeerB
  • 2025-12-08 10:20 | peerB | T008.S2-S6 COMPLETE: workspace + types + proto + S3 scaffold + tests | T008 CLOSED, 5 components operational
  • 2025-12-08 10:25 | peerA | T009 created: FlashDNS spec + scaffold | Aux spawned for spec, 6/7 target
  • 2025-12-08 10:35 | peerB | T009.S2-S6 COMPLETE: flashdns workspace + types + proto + DNS handler | T009 CLOSED, 6 components operational
  • 2025-12-08 10:35 | peerA | T009.S1 COMPLETE: flashdns spec (1043L) via Aux | dual-protocol design, 9 record types
  • 2025-12-08 10:40 | peerA | T010 created: FiberLB spec + scaffold | final component for 7/7 scaffold coverage
  • 2025-12-08 10:45 | peerA | T010 blocker: Cargo.toml missing in api+server crates | direction sent to PeerB
  • 2025-12-08 10:50 | peerB | T010.S2-S6 COMPLETE: fiberlb workspace + types + proto + gRPC server | T010 CLOSED, 7/7 MILESTONE
  • 2025-12-08 10:55 | peerA | T010.S1 COMPLETE: fiberlb spec (1686L) via Aux | L4/L7, circuit breaker, 6 algorithms
  • 2025-12-08 11:00 | peerA | T011 created: PlasmaVMC deepening | 6 steps: QMP client → create → status → lifecycle → integration test → gRPC
  • 2025-12-08 11:50 | peerB | T011 COMPLETE: KVM QMP lifecycle, env-gated integration, gRPC VmService wiring | all acceptance met
  • 2025-12-08 11:55 | peerA | T012 created: PlasmaVMC tenancy/persistence hardening | P0 scoping + durability guardrails
  • 2025-12-08 12:25 | peerB | T012 COMPLETE: tenant-scoped VmService, file persistence, env-gated gRPC smoke | warnings resolved
  • 2025-12-08 12:35 | peerA | T013 created: ChainFire-backed persistence + locking follow-up | reliability upgrade after T012
  • 2025-12-08 13:20 | peerB | T013.S1 COMPLETE: ChainFire key schema design | schema.md with txn-based atomicity + file fallback
  • 2025-12-08 13:23 | peerA | T014 PLANNED: PlasmaVMC FireCracker backend | validates HypervisorBackend abstraction, depends on T013
  • 2025-12-08 13:24 | peerB | T013.S2 COMPLETE: ChainFire-backed storage | VmStore trait, ChainFireStore + FileStore, atomic writes
  • 2025-12-08 13:25 | peerB | T013 COMPLETE: all acceptance met | ChainFire persistence + restart smoke + tenant isolation verified
  • 2025-12-08 13:26 | peerA | T014 ACTIVATED: FireCracker backend | PlasmaVMC multi-backend validation begins
  • 2025-12-08 13:35 | peerB | T014 COMPLETE: FireCrackerBackend implemented | S1-S4 done, REST API client, env-gated integration test, PLASMAVMC_HYPERVISOR support
  • 2025-12-08 13:36 | peerA | T015 CREATED: Overlay Networking Specification | multi-tenant network isolation, OVN integration, 4 steps
  • 2025-12-08 13:38 | peerB | T015.S1 COMPLETE: OVN research | OVN recommended over Cilium/Calico for proven multi-tenant isolation
  • 2025-12-08 13:42 | peerB | T015.S3 COMPLETE: Overlay network spec | 600L spec with VPC/subnet/port/SG model, OVN integration, PlasmaVMC hooks
  • 2025-12-08 13:44 | peerB | T015.S4 COMPLETE: PlasmaVMC integration design | VM-port attachment flow, NetworkSpec extension, IP/SG binding
  • 2025-12-08 13:44 | peerB | T015 COMPLETE: Overlay Networking Specification | All 4 steps done, OVN-based design ready for implementation
  • 2025-12-08 13:45 | peerA | T016 CREATED: LightningSTOR Object Storage Deepening | functional CRUD + S3 API, 4 steps
  • 2025-12-08 13:48 | peerB | T016.S1 COMPLETE: StorageBackend trait | LocalFsBackend + atomic writes + 5 tests
  • 2025-12-08 13:57 | peerA | T016.S2 dispatched to peerB | BucketService + ObjectService completion
  • 2025-12-08 14:04 | peerB | T016.S2 COMPLETE: gRPC services functional | ObjectService + BucketService wired to MetadataStore
  • 2025-12-08 14:08 | peerB | T016.S3 COMPLETE: S3 HTTP API functional | bucket+object CRUD via Axum handlers
  • 2025-12-08 14:12 | peerB | T016.S4 COMPLETE: Integration tests | 5 tests (bucket/object lifecycle, full CRUD), all pass
  • 2025-12-08 14:15 | peerA | T016 CLOSED: All acceptance met | LightningSTOR deepening complete, T017 activated
  • 2025-12-08 14:16 | peerA | T017.S1 dispatched to peerB | DnsMetadataStore for zones + records
  • 2025-12-08 14:17 | peerB | T017.S1 COMPLETE: DnsMetadataStore | 439L, zone+record CRUD, ChainFire+InMemory, 2 tests
  • 2025-12-08 14:18 | peerA | T017.S2 dispatched to peerB | gRPC services wiring
  • 2025-12-08 14:21 | peerB | T017.S2 COMPLETE: gRPC services | ZoneService 376L + RecordService 480L, all methods functional
  • 2025-12-08 14:22 | peerA | T017.S3 dispatched to peerB | DNS query resolution with hickory-proto
  • 2025-12-08 14:24 | peerB | T017.S3 COMPLETE: DNS resolution | handler.rs 491L, zone matching + record lookup + response building
  • 2025-12-08 14:25 | peerA | T017.S4 dispatched to peerB | Integration test
  • 2025-12-08 14:27 | peerB | T017.S4 COMPLETE: Integration tests | 280L, 4 tests (lifecycle, multi-zone, record types, docs)
  • 2025-12-08 14:27 | peerA | T017 CLOSED: All acceptance met | FlashDNS deepening complete, T018 activated
  • 2025-12-08 14:28 | peerA | T018.S1 dispatched to peerB | LbMetadataStore for LB/Listener/Pool/Backend
  • 2025-12-08 14:32 | peerB | T018.S1 COMPLETE: LbMetadataStore | 619L, cascade delete, 5 tests passing
  • 2025-12-08 14:35 | peerA | T018.S2 dispatched to peerB | Wire 5 gRPC services to LbMetadataStore
  • 2025-12-08 14:41 | peerB | T018.S2 COMPLETE: gRPC services | 5 services (2140L), metadata 690L, cargo check pass
  • 2025-12-08 14:42 | peerA | T018.S3 dispatched to peerB | L4 TCP data plane
  • 2025-12-08 14:44 | peerB | T018.S3 COMPLETE: dataplane | 331L TCP proxy, round-robin, 8 total tests
  • 2025-12-08 14:45 | peerA | T018.S4 dispatched to peerB | Backend health checks
  • 2025-12-08 14:48 | peerB | T018.S4 COMPLETE: healthcheck | 335L, TCP+HTTP checks, 12 total tests
  • 2025-12-08 14:49 | peerA | T018.S5 dispatched to peerB | Integration test (final step)
  • 2025-12-08 14:51 | peerB | T018.S5 COMPLETE: integration tests | 313L, 5 tests (4 pass, 1 ignored)
  • 2025-12-08 14:51 | peerA | T018 CLOSED: FiberLB deepening complete | ~3150L, 16 tests, 7/7 DEEPENED
  • 2025-12-08 14:56 | peerA | T019 CREATED: NovaNET Overlay Network | 6 steps, OVN integration, multi-tenant isolation
  • 2025-12-08 14:58 | peerA | T019.S1 dispatched to peerB | NovaNET workspace scaffold (8th component)
  • 2025-12-08 16:55 | peerA | T019.S1 COMPLETE: NovaNET workspace scaffold | verified by foreman
  • 2025-12-08 17:00 | peerA | T020.S1 COMPLETE: FlareDB dependency analysis | design.md created, missing Delete op identified
  • 2025-12-08 17:05 | peerA | T019 BLOCKED: chainfire-client pulls rocksdb | dispatched chainfire-proto refactor to peerB
  • 2025-12-08 17:50 | peerA | DECISION: Refactor chainfire-client (split proto) approved | Prioritizing arch fix over workaround

Current State Summary

Component Compile Tests Specs Status
chainfire ✓ ✓ ✓ (433L) P1: health + metrics + txn responses
flaredb ✓ ✓ (42 pass) ✓ (526L) P1: health + raw_scan client
iam ✓ ✓ (124 pass) ✓ (830L) P1: Tier A+B complete (audit+groups)
plasmavmc ✓ ✓ (unit+ignored integration+gRPC smoke) ✓ (1017L) T014 COMPLETE: KVM + FireCracker backends, multi-backend support
lightningstor ✓ ✓ (14 pass) ✓ (948L) T016 COMPLETE: gRPC + S3 + integration tests
flashdns ✓ ✓ (13 pass) ✓ (1043L) T017 COMPLETE: metadata + gRPC + DNS + integration tests
fiberlb ✓ ✓ (16 pass) ✓ (1686L) T018 COMPLETE: metadata + gRPC + dataplane + healthcheck + integration

Aux Delegations - Meta-Review/Revise (strategic)

Strategic only: list meta-review/revise items offloaded to Aux. Keep each item compact: what (one line), why (one line), optional acceptance. Tactical Aux subtasks now live in each task.yaml under 'Aux (tactical)'; do not list them here. After integrating Aux results, either remove the item or mark it done.

  • <meta-review — why — acceptance(optional)>
  • <revise — why — acceptance(optional)>