- Replace form_urlencoded with RFC 3986 compliant URI encoding - Implement aws_uri_encode() matching AWS SigV4 spec exactly - Unreserved chars (A-Z,a-z,0-9,-,_,.,~) not encoded - All other chars percent-encoded with uppercase hex - Preserve slashes in paths, encode in query params - Normalize empty paths to '/' per AWS spec - Fix test expectations (body hash, HMAC values) - Add comprehensive SigV4 signature determinism test This fixes the canonicalization mismatch that caused signature validation failures in T047. Auth can now be enabled for production. Refs: T058.S1
378 lines
12 KiB
Markdown
378 lines
12 KiB
Markdown
# CreditService Specification
|
|
|
|
> Version: 1.0 | Status: Draft | Last Updated: 2025-12-11
|
|
|
|
## 1. Overview
|
|
|
|
### 1.1 Purpose
|
|
CreditService is a centralized credit/quota management system that acts as the "bank" for PhotonCloud. It manages project-based resource usage accounting and billing, providing admission control for resource creation requests and periodic billing based on usage metrics from NightLight.
|
|
|
|
### 1.2 Scope
|
|
- **In scope**:
|
|
- Project wallet/balance management
|
|
- Admission control (pre-creation checks)
|
|
- Usage-based billing via NightLight integration
|
|
- Transaction logging and audit trail
|
|
- Quota enforcement
|
|
- **Out of scope**:
|
|
- Payment processing (external system)
|
|
- Pricing configuration (admin API, separate)
|
|
- User-facing billing UI (separate frontend)
|
|
|
|
### 1.3 Design Goals
|
|
- **Multi-tenant**: Strict project isolation with org hierarchy
|
|
- **High availability**: Distributed storage via ChainFire/FlareDB
|
|
- **Low latency**: Admission control must not add >10ms to resource creation
|
|
- **Auditability**: Complete transaction history
|
|
- **Integration**: Seamless with IAM, NightLight, and all resource services
|
|
|
|
## 2. Architecture
|
|
|
|
### 2.1 Crate Structure
|
|
```
|
|
creditservice/
|
|
├── crates/
|
|
│ ├── creditservice-types/ # Core types (Wallet, Transaction, Quota)
|
|
│ ├── creditservice-proto/ # gRPC proto definitions
|
|
│ ├── creditservice-api/ # gRPC service implementations
|
|
│ ├── creditservice-server/ # Server binary
|
|
│ └── creditservice-client/ # Client library
|
|
├── proto/
|
|
│ └── creditservice.proto
|
|
└── Cargo.toml
|
|
```
|
|
|
|
### 2.2 Data Flow
|
|
```
|
|
Resource Service (PlasmaVMC, etc.)
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ CreditService │
|
|
│ ┌─────────────┐ ┌──────────────┐ ┌────────────┐ │
|
|
│ │ Admission │───▶│ Wallet │◀───│ Billing │ │
|
|
│ │ Control │ │ Manager │ │ Batch │ │
|
|
│ └─────────────┘ └──────────────┘ └────────────┘ │
|
|
│ │ │ ▲ │
|
|
│ ▼ ▼ │ │
|
|
│ ┌─────────────────────────────────────────────┤ │
|
|
│ │ ChainFire/FlareDB │ │
|
|
│ └─────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────┘
|
|
│
|
|
NightLight │
|
|
(Metrics) │
|
|
```
|
|
|
|
### 2.3 Dependencies
|
|
| Crate | Purpose |
|
|
|-------|---------|
|
|
| tokio | Async runtime |
|
|
| tonic | gRPC framework |
|
|
| chainfire-client | Distributed KV for wallets |
|
|
| flaredb-client | Metadata storage (optional) |
|
|
| iam-client | Authentication/authorization |
|
|
| nightlight-client | Usage metrics (NightLight) |
|
|
| rust_decimal | Precise credit calculations |
|
|
|
|
## 3. API
|
|
|
|
### 3.1 gRPC Services
|
|
|
|
```protobuf
|
|
syntax = "proto3";
|
|
package creditservice.v1;
|
|
|
|
// CreditService provides credit/quota management
|
|
service CreditService {
|
|
// Wallet operations
|
|
rpc GetWallet(GetWalletRequest) returns (GetWalletResponse);
|
|
rpc CreateWallet(CreateWalletRequest) returns (CreateWalletResponse);
|
|
rpc TopUp(TopUpRequest) returns (TopUpResponse);
|
|
rpc GetTransactions(GetTransactionsRequest) returns (GetTransactionsResponse);
|
|
|
|
// Admission Control (called by resource services)
|
|
rpc CheckQuota(CheckQuotaRequest) returns (CheckQuotaResponse);
|
|
rpc ReserveCredits(ReserveCreditsRequest) returns (ReserveCreditsResponse);
|
|
rpc CommitReservation(CommitReservationRequest) returns (CommitReservationResponse);
|
|
rpc ReleaseReservation(ReleaseReservationRequest) returns (ReleaseReservationResponse);
|
|
|
|
// Billing (internal)
|
|
rpc ProcessBilling(ProcessBillingRequest) returns (ProcessBillingResponse);
|
|
|
|
// Quota management
|
|
rpc SetQuota(SetQuotaRequest) returns (SetQuotaResponse);
|
|
rpc GetQuota(GetQuotaRequest) returns (GetQuotaResponse);
|
|
}
|
|
```
|
|
|
|
### 3.2 Admission Control Flow
|
|
```
|
|
1. Resource Service receives creation request
|
|
2. Resource Service calls CheckQuota(project_id, resource_type, quantity)
|
|
3. If approved:
|
|
a. ReserveCredits(project_id, estimated_cost) -> reservation_id
|
|
b. Create resource
|
|
c. CommitReservation(reservation_id, actual_cost)
|
|
4. If failed:
|
|
a. ReleaseReservation(reservation_id)
|
|
```
|
|
|
|
### 3.3 Client Library
|
|
```rust
|
|
use creditservice_client::Client;
|
|
|
|
let client = Client::connect("http://creditservice:8090").await?;
|
|
|
|
// Check if resource can be created
|
|
let check = client.check_quota(CheckQuotaRequest {
|
|
project_id: "proj-123".into(),
|
|
resource_type: ResourceType::VmInstance,
|
|
quantity: 1,
|
|
estimated_cost: Some(Decimal::new(100, 2)), // 1.00 credits
|
|
}).await?;
|
|
|
|
if check.allowed {
|
|
// Reserve and create
|
|
let reservation = client.reserve_credits(ReserveCreditsRequest {
|
|
project_id: "proj-123".into(),
|
|
amount: Decimal::new(100, 2),
|
|
description: "VM instance creation".into(),
|
|
}).await?;
|
|
|
|
// ... create resource ...
|
|
|
|
client.commit_reservation(CommitReservationRequest {
|
|
reservation_id: reservation.id,
|
|
actual_amount: Decimal::new(95, 2), // actual cost
|
|
}).await?;
|
|
}
|
|
```
|
|
|
|
## 4. Data Models
|
|
|
|
### 4.1 Core Types
|
|
|
|
```rust
|
|
/// Project wallet containing credit balance
|
|
pub struct Wallet {
|
|
/// Project ID (primary key)
|
|
pub project_id: String,
|
|
/// Organization ID (for hierarchy)
|
|
pub org_id: String,
|
|
/// Current available balance
|
|
pub balance: Decimal,
|
|
/// Reserved credits (pending reservations)
|
|
pub reserved: Decimal,
|
|
/// Total credits ever deposited
|
|
pub total_deposited: Decimal,
|
|
/// Total credits consumed
|
|
pub total_consumed: Decimal,
|
|
/// Wallet status
|
|
pub status: WalletStatus,
|
|
/// Creation timestamp
|
|
pub created_at: DateTime<Utc>,
|
|
/// Last update timestamp
|
|
pub updated_at: DateTime<Utc>,
|
|
}
|
|
|
|
pub enum WalletStatus {
|
|
Active,
|
|
Suspended, // Insufficient balance
|
|
Closed,
|
|
}
|
|
|
|
/// Credit transaction record
|
|
pub struct Transaction {
|
|
pub id: Uuid,
|
|
pub project_id: String,
|
|
pub transaction_type: TransactionType,
|
|
pub amount: Decimal,
|
|
pub balance_after: Decimal,
|
|
pub description: String,
|
|
pub resource_id: Option<String>,
|
|
pub created_at: DateTime<Utc>,
|
|
}
|
|
|
|
pub enum TransactionType {
|
|
TopUp, // Credit addition
|
|
Reservation, // Temporary hold
|
|
Charge, // Actual consumption
|
|
Release, // Reservation release
|
|
Refund, // Credit return
|
|
BillingCharge, // Periodic billing
|
|
}
|
|
|
|
/// Credit reservation (2-phase commit)
|
|
pub struct Reservation {
|
|
pub id: Uuid,
|
|
pub project_id: String,
|
|
pub amount: Decimal,
|
|
pub status: ReservationStatus,
|
|
pub description: String,
|
|
pub expires_at: DateTime<Utc>,
|
|
pub created_at: DateTime<Utc>,
|
|
}
|
|
|
|
pub enum ReservationStatus {
|
|
Pending,
|
|
Committed,
|
|
Released,
|
|
Expired,
|
|
}
|
|
|
|
/// Resource quota limits
|
|
pub struct Quota {
|
|
pub project_id: String,
|
|
pub resource_type: ResourceType,
|
|
pub limit: i64,
|
|
pub current_usage: i64,
|
|
}
|
|
|
|
pub enum ResourceType {
|
|
VmInstance,
|
|
VmCpu,
|
|
VmMemoryGb,
|
|
StorageGb,
|
|
NetworkPort,
|
|
LoadBalancer,
|
|
DnsZone,
|
|
// ... extensible
|
|
}
|
|
```
|
|
|
|
### 4.2 Storage Format
|
|
- **Engine**: ChainFire (for HA) or FlareDB (for scale)
|
|
- **Serialization**: Protocol Buffers
|
|
- **Key format**:
|
|
```
|
|
/credit/wallet/{project_id} -> Wallet
|
|
/credit/txn/{project_id}/{txn_id} -> Transaction
|
|
/credit/rsv/{reservation_id} -> Reservation
|
|
/credit/quota/{project_id}/{type} -> Quota
|
|
```
|
|
|
|
## 5. Configuration
|
|
|
|
### 5.1 Config File Format (TOML)
|
|
```toml
|
|
[service]
|
|
api_addr = "0.0.0.0:8090"
|
|
metrics_addr = "0.0.0.0:9090"
|
|
|
|
[storage]
|
|
# ChainFire for HA, FlareDB for scale
|
|
backend = "chainfire" # or "flaredb"
|
|
endpoints = ["chainfire-1:2379", "chainfire-2:2379", "chainfire-3:2379"]
|
|
|
|
[billing]
|
|
# Billing interval in seconds
|
|
interval_seconds = 3600 # hourly
|
|
# NightLight endpoint for usage metrics
|
|
nightlight_endpoint = "http://nightlight:8080"
|
|
|
|
[reservation]
|
|
# Reservation expiry (uncommitted reservations)
|
|
expiry_seconds = 300 # 5 minutes
|
|
|
|
[auth]
|
|
# IAM endpoint for token validation
|
|
iam_endpoint = "http://iam:8080"
|
|
|
|
[tls]
|
|
enabled = true
|
|
cert_file = "/etc/creditservice/tls.crt"
|
|
key_file = "/etc/creditservice/tls.key"
|
|
ca_file = "/etc/creditservice/ca.crt"
|
|
```
|
|
|
|
### 5.2 Environment Variables
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| CREDITSERVICE_API_ADDR | 0.0.0.0:8090 | API listen address |
|
|
| CREDITSERVICE_STORAGE_BACKEND | chainfire | Storage backend |
|
|
| CREDITSERVICE_BILLING_INTERVAL | 3600 | Billing interval (seconds) |
|
|
| CREDITSERVICE_NIGHTLIGHT_ENDPOINT | - | NightLight endpoint |
|
|
|
|
## 6. Security
|
|
|
|
### 6.1 Authentication
|
|
- mTLS for service-to-service communication
|
|
- IAM token validation for API requests
|
|
|
|
### 6.2 Authorization
|
|
- **Wallet operations**: Requires `credit:wallets:*` permission on project scope
|
|
- **Admission control**: Service accounts with `credit:admission:check` permission
|
|
- **Billing**: Internal service account only
|
|
|
|
### 6.3 Multi-tenancy
|
|
- All operations scoped to project_id
|
|
- Org hierarchy enforced (org admin can manage project wallets)
|
|
- Cross-project access denied at API layer
|
|
|
|
## 7. Operations
|
|
|
|
### 7.1 Deployment
|
|
- **Single node**: For development/testing
|
|
- **Cluster mode**: ChainFire backend for HA
|
|
|
|
### 7.2 Monitoring
|
|
Prometheus metrics exposed:
|
|
- `creditservice_wallet_balance{project_id}` - Current balance
|
|
- `creditservice_transactions_total{type}` - Transaction counts
|
|
- `creditservice_admission_latency_seconds` - Admission control latency
|
|
- `creditservice_billing_charges_total` - Billing charges processed
|
|
|
|
### 7.3 Health Endpoints
|
|
- `GET /healthz` - Liveness probe
|
|
- `GET /readyz` - Readiness probe (storage connectivity)
|
|
|
|
## 8. NightLight Integration
|
|
|
|
### 8.1 Usage Metrics Query
|
|
CreditService queries NightLight for resource usage:
|
|
```promql
|
|
# VM CPU hours per project
|
|
sum by (project_id) (
|
|
increase(vm_cpu_seconds_total[1h])
|
|
) / 3600
|
|
|
|
# Storage GB-hours
|
|
sum by (project_id) (
|
|
avg_over_time(storage_bytes_total[1h])
|
|
) / (1024^3)
|
|
```
|
|
|
|
### 8.2 Billing Batch Process
|
|
1. Query NightLight for usage metrics per project
|
|
2. Calculate charges based on pricing rules
|
|
3. Deduct from project wallet
|
|
4. Log transactions
|
|
5. Suspend wallets with negative balance
|
|
|
|
## 9. Error Codes
|
|
|
|
| Code | Meaning |
|
|
|------|---------|
|
|
| INSUFFICIENT_CREDITS | Not enough credits for operation |
|
|
| QUOTA_EXCEEDED | Resource quota limit reached |
|
|
| WALLET_SUSPENDED | Wallet is suspended |
|
|
| WALLET_NOT_FOUND | Project wallet does not exist |
|
|
| RESERVATION_EXPIRED | Credit reservation has expired |
|
|
| INVALID_AMOUNT | Amount must be positive |
|
|
|
|
## Appendix
|
|
|
|
### A. Pricing Model (Example)
|
|
| Resource | Unit | Price (credits/hour) |
|
|
|----------|------|---------------------|
|
|
| VM CPU | vCPU | 0.01 |
|
|
| VM Memory | GB | 0.005 |
|
|
| Storage | GB | 0.001 |
|
|
| Network Egress | GB | 0.05 |
|
|
|
|
### B. Glossary
|
|
- **Wallet**: Project-level credit account
|
|
- **Admission Control**: Pre-creation check for sufficient credits/quota
|
|
- **Reservation**: Temporary credit hold during resource creation
|
|
- **Billing Batch**: Periodic process to charge for usage
|