Dev Box

Bet

Having a relatively beefy server with state that's always on is just too useful. I should make one.

• It shouldn't be infra like a CI/CD runner
• It should be a suitable usage entrypoint via TTYD etc
• It should be trivial to nuke and set up again ✓
  • Terraform + cloud-init handles packages/tools, bazel run :deploy -- nuke for full rebuild
  • SSH keys injected at apply-time from local ~/.ssh/

Next Steps

None - core functionality complete!

Definition of Done

• Dev box running and responding on https://lab.tailf2e1e.ts.net
• Bazelified automation for nuking and recreating (bazel run //homelab/hosts/lab:deploy -- nuke)

Log

2025-12-26

TTYD web terminal complete via Tailscale.

Approach change: Dropped public gateway (Caddy) in favor of Tailscale-native access.

• VPN-only by design - no public exposure
• Clean URLs via MagicDNS: https://lab.tailf2e1e.ts.net
• TLS handled automatically by Tailscale

TTYD setup (cloud-init.yaml.tftpl):

• Installs ttyd binary from GitHub releases
• Systemd service runs ttyd -W -p 7681 tmux new-session -A -s main
• Persistent tmux session survives browser disconnects
• -W flag required for writable terminal (read-only by default!)

Tailscale integration:

• tailscale up --auth-key=... --hostname=lab for auto-registration
• tailscale serve --bg --https=443 7681 exposes TTYD on port 443
• Auth key: reusable + ephemeral (old nodes auto-expire)
• API key: used by deploy.sh to delete old node before nuke (avoids lab-1, lab-2 accumulation)

Environment variables required:

• PROXMOX_VE_API_TOKEN - Proxmox access
• TF_VAR_ubuntu_password - VM console password
• TF_VAR_tailscale_auth_key - VM auto-registration
• TAILSCALE_API_KEY - Node cleanup on destroy/nuke

2025-12-25

Core infrastructure complete in homelab/hosts/lab/. Full nuke+recreate in ~45 seconds.

Terraform config (main.tf):

• VM: 16 cores, 32GB RAM, cloned from ubuntu-24.04-template
• Storage: 100GB NVMe boot disk + 1TB ZFS data disk
• Pinned MAC 02:00:00:00:1A:BB for stable IP assignment
• Cloud-init via snippet upload (requires SSH to Proxmox host)
• QEMU agent timeout set to 60s (default 15min was too long)

Cloud-init (cloud-init.yaml.tftpl):

• Terraform template - injects SSH key and password at apply time
• User config (ubuntu user, sudo, SSH keys, password for console access)
• Packages: qemu-guest-agent, curl, git, htop, jq, tmux, zsh, ripgrep, fd, fzf, build-essential
• Starship prompt, auto-start guest agent
• Auto-formats and mounts 1TB disk to /data

Bazel integration (BUILD.bazel + deploy.sh):

• Cross-platform Terraform binary (macOS arm64/x86_64, Linux x86_64)
• bazel run //homelab/hosts/lab:deploy -- {init|plan|apply|destroy|nuke}
• nuke = destroy + apply for full rebuild

Issues resolved:

• QEMU agent timeout: Added qemu-guest-agent package + immediate systemctl enable
• Snippets storage: Created dedicated snippets Directory storage in Proxmox (one-time setup)
• SSH auth for snippets: Provider needs SSH access to Proxmox for snippet uploads (API doesn't support it)
• User/key injection: Moved from user_account block to cloud-init template to avoid conflicts

2025-12-24

Decided on approach:

• VM lifecycle: Terraform + Proxmox provider. Declarative, fits "nuke and recreate" model well.
• VM setup: Cloud-init for bootstrap. Simpler than Ansible, sufficient for initial needs.
• Key management: Local keys injected at apply-time. Pragmatic, can harden later if needed.
• Bazel integration: Bazel validates config, shell script wraps actual terraform apply (matches existing deploy patterns).
• Networking: Pin MAC address in Terraform config. Ubiquiti frozen IPs are tied to MAC, not hostname—without this, recreated VMs get new MACs and lose their static IP assignment.