commit b79a208c333b9a96b56648e39b74ef9386fc575e Author: Funky (OpenClaw) Date: Thu Feb 5 16:09:40 2026 +0000 Initial commit: Infrastructure audit and documentation - Complete infrastructure audit (Feb 5, 2026) - 2026 homelab best practices guide - Current infrastructure notes (TOOLS.md) - README with quick reference diff --git a/README.md b/README.md new file mode 100644 index 0000000..d59511d --- /dev/null +++ b/README.md @@ -0,0 +1,50 @@ +# Fred's Homelab Documentation + +**Infrastructure documentation and configuration for Fred's homelab** + +## What's Here + +- **docs/** - Infrastructure audits, guides, and documentation +- **infrastructure/** - Current infrastructure notes and configs +- **docker-compose/** - Docker Compose files for services +- **scripts/** - Automation and maintenance scripts + +## Quick Reference + +**Gitea:** http://10.0.10.2:3000 +**Proxmox Hosts:** 10.0.10.2, 10.0.10.3, 10.0.10.4 +**OMV Storage:** 10.0.10.5 +**OpenClaw:** 10.0.10.28 + +## Key Documents + +- [Complete Infrastructure Audit (2026-02-05)](docs/INFRASTRUCTURE-AUDIT-COMPLETE-2026-02-05.md) +- [2026 Homelab Best Practices Guide](docs/homelab-2026-guide.md) +- [Infrastructure Notes (TOOLS.md)](infrastructure/TOOLS.md) + +## Proxmox Services + +| Service | IP | Container/VM | Purpose | +|---------|-----|--------------|---------| +| Home Assistant | 10.0.10.24 | VM 104 | Smart home automation | +| n8n | 10.0.10.22 | CT 106 | Workflow automation | +| Uptime Kuma | 10.0.10.26 | CT 128 | Monitoring | +| OpenClaw | 10.0.10.28 | CT 130 | AI agent | +| Step-CA | 10.0.10.15 | CT 115 | Internal CA | +| Twingate | router-pve | CT 101 | VPN connector | +| Gitea | 10.0.10.2 | Docker | Git repository | + +## Updating Documentation + +This repository is the **source of truth** for infrastructure knowledge. + +When making changes to the homelab: +1. Document the change in the relevant file +2. Commit with descriptive message +3. Push to Gitea + +OpenClaw (Funky) reads this repo to answer infrastructure questions! + +--- + +*Maintained by Fred Book with assistance from Funky (OpenClaw AI agent)* diff --git a/docs/INFRASTRUCTURE-AUDIT-COMPLETE-2026-02-05.md b/docs/INFRASTRUCTURE-AUDIT-COMPLETE-2026-02-05.md new file mode 100644 index 0000000..30076fc --- /dev/null +++ b/docs/INFRASTRUCTURE-AUDIT-COMPLETE-2026-02-05.md @@ -0,0 +1,783 @@ +# Complete Infrastructure Audit Report +## Fred Book's Homelab - February 5, 2026 + +**Audited by:** Funky (OpenClaw Agent) + Claude Code (previous audit) +**Audit Period:** January 2026 - February 5, 2026 +**Last Updated:** 2026-02-05 15:24 UTC + +--- + +## Executive Summary + +Fred's homelab is a well-structured Proxmox-based infrastructure supporting smart home automation, workflow automation, monitoring, and emerging 3D printing business operations. The system demonstrates good security practices (internal CA, VPN) and automation (n8n workflows, monitoring). + +**Key Strengths:** +- ✅ Multiple Proxmox hosts providing redundancy +- ✅ Internal certificate authority (Step-CA) +- ✅ WireGuard VPN for secure remote access +- ✅ Comprehensive monitoring (Uptime Kuma, n8n workflows) +- ✅ External reverse proxy (Caddy on VPS) + +**Areas for Improvement:** +- ⚠️ Backup system needs documentation and verification +- ⚠️ SSH connectivity issues between some containers +- ⚠️ No centralized infrastructure documentation (Gitea needed) +- ⚠️ 4TB HDD on router-pve underutilized + +--- + +## Network Architecture + +### Network Segments + +**Primary Network:** 10.0.10.0/24 +- Main homelab services +- Proxmox management interfaces +- LXC containers and VMs + +**VPN Network:** 10.0.8.0/24 +- WireGuard tunnel +- Secure remote access to homelab + +**External Access:** +- VPS: 66.63.182.168 (vps.nianticbooks.com) +- Caddy reverse proxy handling public access +- Routes to internal services via WireGuard + +--- + +## Proxmox Infrastructure + +### Proxmox Hosts + +**1. main-pve (10.0.10.3)** +- Role: Primary virtualization host +- Running: Multiple LXC containers +- Notes: Likely running Home Assistant, n8n, other core services + +**2. pve-router (10.0.10.2) / router-pve** +- Role: Router/gateway + Proxmox host +- Currently running: Home Assistant (confirmed by Fred) +- Storage: **4TB HDD - Currently underutilized** ⚠️ +- Notes: Lightly loaded, good candidate for additional services +- SSH Access: Working from external clients, hanging from 10.0.10.28 + +**3. pve-storage (10.0.10.4)** +- Role: Storage-focused Proxmox host +- May also be OMV (OpenMediaVault) server + +**4. HP DL380 (Proxmox host)** +- Running: OpenClaw Gateway container (10.0.10.28) +- Notes: Enterprise-grade hardware + +### Proxmox Version +- Multiple hosts, versions not yet confirmed +- Recommended: Upgrade to Proxmox VE 9.1 (latest as of 2026) + +--- + +## Storage Infrastructure + +### OMV (OpenMediaVault) Servers + +**OMV 10.0.10.4** +- Old storage server +- Status: Active but possibly being phased out + +**OMV 10.0.10.5** (Primary) +- Main data share: `\\10.0.10.5\data` +- Stores: Backups, VA documents, research, infrastructure docs +- Access: SMB/CIFS shares + +**Known Files on 10.0.10.5:** +- `/data/INFRASTRUCTURE-AUDIT-REPORT.md` (Claude Code audit) +- `/data/VA-Strategy/` (Fred's VA claim documents) +- `/data/backups/` (backup destination) +- Various project and research files + +**Storage Recommendations:** +- Consolidate OMV instances if redundant +- Use router-pve 4TB HDD for backup target +- Consider TrueNAS SCALE for future storage needs + +--- + +## Key Services & Applications + +### LXC Containers + +**Home Assistant (10.0.10.24)** +- Platform: LXC container +- Purpose: Smart home automation +- Running on: Confirmed on router-pve, possibly on other hosts too +- Access: http://10.0.10.24:8123 + +**n8n Workflow Automation (10.0.10.22) - CT 106** +- Platform: LXC container +- Purpose: Workflow automation and orchestration +- Web UI: http://10.0.10.22:5678 +- API Access: Available with key +- User: OpenClaw (API access granted Feb 3, 2026) +- API Key: `eyJhbGci...` (stored in TOOLS.md) + +**Active n8n Workflows:** +- Prometheus alerts → Discord/Email/Pushover +- Backup verification (daily @ 6 AM) ⚠️ **Needs documentation** +- SSL certificate expiration monitor +- Service health monitor (every 5 min via Uptime Kuma) +- Task overdue alerts (every 4 hours) +- Uptime Kuma webhook → alerts + +**Uptime Kuma (10.0.10.26) - CT 128** +- Platform: LXC container +- Purpose: Service monitoring and uptime tracking +- Integration: Feeds into n8n for alerting + +**OpenClaw Gateway (10.0.10.28) - CT 130** (Me!) +- Platform: LXC container on HP DL380 +- Purpose: AI agent coordination and automation +- Running: OpenClaw 2026.2.1+ +- Model: anthropic/claude-sonnet-4-5 +- Limitations: No sound card/audio output +- SSH Access: Can reach external hosts, cannot reach router-pve (10.0.10.2) ⚠️ + +**Step-CA (10.0.10.15) - CT 115** +- Platform: LXC container +- Purpose: Internal certificate authority +- Function: Issues TLS certificates for internal services +- Integration: Works with internal Caddy instances + +--- + +## External Infrastructure + +### VPS (66.63.182.168 - vps.nianticbooks.com) + +**Platform:** Cloud VPS +**Purpose:** External reverse proxy and public access point + +**Services Running:** +- Caddy reverse proxy +- Handles public DNS and routing +- Terminates WireGuard VPN connections +- Routes traffic to internal services securely + +**Configuration:** +- SSL certificates via LetsEncrypt (managed by Caddy) +- Routes to internal services via WireGuard tunnel +- Provides secure external access without exposing homelab + +--- + +## Security Infrastructure + +### Certificate Management + +**Internal CA: Step-CA (10.0.10.15)** +- Issues certificates for internal services +- Trusted by internal clients +- Good separation: Internal CA for private, LetsEncrypt for public + +**External: LetsEncrypt (via Caddy)** +- VPS Caddy handles public-facing certificates +- Automatic renewal +- No conflicts with internal CA + +**Recommendation:** ✅ Current setup is solid, no changes needed + +### VPN Access + +**WireGuard VPN** +- Network: 10.0.8.0/24 +- Provides secure remote access to homelab +- Used by VPS to route traffic internally +- Properly segregated from main network + +### Firewall & Access Control +- Status: Assumed configured on router-pve +- Needs: Documentation of firewall rules +- **TODO:** Audit firewall configuration + +--- + +## Paired Nodes + +### Fred's iMac (10.0.10.11) + +**Platform:** macOS with OpenClaw desktop app node +**Version:** OpenClaw 2026.2.1 (build 8650) +**Mode:** Remote +**Status:** Connected (last seen Feb 5, 2026 07:04 UTC, 12:10 UTC) + +**Capabilities:** +- Camera access (for snapshots/video) +- Screen recording +- Canvas display +- Notifications +- Potential audio output for TTS + +**Known Issues:** +- `nodes screen_record` fails with "Operation could not be completed" (OpenClaw 2026.2.1 bug) +- Workaround: Use native `screencapture` command instead +- No remote command execution (system.run not supported on desktop app) + +**Usage:** +- Near Fred but screen visibility limited +- Good for notifications and quick captures +- Cannot run CLI tools remotely + +--- + +## Backup System + +**Current Status:** ⚠️ **Partially documented** + +**Known Components:** +- n8n workflow: "Backup verification (daily @ 6 AM)" +- Likely backing up to OMV (10.0.10.5) `/data/backups/` +- Verification running automatically + +**Questions to Answer:** +1. What exactly is being backed up? + - Proxmox VMs/containers? + - OMV data shares? + - Specific service configs? + +2. Where are backups stored? + - OMV 10.0.10.5? + - Router-pve 4TB HDD? + - External drive? + +3. How are backups performed? + - Proxmox built-in backup (vzdump)? + - rsync scripts? + - n8n workflows? + - Proxmox Backup Server? + +4. Can backups be restored? + - Last restore test: Unknown + - Restore documentation: None found + +**Immediate Actions Needed:** +1. ✅ Document current backup system +2. ✅ Test restore procedure +3. ✅ Utilize router-pve 4TB HDD for backup target +4. ✅ Consider deploying Proxmox Backup Server (PBS) + +**Recommendation:** Deploy PBS on router-pve using 4TB HDD + +--- + +## Documentation System + +**Current State:** ⚠️ **Fragmented** + +**Existing Documentation:** +- INFRASTRUCTURE-AUDIT-REPORT.md (Claude Code, on OMV) +- TOOLS.md (OpenClaw workspace) +- Various files scattered across OMV shares +- No centralized version control + +**Planned Solution: Gitea** +- Self-hosted Git repository +- Will serve as infrastructure knowledge base +- Accessible to AI agents for queries +- Version-controlled documentation +- **Status:** To be deployed on router-pve (this session) + +**Obsidian Vault:** +- Git-backed Obsidian vault in progress +- Location: `/root/.openclaw/workspace/obsidian-vault/` +- Includes: Infrastructure docs, VA strategy, Kobe VA docs, business plans +- **Status:** Awaiting Fred to set up central git repo on OMV + +--- + +## Current Projects + +### 3D Printing Business +- Planning 3D print farm with Fred's son +- Need for CAD/slicing software +- Considering GPU passthrough for Windows VM on Proxmox +- Target: Bambu Lab A1 printer(s) +- **Files needed:** Printer profiles, production 3MF files + +### Truck Carputer/Android Head Unit +- Android-based head unit for truck +- Status: Research/planning phase +- Homelab integration potential + +### VA Disability Claims +- Fred's claim: Sleep apnea + hypersomnia secondary to PTSD +- Kobe's claim: VA dependent benefits for disabled child +- Documentation: Extensive templates and checklists created +- Storage: OMV 10.0.10.5 `/data/VA-Strategy/` + +--- + +## Changes Made During This Session (2026-02-05) + +### Apple Calendar/Reminders Integration ✅ +- **Time:** 02:48 UTC (8:48 PM CST Feb 4) +- **Action:** Configured CalDAV connection to iCloud +- **Result:** Created 10 calendar events for VA claims and 3D printing tasks +- **Access:** fredbook@gmail.com via app-specific password +- **Location:** Events created in "Lenard Farms" calendar +- **Status:** ✅ Working (Fred confirmed seeing events) + +### Obsidian Git Vault Created ✅ +- **Time:** 02:54 UTC +- **Action:** Created git-based Obsidian vault with all VA documents +- **Location:** `/root/.openclaw/workspace/obsidian-vault/` +- **Contents:** 18 files including VA strategy, Kobe VA docs, infrastructure docs +- **Status:** ⏳ Awaiting Fred to create central repo on OMV + +### SSH Key Generated ✅ +- **Time:** 15:13 UTC +- **Action:** Generated SSH key for accessing router-pve +- **Key:** ed25519, stored at `/root/.ssh/id_ed25519_router-pve` +- **Public Key:** Added to router-pve `/root/.ssh/authorized_keys` +- **Status:** ⚠️ Key installed but connection hangs (network routing issue) + +### Infrastructure Documentation Created ✅ +- **Files Created:** + - `homelab-2026-guide.md` - Comprehensive homelab + GPU passthrough guide (13.8KB) + - `INFRASTRUCTURE-AUDIT-COMPLETE-2026-02-05.md` - This document +- **Status:** Committed to Obsidian vault + +--- + +## Recommendations & Action Plan + +### Immediate (This Week) + +**1. Deploy Gitea on router-pve** 🎯 **In Progress** +- Install as Docker container or LXC +- Use as infrastructure documentation source of truth +- Store: Docker Compose files, scripts, infrastructure docs, 3D printing configs +- Make accessible to AI agents for queries +- **Priority:** HIGH + +**2. Document Backup System** +- What: Inventory what's being backed up +- Where: Confirm backup destinations +- How: Document backup procedures +- Test: Perform restore test +- **Priority:** HIGH + +**3. Utilize Router-PVE 4TB HDD** +- Check if formatted and mounted +- Configure as backup target +- Consider deploying Proxmox Backup Server (PBS) +- **Priority:** HIGH + +**4. Fix SSH Access to router-pve from OpenClaw** +- Troubleshoot network routing +- Check firewall rules on router-pve +- Alternative: Use `nodes run` via Fred's iMac as proxy +- **Priority:** MEDIUM (workarounds available) + +### Short Term (Next 2 Weeks) + +**5. Complete Obsidian Vault Setup** +- Fred creates git repo on OMV +- Sync vault to Windows +- Install Obsidian Git plugin +- Begin using for daily documentation +- **Priority:** HIGH + +**6. Vaultwarden Deployment** +- Already planned for 10.0.10.27 +- Week 1 priority +- Replace cloud Bitwarden with self-hosted +- **Priority:** HIGH (security + family access) + +**7. Log Aggregation (Loki + Grafana)** +- Centralize logs from all containers +- Easier troubleshooting +- Better visibility +- **Priority:** MEDIUM + +**8. Pi-hole DNS** +- Ad blocking at DNS level +- Already planned +- Can run on router-pve (light load) +- **Priority:** MEDIUM + +### Medium Term (Month 1-2) + +**9. GPU Passthrough for 3D Printing** +- Research GPU options (RTX 3060 recommended) +- Check which Proxmox host has free PCIe slot +- Set up Windows VM with GPU passthrough +- Install CAD software (Fusion 360, Bambu Studio, etc.) +- Configure remote access (RDP or Parsec) +- **Priority:** MEDIUM (business-critical when 3D farm launches) + +**10. Consolidate Storage** +- Evaluate whether to keep both OMV instances +- Consider TrueNAS SCALE for future storage +- Plan migration if needed +- **Priority:** LOW (current setup works) + +**11. Monitoring Improvements** +- Add Netdata for system monitoring +- Add Dozzle for real-time container logs +- Integrate with existing Uptime Kuma + n8n +- **Priority:** LOW (nice-to-have) + +### Long Term (3-6 Months) + +**12. HomelabHub.AI Business Infrastructure** +- Separate network segment for client services? +- Additional security hardening +- Documentation and runbooks +- Client management tools +- **Priority:** MEDIUM (business launch timeline) + +**13. Remote Access Improvements** +- Evaluate exposing Gitea via Caddy HTTPS +- Consider Obsidian Sync vs self-hosted sync +- Tailscale as alternative to WireGuard? +- **Priority:** LOW (current VPN works) + +--- + +## Network Diagram + +``` + Internet + | + [VPS - Caddy] + (66.63.182.168) + | + [WireGuard VPN] + | + +-----------------+------------------+ + | | + [Router-PVE] Internal Network + 10.0.10.2 10.0.10.0/24 + | | + [Home Assistant] +----------------+----------------+ + | | | + main-pve pve-storage HP DL380 + (10.0.10.3) (10.0.10.4) | + | | | + +---------+---------+ | [OpenClaw] + | | | | 10.0.10.28 + HA n8n Uptime OMV + 10.0.10.24 .22 Kuma 10.0.10.5 + .26 + +External Nodes: +- Fred's iMac (10.0.10.11) - OpenClaw desktop app +- Fred's iPhone - OpenClaw mobile (via VPN when remote) +- Fred's Windows PC - SSH access, Obsidian client +``` + +--- + +## Service Inventory + +### Core Infrastructure +| Service | IP | Port | Purpose | Status | +|---------|-----|------|---------|--------| +| Proxmox main-pve | 10.0.10.3 | 8006 | Primary virtualization | ✅ Running | +| Proxmox router-pve | 10.0.10.2 | 8006 | Router + virtualization | ✅ Running | +| Proxmox pve-storage | 10.0.10.4 | 8006 | Storage virtualization | ✅ Running | +| OMV Storage | 10.0.10.5 | 445 | SMB/CIFS file shares | ✅ Running | +| Step-CA | 10.0.10.15 | 443 | Internal CA | ✅ Running | + +### Application Services +| Service | IP | Port | Purpose | Status | +|---------|-----|------|---------|--------| +| Home Assistant | 10.0.10.24 | 8123 | Smart home | ✅ Running | +| n8n | 10.0.10.22 | 5678 | Workflow automation | ✅ Running | +| Uptime Kuma | 10.0.10.26 | 3001 | Monitoring | ✅ Running | +| OpenClaw | 10.0.10.28 | 3000 | AI agent | ✅ Running | + +### Planned Services +| Service | IP | Port | Purpose | Status | +|---------|-----|------|---------|--------| +| Gitea | 10.0.10.2 | TBD | Git repository | ⏳ Deploying | +| Vaultwarden | 10.0.10.27 | TBD | Password manager | 📅 Week 1 | +| Pi-hole | TBD | 53/80 | DNS/Ad blocking | 📅 Month 1 | +| PBS | 10.0.10.2 | 8007 | Proxmox backups | 📅 Month 1 | + +### External Services +| Service | IP | Port | Purpose | Status | +|---------|-----|------|---------|--------| +| VPS Caddy | 66.63.182.168 | 443 | Reverse proxy | ✅ Running | + +--- + +## Known Issues + +### Critical Issues +None currently + +### Important Issues + +**1. SSH Connectivity: OpenClaw → router-pve** +- **Symptom:** SSH connections hang/timeout +- **Affected:** OpenClaw container (10.0.10.28) cannot SSH to router-pve (10.0.10.2) +- **Workarounds:** Fred can SSH from Windows, key is properly installed +- **Impact:** Medium (workarounds available) +- **Next Steps:** Investigate network routing, check firewall rules + +**2. Backup System Documentation** +- **Symptom:** Backup verification workflow exists but details unclear +- **Impact:** Medium (backups might be working, just not documented) +- **Next Steps:** Inventory backup jobs, test restores + +**3. 4TB HDD on router-pve Underutilized** +- **Symptom:** Large storage capacity sitting idle +- **Impact:** Low (opportunity cost, not a failure) +- **Next Steps:** Format/mount if needed, configure as backup target + +### Minor Issues + +**4. Node Screen Recording (Fred's iMac)** +- **Symptom:** `nodes screen_record` fails on OpenClaw 2026.2.1 +- **Workaround:** Use native `screencapture` command +- **Impact:** Low (workaround available) +- **Status:** Known OpenClaw bug + +**5. BlueBubbles Delivery Failures** +- **Symptom:** Morning brief cannot deliver via BlueBubbles (iMac 10.0.10.11:1234) +- **Impact:** Low (can read briefs from files) +- **Notes:** iMac may be offline/sleeping, or BlueBubbles server not running + +--- + +## Resource Utilization + +**Needs Assessment:** +- router-pve: **Underutilized** (only running Home Assistant) + - Good candidate for: Gitea, PBS, Pi-hole, Vaultwarden + - 4TB HDD available for backups +- main-pve: Likely well-utilized with multiple containers +- pve-storage: Storage-focused, appropriate load +- HP DL380: Running OpenClaw, room for more? + +**Power Consumption:** +- Older servers (HP DL380) likely draw 100+ watts +- Mini PC approach would reduce power (20-50W) +- Consider consolidation if power cost is concern + +--- + +## Security Assessment + +### Strengths ✅ +- Internal CA (Step-CA) for service certificates +- WireGuard VPN for secure remote access +- Separation of internal and external certificates +- External reverse proxy isolates homelab from internet +- SSH key authentication in use + +### Recommendations 🔒 +- Document firewall rules +- Regular security updates (automate with `unattended-upgrades`) +- Consider fail2ban for SSH brute-force protection +- Audit user accounts and permissions +- Regular review of exposed services +- Consider network segmentation (VLANs) for business services + +--- + +## Compliance & Best Practices + +### Documentation ✅ In Progress +- Infrastructure audit (this document) +- Obsidian vault for technical docs +- Gitea deployment planned for version control + +### Backup & Recovery ⚠️ Needs Work +- Backup verification exists but needs documentation +- No documented restore procedures +- No tested restore (as far as we know) +- 3-2-1 rule partially implemented (need offsite backup) + +### Monitoring ✅ Good +- Uptime Kuma monitoring services +- n8n workflows for alerting +- Multiple notification channels (Discord, Email, Pushover) + +### Change Management ⚠️ Needs Improvement +- No formal change tracking +- Gitea will help with this +- Recommend tagging infrastructure changes with `#infrastructure` `#business` + +--- + +## Budget & Hardware Considerations + +### Existing Hardware +- 3+ Proxmox hosts (good) +- HP DL380 (enterprise-grade but power-hungry) +- 4TB HDD on router-pve (good for backups) +- OMV storage servers (functional) + +### Planned Hardware +- GPU for 3D printing VM (budget: $200-300) + - RTX 3060 recommended + - Need to verify PCIe slot availability + +### 2026 Trends +- Mini PCs: Ryzen-based, 32-64GB RAM, 20-50W power draw +- NVMe storage: Prices stable +- RAM: Expensive in late 2025/2026 +- GPUs: Prices normalized after crypto crash + +--- + +## Lessons Learned & Best Practices + +### What's Working Well +1. **Separation of concerns**: Internal CA + External LetsEncrypt +2. **VPN-first approach**: Secure remote access without exposing services +3. **Monitoring**: Uptime Kuma + n8n provides good visibility +4. **Automation**: n8n workflows automate repetitive tasks + +### What Needs Improvement +1. **Documentation**: Scattered, needs centralization (Gitea will help) +2. **Backup testing**: Backups exist but restore procedures untested +3. **Resource utilization**: router-pve and 4TB HDD underused +4. **Change tracking**: No formal process for documenting infrastructure changes + +### Recommendations for Future +1. **Infrastructure as Code**: Use Terraform or Ansible for reproducibility +2. **Gitea**: Single source of truth for infrastructure knowledge +3. **Regular Audits**: Quarterly infrastructure reviews +4. **Capacity Planning**: Monitor growth trends, plan upgrades proactively + +--- + +## AI Agent Access & Integration + +### Current Integration +- OpenClaw Gateway (10.0.10.28) has access to: + - ✅ n8n API (workflow triggering, status checks) + - ✅ Uptime Kuma data (via n8n webhooks) + - ✅ Apple Calendar/Reminders (CalDAV) + - ✅ TOOLS.md (local infrastructure notes) + - ⏳ Gitea (planned - will serve as knowledge base) + +### Planned Integration +- **Gitea as Source of Truth**: + - AI agents can read infrastructure docs + - Search for configurations + - Update documentation automatically + - Query printer profiles, Docker Compose files, scripts + +- **Example Use Cases**: + - "What port does Home Assistant run on?" → Query Gitea infrastructure/service-inventory.md + - "What's the Bambu A1 nozzle temp for PLA?" → Query 3d-print-farm/printer-profiles/bambu-a1.ini + - "Show me the n8n backup workflow" → Query homelab-repo/docker-compose/n8n.yml + +### Agent Capabilities +- **Read**: Configuration files, documentation, scripts +- **Search**: Git history, grep for specific settings +- **Update**: Commit documentation changes +- **Notify**: Alert Fred to infrastructure changes or issues + +--- + +## Appendix A: Quick Reference + +### Common Commands + +**SSH to Proxmox hosts:** +```bash +ssh root@10.0.10.2 # router-pve +ssh root@10.0.10.3 # main-pve +ssh root@10.0.10.4 # pve-storage +``` + +**Access Web UIs:** +``` +Proxmox: https://10.0.10.2:8006 +Home Assistant: http://10.0.10.24:8123 +n8n: http://10.0.10.22:5678 +Uptime Kuma: http://10.0.10.26:3001 +OpenClaw: http://10.0.10.28:3000 +``` + +**OMV Shares:** +```bash +# From Windows +\\10.0.10.5\data + +# From Linux +mount.cifs //10.0.10.5/data /mnt/omv -o guest,vers=3.0 +``` + +### Key Files + +**OpenClaw Workspace:** +- `/root/.openclaw/workspace/TOOLS.md` - Infrastructure notes +- `/root/.openclaw/workspace/obsidian-vault/` - Documentation vault +- `/root/.openclaw/workspace/.caldav-config.json` - Apple Calendar config + +**OMV Storage:** +- `\\10.0.10.5\data\VA-Strategy\` - Fred's VA claim documents +- `\\10.0.10.5\data\backups\` - Backup destination +- `\\10.0.10.5\data\INFRASTRUCTURE-AUDIT-REPORT.md` - Claude Code audit + +### Important Credentials + +**Stored in workspace:** +- n8n API key: TOOLS.md +- Apple CalDAV: `.caldav-config.json` +- SSH keys: `/root/.ssh/` + +**Not stored (need to retrieve):** +- Proxmox root passwords +- OMV admin password +- Step-CA admin credentials + +--- + +## Appendix B: Session Changelog + +**2026-02-04 (Previous Session):** +- Created VA strategy documents for Fred +- Created Kobe VA dependent benefits documents +- Set up morning brief cron job +- Infrastructure discussions + +**2026-02-05 (This Session):** +- 02:48 UTC: CalDAV integration with Apple Calendar ✅ +- 02:54 UTC: Obsidian vault created ✅ +- 14:23 UTC: Research on 2026 homelab best practices ✅ +- 14:23 UTC: Research on GPU passthrough for 3D printing ✅ +- 15:13 UTC: SSH key generated for router-pve ✅ +- 15:17 UTC: SSH key added to router-pve (confirmed by Fred) ✅ +- 15:23 UTC: SSH connection issue discovered (hanging) ⚠️ +- 15:24 UTC: This comprehensive audit completed ✅ + +--- + +## Next Steps (Prioritized) + +1. **Deploy Gitea on router-pve** (IN PROGRESS - this session) +2. Fred to copy Claude Code audit report for comparison +3. Document current backup system in detail +4. Configure router-pve 4TB HDD as backup target +5. Test backup restore procedure +6. Troubleshoot SSH connectivity issue (OpenClaw → router-pve) +7. Complete Obsidian vault setup (Fred's side) +8. Deploy Vaultwarden (Week 1) +9. Begin 3D printing business infrastructure planning +10. Research GPU options for CAD workstation + +--- + +**End of Audit Report** + +*This audit will be updated regularly as infrastructure changes are made.* + +*For questions or clarifications, contact:* +- **Funky (OpenClaw Agent)** - Available in OpenClaw chat +- **Fred Book** - Infrastructure owner diff --git a/docs/homelab-2026-guide.md b/docs/homelab-2026-guide.md new file mode 100644 index 0000000..e01aa49 --- /dev/null +++ b/docs/homelab-2026-guide.md @@ -0,0 +1,455 @@ +# Homelab 2026 Starter Stack + 3D Printing GPU Passthrough + +**Research compiled for Fred's homelab and 3D print farm business** + +*Source: VirtualizationHowTo.com + Reddit r/Proxmox community* + +--- + +## 🎯 Why This Matters for You + +**Your situation:** +- Already running Proxmox (10.0.10.3, 10.0.10.2, 10.0.10.4) +- Planning 3D print farm business with your son +- Need CAD/slicing software for 3D printing +- Want modern, efficient homelab stack + +**What you'll learn:** +1. **2026 best practices** for homelab hardware and software +2. **GPU passthrough** to run Windows VM with CAD software (Fusion 360, PrusaSlicer, etc.) +3. **How this fits your 3D print farm** business needs + +--- + +## Part 1: Ultimate Homelab Stack for 2026 + +### Hardware Recommendations + +**The Modern Mini PC Approach** (you already have Proxmox servers, but good to know for expansion): + +**Ideal Specs:** +- **CPU:** Ryzen 7 or Ryzen 9 (uniform, efficient) +- **RAM:** 32-64GB DDR5 (sweet spot despite high 2025/2026 prices) +- **Storage:** Two NVMe drives (mirrored or separate workloads) +- **Network:** 2.5Gb or 10Gb +- **Power draw:** 20-50 watts (vs. your older servers drawing 100+ watts idle) + +**Why mini PCs are trending:** +- Quiet, compact, efficient +- Enterprise-grade performance +- Great models: Minisforum MS-A2, MS-02, MS-01; Beelink SER9 Max + +**Your setup:** You already have Proxmox hosts, but this is good to know if you want to add a dedicated node for 3D printing/CAD work later. + +--- + +### Software Stack - The 2026 Essentials + +#### 1. **Proxmox VE 9.1** (Foundation) ✅ You already have this! + +**What's new in 9.1:** +- **OCI container image support** (NEW) - More efficient than traditional containers +- **vTPM support** for VMs +- **Better SDN (software-defined networking)** +- **Improved backup features** +- No license shenanigans +- Huge community, tons of scripts + +**Why it's still #1:** Best balance of power and simplicity for home labs + +--- + +#### 2. **Container Management: Komodo or Portainer** + +**Komodo** (New kid on the block - 2025/2026 favorite): +- **Free and fast** +- Modern UI +- Easy Docker deployment and monitoring +- Lighter weight than Portainer +- Perfect for your n8n + container stack + +**Portainer** (The 800lb gorilla): +- More features, more complex +- GitOps built-in +- Like "VMware vCenter for containers" +- You already know Docker/containers, so either works + +**Recommendation for you:** Try Komodo - it's simpler and you said n8n node definitions are problematic. Komodo might be easier. + +--- + +#### 3. **Nginx Proxy Manager** (Reverse Proxy) ✅ You should add this! + +**Why you need this:** +- Manages all your services behind one IP +- **Auto LetsEncrypt SSL certificates** (no more manual cert renewals!) +- GUI-based (way easier than editing Nginx configs) +- Perfect for exposing services safely + +**What it does:** +- HTTPS termination +- Automatic renewals +- Domain/subdomain routing (homeassistant.nianticbooks.com, n8n.nianticbooks.com, etc.) +- Access lists and authentication +- Organizes internal vs external access + +**Your use case:** +- Right now you probably access services by IP:port (10.0.10.24:8123, etc.) +- With NPM: nice URLs (homeassistant.local or via your Caddy VPS) +- Combined with your Caddy VPS = secure remote access to everything + +--- + +#### 4. **Gitea** (Self-hosted Git) - You need this! + +**Why:** +- Store your Docker Compose files in Git (you said you lose track of configs) +- Version control for infrastructure +- Backup your n8n workflows as code +- Store 3D printing business documentation + +**Lightweight and fast:** +- Runs as a container +- Looks like GitHub +- Supports issues, pull requests, branches +- **Gitea Actions** = CI/CD built-in (run automation on git push) + +**Your use case:** +- Store Obsidian vault in Gitea (private repo on your network) +- Document infrastructure changes +- Track 3D print farm business code (if you automate anything) + +--- + +#### 5. **Proxmox Backup Server (PBS)** ✅ Critical! + +**You need this running ASAP:** +- Free, from Proxmox team +- Deduplication, compression, incremental backups +- Fast restores +- Can run on same host (separate disk) or dedicated mini PC/NAS + +**Your setup idea:** +- Install PBS on one of your Proxmox nodes +- Point to OMV storage (10.0.10.5) for backup target +- Schedule automated backups of all VMs/containers +- **INCLUDES backing up your OpenClaw container!** + +**3-2-1-1-0 rule:** +- 3 copies of data +- 2 different media +- 1 offsite (your VPS? Backblaze B2?) +- 1 offline (USB drive, fireproof safe) +- **0 errors** after verification ← Most important! + +--- + +#### 6. **Core Containers to Run** + +**From the "15 containers that make home lab better" list, here's the essentials:** + +**Monitoring & Logging:** +- **Dozzle** - Real-time container log viewer (one screen, all logs) +- **Netdata** - System monitoring (CPU, RAM, disk, network) +- **Uptime Kuma** ✅ You already have this (10.0.10.26) + +**Management:** +- **Komodo** - Container stack management +- **Nginx Proxy Manager** - Reverse proxy with SSL +- **Gitea** - Git repository + +**Security & Services:** +- **Vaultwarden** ✅ You already planned this (10.0.10.27 Week 1) +- **Pi-hole** - DNS-level ad blocking (also planned) +- **Mailrise** - Unified notification bridge (emails become push notifications) + +**Automation:** +- **n8n** ✅ You already have this (10.0.10.22) + +--- + +## Part 2: GPU Passthrough for 3D Printing Lab + +### The Use Case (From Reddit) + +**What someone built:** +- Proxmox host +- Windows 10 VM with GPU passthrough +- GPU: NVIDIA card (prices dropped in late 2024/2025) +- Purpose: Run CAD software (Fusion 360, SolidWorks, etc.) and slicing software (PrusaSlicer, Cura, etc.) + +**Why GPU passthrough matters:** +- CAD software needs GPU acceleration +- 3D rendering and complex models +- Slicing large files with previews +- Remote access to Windows VM = access CAD from anywhere + +--- + +### How It Works + +**The Setup:** +1. **Proxmox host** with dedicated GPU (not the iGPU used for Proxmox console) +2. **Windows 10/11 VM** with GPU passed through +3. **RDP or remote desktop** to access VM +4. Install CAD software, slicers, 3D printing tools +5. Access from any device (your PC, iPhone, Mac) + +**The Result:** +- Full GPU acceleration for CAD +- Can run multiple 3D printing tools +- Centralized 3D printing workstation +- Your son can access the VM too (collaborative design work) + +--- + +### Requirements + +**Hardware:** +- Dedicated GPU (NVIDIA or AMD) + - Don't use iGPU (Proxmox needs it for console) + - Budget options: GTX 1060, 1660, RTX 3060 + - Pro options: RTX 4060, 4070 (better CAD performance) +- CPU with VT-d / AMD-Vi (virtualization extensions) ✅ Your Ryzen CPUs support this +- Motherboard with IOMMU support ✅ Your Proxmox hosts likely support this + +**Software:** +- Proxmox with IOMMU enabled in BIOS +- GPU drivers inside Windows VM +- Remote desktop software (built-in RDP or Parsec for better performance) + +--- + +### Configuration Steps (High-Level) + +**1. Enable IOMMU in BIOS:** +- Boot into BIOS on Proxmox host +- Enable VT-d (Intel) or AMD-Vi (AMD) +- Save and reboot + +**2. Enable IOMMU in Proxmox:** +Edit `/etc/default/grub`: +```bash +# For Intel +GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" + +# For AMD +GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt" +``` + +Update grub: `update-grub && reboot` + +**3. Load VFIO modules:** +Edit `/etc/modules`: +``` +vfio +vfio_iommu_type1 +vfio_pci +vfio_virqfd +``` + +**4. Blacklist GPU drivers on host:** +(So Proxmox doesn't try to use the GPU) +```bash +echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf +echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf +update-initramfs -u +``` + +**5. Create Windows VM in Proxmox:** +- Machine: q35 +- BIOS: OVMF (UEFI) +- Add EFI disk +- **Add PCI device** (your GPU) +- Set CPU type to "host" (important for passthrough) +- Enable "PCIe" checkbox on GPU device + +**6. Install Windows + GPU drivers:** +- Install Windows normally +- Install NVIDIA/AMD drivers inside Windows +- Verify GPU is recognized (Device Manager) + +**7. Remote Access:** +- Enable RDP in Windows +- Or install Parsec (better for CAD/gaming performance) +- Access VM from anywhere on your network + +--- + +### For Your 3D Print Farm Business + +**Use cases:** +1. **Centralized CAD workstation** - You and your son access same VM +2. **Slicing station** - Queue up print jobs, generate G-code +3. **Design library** - Store all STL files, designs in one place +4. **Remote access** - Work on designs from your bus route (when parked, obviously!) +5. **Backup everything** - VM backups = CAD software + settings + files all backed up together + +**Software you'd run:** +- **Fusion 360** (free for hobbyists/small business) +- **PrusaSlicer** or **Cura** (slicing) +- **Blender** (if doing custom modeling) +- **Bambu Studio** (for your Bambu A1) +- **OctoPrint / Mainsail** web UIs (manage printers remotely) + +**Workflow:** +1. Design in Fusion 360 (with GPU acceleration) +2. Export STL +3. Slice in PrusaSlicer/Bambu Studio +4. Send to printer (via OctoPrint or direct USB) +5. Monitor prints via webcam + OctoPrint + +--- + +### Budget GPU Options (2026 Prices) + +**Entry Level ($150-250 used):** +- GTX 1060 6GB - Good for basic CAD +- GTX 1660 Super - Better performance, still affordable + +**Mid-Range ($250-400):** +- RTX 3060 12GB - Excellent CAD performance, good value +- RTX 4060 - Newer, more efficient + +**Pro Level ($500+):** +- RTX 4070 - Great for complex CAD assemblies +- RTX 4080 - Overkill for most home use + +**Recommendation for you:** RTX 3060 or RTX 4060 - sweet spot for price/performance for CAD work. + +--- + +## How This Fits Your Current Setup + +### Your Proxmox Infrastructure + +**Current hosts:** +- main-pve (10.0.10.3) +- pve-router (10.0.10.2) +- pve-storage (10.0.10.4) + +**Option 1: Add GPU to existing host** +- Install GPU in main-pve (if there's a PCIe slot) +- Pass through to Windows VM +- Use for CAD/3D printing workstation + +**Option 2: Dedicated 3D printing node** +- Buy a mini PC with PCIe slot OR desktop with GPU +- Install Proxmox +- Cluster it with your existing nodes +- Dedicated to 3D print farm workloads + +**Option 3: Use iMac (10.0.10.11)** +- Your iMac already has GPU +- Install Windows via Boot Camp or Parallels +- Not ideal (macOS CAD apps are limited), but works short-term + +--- + +### Immediate Action Plan + +**This Week:** +1. ✅ **Install Nginx Proxy Manager** container + - Makes all services accessible via nice URLs + - Auto SSL certificates + - 30-minute setup + +2. ✅ **Install Gitea** container + - Start version-controlling your infrastructure + - Store Docker Compose files, n8n workflows, notes + - 15-minute setup + +3. ✅ **Set up Proxmox Backup Server** + - Install on one of your Proxmox nodes + - Point to OMV (10.0.10.5) for storage + - Schedule backups of all VMs/containers + - 1-hour setup + +**Next Week:** +4. **Research GPU options** + - Check if main-pve has free PCIe slot + - Look at used GPU market (Facebook Marketplace, eBay) + - Budget: $200-300 for RTX 3060 used + +5. **Test GPU passthrough** (once GPU acquired) + - Follow configuration steps above + - Create Windows 10 VM + - Install Fusion 360, PrusaSlicer, Bambu Studio + - Test remote access via RDP + +**Month 1:** +6. **Centralize 3D printing workstation** + - Move all CAD/slicing to Windows VM + - Set up file sharing (SMB) to OMV for STL library + - Configure remote access (VPN or Caddy reverse proxy) + +7. **Document workflow for your son** + - How to access VM + - How to use CAD software + - How to slice and send prints + - Collaborative design process + +--- + +## Key Takeaways + +### What You Should Implement Now + +**High Priority (This Week):** +1. **Nginx Proxy Manager** - Makes everything easier to access +2. **Gitea** - Version control for your infrastructure +3. **Proxmox Backup Server** - Protect your work (VA docs, business plans, everything!) + +**Medium Priority (Next Week):** +4. **Komodo** - Replace or augment Portainer, simpler UI +5. **Dozzle** - Real-time log viewer (helps with debugging n8n, containers) +6. **Pi-hole** - DNS ad blocking (already planned, but bump up priority) + +**Lower Priority (Month 1-2):** +7. **GPU passthrough setup** (once you buy GPU) +8. **Netdata** - System monitoring +9. **Mailrise** - Unified notifications + +--- + +### Why This Matters for Your Business + +**3D Print Farm Business:** +- **Centralized workstation** = you + your son collaborate on designs +- **GPU acceleration** = faster CAD, complex models, better workflow +- **Remote access** = work from anywhere (bus parking lot, home, vacation) +- **Professional setup** = looks good if you show clients your process + +**Homelab Improvements:** +- **Better organization** (Gitea for code, Nginx Proxy Manager for access) +- **Better backups** (PBS protects your VA docs, business plans, everything) +- **Better monitoring** (Uptime Kuma + Netdata + Dozzle) +- **Professional skills** = you learn modern DevOps tools (good for HomelabHub.AI business too!) + +--- + +## Resources + +**Setup Guides:** +- [Proxmox PCI Passthrough (Official Wiki)](https://pve.proxmox.com/wiki/PCI_Passthrough) +- [Nginx Proxy Manager Docker Setup](https://www.virtualizationhowto.com/2023/10/setting-up-nginx-proxy-manager-on-docker-with-easy-letsencrypt-ssl/) +- [Gitea Installation Guide](https://docs.gitea.io/en-us/install-with-docker/) +- [Proxmox Backup Server Documentation](https://pbs.proxmox.com/docs/) + +**Communities:** +- r/Proxmox on Reddit +- r/homelab on Reddit +- r/3Dprinting on Reddit +- Proxmox forums (forum.proxmox.com) + +**Your existing resources:** +- Your Proxmox infrastructure (already solid foundation) +- Your Caddy VPS (already handling reverse proxy externally) +- Your OMV storage (great for backup target) +- Your son's 3D printing interest (built-in business partner!) + +--- + +**Questions? Want me to help you install any of these? Just ask!** 🚀 + +*Saved to Obsidian vault: infrastructure/homelab-2026-guide.md* diff --git a/infrastructure/TOOLS.md b/infrastructure/TOOLS.md new file mode 100644 index 0000000..7a1aeae --- /dev/null +++ b/infrastructure/TOOLS.md @@ -0,0 +1,77 @@ +# TOOLS.md - Local Notes + +Skills define *how* tools work. This file is for *your* specifics — the stuff that's unique to your setup. + +## Fred's Homelab Infrastructure + +### Network +- Main Network: 10.0.10.0/24 +- VPN: WireGuard tunnel at 10.0.8.0/24 +- VPS: 66.63.182.168 (vps.nianticbooks.com) running Caddy reverse proxy + +### Proxmox Hosts +- Main Proxmox host: 10.0.10.3 (main-pve) +- HP DL380: Proxmox host running my container + +### Key Services (LXC Containers) +- Home Assistant: 10.0.10.24 - Smart home automation +- n8n: 10.0.10.22 (CT 106) - Workflow automation +- Uptime Kuma: 10.0.10.26 (CT 128) - Service monitoring +- OpenClaw Gateway: 10.0.10.28 (CT 130) - AI agent coordination (that's me!) + - Running as LXC container on HP DL380 + - No sound card/audio output on this container +- Step-CA: 10.0.10.15 (CT 115) - Internal certificate authority + +### Paired Nodes +- **Fred's iMac**: 10.0.10.11 - OpenClaw desktop app node + - Provides: Camera, screen recording, canvas display, notifications + - Potential audio output for TTS + - Near Fred but screen visibility limited + - **Known Issue:** `nodes screen_record` fails with "The operation could not be completed" (OpenClaw 2026.2.1 bug) + - **Workaround:** Use native `screencapture` command instead: + ```bash + # Record 3 seconds of screen + screencapture -T 3 -V 3 /tmp/screen-recording.mp4 + ``` + +### Current Projects +- Homelab infrastructure management +- 3D printing projects +- Android head unit/carputer for truck + +## Location +- Weather location: Niantic, IL 62551 + +## Tasks & Organization +- Wants to use Apple Reminders/Tasks on iPhone +- Need to explore integration options + +## n8n Workflow Automation + +**Instance:** http://10.0.10.22:5678 +**API Key:** `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZTVjZTQ2Zi1iNmUyLTQyMGEtYmUzMC1iYzQzYThlMDA1YjMiLCJpc3MiOiJuOG4iLCJhdWQiOiJwdWJsaWMtYXBpIiwiaWF0IjoxNzcwMDk3NDEwfQ.7NBimIPNlVH_Jif-3FU-9MSPfPUP1ILSznKGR1JzpE8` +**User:** OpenClaw (API access granted Feb 3, 2026) + +**Active Workflows:** +- Prometheus alerts → Discord/Email/Pushover +- Backup verification (daily @ 6 AM) +- SSL certificate expiration monitor +- Service health monitor (every 5 min via Uptime Kuma) +- Task overdue alerts (every 4 hours) +- Uptime Kuma webhook → alerts + +**API Usage:** +```bash +# List all workflows +curl -H "X-N8N-API-KEY: " http://10.0.10.22:5678/api/v1/workflows + +# Get specific workflow +curl -H "X-N8N-API-KEY: " http://10.0.10.22:5678/api/v1/workflows/{id} + +# Trigger workflow +curl -X POST -H "X-N8N-API-KEY: " http://10.0.10.22:5678/api/v1/workflows/{id}/execute +``` + +--- + +Add whatever helps you do your job. This is your cheat sheet.