HaloHabitat Infrastructure Repository

Project Stage: Initial Bootstrap & OS Configuration Target: Debian 13 servers managed with Ansible Deployment Method: Docker via SSH Remote Context

Overview

This repository contains the Infrastructure-as-Code for HaloHabitat using Ansible. It provides automated, reproducible server configuration starting from a fresh Debian 13 instance.

What This Repository Does

1. Bootstraps fresh Debian 13 servers - Ensures Python and prerequisites are available
2. Configures the operating system - System packages, security updates, SSH hardening
3. Sets up Docker - Installs Docker Engine, Compose Plugin, and containerd from official sources
4. Creates a deploy user - Enables safe, non-root Docker operations via SSH
5. Automates security updates - Configures unattended-upgrades with scheduled reboots
6. Enables remote Docker - Prepares the server for Docker CLI access via SSH

The repository does NOT yet include:

• Application services (Traefik, Forgejo, Logto, PostgreSQL, MongoDB, etc.)
• TLS/HTTPS or Let's Encrypt certificates
• DNS configuration
• Hetzner Firewall rules or Private Networks
• Backup systems
• Monitoring/alerting infrastructure
• Terraform/OpenTofu for infrastructure provisioning
• CI/CD pipelines

Application deployment will be added in future phases.

Architecture

Control Flow

┌─────────────────────────────────────────────────────────────────┐
│ Your Local Machine (Control Node)                               │
│  - Ansible installed                                            │
│  - SSH private key                                              │
│  - Playbook files                                               │
└────────────────────────┬────────────────────────────────────────┘
                         │
                    SSH connection
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│ Debian 13 Server (Target Node)                                  │
│  - No Ansible installed (unnecessary)                           │
│  - Only Python 3 required (auto-installed if missing)           │
│  - SSH server running                                           │
│                                                                  │
│  After Ansible runs:                                            │
│  ├─ SSH with key-based auth only (password disabled)            │
│  ├─ docker user with docker group membership                    │
│  ├─ Docker Engine + Compose Plugin                              │
│  ├─ Automatic security updates (reboot at 03:00 if needed)      │
│  └─ Ready for Docker deployments via SSH                        │
└────────────────────────────────────────────────────────────────┘

Deployment Architecture

┌─────────────────────────────────────────────────────────────┐
│ Local Machine                                               │
│ ├─ Docker CLI                                               │
│ ├─ Compose files (in this repo)                             │
│ └─ SSH key                                                  │
└────────────────────┬────────────────────────────────────────┘
                     │
            SSH Remote Context
        (docker context create ... --docker ssh://)
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│ Server Docker Daemon                                        │
│ ├─ Listens on Unix socket only (no TCP ports exposed)       │
│ ├─ Access via SSH only (deploy user)                        │
│ ├─ Runs application containers                              │
│ └─ Managed by Docker Compose from local CLI                 │
└─────────────────────────────────────────────────────────────┘

Why Ansible Runs Locally (Not on Server)

• Simpler - No agent to manage on the target
• Safer - No persistent Ansible service that could be compromised
• Flexible - Easy to manage multiple servers from a central location
• Standard - This is Ansible's agentless design philosophy

Prerequisites

On Your Local Machine

• Ansible 2.9+ (2.10+ recommended)

  ansible --version
  

• SSH client (standard on macOS/Linux, built into Windows 10+)

  ssh -V
  

• SSH private key with access to your server as root

  • The key is typically ~/.ssh/id_rsa or similar
  • Test access: ssh -i /path/to/key root@SERVER_IP

• Git (for cloning/managing this repository)

  git --version
  

On Your Server

• Debian 13 (fresh installation)
• SSH access as root (available by default on Hetzner)
• Python 3 (auto-installed by bootstrap if missing)

What You DON'T Need on the Server

• ❌ Ansible (not required, runs locally)
• ❌ Docker (we'll install it)
• ❌ Special SSH ports or firewall rules (uses standard port 22)
• ❌ Pre-configured deploy user (we'll create it)

Quick Start

1. Clone This Repository

git clone https://github.com/happy-ops-org/infrastructure.git
cd infrastructure

2. Update the Inventory

Edit ansible/inventory/primary.yml and set your server's IP:

all:
  children:
    primary:
      hosts:
        halo-primary:
          ansible_host: 192.0.2.1 # ← Change this to your actual IP
          ansible_user: root
          ansible_port: 22

3. Verify SSH Access

ssh -i ~/.ssh/your_key root@192.0.2.1

(Should connect without errors; exit with exit)

4. Run the Bootstrap

cd ansible

# Syntax check (optional but recommended)
ansible-playbook -i inventory/primary.yml primary.yml --syntax-check

# Run the full playbook
ansible-playbook -i inventory/primary.yml primary.yml -k

# Note: -k prompts for SSH password if you use password auth
# Omit -k if using SSH key authentication

5. Verify Deployment

# Test SSH as deploy user
ssh deploy@192.0.2.1

# On the server, test Docker
docker version
docker compose version
docker run --rm hello-world

# Exit
exit

6. Set Up Docker Remote Context (Local)

# Create a local Docker context for remote deployment
docker context create halohabitat-prod \
  --docker "host=ssh://deploy@192.0.2.1"

# Test it
docker --context halohabitat-prod info

See docs/docker-remote.md for detailed Docker remote setup.

Repository Structure

infrastructure/
├── README.md                          # This file
│
├── ansible/
│   ├── ansible.cfg                   # Ansible configuration
│   ├── primary.yml                   # Main orchestration playbook
│   ├── bootstrap.yml                 # Python/Ansible prerequisites
│   ├── base.yml                      # Base system configuration
│   ├── docker.yml                    # Docker installation
│   │
│   ├── inventory/
│   │   ├── primary.yml                # Primary server
│   │   └── replica.yml                # Replica server (not yet configured)
│   │
│   ├── group_vars/
│   │   └── primary.yml                # Primary group variables
│   │
│   ├── host_vars/
│   │   └── primary.yml               # Primary host-specific variables
│   │
│   └── roles/
│       ├── bootstrap/                # Ensure Python 3 is installed
│       ├── base/                     # System packages, updates
│       ├── ssh/                      # SSH hardening
│       ├── deploy-user/              # Create deploy user
│       ├── unattended-upgrades/      # Automatic security updates
│       └── docker/                   # Docker Engine and Compose
│
├── compose/
│   ├── primary/
│   │   └── README.md                 # Primary server deployment guide
│   └── replica/
│       └── README.md                 # Replica server (future)
│
├── docs/
│   ├── bootstrap.md                  # Detailed bootstrap procedure
│   ├── docker-remote.md              # Docker SSH context setup
│   └── operations.md                 # Day-to-day operations guide

Installation Process (Detailed)

Step 1: Install Ansible Locally

# macOS (Homebrew)
brew install ansible

# Ubuntu/Debian
sudo apt-get install ansible

# Python pip (universal)
pip install ansible

# Verify
ansible --version
# Should show 2.9 or later

Step 2: Prepare Your SSH Key

# Generate a new key (if you don't have one)
ssh-keygen -t ed25519 -C "halohabitat" -f ~/.ssh/halohabitat_key

# Add it to your SSH agent
ssh-add ~/.ssh/halohabitat_key

# Verify it works
ssh -i ~/.ssh/halohabitat_key root@SERVER_IP "echo Success"

Step 3: Update Inventory

In ansible/inventory/primary.yml:

all:
  children:
    primary:
      hosts:
        halo-primary:
          ansible_host: YOUR_SERVER_IP_HERE
          ansible_user: root
          ansible_port: 22

Step 4: Run Playbook

cd ansible

# First time: syntax check
ansible-playbook -i inventory/primary.yml primary.yml --syntax-check

# Run the playbook (this takes 2-5 minutes)
ansible-playbook -i inventory/primary.yml primary.yml

The playbook will:

1. Check for Python 3 (install if missing)
2. Update system packages
3. Configure SSH (disable password auth)
4. Create deploy user
5. Install Docker from official repositories
6. Configure automatic security updates
7. Set up the system for remote Docker access

Step 5: Verify Everything

# SSH as deploy user (should work with your key)
ssh deploy@SERVER_IP

# Once connected, test Docker:
docker ps
docker compose version
docker run --rm hello-world

# Exit
exit

# Test password auth is disabled (should fail)
ssh -o PubkeyAuthentication=no deploy@SERVER_IP

Key Features

✅ Bootstrap Without Prerequisites

The bootstrap playbook handles servers with no Python installed:

# First run: checks for Python 3
# If missing: automatically installs python3 and python3-apt
# Then: switches to normal Ansible modules

✅ Idempotent Configuration

Run the playbook multiple times—second run shows no changes:

# First run
ansible-playbook -i inventory/prod.yml playbooks/primary.yml
# changed=25, ok=10

# Second run
ansible-playbook -i inventory/prod.yml playbooks/primary.yml
# changed=0, ok=35  ← Idempotent!

✅ Safe SSH Hardening

• ✅ Disables password authentication
• ✅ Keeps key-based auth working
• ✅ Root login via key still allowed (for bootstrap + emergency)
• ✅ SSH config validation before restart
• ❌ No aggressive changes that risk lockout

✅ Docker from Official Sources

• Uses Docker's official Debian repository
• Installs: Engine, CLI, containerd, buildx, Compose
• No Docker TCP ports exposed (Unix socket only)
• Ready for remote access via SSH

✅ Automatic Security Updates

• Enabled via unattended-upgrades
• Updates install automatically, quietly
• Reboots at 03:00 if needed
• No manual intervention required

✅ Deploy User Management

root (Ansible access, emergency admin)
  ↓
deploy (Day-to-day operations)
  ├─ Docker group membership (no sudo for docker commands)
  ├─ Full sudo access (when needed)
  ├─ SSH key authentication only
  └─ Cannot use password login

Configuration Details

SSH Configuration

Location: /etc/ssh/sshd_config Key Settings:

PasswordAuthentication no          # Disable password login
PubkeyAuthentication yes           # Keep key auth enabled
PermitRootLogin prohibit-password  # Root login with key only

Validation: sshd -t before restart (prevents lockout)

Docker Configuration

Daemon Config: /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "userland-proxy": false
}

Access: Unix socket only (/var/run/docker.sock) Remote Access: SSH context (ssh://deploy@SERVER_IP)

Automatic Updates

Config Files:

• /etc/apt/apt.conf.d/50unattended-upgrades
• /etc/apt/apt.conf.d/02periodic

Schedule:

• Daily update check at 6 AM
• Security updates: automatic
• Standard updates: automatic with minimal steps
• Reboot: 03:00 UTC (if needed)

Logs:

# View update history
ssh deploy@SERVER_IP sudo journalctl -u unattended-upgrades

# Check for pending reboots
ssh deploy@SERVER_IP sudo needrestart -r l

Idempotency Testing

Idempotency is critical: the playbook should be safe to run repeatedly.

Test Procedure

# First run (changes system)
ansible-playbook -i inventory/prod.yml playbooks/primary.yml

# Wait for completion, check output for any failures

# Second run (should make no changes)
ansible-playbook -i inventory/prod.yml playbooks/primary.yml

# Expected output:
# changed=0    unreachable=0    failed=0

What This Means

• changed=0 → System already in desired state
• ok=35 → 35 checks passed, all conditions met
• No unnecessary restarts → Services only restart on actual config changes
• Safe to automate → Can run from cron or CI/CD without side effects

Deployment Workflow

Once bootstrap is complete:

1. Local Workflow

# Create/update docker-compose.yml locally
cat > docker-compose.yml << 'EOF'
services:
  app:
    image: myapp:latest
    ports:
      - "8080:8080"
EOF

# Deploy via Docker context
docker context use halohabitat-prod
docker compose up -d

# Verify
docker compose ps
docker compose logs -f

2. CI/CD Workflow (Future)

# In your CI/CD pipeline
deploy:
  script:
    - docker context create prod --docker "host=ssh://deploy@$SERVER_IP"
    - docker context use prod
    - docker compose up -d

Roadmap & Future Phases

Phase 1 (Current): OS & Docker Bootstrap ✅

• Fresh Debian 13 bootstrap
• SSH hardening (key-only auth)
• Deploy user creation
• Docker installation
• Automatic updates
• Remote Docker context ready

Phase 2 (Next): Infrastructure Services

• Traefik (reverse proxy, TLS)
• Forgejo (Git hosting)
• PostgreSQL (database)
• Automated SSL/TLS certificates

Phase 3: Application Services

• HaloHabitat Backend
• HaloHabitat Frontend
• Logto (authentication)
• Odoo Connector

Phase 4: Advanced Infrastructure

• Monitoring/Alerting (Prometheus, Grafana)
• Log Aggregation (ELK Stack, Loki)
• Backup systems
• Secrets management (1Password integration)
• Load balancing / HA setup
• Replica server configuration

Phase 5: Operations

• CI/CD pipeline integration
• Automated deployments
• Health checks and auto-recovery
• Documentation automation

Secrets Management

Current Status

No secrets are managed by Ansible in this phase.

Future (Phase 2+)

Secrets will be sourced from 1Password using:

• 1Password CLI for local operations
• Ansible 1Password lookup plugin
• Environment variable injection during deployment

Best Practices (Starting Now)

# NEVER commit:
❌ AWS keys
❌ Database passwords
❌ API tokens
❌ Private keys

# ALWAYS:
✅ Use .gitignore for sensitive files
✅ Store secrets in 1Password
✅ Pass secrets via environment variables
✅ Document where secrets come from

Networking & Firewall

Current Configuration

• SSH: Port 22 (open)
• Docker TCP: Not exposed (no 2375/2376)
• Firewall: External (Hetzner), no host-based FW configured

Future (Not in Scope Yet)

• Private network setup (Hetzner vNIC)
• Host firewall rules (ufw/nftables)
• Load balancer configuration
• DNS entries

Scaling to Multiple Servers

Replica Server (Planned)

The repository structure already supports future replicas:

# Future: ansible/inventory/primary.yml
all:
  children:
    primary:
      hosts:
        halo-primary:
          ansible_host: PROD_IP
    replica:
      hosts:
        halo-replica:
          ansible_host: REPLICA_IP

Provisioning a Replica

# Same bootstrap process, just update inventory
# and switch context:

docker context create halohabitat-replica \
  --docker "host=ssh://deploy@REPLICA_IP"

Troubleshooting

"Python is not installed on this host"

The bootstrap step should handle this automatically. If not:

ssh root@SERVER_IP
apt-get update && apt-get install -y python3 python3-apt
exit

# Then re-run playbook
ansible-playbook -i inventory/prod.yml playbooks/primary.yml

"Permission denied (publickey)"

• Verify SSH key is correct: ssh -i ~/.ssh/your_key root@SERVER_IP
• Check key permissions: ls -la ~/.ssh/your_key (should be 600)
• Test with verbose output: ssh -vvv -i ~/.ssh/your_key root@SERVER_IP

"SSH port 22: Connection refused"

• Server not running or not reachable
• Check Hetzner console
• Verify correct IP address
• Test from different network (some ISPs block port 22)

Playbook hangs on a task

• Add -vvv for verbose output: ansible-playbook ... -vvv
• Check server load: ssh root@SERVER_IP uptime
• Look for password prompts (may need -k flag)

Docker context can't connect

# Test Docker access
docker --context halohabitat-prod ps

# Debug SSH connection
ssh -vvv deploy@SERVER_IP docker ps

# Verify deploy user is in docker group
ssh deploy@SERVER_IP groups
# Should include: docker

See docs/operations.md for more detailed troubleshooting.

Documentation

• Bootstrap Guide - Detailed setup procedure
• Docker Remote Setup - SSH context configuration
• Operations Guide - Day-to-day server management

Contributing & Maintenance

Testing Changes

# Syntax check
ansible-playbook -i inventory/primary.yml primary.yml --syntax-check

# Lint (if ansible-lint is installed)
ansible-lint ansible/

# Run against a replica server first
ansible-playbook -i inventory/replica.yml primary.yml

Idempotency Check

Always verify idempotency before committing:

# Run twice on a test server
ansible-playbook -i inventory/primary.yml primary.yml
ansible-playbook -i inventory/primary.yml primary.yml

# Second run should show changed=0

Adding New Roles

1. Create role directory: mkdir ansible/roles/my-role/{tasks,handlers,templates}
2. Create tasks/main.yml with role tasks
3. Import in appropriate playbook
4. Test idempotency
5. Document in README

License & Attribution

This infrastructure is part of the HaloHabitat project.

Support & Questions

For issues or questions:

1. Check docs/operations.md for troubleshooting
2. Review docs/bootstrap.md for detailed procedures
3. Consult Ansible documentation: https://docs.ansible.com/
4. Review Docker documentation: https://docs.docker.com/

FAQ

Q: Why don't we install Ansible on the server? A: Ansible's agentless design is simpler, safer, and more flexible. We manage the server remotely from our local machine.

Q: Can we change the SSH port? A: Currently no, to avoid accidental lockout during early development. This can be added in a future security hardening phase.

Q: What if the server doesn't have Python? A: The bootstrap playbook handles this automatically using Ansible's raw module.

Q: Can we use password SSH authentication? A: Not after bootstrap—it's disabled for security. Use SSH keys only.

Q: Can we deploy applications now? A: Not yet. This bootstrap only prepares the OS and Docker. Application deployment will be added in Phase 2.

Q: How do we manage secrets? A: Currently none are needed. In the future, secrets will come from 1Password.

Q: Can we run Ansible from the server itself? A: Possible but unnecessary. Remote execution from your local machine is the recommended approach.

---

Last Updated: 2024-09-13 Target OS: Debian 13 Ansible Version: 2.9+ Status: Initial Bootstrap Phase ✅