theme: dracula paginate: true marp: true size: 4K

footer: Repo: github.com/mr-karan/hydra

Self Hosting with Nomad

My experiences of running and managing self hosted applications using Nomad.

Karan Sharma mrkaran.dev

whoami

👨‍💻 Works at Zerodha

📓 Blogs about things I find interesting

📈 Interested in Self Hosting

Why (I) Self Host

• Break from the Big Tech Co

Why (I) Self Host

• Break from the Big Tech Co
• Own your data

Why (I) Self Host

• Break from the Big Tech Co
• Own your data
• No lock ins for data which is critical

Why (I) Self Host

• Break from the Big Tech Co
• Own your data
• No lock ins for data which is critical
• Chance to contribute to OSS

Why (I) Self Host

• Break from the Big Tech Co
• Own your data
• No lock ins for data which is critical
• Chance to contribute to OSS
• Experiment and learn

My Setup

Servers

• DigitalOcean Droplet (2vCPU, 4GB RAM, blr1 Region)

Infra Tools

• Ansible
• Terraform
• Nomad

Ansible

• Boostrap the server
  • Harden SSH. User, Shell setups.
  • Install node-exporter, docker, tailscale.

Terraform

• DigitalOcean infra

  • Droplet
  • Firewalls
  • SSH Keys, Volumes, Floating IPs etc.

• Cloudflare DNS records

Nomad

• Simple workload orchestrator and scheduler
• Run workloads with multiple task drivers (not just docker containers)

Why Nomad

• Was using K8s before - went down the deep complexity hell
• Deployed my first app in a few minutes
• Single binary exectubale - with a UI

Nomad Agent

• Server takes the scheduling decisions

  • For HA, run 3/5/7 nodes. (Raft consensus)

• Client runs the actual task given by the server

  • Interacts with task plugins like docker etc

Running Nomad

• Grab the binary
• nomad agent -dev -> starts in dev mode. Great for local testing
• Configure server.hcl/client.hcl

Jobspec

• A deployment file is called "Jobspec". Think of docker-compose.yml
• Specify all possible things to run that app in one file
  • Job -> Group -> Task
  • Artifact (S3/GitHub Releases/Remote config files)
  • Networking options
  • Volume mounts

Deploying Gitea

job "gitea" {
  datacenters = ["hydra"]
  type        = "service"
  group "app" {
    count = 1
    network {
      port "http" {
        to = 3000
      }
      port "ssh" {
        to           = 22
        static       = 4222
        host_network = "tailscale"
      }
    }

Deploying Gitea

    task "web" {
      driver = "docker"
      config {
        image = "gitea/gitea:latest"
        ports = ["http", "ssh"]
        mount {
          type   = "bind"
          source = "local/gitea.ini"
          target = "/data/gitea/conf/app.ini"
        }
      }
      resources {
        cpu    = 200
        memory = 300
      }
    }

Deploying Gitea

      service {
        provider = "nomad"
        name     = "gitea-web"
        tags     = ["gitea", "web"]
        port     = "http"
      }
      service {
        provider = "nomad"
        name     = "gitea-ssh"
        tags     = ["gitea", "ssh"]
        port     = "ssh"
      }

Exploring the UI

Networking

• Tailscale for "mesh network"

  • Based on Wireguard VPN.
  • Authenticated sessions only.
  • Expose services on an interbal network and access them on all devices

• Tailscale uses 100.64.0.0/10 subnet from the Carrier Grade NAT (CGNAT) space.

Networking

• Caddy as a proxy for all services.

  • Running 2 instances of Caddy.

    • Private: Listens on Tailscale Interface.
    • Public: Listens on DO's public IPv4 Interface.

  • Automatic SSL with ACME DNS challenge

    • Built my own image: https://github.com/mr-karan/caddy-plugins-docker

Networking

Dont expose to the world

doggo adguard.mrkaran.dev     
NAME                	TYPE	CLASS	TTL	ADDRESS       	NAMESERVER   
adguard.mrkaran.dev.	A   	IN   	23s	100.111.91.100	127.0.0.1:53

unless required

doggo doggo.mrkaran.dev
NAME              	TYPE	CLASS	TTL	ADDRESS       	NAMESERVER   
doggo.mrkaran.dev.	A   	IN   	25s	172.67.187.239	127.0.0.1:53	
doggo.mrkaran.dev.	A   	IN   	25s	104.21.7.168  	127.0.0.1:53	

Storage

• Enable snapshots for volumes provided by cloud provider.
• Use separate DB instances for different applications.

Backups

• Restic
  • Periodic Job in Nomad.
  • Single vault with everything inside /data.
  • All applications mount inside /data folder.
  • Upload to Backblaze B2.

Extensibility

• Plugs into other Hashicorp tools very well
  • Native integration with Consul Connect for ACLs
  • Fetch secrets from Vault
  • Deploy using Waypoint

Services I run

• Adguard (DNS)
• Gitea (Git)
• Joplin Sync Server (Notes app)
• Miniflux (RSS reader)
• Plausible (Website Analytics)
• Grafana/Prometheus (Monitoring)
• Nextcloud (Documents/Photos)
• doggo.mrkaran.dev (DNS resolver)

Monitoring

• Grafana
• Prometheus
• Telegraf to collect home ISP stats
  • Ping Input plugin
  • DNS Input Plugin

Security

• If it should not be public facing, don't expose to WWW.

  • Prefer to use a VPN or mesh network instead of IP whitelists.
  • Tighter Firewall rules otherwise.

• Adguard, Gitea, etc Admin interfaces must always be protected with strong passwords.

• Periodic updates to App and OS.

Takeaways

• Don't get overwhelmed by choices. Pick something really simple (like Adguard/Pi-hole) and host it.
• Aim for simplicity

Resources

• Mon School - Self Hosting 101 Course

• r/selfhosted

  • Beginner friendly wiki: https://wiki.r-selfhosted.com/

• github.com/awesome-selfhosted

• learn.hashicorp.com/nomad

Thank You