Introduction
I have used many other peoples work to put this together, a key reference being the following: https://omarghader.github.io/haproxy-letsencrypt-docker-certbot-certificates/
On my home server I have several applications I want to access outside of the home for example; home-assistant and wordpress.
In order to expose these applications to the internet I had forwarded the ports of these on my router. On my server I have a dedicated VM for each application, the diagram below shows the setup.
As-Is Lab State
Existing Home-Assistant Setup
- Using https, so the traffic was encrypted and secure
- Using the default port for home-assistant which is 8123, which is annoying as you have to put the port number in the url to access it.
- LetsEncrypt plugin on home-assistant to manage the certificates needed for https
- LetsEncrypt needs to check the webserver when managing certificates so port 80 was also forwarded to home-assistant.
Existing WordPress Setup
- Using http, so the traffic was not encrypted, bad news for keeping usernames and passwords and traffic secure from hackers
- Using the default port 8080, same problem with home-assistant for me that you have to put the port number on to reach it.
Challenges with my implementation
- Due to the way letsencrypt works it makes a request to the webserver to confirm its there as part of generating and renewing certificates. Letsencrypt makes this request by default over port 80 .With my setup if i wanted to also have a certificate and https for wordpress, I would need to manually switch the port forwarding on my router between home-assistant or wordpress to get the certificates to be issued and updated on both.
- LetsEncrypt on home-assistant does not automatically renew certificates, you have to start the plugin once at least every 90 days and then restart home-assistant to update the certificate, I could have used an automation in home-assistant to manage this, however I wanted something a bit better.
- No secure access for wordpress using https and wanted to secure it.
To-Be Lab state
By introducing a reverse proxy I’m able to put my home-assistant and wordpress behind a single port number and depending on the web URL divert traffic to the correct VM.
I can also then manage the certificates for home-assistant and wordpress on the reverse proxy rather than each application needing to have its own set of certificates.
The diagram below shows roughly what I was aiming for:
Target State Setup
Proxmox is the hypervisor that I use to host the VMs – I’m not going to go into any detail on proxmox here in this post…
As part of the improvements here and introducing HAProxy, I wanted to move towards using containers and Proxmox doesn’t have an out of the box way to support Docker images. For Docker I needed a virtual machine. Once I have Docker on the VM I can then run many different containers on the one VM.
Docker also allows me to easily test different services in containers without needing to provision VMs all the time, reducing the overhead and resources on my home server.
So my setup was going to be like this:
- Proxmox Server
- Ubuntu VM
- To Store
- Config files and scripts for management
- Running Docker
- HAProxy Docker Container (Reserse proxy)
- Nginx Docker Container (To handle letsencrypt certificate renewals)
- To Store
- Ubuntu VM
Linux VM
First I setup a VM, I used Ubuntu. Setting up a Ubuntu VM on Proxmox is not part of my post. https://www.snel.com/support/ubuntu-vm-in-proxmox-and-networking-setup/ – This link may help you, or otherwise google it!
Install Docker
Second I installed Docker on the VM. Installing docker is also not part of my post, but the docker documentation here might be useful: https://docs.docker.com/engine/install/ubuntu/
Configuration File Structure
This is a key step making sure that there is a place to define and configure everything needed.
I created a folder in /home/<user> called dockerconfig, replace user with the name of the user you are logged in as, for me it was robert, so ill put this in the examples:
mkdir /home/robert/dockerconfigThe structure of the dockerconfig folder was as follows:
dockerconfig
|--certs
|--haproxy
|--letsencrypt
|--webroot
|--dockercompose.yaml
|--create-cert.sh
|--renew-certs.shUse the mkdir command to create the folders; certs, haproxy, letsencrypt, webroot
- certs: Where the https certificates will be stored
- haproxy: Where the config for haproxy is stored
- letsencrypt: Where the letsencrypt certificate files are stored and processed
- webroot: Where challenge files from letsencrypt get stored for nginx
HAProxy Configuration
Next we need to define the HAProxy configuration so that haproxy knows how to route the traffic to the wordpress and home-assistant applications.
Create a new config file:
nano /home/robert/dockerconfig/haproxy/haproxy.cfgUse the following content, making sure to change the domain to your actual domain (blog.domain, ha.domain):
global
log stdout format raw local0
daemon
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-
resolvers docker_resolver
nameserver dns 127.0.0.11:53
defaults
log global
mode http
option httplog
option dontlognull
frontend http
bind *:80
mode http
# if this is an ACME request to proof the domain ownder, then redirect to nginx-certbot server
acl is_well_known path_beg -i /.well-known/
# else redirect the traffic to https
redirect scheme https code 301 if !is_well_known !{ ssl_fc }
use_backend letsencrypt if is_well_known
frontend stats
bind *:8404
stats enable
stats uri /
stats refresh 10s
backend letsencrypt
server letsencrypt nginx-certbot:80 resolvers docker_resolver check init-addr none
frontend https
bind *:443 ssl crt /usr/local/etc/certs/
http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"
default_backend mybackend
acl ACL_blog hdr(host) -i blog.domain
use_backend blog if ACL_blog
# ACL for "ha"
acl ACL_ha hdr(host) -i ha.domain
use_backend home-assistant if ACL_ha
backend home-assistant
server ha 10.0.0.212:8123 ssl verify none
backend blog
server blog 10.0.0.213:80
To save the file: Press ctrl + x
Temporary Certificate
This is important to get things going
Making sure you are in the ‘dockerconfig’ directory:
sudo apt-get install openssl
openssl req -nodes -x509 -newkey rsa:2048 -keyout test.key -out test.crt -days 30
cat test.key test.crt > ./certs/test.pemDefine Docker Compose and Start HAProxy
Next I needed to create the docker compose file to define the containers to provision and run in docker.
nano /home/robert/dockerconfig/dockercompose.yamlOnce the editor is up this is my file example:
services:
proxy:
image: haproxy:latest
restart: always
volumes:
- ./haproxy:/usr/local/etc/haproxy:ro
- ./certs:/usr/local/etc/certs:ro
ports:
- 80:80
- 443:443
- 8404:8404
nginx-certbot:
image: nginx
restart: always
container_name: nginx-certbot
volumes:
- ./webroot:/usr/share/nginx/htmlTo save the file: Press ctrl + x
Start the docker containers making sure you are in the /home/robert/dockerconfig directory:
docker-compose up -dCreate certificate using LetsEncrypt Certbot
The scripts I use are mostly a lift from: https://omarghader.github.io/haproxy-letsencrypt-docker-certbot-certificates/
I have made a few subtle modifications to suit my needs.
There is a single certificate generated which contains alternative sub domains (blog and ha). I’ve hard coded these sub domains in the script because it was simplest for my needs. The result of the single cert is any of the sub domains defined in the certificate will also be covered by it, this reduces the need to manage separate certificates.
This script will generate the certificate from LetsEncrypt
Create the script using nano
nano /home/robert/dockerconfig/create-certs.shContents of the script:
#!/bin/bash
set -e
echo "Starting create new certificate..."
if [ "$#" -lt 2 ]; then
echo "Usage: ... <domain> <email> [options]"
exit
fi
DOMAIN=$1
EMAIL=$2
OPTIONS=$3
docker run --rm \
-v $PWD/letsencrypt:/etc/letsencrypt \
-v $PWD/webroot:/webroot \
certbot/certbot \
certonly --webroot -w /webroot \
-d $DOMAIN \
-d blog.$DOMAIN \
-d ha.$DOMAIN \
--email $EMAIL \
--non-interactive \
--agree-tos \
$3
# Merge private key and full chain in one file and add them to haproxy certs folder
function cat-cert() {
dir="./letsencrypt/live/$1"
cat "$dir/privkey.pem" "$dir/fullchain.pem" > "./certs/$1.pem"
}
# Run merge certificate for the requested domain name
cat-cert $DOMAINMake sure you save and make the script executable:
chmod +x create-certs.shExecute for testing by using the staging parameter, remove this for production use:
./create-cert yourdomain.com youremail.com --stagingRenew Certificate
This script renews an existing LetsEncrypt signed certificate
Create the script by using nano
nano /home/robert/dockerconfig/renew-certs.sh#!/bin/bash
set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd $DIR
echo "$(date) About to renew certificates" >> letsencrypt-renew.log
docker run \
-i \
--rm \
--name certbot \
-v $PWD/letsencrypt:/etc/letsencrypt \
-v $PWD/webroot:/webroot \
certbot/certbot \
renew -w /webroot
echo "$(date) Cat certificates" >> letsencrypt-renew.log
function cat-cert() {
dir="./letsencrypt/live/$1"
cat "$dir/privkey.pem" "$dir/fullchain.pem" > "./certs/$1.pem"
}
for dir in ./letsencrypt/live/*; do
if [[ "$dir" != *"README" ]]; then
cat-cert $(basename "$dir")
fi
done
echo "$(date) Reload haproxy" >> letsencrypt-renew.log
docker restart dockerconfig-proxy-1
echo "$(date) Done" >> letsencrypt-renew.logMake sure you save and make the script executable
chmod +x renew-certs.shAutomated certificate renewal
Using cron on the ubuntu VM, once a month the renew certificates script runs,
To start you need to elevate your permissions to root:
sudo suThen executing this command will add a line to the cron config file to execute the script to renew certifcates:
echo "0 0 1 * * /home/robert/dockerconfig/renew-certs.sh" >> /etc/crontabUseful Docker Commands
#Run haproxy
sudo docker run -d \
--name haproxy \
--net mynetwork \
-v $(pwd):/usr/local/etc/haproxy:ro \
-p 80:80 \
-p 8404:8404 \
haproxytech/haproxy-alpine:2.4
#Find all running containers
sudo docker ps
#Find all running and stopped containers
sudo docker ps -a
#Start containers that are stopped
sudo docker start $(docker ps -a -q -f status=exited)
#Auto start the haproxy container
sudo docker run --restart=always -d haproxytech/haproxy-alpine:2.4
#Reload haproxy config
sudo docker kill -s HUP haproxyHAProxy Config Gotcha when using DNS
References:
https://www.haproxy.com/documentation/haproxy-configuration-tutorials/dns-resolution
The init-addr field supports three settings
- last
- libc
- none
If the backend server is using a dns name, and haproxy cannot reach it when it starts, it will not startup. This is useless for homelabs when you are testing, so use init-addr none, in order to get a successful startup.
backend webservers
server s1 hostname1.example.com:80 check resolvers mynameservers init-addr noneReferences
https://www.haproxy.com/blog/how-to-run-haproxy-with-docker
https://stackoverflow.com/questions/38221463/command-for-restarting-all-running-docker-containers
https://www.haproxy.com/blog/how-to-map-domain-names-to-backend-server-pools-with-haproxy
https://omarghader.github.io/haproxy-letsencrypt-docker-certbot-certificates -Key link for certificate management
Leave a Reply