25 Mar 2021 30 Mar 2021

Automating the setup of multi server Wireguard mesh using ansible

Recently, I have been experimenting with a multi-region architecture, with a couple servers in multiple continents, fronted by GeoDNS to route the users to the closest one.

These servers had to talk to each other, as there is still a single database primary.

Since this is a side project, I am using cheap VPS servers from multiple hosters: Scaleway & Hetzner in Europe, Vultr & Digital Ocean for other regions, ….

This excluded cloud-specific solutions, e.g. AWS’s multi region VPC peering.

Instead, I resorted to using Wireguard to secure the inter-server exchanges.

Wireguard is a very popular VPN, and there are many quality tutorials on how to setup a mesh between multiple servers, such as:

Those were great to get started, but I needed to automate the whole procedure using ansible. I managed to do that, and I thought that explaining how I did could prove useful to others, especially with the preshared keys setup, which was a bit challenging.

Compatibility

The playbook works on Ubuntu, but could very easily be adapted to other distributions. It was tested with Ubuntu 18.04 and 20.04.

The inventory

Here’s an example inventory. In this particular case it lists 3 VMs from Hetzner.

all:
  hosts:
    A:
      ansible_ssh_user: root
      ansible_host: xxx.xx.xxx.x
      ansible_ssh_port: 22

      wireguard_ip: 10.0.1.100

    B:
      ansible_ssh_user: root
      ansible_host: xxx.xx.xxx.xx
      ansible_ssh_port: 22

      wireguard_ip: 10.0.1.101

    C:
      ansible_ssh_user: root
      ansible_host: xx.xxx.xx.xxx
      ansible_ssh_port: 22

      wireguard_ip: 10.0.1.102


  vars:
    ansible_become_method: su

    wireguard_mask_bits: 8
    wireguard_port: 51871

Besides the ansible variables (ansible_ssh_user, ansible_host, ansible_ssh_port, …), some wireguard specific variables need to be configured:

wireguard_ip for every host. a class A private IP adresses in this example.
wireguard_mask_bits global variable. 8 in this case, which corresponds to the number of bits in the network prefix, i.e. what comes after the / in the CIDR notation.
wireguard_port global variable. 51871 in this case, which corresponds to the wireguard UDP port.

The playbook

I chose to configure the mesh using a flat playbook, without roles.

Here’s how it looks like:

---
- hosts: all
  any_errors_fatal: true
  gather_facts: yes
  tasks:
    - name: update packages
      apt:
        update_cache: yes
        cache_valid_time: 3600
      become: yes
    - ... other tasks

The next tasks will be presented one by one.

1. Installing wireguard

Self explanatory: Wireguard needs to be installed in every host.

- name: Install wireguard
  apt:
    name: wireguard
    state: present
  become: yes

2. Generating the public/private keypairs

For every host, it not already done, generate a public/private keypair and store them in /etc/wireguard/publickey and /etc/wireguard/privatekey, respectively.

Also, register both in wireguard_public_key and wireguard_private_key variables.

- name: Generate Wireguard keypair
  shell: wg genkey | tee /etc/wireguard/privatekey | wg pubkey | tee /etc/wireguard/publickey
  args:
    creates: /etc/wireguard/privatekey
  become: yes

- name: register private key
  shell: cat /etc/wireguard/privatekey
  register: wireguard_private_key
  changed_when: false
  become: yes

- name: register public key
  shell: cat /etc/wireguard/publickey
  register: wireguard_public_key
  changed_when: false
  become: yes

Notice the creates which tells ansible to skip executing wg genkey if the key pair was already generated. Idempotent playbook FTW !

3. Generating the pre-shared keys

As an additional layer of security, we’ll also generate pre-shared keys so that Wireguard can mix it with the public key cryptography.

Here’s where things get tricky: we need to generate 1 pre-shared key per server-pair.

For example, given 3 server A, B and C, we need 3 pre-shared keys:

One for A <-> B
One for A <-> C
One for B <-> C

So, unlike the public/private keypairs, a simple with_items: groups['all'] won’t do.

It took me a while to figure out how to achieve this with ansible.

What I did was the following:

- name: generate Preshared keyskeypair
  shell: "wg genpsk > /etc/wireguard/psk-{{ item }}"
  args:
    creates: "/etc/wireguard/psk-{{ item }}"
  when: inventory_hostname < item
  with_items: "{{ groups['all'] }}"
  become: yes

- name: register preshared key
  shell: "cat /etc/wireguard/psk-{{ item }}"
  register: wireguard_preshared_key
  changed_when: false
  when: inventory_hostname < item
  with_items: "{{ groups['all'] }}"
  become: yes

The playbook already iterates over hosts. The task will also iterate over the hosts, and will only generate a pre-shared key if the host name the playbook is currently running on is lexicographically before the host name the task is iterating on.

Whew. That was a mouthful.

Back to the 3 host example with A, B and C:

The playbook start iterating over those 3:

host=A
- the task iterates over the hosts:
  - item=A, A < A does not hold, skip
  - item=B, A < B does hold, generate a pre-shared key and store it in /etc/wireguard/psk-B in host A
  - item=C, A < C does hold, generate a pre-shared key and store it in /etc/wireguard/psk-C in host A
host=B
- the task iterates over the hosts:
  - item=A, B < A does not hold, skip
  - item=B, B < B does not hold, skip
  - item=C, B < C does hold, generate a pre-shared key and store it in /etc/wireguard/psk-C in host B
host=C
- the task iterates over the hosts:
  - item=A, C < A does not hold, skip
  - item=B, C < B does not hold, skip
  - item=B, C < C does not hold, skip

As can be seen above, we do indeed end up with 3 pre-shared keys.

Next the playbook reads those generated keys and registers them in the wireguard_preshared_key variable.

Ansible, when faced with a register combined with with_items, will populate the variable as a list with the following structure:

in host A:

{
  results: [
    { item: A, skipped: true},
    { item: B, "stdout": "pre-shared-key A<->B"},
    { item: C, "stdout": "pre-shared-key A<->C"}
  ]
}

in host B:

{
  results: [
    { item: A, skipped: true},
    { item: B, skipped: true},
    { item: C, "stdout": "pre-shared-key B<->C"}
  ]
}

in host C:

{
  results: [
    { item: A, skipped: true},
    { item: B, skipped: true},
    { item: C, skipped: true}
  ]
}

The playbook next massages this datastructure to transform it into a dictionary:

- name: massage preshared keys
  set_fact: "wireguard_preshared_keys={{ wireguard_preshared_keys|default({}) | combine( {item.item: item.stdout} ) }}"
  when: item.skipped is not defined
  with_items: "{{ wireguard_preshared_key.results }}"
  become: yes

The new wireguard_preshared_keys variable has the following structure:

in host A:

{
  B: "pre-shared-key A<->B"
  C: "pre-shared-key A<->C"
}

in host B:

{
  C: "pre-shared-key B<->C"
}

in host C:

undefined

This will make it easier to consume in the next steps.

4. Configuring the wg0 network

Using systemd-network:

- name: Setup wg0 device
  template:
    src: ./templates/systemd.netdev
    dest: /etc/systemd/network/99-wg0.netdev
    owner: root
    group: systemd-network
    mode: 0640
  become: yes
  notify: systemd network restart

- name: Setup wg0 network
  template:
    src: ./templates/systemd.network
    dest: /etc/systemd/network/99-wg0.network
    owner: root
    group: systemd-network
    mode: 0640
  become: yes
  notify: systemd network restart

The playbook configures:

A. `wg0` virtual network device

The network device descriptor /etc/systemd/network/99-wg0.netdev is created using the following (jinja2) template:

[NetDev]
Name=wg0
Kind=wireguard
Description=WireGuard tunnel wg0

[WireGuard]
ListenPort={{ wireguard_port }}
PrivateKey={{ wireguard_private_key.stdout }}

{% for peer in groups['all'] %}
{% if peer != inventory_hostname %}

[WireGuardPeer]
PublicKey={{ hostvars[peer].wireguard_public_key.stdout }}
PresharedKey={{ wireguard_preshared_keys[peer] if inventory_hostname < peer else hostvars[peer].wireguard_preshared_keys[inventory_hostname] }}
AllowedIPs={{ hostvars[peer].wireguard_ip }}/32
Endpoint={{ hostvars[peer].ansible_host }}:{{ wireguard_port }}
PersistentKeepalive=25
{% endif %}
{% endfor %}

The template:

configures the wireguard UDP port
configures the wireguard priate key (of the current host)
for each (other) hosts
- add a [WireGuardPeer] entry
- configure PublicKey with the peer’s public key, retrieved using hostvars[peer].wireguard_public_key.stdout
- configure AllowedIPs with the peer’s private/Wireguard IP, retrieved using {{ hostvars[peer].wireguard_ip }}
- configure Endpoint with the peer’s public IP, retrieved using {{ hostvars[peer].ansible_host }}::{{ wireguard_port }}
- configure the PresharedKey <- this is again the tricky part
- configure the PersistentKeepalive with 25 (seconds) <- this is important or else the inter-node connections will frequently drop, with the only way of restoring them being the hosts mutually pinging each other

For the preshared key, the template uses the following expression to extract the correct key from the correct host:

wireguard_preshared_keys[peer] if inventory_hostname < peer
 else hostvars[peer].wireguard_preshared_keys[inventory_hostname]

B. `wg0` virtual network

The network descriptor /etc/systemd/network/99-wg0.network is created using the following (jinja2) template:

[Match]
Name=wg0

[Network]
Address={{ wireguard_ip }}/{{ wireguard_mask_bits }}

5. Restart systemd-networkd

As a handler:

handlers:
  - name: systemd network restart
    service:
      name: systemd-networkd
      state: restarted
      enabled: yes
    become: yes

Bonus points: UFW

To lock down the system, enable UFW with a default policy of reject, and only accept traffic from the inventory hosts wireguard IPs:

- name: Allow SSH in UFW
  ufw:
    rule: allow
    port: "{{ ansible_ssh_port }}"
    proto: tcp
  become: yes

- name: Set ufw logging
  ufw:
    logging: "on"
  become: yes

- name: inter-node Wireguard UFW connectivity
  ufw:
    rule: allow
    src: "{{ hostvars[item].wireguard_ip }}"
  with_items: "{{ groups['all'] }}"
  become: yes and item != inventory_hostname

- name: Reject everything and enable UFW
  ufw:
    state: enabled
    policy: reject
    log: yes
  become: yes

Do not forget to open the SSH port or else you’ll get locked out of the machines

The full picture

You can find the whole playbook in this companion Github repository

But here it is anyway, to save you a click:

---
- hosts: all
  any_errors_fatal: true
  gather_facts: yes
  tasks:
    - name: update packages
      apt:
        update_cache: yes
        cache_valid_time: 3600
      become: yes

    - name: Allow SSH in UFW
      ufw:
        rule: allow
        port: "{{ ansible_ssh_port }}"
        proto: tcp
      become: yes

    - name: Set ufw logging
      ufw:
        logging: "on"
      become: yes

    - name: inter-node Wireguard UFW connectivity
      ufw:
        rule: allow
        src: "{{ hostvars[item].wireguard_ip }}"
      with_items: "{{ groups['all'] }}"
      become: yes and item != inventory_hostname

    - name: Reject everything and enable UFW
      ufw:
        state: enabled
        policy: reject
        log: yes
      become: yes

    - name: Install wireguard
      apt:
        name: wireguard
        state: present
      become: yes

    - name: Generate Wireguard keypair
      shell: wg genkey | tee /etc/wireguard/privatekey | wg pubkey | tee /etc/wireguard/publickey
      args:
        creates: /etc/wireguard/privatekey
      become: yes

    - name: register private key
      shell: cat /etc/wireguard/privatekey
      register: wireguard_private_key
      changed_when: false
      become: yes

    - name: register public key
      shell: cat /etc/wireguard/publickey
      register: wireguard_public_key
      changed_when: false
      become: yes

    - name: generate Preshared keyskeypair
      shell: "wg genpsk > /etc/wireguard/psk-{{ item }}"
      args:
        creates: "/etc/wireguard/psk-{{ item }}"
      when: inventory_hostname < item
      with_items: "{{ groups['all'] }}"
      become: yes

    - name: register preshared key
      shell: "cat /etc/wireguard/psk-{{ item }}"
      register: wireguard_preshared_key
      changed_when: false
      when: inventory_hostname < item
      with_items: "{{ groups['all'] }}"
      become: yes

    - name: massage preshared keys
      set_fact: "wireguard_preshared_keys={{ wireguard_preshared_keys|default({}) | combine( {item.item: item.stdout} ) }}"
      when: item.skipped is not defined
      with_items: "{{ wireguard_preshared_key.results }}"
      become: yes

    - name: Setup wg0 device
      template:
        src: ./templates/systemd.netdev
        dest: /etc/systemd/network/99-wg0.netdev
        owner: root
        group: systemd-network
        mode: 0640
      become: yes
      notify: systemd network restart

    - name: Setup wg0 network
      template:
        src: ./templates/systemd.network
        dest: /etc/systemd/network/99-wg0.network
        owner: root
        group: systemd-network
        mode: 0640
      become: yes
      notify: systemd network restart

  handlers:
    - name: systemd network restart
      service:
        name: systemd-networkd
        state: restarted
        enabled: yes
      become: yes

This is Jawher Moussa's blog