Access cluster's internal services with WireGuard

Context and need

Some services on a k8s cluster are meant to be publicly accessible, others like dashboards administration consoles or monitoring systems do not. Depending on where your cluster is and whether your cloud provides private networks, the only way to access those consoles would be through an ingress endpoint: an inelegant and risky solution. Here is an WireGard based alternative my sensei taught me. It uses kilo, a WireGard overlay designed for Kubernetes by L. Marín (aka squat). With this set up, you'll be able to access from your laptop, say longhorn dashboard, simply by visiting http://longhorn-frontend.longhorn-system.svc.cluster.local/dashboard

Requirements

Server

  • WireGuard installed
  • a public IP
  • port 51820 opened
  • firewall allowing postrouting in masquerade mode (--add-masquerade with firewalld see also MASQUERADE with iptables)

Client

  • WireGuard installed
  • a public IP
  • openresolv installed (for DNS resolutions inside the VPN network)

WireGard Kilo general idea

Client keys generation

First we need a pair of keys 

sudo -i
mkdir -p /etc/wireguard/keys && /etc/wireguard/keys
umask 077
wg genkey | tee privatekey | wg pubkey > publickey

For the rest of the tutorial, this will be the

  • client's secret key: xxxxxxTheClientSecretKeyxxxxxxxx=
  • client's public key: xxxxxxTheClientPublicKeyxxxxxxxx=

Server setting

All you really need to do is to:

  1. install kilo with one of the many available manifests (here I'll use the one for k3s)
  2. check that the relevant network interfaces have been created
  3. Define a peer using the client's public key
  4. retrieve the server's public key (to set the client up)
kubectl apply -n kube-system -f https://github.com/squat/kilo/blob/master/manifests/kilo-k3s.yaml

This should create two network interfaces (kilo0 and tunl0)

ip addr
......
22: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.4.0.1/16 brd 10.4.255.255 scope global kilo0
       valid_lft forever preferred_lft forever
23: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1430 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.42.0.1/32 brd 10.42.0.1 scope global tunl0
       valid_lft forever preferred_lft forever

Then, we create a kilo-peers.yml

apiVersion: kilo.squat.ai/v1alpha1
kind: Peer
metadata:
  name: squat
spec:
  allowedIPs:
  - 10.0.0.2/32
  publicKey: xxxxxxTheClientPublicKeyxxxxxxxx=
  persistentKeepalive: 10

kubectl apply -n kube-system -f path/to/kilo-peers.yml

Finally, let's get this server's public keys using wg

sudo wg
interface: kilo0
  public key: yyyyyyTheServerPublicKeyyyyyyyyy=
  private key: (hidden)
  listening port: 51820

peer: xxxxxxTheClientPublicKeyxxxxxxxx=
  allowed ips: 10.0.0.2/32

client setting

Now that we have a server public key, we can create a kilo0 interface by editing /etc/wireguard/kilo0.conf. Concerning AllowedIPs, those are the default CIDR of

  • pods: 10.42.0.0/24
  • services: 10.43.0.0/16
  • kilo: 10.4.0.1/16
[Interface]
PrivateKey = xxxxxxTheClientSecretKeyxxxxxxxx=
Address = 10.0.0.2/32   # this will be your subnet in this network
DNS = 10.43.0.10        # DNS server's IP in the cluster
# server
[Peer]
PublicKey = yyyyyyTheServerPublicKeyyyyyyyyy=
Endpoint = 111.11.11.111:51820
AllowedIPs = 10.42.0.0/24, 10.43.0.0/16, 10.4.0.1/16 
persistentKeepalive = 12

persistentKeepalive is indispensible. Without it, no handshake will ever happen

Let's then start the interface

sudo wg-quick up kilo0

and check that everything is in order

ip addr
......
57: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.0.0.2/32 scope global kilo0
       valid_lft forever preferred_lft forever

Test

Find any service (or pod, for that matter) and try to access it with your web browser. 

kubectl get svc -A
....
longhorn-system        longhorn-frontend                    ClusterIP   10.43.68.163    <none>        80/TCP
....

With this set up, I can easily access longhorn's dashboard by visiting either:

  • http://longhorn-frontend.longhorn-system.svc.cluster.local
  • http://http://10.43.68.163

Troubleshooting

First check that both ends managed to perform a handshake. Typically, you should see that

  • both
  • the server has identified the client's public IP
  • public keys match

Server

sudo wg
interface: kilo0
  public key: yyyyyyTheServerPublicKeyyyyyyyyy=
  private key: (hidden)
  listening port: 51820

peer: xxxxxxTheClientPublicKeyxxxxxxxx=
  endpoint: 22.222.22.2:9652
  allowed ips: 10.0.0.2/32
  latest handshake: 5 seconds ago
  transfer: 1.18 MiB received, 20.94 MiB sent

Client

sudo wg
interface: kilo0
  public key: xxxxxxTheClientPublicKeyxxxxxxxx=
  private key: (hidden)
  listening port: 43573

peer: yyyyyyTheServerPublicKeyyyyyyyyy=
  endpoint: 111.11.11.111:51820
  allowed ips: 10.42.0.0/24, 10.43.0.0/16, 10.4.0.1/32
  latest handshake: 1 minute, 16 seconds ago
  transfer: 21.58 MiB received, 1.22 MiB sent
  persistent keepalive: every 12 seconds

If this is the case and you are still unable to access a cluster's service, try to ping the server's kilo0 CIDR. 

ping 10.4.0.1
PING 10.4.0.1 (10.4.0.1) 56(84) bytes of data.
64 bytes from 10.4.0.1: icmp_seq=1 ttl=64 time=40.9 ms
64 bytes from 10.4.0.1: icmp_seq=2 ttl=64 time=42.2 ms
64 bytes from 10.4.0.1: icmp_seq=3 ttl=64 time=42.7 ms
64 bytes from 10.4.0.1: icmp_seq=4 ttl=64 time=42.3 ms

If you do not get an answer, check on the server that kilo0 is up and running. If you do get an answer, the problem is probably caused by your firewall not allowing postrouting