Some services on a k8s cluster are meant to be publicly accessible, others like dashboards administration consoles or monitoring systems do not. Depending on where your cluster is and whether your cloud provides private networks, the only way to access those consoles would be through an ingress endpoint: an inelegant and risky solution. Here is an WireGard based alternative my sensei taught me. It uses kilo, a WireGard overlay designed for Kubernetes by L. MarĂn (aka squat). With this set up, you'll be able to access from your laptop, say longhorn dashboard, simply by visiting http://longhorn-frontend.longhorn-system.svc.cluster.local/dashboard
--add-masquerade
with firewalld see also MASQUERADE
with iptables)openresolv
installed (for DNS resolutions inside the VPN network)First we need a pair of keys
sudo -i
mkdir -p /etc/wireguard/keys && /etc/wireguard/keys
umask 077
wg genkey | tee privatekey | wg pubkey > publickey
For the rest of the tutorial, this will be the
xxxxxxTheClientSecretKeyxxxxxxxx=
xxxxxxTheClientPublicKeyxxxxxxxx=
All you really need to do is to:
kilo
with one of the many available manifests (here I'll use the one for k3s)kubectl apply -n kube-system -f https://github.com/squat/kilo/blob/master/manifests/kilo-k3s.yaml
This should create two network interfaces (kilo0
and tunl0
)
ip addr
......
22: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.4.0.1/16 brd 10.4.255.255 scope global kilo0
valid_lft forever preferred_lft forever
23: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1430 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 10.42.0.1/32 brd 10.42.0.1 scope global tunl0
valid_lft forever preferred_lft forever
Then, we create a kilo-peers.yml
apiVersion: kilo.squat.ai/v1alpha1
kind: Peer
metadata:
name: squat
spec:
allowedIPs:
- 10.0.0.2/32
publicKey: xxxxxxTheClientPublicKeyxxxxxxxx=
persistentKeepalive: 10
kubectl apply -n kube-system -f path/to/kilo-peers.yml
Finally, let's get this server's public keys using wg
sudo wg
interface: kilo0
public key: yyyyyyTheServerPublicKeyyyyyyyyy=
private key: (hidden)
listening port: 51820
peer: xxxxxxTheClientPublicKeyxxxxxxxx=
allowed ips: 10.0.0.2/32
Now that we have a server public key, we can create a kilo0
interface by editing /etc/wireguard/kilo0.conf
. Concerning AllowedIPs, those are the default CIDR of
10.42.0.0/24
10.43.0.0/16
10.4.0.1/16
[Interface]
PrivateKey = xxxxxxTheClientSecretKeyxxxxxxxx=
Address = 10.0.0.2/32 # this will be your subnet in this network
DNS = 10.43.0.10 # DNS server's IP in the cluster
# server
[Peer]
PublicKey = yyyyyyTheServerPublicKeyyyyyyyyy=
Endpoint = 111.11.11.111:51820
AllowedIPs = 10.42.0.0/24, 10.43.0.0/16, 10.4.0.1/16
persistentKeepalive = 12
persistentKeepalive
is indispensible. Without it, no handshake will ever happen
Let's then start the interface
sudo wg-quick up kilo0
and check that everything is in order
ip addr
......
57: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.0.0.2/32 scope global kilo0
valid_lft forever preferred_lft forever
Find any service (or pod, for that matter) and try to access it with your web browser.
kubectl get svc -A
....
longhorn-system longhorn-frontend ClusterIP 10.43.68.163 <none> 80/TCP
....
With this set up, I can easily access longhorn's dashboard by visiting either:
First check that both ends managed to perform a handshake. Typically, you should see that
Server
sudo wg
interface: kilo0
public key: yyyyyyTheServerPublicKeyyyyyyyyy=
private key: (hidden)
listening port: 51820
peer: xxxxxxTheClientPublicKeyxxxxxxxx=
endpoint: 22.222.22.2:9652
allowed ips: 10.0.0.2/32
latest handshake: 5 seconds ago
transfer: 1.18 MiB received, 20.94 MiB sent
Client
sudo wg
interface: kilo0
public key: xxxxxxTheClientPublicKeyxxxxxxxx=
private key: (hidden)
listening port: 43573
peer: yyyyyyTheServerPublicKeyyyyyyyyy=
endpoint: 111.11.11.111:51820
allowed ips: 10.42.0.0/24, 10.43.0.0/16, 10.4.0.1/32
latest handshake: 1 minute, 16 seconds ago
transfer: 21.58 MiB received, 1.22 MiB sent
persistent keepalive: every 12 seconds
If this is the case and you are still unable to access a cluster's service, try to ping
the server's kilo0
CIDR.
ping 10.4.0.1
PING 10.4.0.1 (10.4.0.1) 56(84) bytes of data.
64 bytes from 10.4.0.1: icmp_seq=1 ttl=64 time=40.9 ms
64 bytes from 10.4.0.1: icmp_seq=2 ttl=64 time=42.2 ms
64 bytes from 10.4.0.1: icmp_seq=3 ttl=64 time=42.7 ms
64 bytes from 10.4.0.1: icmp_seq=4 ttl=64 time=42.3 ms
If you do not get an answer, check on the server that kilo0
is up and running. If you do get an answer, the problem is probably caused by your firewall not allowing postrouting