Company infrastructure is distributed in different locations. Whether it is the increasing usage of cloud providers such as AWS or Azure, the multiplication of local decentralized offices, employees hitting the road and working from remote locations or the advent of the Internet of Things (IoT), more and more internal data flows are crossing untrusted communication networks. We can leverage IPsec VPNs to secure these communications.
In today’s blog post, we’ll see how to easily deploy 6WIND Turbo IPsec vRouter as a high-availability IPsec VPN Concentrator cluster sustaining 10k tunnels and 20 Gbps of secured IPsec traffic.
In this post, we’ll focus on the detailed configuration of 6WIND Turbo IPsec vRouter in the following use case:
In this setup, we emulate the connection of a farm of uCPEs to our VPN Concentrator cluster.
An IXIA traffic generator sends traffic in both directions. The packets are ciphered between uCPEs and the VPN Concentrator.
The configuration of 6WIND Turbo IPsec nodes involves the following features:
- VRRP interfaces
- IKE and IPsec synchronization over a dedicated link
- Active/Hot Standby high availability for seamless failover
- Advanced monitoring with metrics collected and sent to an EMS
- High performance ciphering and deciphering
- Enhanced scalability allowing 10k IPsec VPNs
In our case we deployed two 6WIND Turbo IPsec instances as VMs, each with 5 cores / 10 threads and 8GB RAM.
System And Network
Let’s first focus on the fast path configuration under “system”:
system fast-path port pci-b0s4 port pci-b0s5 core-mask fast-path 2-9 .. advanced nb-rxd 4096 nb-txd 4096 .. ..
We dedicate 4 cores/8 threads to the data plane (cores 2 to 9), leaving 1 core/2 threads for the control plane.
LAN and WAN interfaces handled by the fast path are configured with static addresses inside the main vrf, and the dedicated synchronization link as well.
vrf main interface physical wan01 port pci-b0s4 enabled true description "Concentrator public interface" ipv4 address 192.168.0.11/24 .. .. physical lan01 port pci-b0s5 enabled true description "Concentrator internal interface" ipv4 address 192.168.1.11/24 .. .. physical sync01 port pci-b0s6 description "HA synchronization interface" ipv4 address 192.168.100.1/30 .. ..
On top of LAN and WAN interfaces, we configure 2 VRRP interfaces that are shared with the secondary node:
vrf main interface vrrp vrrp0 description "Public VIP" version 3 link-interface wan01 garp-delay 1 use-vmac true vmac-xmit-base true vrid 10 priority 200 preempt false advertisement-interval 100 track-fast-path true unicast-peer 192.168.0.12 virtual-address 250.0.0.2/24 .. vrrp vrrp1 description "Internal VIP" version 3 link-interface lan01 garp-delay 1 use-vmac true vmac-xmit-base true vrid 11 priority 200 preempt false advertisement-interval 100 track-fast-path true unicast-peer 192.168.1.12 virtual-address 126.96.36.199/24 .. ..
They are handling the virtual IP addresses that the outside world knows for this fully redundant VPN concentrator cluster.
We also configure static routes to send deciphered traffic back to the traffic generator and the ciphered traffic towards the uCPEs.
routing static ipv4-route 0.0.0.0/0 next-hop 192.168.0.1 .. ipv4-route 172.26.0.0/24 next-hop 188.8.131.52 .. ipv4-route 172.27.0.0/24 next-hop 184.108.40.206 .. ipv4-route 172.28.0.0/24 next-hop 220.127.116.11 .. ipv4-route 172.29.0.0/24 next-hop 18.104.22.168 .. .. ..
IKE and IPsec configuration
We enable the IKE control plane in the main VRF. In addition to the ciphering parameters, a few tuning parameters ensure the scalability and the security of the VPN concentrator.
Instead of manually defining the configuration of each IPsec VPN, 6WIND Turbo IPsec allows the definition of templates that can be reused in different VPN definitions.
The configuration parameters of the 2 IKE phases are respectively handled by the ike-policy-template and ipsec-policy-template in our configuration:
ike-policy-template vpnc_6wind_ike ike-proposal 1 aead-alg aes128-gcm-96 prf-alg hmac-sha1 dh-group modp1024 .. unique-sa replace rekey-time 0 mobike true .. ipsec-policy-template vpnc_6wind_esp esp-proposal 1 aead-alg aes128-gcm-128 esn true .. start-action none close-action none replay-window 4096 rekey-time 0 ..
Both phases use the AES-GCM algorithm with a key size of 128 bits in our case.
We can now define the IPsec VPN characteristics by referencing these templates:
vpn vpnc_6wind ike-policy template vpnc_6wind_ike .. ipsec-policy template vpnc_6wind_esp .. local-address 250.0.0.2 local-id vpnc.6wind.com security-policy sp_vpnc1 local-ts subnet 172.26.0.0/24 remote-ts subnet 172.16.0.0/24 .. security-policy sp_vpnc2 local-ts subnet 172.27.0.0/24 remote-ts subnet 172.17.0.0/24 .. security-policy sp_vpnc3 local-ts subnet 172.28.0.0/24 remote-ts subnet 172.18.0.0/24 .. security-policy sp_vpnc4 local-ts subnet 172.29.0.0/24 remote-ts subnet 172.19.0.0/24 .. ..
The templates take care of all the cryptographic parameters. We only define the local IP address here, allowing connections to this address from any IP address. Note that the address defined here is the virtual IP address of the public VRRP interface built on top of the WAN interface. Additionally, we define the security policies with local and remote traffic selector subnets: initiators are allowed to negotiate the actual subnets protected by the Security Policies as subsets of these subnets.
We must ensure that the same node receives traffic on the LAN and the WAN interfaces and that this node is actually the one active from an IPsec point of view.
That’s why we define that the VRRP interfaces are grouped, thus binding their fate: each state transition of one interface implies the same transition for the other. This ensures that both virtual IP addresses are held by the same node at each time:
vrf main vrrp group group1 instance vrrp0 instance vrrp1 notify-ha-group ha-group1 .. .. ..
This VRRP group notifies the ha-group ha-group1 of each state transition. As we will see in the next paragraph, the IKE/IPsec HA daemon listens to these notifications via the listen-ha-group ha-group1 configuration item. As a result, the IKE daemon of each node is aware of the status of its respective sibling and knows if it should send or receive updates from the other node.
IKE and IPsec
To achieve the high-availability goal, we must keep the standby node synchronized with the active one regarding:
- The currently established IPsec VPNs (installed Security Associations and the negotiated secrets)
- Their current state (input and output sequence numbers of packets exchanged within a given IPsec VPN)
We enable this synchronization with the following configuration:
vrf main ike ha listen-ha-group ha-group1 node-id 1 interface sync01 local-address 192.168.100.1 remote-address 192.168.100.2 seqnum-sync oseq-shift 65536 sync-period-time 10s sync-period-packets 2 .. ..
We first define the communication channel between both nodes with the local interface and remote- and local-addresses involved in the synchronization channel. Each node has its unique node-id. The configuration of vpnc2 is consequently the following:
node-id 2 interface sync01 local-address 192.168.100.2 remote-address 192.168.100.1
This synchronization process ensures that each VPN established on the active node is also propagated to the standby one.
6WIND Turbo IPsec collects and sends advanced KPIs to a remote Element Management System (EMS) running InfluxDB, a time series database, which can be displayed by an analytics frontend such as Grafana. More information is at https://github.com/6WIND/supervision-grafana.
The configuration is pretty straight-forward as we only need to enable the KPI retrieval at the system level and to configure the EMS target in the main VRF:
vrf main kpi telegraf influxdb-output url http://192.168.255.2:8086 database telegraf .. .. .. system kpi enabled true .. ..
Operation and resiliency
Let’s first establish a few IPsec VPNs by starting the traffic. We start the IXIA traffic generator and see that we can reach 18.9 Gbps of clear text traffic. With the overhead induced by the chosen IPsec algorithm this results in 20 Gbps of IPsec traffic between the uCPEs and the VPN concentrator:
We can monitor these IPsec VPNs from the VPN concentrator point of view:
Commands are available to list the established IPsec VPNs, get the total number of VPNs and retrieve the details of a particular one either from its originating IP address or from its originating ID.
We can also check that the traffic exchanged on the WAN interface is encrypted:
The monitoring dashboard available on the EMS provides an overview of the status:
Scaling to 10k VPNs
Let’s now establish 10k additional IPsec VPNs. We can follow the progress on the dashboard:
The 10k VPNs are successfully negotiated while dealing with 20 Gbps of IPsec traffic.
Let’s take a closer look at the initial state of the cluster by comparing the configuration and the status of vpnc1 (left) and vpnc2 (right):
The VRRP interfaces are declared with a higher priority on vpnc1, ensuring that this node becomes master initially. As a consequence, vpnc1 holds both WAN and LAN virtual IP addresses.
We then trigger the failover by simulating a temporary failure of the LAN network connection of vpnc1. Due to the grouping of the VRRP interfaces, vpnc2 takes over both virtual IP addresses:
As both nodes synchronized the IPsec VPN states, vpnc2 could take over the traffic as soon as the failover occurred. From an external point of view, nothing has changed:
- LAN and WAN IP addresses have continuously been reachable
- The IPsec traffic has been ciphered and deciphered without interruption
The matching dashboard on vpnc1:
And the matching dashboard on vpnc2:
Using 6WIND Turbo IPsec vRouter to implement a VPN concentrator solution will bring multiple benefits:
- 6WIND vRouter can run in any environment including commercial-off-the-shelf (COTS) servers and common virtualization infrastructures
- 6WIND vRouter management is NETCONF/Yang-based for easy automation
- High IPsec throughput scaling with the number of cores allocated
- Scales to several thousands of IPsec VPNs with a high establishment rate
- Protection mechanisms against DDoS attacks
- VRRP and IKE/IPsec High Availability for seamless failover
- Advanced monitoring through KPIs exported to an analytics dashboard
Feel free to request an evaluation as well. We will be happy to hear from you.
This article was written by Aurélien DEGEORGES, L2 support engineer at 6WIND.