Build HA services in a VPC by using Keepalived

Overview

Keepalived is usually used to build high availability (HA) services such as NGINX, HAProxy, MySQL, and DNS. The use of Keepalived in a VPC is slightly different from that in a traditional network due to differences between VPCs and traditional networks. This topic introduces Keepalived and describes the differences of applying Keepalived on KEC instances and a physical network.

The following example describes how to configure unicast VRRP packets for VRRP instances that are based on Keepalived.

Implementation of master/backup HA clusters

Typically, a master/backup HA cluster contains two servers. One server acts as the master node and is in active state, and the other server acts as the backup node and is in standby state. The two nodes share the same virtual IP address (VIP). The VIP only takes effect on the master node. When the master node fails, the standby node takes over the VIP and continues to provide services.

Differences between VPCs and physical networks

In a traditional physical network, the master/backup state can be negotiated through VRRP of Keepalived. The master and backup nodes send heartbeat messages in multicast or unicast mode to elect the master node. The master node periodically sends gratuitous ARP packets to update the MAC address table or terminal ARP table of the uplink switch, triggering the migration of the VIP to the master node.

Kingsoft Cloud VPC supports deploying Keepalived to build HA master/backup clusters. Compared with physical networks, the main difference is that the VIP takes effect in the VPC domain after migration only when the API is called to adjust the route.

Deploy Keepalived in a VPC

Prerequisites

The KEC instance must be able to access the Internet (through EIP or NAT).

Step 1 Select a VIP from a VPC

After you create a subnet in a VPC, you can select any unused IP address in the subnet as the VIP. You can view the IP addresses that are in use on the subnet page of the VPC console.

Note:

1. It is recommended that you select an IP address with a larger number in the subnet as the VIP to avoid the situation that the VIP is occupied when a new KEC instance is created.
2. Kingsoft Cloud provides the feature of reserved IP address.
3. A VIP will not be automatically configured on any KEC instance, and the VIP takes effect only when it is used with Keepalived.

The deployment environment example is as follows. You need to modify the data based on your own network configuration.

VPC 172.18.0.0/16

KEC1 172.18.0.10

KEC2 172.18.0.20

Reserved VIP 172.18.0.253

Step 2 Prepare required software

1. Install Keepalived (version 1.3.5 or higher) on the master and backup KEC instances. For CentOS, run the yum –y install keepalived command.

2. Prepare scripts notify_action.sh and nexthop.py as well as the Keepalived configuration file.

   Step 1: Clone the repository at https://github.com/KscSDK/ksc-sdk-python.git.

   Step 2: Run the python setup.py install command to install the Kingsoft Cloud SDK. In some environments, you need to run the yum install python-pip or pip install -U pip setuptools command for update.

   Step 3: Copy nexthop.py, keepalived.conf, and notify_action.sh from ksc-sdk-python/examples to the /etc/keepalived directory.

3. Create the executable file chmod +x notify_action.sh.

Step 3. Modify keepalived.conf.

        0) state               The initial role. Set the value to BACKUP.
        1) interface           Set the value to the name of the local NIC card. Example: eth0.
        2) priority            Set the same integer value for both nodes. Example: 50.
        3) unicast_src_ip      Set the value to the private IP address of the local end.
        4) unicast_peer        Set the value to the private IP address of the peer end.
        5) virtual_ipaddress   Set the value to the private VIP.
        6) track_interface     Set the value to the name of the local NIC card. Example: eth0.

Note: It is important to configure the unicast mode, that is, to specify the IP address of the peer end.
To configure the unicast mode, remove vrrp_strict from the configuration file.

Step 4. Modify nexthop.py.

nexthop.py is the program that is used for master/backup switching based on the API. When the master/backup configuration in nexthop.py is modified, Keepalived triggers the notify_action.sh program to call the API action of modifying VPC routes. The VIP then directs to another KEC instance.

1. Obtain API access keys: Log in to the Kingsoft Cloud console. Move the pointer over the account in the upper-right corner and then select accesskeys from the drop-down list that appears. You can create new access keys or use existing access keys.
2. The VPC ID and the IDs of the master and backup instances can be obtained from the Kingsoft Cloud console.
3. Modify the following part in nexthop.py.

##################Begin####################
region='cn-beijing-6'   #The Kingsoft Cloud region where the cloud service is located.
vpcId = '1858a08a-6cc9-4278-8d0c-d536f441fe8e'  #The VPC ID.
ks_access_key_id = 'your access key ID'
ks_secret_access_key = 'your secret access key'
vip = "172.18.0.253" #Your local private VIP.
DestinationCidrBlock = '172.18.0.253/32' #The VIP.
thisInstanceId = '1cf963ff-7847-4859-8462-5405f0facc1d' #The ID of the current KEC instance.
thatInstanceId = 'b141da5f-8e3e-44c0-ac0f-a0feccba78c7' #The ID of the KEC instance before migration.
interface = {"eth0":"172.18.0.13"} #The primary NIC and IP address of the current KEC instance.
##################End######################

Step 5. Check whether the VIP and public IP address are switched normally during master/backup switching.

1. Start Keepalived by running the /etc/init.d/keepalived start, systemctl start keepalived, or service keepalived start command. Note: For the production environment, add the startup method to /etc/rc.local to enable automatic start.
2. Verify the disaster recovery effect of master/backup switching. You can simulate a KEC instance failure by restarting the Keepalived process or a KEC instance to check whether the VIP can be migrated. Corresponding logs will be recorded in /var/log/keepalived.log. By pinging the VIP, you can obtain the interval from network interruption to recovery.

Note:
1. Because route redirection is implemented asynchronously by using the API, the route takes effect within 1 minute and access
to the VIPs across subnets may failed within that period. VIP access in the same subnet is only affected by the switching
time of Keepalived.
2. Script logs will be written to /var/log/keealived.log. Logs take up your disk space. You can use tools such as logrotate
to handle log accumulation. Keepalived process logs are written to /var/log/message.