Shutdown/Startup Procedure - Redundancy - Controlled Failover and Failback

Introduction

A controlled fail-over might be required when one or more VM-Hosts in the cluster must be taken down for maintenance. It might then also be required to proceed with a failback to the original machines after starting the first VM-Host again, to take the second one down for maintenance.

Prerequisites

Your VM-Cluster must have at least two nodes, in which a redundant jtel cluster is running. Before shutting down, an active VM-Host and the active jtel vms within must be chosen. All activity on the jtel ACD is moved to the active VM-Host, which enables you to shutdown the inactive Host containing only standby-machines from your redundant jtel ACD cluster 

A redundant jtel Cluster within your VM-Hosts may look like the example architecture from Shutdown/Startup Procedure - Large V3 - Redundant Databases + Load Balancing + Storage

Example

Explanation

To begin shutdown procedures, the active side is chosen. In the table below the active side is VM-Host 1. We will move it to VM-Host 2 to illustrate a controlled fail-over and do a failback to VM-Host 1 afterwards.

Expected Normal Operation Status

This table describes the systems expected status under fully redundant operation. Redundant components are active on VM-Host 1 and on standby at VM-Host 2.

Shutdown

Various steps are required before the virtual machines can be shutdown.

Step 1 - Backups

At least a backup of the active database, in this case acd-dbm1 is required. If the capacity on your VM-Hosts is enough, snapshots of critical machines are also beneficial to add additional safety, but not required. The critical machines are:

• acd-dbm1
• acd-dbm2
• acd-lb1
• acd-lb2
• acd-store1
• acd-store2

Step 2 - Deactivate Monitoring

If monitoring is installed on your system, schedule a downtime for 2 hours for the machines on VM-Host 2. This is the approximate time this procedure will take to conclude. Extend if needed. The downtime for the machines on VM-Host 1 should be set to however long they will be inactive.

If your monitoring server is currently running on VM-Host 1, it has to be moved to a different VM-Host.

Step 3 - Shutting down all software

Step 4 - Check for active sessions

Checking for active database-sessions on the database master is a precaution, but necessary to ensure that all services are stopped and no activity is present on the entire system.

Checking can either be done on the HaProxy admin page on acd-lb1, by checking the current session count for acd-dbm1, or it can be done on acd-dbm1 within the MySQL terminal, by typing the following command:

# The expected output should contain only replication status events
SHOW PROCESSLIST \G

Step 5 - Manual Failover to acd-store2

To execute a manual failover, the pcs cluster node acd-store1 is temporarily set into standby, which will cause acd-store2 to become the primary node. After acd-store2 is primary, an unstandby command is executed. After this, acd-store2 will be the primary node and acd-store1 will be secondary.

Execute the following commands on acd-store1:

# Set acd-store1 to standby
pcs node standby acd-store1
# Check if acd-store2 was switched to primary
pcs status
# Set acd-store1 back to unstandby
pcs node unstandby acd-store1
# Check if acd-store2 is primary, and acd-store1 is secondary
pcs status

Step 6 - Manual Failover to acd-lb2

To execute a manual failover, the pcs cluster node acd-lb1 is temporarily set into standby, which will cause acd-lb2 to become the primary node. After acd-lb2 is primary, an unstandby command is executed. After this, acd-lb2 will be the primary node and acd-lb1 will be secondary.

Execute the following commands on acd-lb1:

# Set acd-lb1 to standby
pcs node standby acd-lb1
# Check if acd-lb2 was switched to primary
pcs status
# Set acd-lb1 back to unstandby
pcs node unstandby acd-lb1
# Check if acd-lb2 is primary, and acd-lb2 is secondary
pcs status

Step 7 - Configure HaProxy

Access the HaProxy admin page for both acd-lb1 and acd-lb2. The primarily important machine to configure is acd-lb2, but doing it on both HaProxys will ensure that the configuration is the same in case an accidental fail-over happens before the machines on VM-Host 1 are shutdown.

Configure the status "MAINT" for all machines on VM-Host 1

Step 8 -  Check the AcdGroupDistribute Daemon

If the Daemon AcdGroupDistribute.r5 is running on acd-tel1, it must be started on acd-tel2 to ensure that calls to acd-groups will still be routed by the routing-algorithm.

Step 9 - Shutdown the virtual machines on VM-Host 1

The correct order in which to shutdown must still be maintained. The following table displays the order

First shutdown acd-tel1, acd-jb1-3 as well as acd-chat1 and acd-api1 in no particular order. You do not have to wait until acd-tel1, acd-jb1-3 as well as acd-chat1 and acd-api1 are down before shutting acd-dbs1/dbr1 down. Wait until acd-dbs1/dbr1 is down until shutting acd-dbm1 down. Wait until acd-dbm1 is down before shutting down acd-lb1. Wait until acd-lb1 is down before shutting down acd-store1:

Step 10 - Check the cluster status

# acd-store2 -> The DRBD will not be synchronized/disconnected | acd-store2 should be primary, and acd-store1 should be offline
pcs status
# acd-lb2 -> acd-lb2 should be primary, and acd-lb1 should be offline
pcs status
# acd-dbm2 -> The replication to acd-dbm1 should be in status "connecting"
SHOW SLAVE STATUS \G

Step 10 - Start Software on all machines on VM-Host 2

Step 11 - Ensure system functionality

Information can be found here: System Health Check

Step 12 - Reactivate monitoring 

The monitoring is now reactivated on all machines on VM-Host 2

If all tests are successful, the system is now running only on VM Host 2 and fully operational.  

Startup

The following steps assume that VM Host 1 has been started, and all jtel VMs on the server stayed turned off. If the jtel vms were mistakenly automatically turned on, problems in the cluster might have occured as a result, and a different may be required.

Step 1 - Backups

Another backup of the database is created. This time acd-dbm2 is the active database.

Step 2 - Deactivate Monitoring

Schedule a downtime for the monitoring on all machines for approximately 2 hours. Extend if needed.

Step 3 - Stop all software on all machines on VM-Host 2

Step 4 - Check for active sessions

Checking for active database-sessions on the database master is a precaution, but necessary to ensure that all services are stopped and no activity is present on the entire system.

Checking can either be done on the HaProxy admin page on acd-lb2, by checking the current session activity on acd-dbm2, or it can be done on acd-dbm2 within the MySQL terminal, by typing the following command:

# The expected output contains only replication status events
SHOW PROCESSLIST \G

Step 5 - Start all virtual machines on VM-Host 1

The correct order in which to start must be maintained. The following table displays the order

First start acd-store1 and wait until it is up. Start acd-lb1 and wait until it is up. Start acd-dbm1 and wait until it is up. Start acd-dbs1/dbr1 and wait until it is up. Afterwards, start acd-jb1-3, acd-tel1 as well as acd-chat1 and acd-api1 in no particular order.

Step 6 - Stop all software on virtual machines on VM-Host 1

Step 7 - Check for active sessions

Checking for active database-sessions on the database master is a precaution, but necessary to ensure that all services are stopped and no activity is present on the entire system.

Checking can either be done on the HaProxy admin page on acd-lb2, by checking the current session activity on acd-dbm2, or it can be done on acd-dbm2 within the MySQL terminal, by typing the following command:

# The expected output contains only replication status events
SHOW PROCESSLIST \G

Step 7 - Check cluster

To prepare for the failback, the cluster needs to be checked after booting up. Execute the following commands to check the status:

# acd-store1 -> The DRBD should be synchronized | acd-store2 should still be primary, and acd-store1 should be secondary
# acd-store2 -> The DRBD should be synchronized | acd-store2 should still be primary, and acd-store1 should be secondary
cat /proc/drbd
pcs status
# acd-lb1 -> acd-lb2 should still be primary, and acd-lb1 should be secondary
# acd-lb2 -> acd-lb2 should still be primary, and acd-lb1 should be secondary
pcs status
# acd-dbm1 -> The replication should be started and synchronized
SHOW SLAVE STATUS \G
# acd-dbm2 -> The replication should be started and synchronized 
SHOW SLAVE STATUS \G
# acd-dbs1 -> The replication should be started and synchronized 
SHOW SLAVE STATUS \G
# acd-dbs2 -> The replication should be started and synchronized 
SHOW SLAVE STATUS \G

Step 5 - Manual Failback to acd-store1

To execute a manual failback , the pcs cluster node acd-store2 is temporarily set into standby, which will cause acd-store1 to become the primary node. After acd-store1 is primary, an unstandby command is executed. After this, acd-store1 will be the primary node and acd-store2 will be secondary.

Execute the following commands on acd-store2:

# Set acd-store2 to standby
pcs node standby acd-store2
# Check if acd-store1 was switched to primary
pcs status
# Set acd-store2 back to unstandby
pcs node unstandby acd-store2
# Check if acd-store1 is primary, and acd-store2 is secondary
pcs status

Step 6 - Manual Failback to acd-lb1

To execute a manual failback, the pcs cluster node acd-lb2 is temporarily set into standby, which will cause acd-lb1 to become the primary node. After acd-lb1 is primary, an unstandby command is executed. After this, acd-lb1 will be the primary node and acd-lb2 will be secondary.

Execute the following commands on acd-lb2:

# Set acd-lb2 to standby
pcs node standby acd-lb2
# Check if acd-lb1 was switched to primary
pcs status
# Set acd-lb2 back to unstandby
pcs node unstandby acd-lb2
# Check if acd-lb1 is primary, and acd-lb2 is secondary
pcs status

Step 7 - Check status on acd-lb1 and Configure HaProxy on acd-lb2

Access the HaProxy admin page for acd-lb1. The status after the failback should be exactly as you found it before starting the Shutdown procedure at Step 4.

Access the HaProxy admin page for acd-lb2. The configuration will only be done on acd-lb2, because the status of all cluster members on VM-Host 1 was set to default after the boot.

On acd-lb2, remove the status "MAINT" for all machines on VM-Host 1 and set them to "READY". Ensure the correct status before continuing.

Step 8 - Start Software on all machines

Step 9 -  Stop the AcdGroupDistribute Daemon on acd-tel2

Since acd-tel1 is active again, if the Daemon AcdGroupDistribute.r5 is was running on acd-tel1 before, it must be stopped on acd-tel2 to ensure that calls to acd-groups will still be routed properly by the routing-algorithm.

Step 11 - Reactivate monitoring 

The monitoring is now reactivated for all machines

Step 12 - Ensure system functionality

Information can be found here: System Health Check

If all tests are successful, the system is now running only on VM Host 2 and fully operational.  

Step 13 - Remove backups

Backups or snapshots can now be deleted