...
A controlled fail-over might be required when one or more VM-Hosts in the cluster must be taken down for maintenance. It might also be required to then proceed with a failback to the original machines.
Prerequisites
Your VM-Cluster must have at least two nodes, in which the jtel cluster is built. Before shutting down, an active VM-Host and the active jtel vms within must be chosen. All activity on the jtel ACD is moved to the active VM-Host, which enables you to shutdown the inactive Host containing only standby-machines from your redundant jtel ACD cluster
...
If your monitoring server is currently running on VM-Host 1, it has to be moved to a different VM-Host.
Step 3 - Shutting down Stop all software on all machines on VM-Host 2
| Machine(s) | Stop what | It is installed if you are using | How to stop |
|---|---|---|---|
| acd-telNtel2 | 8-Server | ACD / IVR | X the cmd file starter window. For service installations, stop the robot5 service. |
| Platform UDP Listener | ACD / IVR | X the cmd file starter window. For service installations, stop the jtel Platform UDP Listener service. | |
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| X the cmd file starter window. For service installations, stop the jtel Presence Aggregator service. | |
| Telephony Connector | A PBX which uses a custom connector:
| X the cmd file starter window. For service installations, stop the service, for example the jtel TAPI service or jtel Innovaphone Service. | |
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Stop the jtelEWSMailService service. | |
| IMAP Connector | E-Mail with an IMAP(S) Server | Stop the jtelIMAPMailService service. | |
| acd-jbNjb4-6 | Wildfly | Anything | sudo systemctl stop wildfly For installations not using systemctl: sudo service wildfly stop |
| acd-chatNchat2 | Chat Server | CHAT | sudo systemctl stop jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger stop |
| acd-apiNapi2 | REST API | REST | sudo systemctl stop jtelrest For installations not using systemctl: sudo service jtelrest stop |
| acd-dbmNdbm2 | Platform UDP Listener | SOAP | sudo systemctl stop jtel-listener For installations not using systemctl: sudo service jtel-listener stop |
Step 4 - Check for active sessions
Checking for active database-sessions on the database master is a precaution, but necessary to ensure that all services are stopped and no activity is present on the entire system.
Checking can either be done on the HaProxy admin page on acd-lb1, by checking the current session activity on acd-dbm1, or it can be done on acd-dbm1 within the MySQL terminal, by typing the following command:
| Code Block |
|---|
# The expected output contains only replication status events
SHOW PROCESSLIST \G |
| Warning |
|---|
Only continue when no sessions are active - If this step is not properly executed, this procedure will fail. |
Step 5 - Start all virtual machines on VM-Host 1
The correct order in which to start must be maintained. The following table displays the order
First start acd-store1 and wait until it is up. Start acd-lb1 and wait until it is up. Start acd-dbm1 and wait until it is up. Start acd-dbs1/dbr1 and wait until it is up. Afterwards, start acd-jb1-3, acd-tel1 as well as acd-chat1 and acd-api1 in no particular order.
| VM-Host | Steps 1 to N | machine |
|---|---|---|
| 1 | 1 | acd-store1 |
| 1 | 2 | acd-lb1 |
| 1 | 3 | acd-dbm1 |
| 1 | 4 | acd-dbs1/dbr1 |
| 1 | 5 | acd-chat1 |
| 1 | 6 | acd-api1 |
| 1 | 7 | acd-jb1 |
| 1 | 8 | acd-jb2 |
| 1 | 9 | acd-jb3 |
| 1 | 10 | acd-tel1 |
Step 5 - Stop all software on virtual machines on VM-Host 1
| Machine(s) | Stop what | It is installed if you are using | How to stop |
|---|---|---|---|
| acd-tel1 | 8-Server | ACD / IVR | X the cmd file starter window. For service installations, stop the robot5 service. |
| Platform UDP Listener | ACD / IVR | X the cmd file starter window. For service installations, stop the jtel Platform UDP Listener service. | |
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| X the cmd file starter window. For service installations, stop the jtel Presence Aggregator service. | |
| Telephony Connector | A PBX which uses a custom connector:
| X the cmd file starter window. For service installations, stop the service, for example the jtel TAPI service or jtel Innovaphone Service. | |
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Stop the jtelEWSMailService service. | |
| IMAP Connector | E-Mail with an IMAP(S) Server | Stop the jtelIMAPMailService service. | |
| acd-jb1-3 | Wildfly | Anything | sudo systemctl stop wildfly For installations not using systemctl: sudo service wildfly stop |
| acd-chat1 | Chat Server | CHAT | sudo systemctl stop jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger stop |
| acd-api1 | REST API | REST | sudo systemctl stop jtelrest For installations not using systemctl: sudo service jtelrest stop |
| acd-dbm1 | Platform UDP Listener | SOAP | sudo systemctl stop jtel-listener For installations not using systemctl: sudo service jtel-listener stop |
Step 4 - Check for active sessions
Checking for active database-sessions on the database master is a precaution, but necessary to ensure that all services are stopped and no activity is present on the entire system.
Checking can either be done on the HaProxy admin page on acd-lb1, by checking the current session activity on acd-dbm1, or it can be done on acd-dbm1 within the MySQL terminal, by typing the following command:
| Code Block |
|---|
# The expected output contains only replication status events
SHOW PROCESSLIST \G |
Step 7 - Check cluster
To prepare for the failback, the cluster needs to be checked after booting up. Execute the following commands to check the status:
| Code Block |
|---|
# acd-store1 -> The DRBD should be synchronized | acd-store2 should still be primary, and acd-store1 should be secondary
# acd-store2 -> The DRBD should be synchronized | acd-store2 should still be primary, and acd-store1 should be secondary
pcs status
# acd-lb1 -> acd-lb2 should still be primary, and acd-lb1 should be secondary
# acd-lb2 -> acd-lb2 should still be primary, and acd-lb1 should be secondary
pcs status
# acd-dbm1 -> The replication should be started and synchronized
SHOW SLAVE STATUS \G
# acd-dbm2 -> The replication shouldbe started and synchronized
SHOW SLAVE STATUS \G
# acd-dbs1 -> The replication shouldbe started and synchronized
SHOW SLAVE STATUS \G
# acd-dbs2 -> The replication should be started and synchronized
SHOW SLAVE STATUS \G |
| Warning |
|---|
Do not continue with the next steps if the above status is not yet reached. For problems with DRBD, visit the following sites DRBD - Maintenance and Resolve Split Brain or Node Errors For problems with database replication, visit the following sites Restore MySQL Master-Master Replication Restore MySQL Master Slave Replication Other helpful sites |
Step 5 - Manual Failback to acd-store1
To execute a manual failback , the pcs cluster node acd-store2 is temporarily set into standby, which will cause acd-store1 to become the primary node. After acd-store1 is primary, an unstandby command is executed. After this, acd-store1 will be the primary node and acd-store2 will be secondary.
Execute the following commands on acd-store2:
| Code Block |
|---|
# Set acd-store2 to standby
pcs cluster standby acd-store2
# Check if acd-store1 was switched to primary
pcs status
# Set acd-store2 back to unstandby
pcs cluster unstandby acd-store2
# Check if acd-store1 is primary, and acd-store2 is secondary
pcs status |
Step 6 - Manual Failback to acd-lb1
To execute a manual failback, the pcs cluster node acd-lb2 is temporarily set into standby, which will cause acd-lb1 to become the primary node. After acd-lb1 is primary, an unstandby command is executed. After this, acd-lb1 will be the primary node and acd-lb2 will be secondary.
Execute the following commands on acd-lb2:
| Code Block |
|---|
# Set acd-lb2 to standby
pcs cluster standby acd-lb2
# Check if acd-lb1 was switched to primary
pcs status
# Set acd-lb2 back to unstandby
pcs cluster unstandby acd-lb2
# Check if acd-lb1 is primary, and acd-lb2 is secondary
pcs status |
Step 7 - Check status on acd-lb1 and Configure HaProxy on acd-lb2
Access the HaProxy admin page for acd-lb1. The status after the failback should be exactly as you found it before starting the Shutdown procedure at Step 4.
| Warning |
|---|
If the above described state is not the current state, something went wrong. Please refer to the troubleshooting pages named above. |
Access the HaProxy admin page for acd-lb2. The configuration will only be done on acd-lb2, because the machines on VM-Host 1 are still set to "MAINT" there.
Remove the status "MAINT" for all machines on VM-Host 1
| Warning |
|---|
When the machine is booted and the HaProxy on acd-lb1 starts again, all "MAINT" statuses will be reset to default ,and the status of each machine will be in the default state. |
Step 8 - Check the AcdGroupDistribute Daemon
If the Daemon AcdGroupDistribute.r5 is running on acd-tel1, it must be started on acd-tel2 to ensure that calls to acd-groups will still be routed by the routing-algorithm.
Step 10 - Start Software on all machines on VM-Host 2
| Machine(s) | Start what | It is installed if you are using | How to stop |
|---|---|---|---|
| acd-tel2 | 8-Server | ACD / IVR | Explorer to shell:startup - start the link to startup_launcher.cmd For service installations, start the robot5 service. |
| Platform UDP Listener | ACD / IVR | Explorer to shell:startup - start the link to startListener.bat For service installations, start the jtel Platform UDP Listener service. | |
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| Explorer to shell:startup - start the link to start-presence-aggregator.cmd For service installations, start the jtel Presence Aggregator service. | |
| Telephony Connector | A PBX which uses a custom connector:
| Explorer to shell:startup - start the link to JTELInnovaphonePBXService.exe or jtelTAPIMonitorService.exe For service installations, start the service, for example the jtel TAPI service or jtel Innovaphone Service. | |
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Start the jtelEWSMailService service. | |
| IMAP Connector | E-Mail with an IMAP(S) Server | Start the jtelIMAPMailService service. | |
| acd-jb4-6 | Wildfly | Anything | sudo systemctl start wildfly For installations not using systemctl: sudo service wildfly start |
| acd-chat2 | Chat Server | CHAT | sudo systemctl start jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger start |
| acd-api2 | REST API | REST | sudo systemctl start jtelrest For installations not using systemctl: sudo service jtelrest start |
| acd-dbm2 | Platform UDP Listener | SOAP | sudo systemctl start jtel-listener For installations not using systemctl: sudo service jtel-listener start |
Step 11 - Reactivate monitoring
The monitoring is now reactivated on all machines on VM-Host 2
Step 12 - Ensure system functionality
Information can be found here: System Health Check
If all tests are successful, the system is now running only on VM Host 2 and fully operational.