...
| Warning |
|---|
Do not attempt this procedure if the systems current state is unknown. Before work can begin, all cluster functions must be validated. If any part of the cluser, for example the DRBD Replication on acd-store is not functioning properly, a failover will might not work and a systemwide shutdown failure of functions could be the result. Information can be found here: System Health Check |
...
A controlled fail-over might be required when one or more VM-Hosts in the cluster must be taken down for maintenance. It might then also be required to then proceed with a failback to the original machines after starting the first VM-Host again, to take the second one down for maintenance.
Prerequisites
Your VM-Cluster must have at least two nodes, in which the a redundant jtel cluster is builtrunning. Before shutting down, an active VM-Host and the active jtel vms within must be chosen. All activity on the jtel ACD is moved to the active VM-Host, which enables you to shutdown the inactive Host containing only standby-machines from your redundant jtel ACD cluster
A redundant jtel Cluster within your VM-Hosts may look like the example architecture from Shutdown/Startup Procedure - Large V3 - Redundant Databases + Load Balancing + Storage
| Info |
|---|
Your VM-Hosts should be configured so that none of the jtel vms turn on automatically after the Host ist started. If any startup-automation is in place, it should be deactivated beforehand or configured to fit within the parameters of this procedure. |
Example
Explanation
To begin shutdown procedures, the active side is chosen. In the table below the active side is VM-Host 1. We will move it to VM-Host 2 to illustrate a controlled fail-over and do a failback to VM-Host 1 afterwards.
| Info |
|---|
For the purposes of illustrating the procedure in this page, acd-telN and acd-jbN are replaced with multiple machines. jtel acd-chat and acd-api servers are also active, as well as a presence-aggregator and IMAP and exchange mail connectors |
Example System Parameters/Expected Normal Operation Status
This table describes the systems expected status under fully redundant operation operation.
| VM-Host | machine | Active |
|---|---|---|
| 1 | acd-tel1 | Yes |
| 2 | acd-tel2 | Yes |
| 1 | acd-jb1 | Yes |
| 1 | acd-jb2 | Yes |
| 1 | acd-jb3 | Yes |
| 2 | acd-jb4 | Yes |
| 2 | acd-jb5 | Yes |
| 2 | acd-jb6 | Yes |
| 1 | acd-chat1 | Yes |
| 2 | acd-chat2 | Yes |
| 1 | acd-api1 | Yes |
| 2 | acd-api2 | Yes |
| 1 | acd-dbs1/dbr1 | Yes |
| 2 | acd-dbs2/dbr2 | Yes |
| 1 | acd-dbm1 | Yes |
| 2 | acd-dbm2 | Standby |
| 1 | acd-lb1 | Yes |
| 2 | acd-lb2 | Standby |
| 1 | acd-store1 | Yes |
| 2 | acd-store2 | Standby |
...
At least a backup of database on acd-dbm1 is required. If the capacity on your VM-Hosts is enough, snapshots of critical machines are also goodbeneficial to add additional safety, but not essentialrequired. The critical machines are:
- acd-dbm1
- acd-dbm2
- acd-lb1
- acd-lb2
- acd-store1
- acd-store2
...
| Machine(s) | Stop what | It is installed if you are using | How to stop |
|---|---|---|---|
| acd-telNtel1+2 | 8-Server | ACD / IVR | X the cmd file starter window. For service installations, stop the robot5 service. |
| Platform UDP Listener | ACD / IVR | X the cmd file starter window. For service installations, stop the jtel Platform UDP Listener service. | |
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| X the cmd file starter window. For service installations, stop the jtel Presence Aggregator service. | |
| Telephony Connector | A PBX which uses a custom connector:
| X the cmd file starter window. For service installations, stop the service, for example the jtel TAPI service or jtel Innovaphone Service. | |
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Stop the jtelEWSMailService service. | |
| IMAP Connector | E-Mail with an IMAP(S) Server | Stop the jtelIMAPMailService service. | |
| acd-jb1-jbN6 | Wildfly | Anything | sudo systemctl stop wildfly For installations not using systemctl: sudo service wildfly stop |
| acd-chatNchat1 | Chat Server | CHAT | sudo systemctl stop jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger stop |
| acd-apiNapi1 | REST API | REST | sudo systemctl stop jtelrest For installations not using systemctl: sudo service jtelrest stop |
| acd-dbmNdbm1 | Platform UDP Listener | SOAP | sudo systemctl stop jtel-listener For installations not using systemctl: sudo service jtel-listener stop |
...
| Code Block |
|---|
# The expected output containsshould contain only replication status events SHOW PROCESSLIST \G |
...
| Warning |
|---|
Only continue when no sessions are active. If this step is not properly executed, this procedure might fail at Step 5 or 6. |
Step 5 - Manual Failover to acd-store2
...
Access the HaProxy admin page for both acd-lb1 and acd-lb2. The primarily important machine to configure is acd-lb2, but the configuration will be the same in case a an accidental fail-over happens during this entire procedure, the configuration will be the samebefore the machines on VM-Host 1 are shutdown.
Configure the status "MAINT" for all machines on VM-Host 1
...
| VM-Host | Steps 1 to N | machine |
|---|---|---|
| 1 | 1 | acd-tel1 |
| 1 | 2 | acd-jb1 |
| 1 | 3 | acd-jb2 |
| 1 | 4 | acd-jb3 |
| 1 | 5 | acd-chat1 |
| 1 | 6 | acd-api1 |
| 1 | 7 | acd-dbs1/dbr1 |
| 1 | 8 | acd-dbm1 |
| 1 | 9 | acd-lb1 |
| 1 | 10 | acd-store1 |
Step 10 - Start Software on all machines on VM-Host 2Check the cluster status
| Code Block |
|---|
# acd-store2 -> The DRBD will not be synchronized | acd-store2 should be primary, and acd-store1 should be offline
pcs status
# acd-lb2 -> acd-lb2 should be primary, and acd-lb1 should be offline
pcs status
# acd-dbm2 -> The replication should be in status "connecting"
SHOW SLAVE STATUS \G
# acd-dbs2 -> The replication should be in status "connecting"
SHOW SLAVE STATUS \G |
Step 10 - Start Software on all machines on VM-Host 2
| Machine(s) | Start | Machine(s) | Start what | It is installed if you are using | How to stop |
|---|---|---|---|---|---|
| acd-tel2 | 8-Server | ACD / IVR | Explorer to shell:startup - start the link to startup_launcher.cmd For service installations, start the robot5 service. | ||
| Platform UDP Listener | ACD / IVR | Explorer to shell:startup - start the link to startListener.bat For service installations, start the jtel Platform UDP Listener service. | |||
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| Explorer to shell:startup - start the link to start-presence-aggregator.cmd For service installations, start the jtel Presence Aggregator service. | |||
| Telephony Connector | A PBX which uses a custom connector:
| Explorer to shell:startup - start the link to JTELInnovaphonePBXService.exe or jtelTAPIMonitorService.exe For service installations, start the service, for example the jtel TAPI service or jtel Innovaphone Service. | |||
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Start the jtelEWSMailService service. | |||
| IMAP Connector | E-Mail with an IMAP(S) Server | Start the jtelIMAPMailService service. | |||
| acd-jb4-6 | Wildfly | Anything | sudo systemctl start wildfly For installations not using systemctl: sudo service wildfly start | ||
| acd-chat2 | Chat Server | CHAT | sudo systemctl start jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger start | ||
| acd-api2 | REST API | REST | sudo systemctl start jtelrest For installations not using systemctl: sudo service jtelrest start | ||
| acd-dbm2 | Platform UDP Listener | SOAP | sudo systemctl start jtel-listener For installations not using systemctl: sudo service jtel-listener start |
...
The following steps assume that VM Host 1 has been booted, and all jtel VMs on the server are stayed turned off. If the jtel vms were mistakenly automatically turned on, problems in the cluster might have occured as a result.
...
Step 2 - Deactivate Monitoring
Schedule a downtime for the monitoring on all machines for approximately 2 hours. Extend if needed.
Schedule a downtime for the monitoring on all machines for approximately 2 hours. Extend if neededIf your monitoring server is currently running on VM-Host 1, it has to be moved to a different VM-Host.
Step 3 - Stop all software on all machines on VM-Host 2
...
Checking can either be done on the HaProxy admin page on acd-lb1lb2, by checking the current session activity on acd-dbm1dbm2, or it can be done on acd-dbm1 dbm2 within the MySQL terminal, by typing the following command:
...
The correct order in which to start must be maintained. The following table displays the order
First start acd-store1 and wait until it is up. Start acd-lb1 and wait until it is up. Start acd-dbm1 and wait until it is up. Start acd-dbs1/dbr1 and wait until it is up. Afterwards, start acd-jb1-3, acd-tel1 as well as acd-chat1 and acd-api1 in no particular order.
| VM-Host | Steps 1 to N | machine |
|---|---|---|
| 1 | 1 | acd-store1 |
| 1 | 2 | acd-lb1 |
| 1 | 3 | acd-dbm1 |
| 1 | 4 | acd-dbs1/dbr1 |
| 1 | 5 | acd-chat1 |
| 1 | 6 | acd-api1 |
| 1 | 7 | acd-jb1 |
| 1 | 8 | acd-jb2 |
| 1 | 9 | acd-jb3 |
| 1 | 10 | acd-tel1 |
Step 5 6 - Stop all software on virtual machines on VM-Host 1
| Machine(s) | Stop what | It is installed if you are using | How to stop |
|---|---|---|---|
| acd-tel1 | 8-Server | ACD / IVR | X the cmd file starter window. For service installations, stop the robot5 service. |
| Platform UDP Listener | ACD / IVR | X the cmd file starter window. For service installations, stop the jtel Platform UDP Listener service. | |
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| X the cmd file starter window. For service installations, stop the jtel Presence Aggregator service. | |
| Telephony Connector | A PBX which uses a custom connector:
| X the cmd file starter window. For service installations, stop the service, for example the jtel TAPI service or jtel Innovaphone Service. | |
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Stop the jtelEWSMailService service. | |
| IMAP Connector | E-Mail with an IMAP(S) Server | Stop the jtelIMAPMailService service. | |
| acd-jb1-3 | Wildfly | Anything | sudo systemctl stop wildfly For installations not using systemctl: sudo service wildfly stop |
| acd-chat1 | Chat Server | CHAT | sudo systemctl stop jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger stop |
| acd-api1 | REST API | REST | sudo systemctl stop jtelrest For installations not using systemctl: sudo service jtelrest stop |
| acd-dbm1 | Platform UDP Listener | SOAP | sudo systemctl stop jtel-listener For installations not using systemctl: sudo service jtel-listener stop |
Step 4 7 - Check for active sessions
...
Checking can either be done on the HaProxy admin page on acd-lb1lb2, by checking the current session activity on acd-dbm1dbm2, or it can be done on acd-dbm1 dbm2 within the MySQL terminal, by typing the following command:
| Code Block |
|---|
# The expected output contains only replication status events SHOW PROCESSLIST \G |
| Warning |
|---|
Only continue when no sessions are active. If this step is not properly executed, this procedure will fail. |
Step 7 - Check cluster
To prepare for the failback, the cluster needs to be checked after booting up. Execute the following commands to check the status:
...
| Warning |
|---|
Do not continue with the next steps if the above status is not yet reachedreached. The sites below may help with troubleshooting any issues. For problems with DRBD, visit the following sites DRBD - Maintenance and Resolve Split Brain or Node Errors For problems with database replication, visit the following sites Restore MySQL Master-Master Replication Restore MySQL Master Slave Replication Other helpful sites |
...
Access the HaProxy admin page for acd-lb1. The status after the failback should be exactly as you found it before starting the Shutdown procedure at Step 4.
| Warning |
|---|
If the above described state is not the current state, something went wrong. Please refer to the troubleshooting pages named above. |
Access the HaProxy admin page for acd-lb2. The configuration will only be done on acd-lb2, because the machines on VM-Host 1 are still set to "MAINT" there.
Remove On acd-lb2, remove the status "MAINT" for all machines on VM-Host 1 and set them to "READY". Ensure the correct status continuing.
| Warning |
|---|
When the machine is booted and the HaProxy on acd-lb1 starts again, all "MAINT" statuses will be reset to default ,and the status of each machine will be in the default state. |
Step 8 - Check the AcdGroupDistribute Daemon
...
. |
Step 10 8 - Start Software on all machines on VM-Host 2
| Machine(s) | Start what | It is installed if you are using | How to stop | ||
|---|---|---|---|---|---|
| acd-tel2tel1+2 | 8-Server | ACD / IVR | Explorer to shell:startup - start the link to startup_launcher.cmd For service installations, start the robot5 service. | ||
| Platform UDP Listener | ACD / IVR | Explorer to shell:startup - start the link to startListener.bat For service installations, start the jtel Platform UDP Listener service. | |||
| Presence Aggregator | A PBX or presence connector which uses the presence aggregator:
| Explorer to shell:startup - start the link to start-presence-aggregator.cmd For service installations, start the jtel Presence Aggregator service. | |||
| Telephony Connector | A PBX which uses a custom connector:
| Explorer to shell:startup - start the link to JTELInnovaphonePBXService.exe or jtelTAPIMonitorService.exe For service installations, start the service, for example the jtel TAPI service or jtel Innovaphone Service. | |||
| Exchange Connector | E-Mail with an Exchange or Office 365 Server | Start the jtelEWSMailService service. | |||
| IMAP Connector | E-Mail with an IMAP(S) Server | Start the jtelIMAPMailService service. | |||
| acd-jb4jb1-6 | Wildfly | Anything | sudo systemctl start wildfly For installations not using systemctl: sudo service wildfly start | ||
| acd-chat1+2 | Chat Server | CHAT | sudo systemctl start jtel-clientmessenger For installations not using systemctl: sudo service jtel-clientmessenger start | ||
| acd-api1+2 | REST API | RESTAnything | sudo systemctl start wildflyjtelrest For installations not using systemctl: sudo service wildfly jtelrest start | ||
| acd-chat2 | Chat Server | dbm1+2 | Platform UDP Listener | SOAPCHAT | sudo systemctl start jtel-clientmessengerlistener For installations not using systemctl: sudo service jtel-clientmessenger listener start |
Step 9 - Stop the AcdGroupDistribute Daemon on acd-
...
sudo systemctl start jtelrest
For installations not using systemctl: sudo service jtelrest start
...
sudo systemctl start jtel-listener
tel2
Since acd-tel1 is active again, if the Daemon AcdGroupDistribute.r5 is was running on acd-tel1 before, it must be stopped on acd-tel2 to ensure that calls to acd-groups will still be routed properly by the routing-algorithm.
...
Step 11 - Reactivate monitoring
The monitoring is now reactivated on for all machines on VM-Host 2
Step 12 - Ensure system functionality
...
If all tests are successful, the system is now running only on VM Host 2 and fully operational.
Step 13 - Remove backups
Backups or snapshots can now be deleted