Under Construction
This site is currently under construction. Contained Information is subject to change.
Introduction
A jtel ACD consists of a minimum 2 of virtual machines and may grow to sizes of 30 or more, in concurrence with for example redundancy or performance requirements. Based on a systems architecture, there are certain dependencies between the services, so a clean startup and shutdown is required and only done in a certain order to prevent problems during and after startup. The following page describes the various possible scenarios for proper shutdown and startup procedures, depending on the systems architecture.
On Premise
Caution for on premise jtel ACD systems:
Shutting down and starting the virtual machines of your jtel ACD back is not part of the jtel service contract and may incur service fees as a result. For further information, or to book a service appointment, please contact us at service@jtel.de
Page Layout
This page is split into three rows, each named after a different type of sizing. The sizing variants are Small, Medium and Large
General
The Hostnames of an ACDs jtel virtual machines may not be in concurrence with the aliases displayed below.
| Alias | Signifies | Shutdown Priority | Startup Priority |
|---|---|---|---|
| acd-dbm | Database Master | Third | Third |
| acd-dbm1 | First Database Master | Third | Third |
| acd-dbm2 | Second Database Master | Third | Third |
| acd-dbs | Database Slave | Second | Fourth |
| acd-dbs1 | First Database Slave | Second | Fourth |
| acd-dbs2 | Second Database Slave | Second | Fourth |
| acd-dbr | Reporting Database | Second | Fourth |
| acd-lb | The Load Balancer | Fourth | Second |
| acd-lb1 | First Load Balancer | Fourth | Second |
| acd-lb2 | Second Load Balancer | Fourth | Second |
| acd-store | The File Storage | Last | First |
acd-tel1 | The Telephony Machine(s) Numbered from 1 ... N | First | Last |
acd-jb1 | The Webserver Machine(s) Numbered from 1 ... N | First | Last |
| acd-api | The REST-API | First | Last |
| acd-chat | Chat and or WhatsApp | First | Last |
| acd-chatbot | Chatbot | First | Last |
Dependencies
All services of the platform are dependent on two central entities:
- The Storage(s)
- The Database(s)
To boot the solution, the storage should always be booted first. If this is external to the solution, continue with the next step. Otherwise, the computer (or in case of redundancy - both computers) is started on which the storage is located. This is usually the load balancer, but for larger systems a separate storage machine could have been implemented. Afterwards, the master database is booted. As soon as its is finished, the slave can be started up. Finally, the telephony servers and web servers can be started up in any order, even simultaneously.
Small V1
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-tel | acd-store/lb/dbm/dbs/dbr/jb |
| 2 | acd-store/lb/dbm/dbs/dbr/jb | acd-tel |
Small V2
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-tel | acd-store/lb/jb |
| 2 | acd-dbm/dbs/dbr | acd-dbm/dbs/dbr |
| 3 | acd-store/lb/jb | acd-tel |
Small V3
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-tel | acd-store/lb |
| 2 | acd-jbN | acd-dbm/dbs/dbr |
| 3 | acd-dbm/dbs/dbr | acd-jbN |
| 4 | acd-store/lb | acd-tel |
Shutdown Dependencies for Small, Medium and Large systems
Small V1
The two computers should be shutdown in the correct order. You do not have to wait until acd-tel is shutdown fully before starting the shutdown on acd-store/lb/dbm/dbs/dbr/jb, but the order should be kept anyhow.
Small V2
First shutdown acd-tel. You do not have to wait until acd-tel is down before starting the shutdown at acd-dbm/dbs/dbr. Wait for acd-dbm/dbs/dbr to be shutdown before starting the shutdown on acd-store/lb/jb.
Small V3
First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before starting the shutdown at acd-dbm/dbs/dbr. Wait for acd-dbm/dbs/dbr to be shutdown before starting the shutdown on acd-store/lb.
Medium V1
First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before starting the shutdown at acd-dbs/dbr. Wait until acd-dbs/dbr is shutdown before starting the shutdown on acd-dbm. Wait until acd-dbm is shutdown before starting the shutdown on acd-store/lb.
Medium V2
First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before starting the shutdown at acd-dbs/dbr. Wait until acd-dbs/dbr is shutdown before starting the shutdown on acd-dbm. Wait until acd-dbm is shutdown before starting the shutdown on acd-lb. Wait until acd-lb is down before starting the shutdown on acd-store
Large V1
When shutting down a redundant database architecture, the inactive machines are shutdown first. Therefore, before a shutdown procedure can begin, the active machines must be known. The table at Large V1 assumes that acd-XXX is currently the active master database and displays the shutdown procedure according to that.
Large V2
When shutting down a redundant load balancing architecture, the inactive machines are shutdown first. Therefore, before a shutdown procedure can begin, the active machines must be known. The table at Large V2 assumes that acd-XXX is currently the active load balancer and displays the shutdown procedure according to that.
Large V3
When shutting down a redundant storage architecture, the inactive machine is shutdown first. Therefore, before a shutdown procedure can begin, the active machine must be known. The table at Large V3 assumes that acd-XXX is currently the active load balancer and displays the shutdown procedure according to that.
Medium V1
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-tel | acd-store/lb |
| 2 | acd-jbN | acd-dbm |
| 3 | acd-dbs/dbr | acd-dbs/dbr |
| 4 | acd-dbm | acd-jbN |
| 5 | acd-store/lb | acd-tel |
Medium V2
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-tel | acd-store |
| 2 | acd-jbN | acd-lb |
| 3 | acd-dbs/dbr | acd-dbm |
| 4 | acd-dbm | acd-dbs/dbr |
| 5 | acd-lb | acd-jbN |
| 6 | acd-store | acd-tel |
Startup Dependencies for Small, Medium and Large systems
Small V1
The two computers must be started in the correct order. Wait until acd-store/lb/dbm/dbs/dbr/jb is up before starting acd-tel.
Small V2
First start acd-store/lb/jb and wait until it is up. Start acd-dbm/dbs/dbr and wait until it is up. Start acd-tel afterwards.
Small V3
First start acd-store/lb and wait until it is up. Start acd-dbm/dbs/dbr and wait until it is up. Start acd-jbN and acd-tel afterwards in no particular order.
Medium V1
First start acd-store/lb and wait until it is up. Start acd-dbm and wait until it is up. Start acd-dbs/dbr and check the MySQL replication status after it is up. Start acd-jbN and acd-tel afterwards in no particular order.
Medium V2
First start acd-store and wait until it is up. Start acd-lb and wait until it is up. Start acd-dbm and wait until it is up. Start acd-dbs/dbr and check the MySQL replication status after it is up. Start acd-jbN and acd-tel afterwards in no particular order.
Large V1
Large V2
Large V3
When starting a Small or Medium system,
Storage (Separates Storage acd-store or Load-Balancer acd-lb).
Wait until Storage is up.
With separate storage, now start the load balancer acd-lb.
There is no need to wait here.
acd-dbm - Database Master .
Wait until DB-Master is up.
acd-dbs - Database Slave- .
Wait until DB slave is up.
Boot acd-jb1 ... acd-jb(x) aswell as acd-tel1 ... acd-tel(x)
Checks
Log on to the Web via the load balancer.
Log on OK?
If Not, check /home/jtel/jboss-(version)/standalone/log/server.log
ESSENTIAL: Logo visible in the web application?
If not, the storage is not up. Check with mount on the web servers. If necessary restore mount /home/jtel/shared Mount.
Call System.
Calls are put through?
If not, check the telephony server.
Look for red messages in the telephony server and act accordingly.
Calls are signalled in the Agent Home or Mini Client?
If not, check the Hazelcast cluster (PlatformListener for telephony and web server).
Large V1 - Redundant Databases
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-telN | acd-store |
| 2 | acd-jbN | acd-lb |
| 3 | acd-dbs1/dbr1 | acd-dbm2 |
| 4 | acd-dbs2/dbr2 | acd-dbm1 |
| 5 | acd-dbm1 | acd-dbs2/dbr2 |
| 6 | acd-dbm2 | acd-dbs1/dbr1 |
| 7 | acd-lb | acd-jbN |
| 8 | acd-store | acd-telN |
Large V2 - Redundant Databases + Load Balancing
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-telN | acd-store |
| 2 | acd-jbN | acd-lb2 |
| 3 | acd-dbs1/dbr1 | acd-lb1 |
| 4 | acd-dbs2/dbr2 | acd-dbm2 |
| 5 | acd-dbm1 | acd-dbm1 |
| 6 | acd-dbm2 | acd-dbs2/dbr2 |
| 7 | acd-lb1 | acd-dbs1/dbr1 |
| 8 | acd-lb2 | acd-jbN |
| 9 | acd-store | acd-telN |
Large V3 - Redundant Databases + Load Balancing + Storage
| Steps 1 to N | Shutdown | Startup |
|---|---|---|
| 1 | acd-telN | acd-store2 |
| 2 | acd-jbN | acd-store1 |
| 3 | acd-dbs1/dbr1 | acd-lb2 |
| 4 | acd-dbs2/dbr2 | acd-lb1 |
| 5 | acd-dbm1 | acd-dbm2 |
| 6 | acd-dbm2 | acd-dbm1 |
| 7 | acd-lb1 | acd-dbs2/dbr2 |
| 8 | acd-lb2 | acd-dbs1/dbr1 |
| 9 | acd-store1 | acd-jbN |
| 10 | acd-store2 | acd-telN |
Startup Dependencies for Large Systems
Shutting down redundant clusters is easier than starting them back up.
Some components must be checked directly after starting up, before work can continue.
Database:
- MySQL Replication between all replication partners
Load Balancing:
- pcs cluster status
Storage:
- pcs cluster status
- drbd status
Startup and Shutdown - Redundant components
If the entire system is shut down, some components may not be immediately available or manual intervention may be necessary.
Storage (Redundant) - CentOS
This checklist is not viable for installations that use the new DRBD construct of Primary/Secondary. It is viable only for constructs that use Primary/Primary
Shut down (only one)
Always shut down one after the other.
With:
pcs status
determine which node is Primary.
On the Secondary Node:
drbdadm secondary jtelshared shutdown now
Then shut down the Primary Node.
shutdown now
Boot up (only one)
After booting the Secondary Node, check the status. It should be Secondary and UpToDate
pcs status
Check Sync status with :
cat /proc/drbd
After the sync is done, if necessary, and is ready:
drbdadm primary jtelshared
Check status with: It should be Primary/Primary + UpToDate/UpToDate
cat /proc/drbd
Shut down (both)
With:
pcs status
determine which node is active (the node on which Samba and the virtual IP is running). This is the Primary Node.
Set the secondary Node to secondary with:
drbdadm secondary jtelshared shutdown now
On the Secondary Node first, then Primary:
shutdown now
Start up (both)
Boot the Primary Node first
Check DRBD sync Status and pcs cluster status
Check with:
pcs status cat /proc/drbd
The Primary Node should be primary, and Secondary Node should be secondary. Disc should be UpToDate
Then on Secondary Node
drbdadm primary jtelshared
Check with:
cat /proc/drbd
Both nodes should be set to Primary/Primary + UpToDate/UpToDate
Then check if everything works a final time:
pcs status cat /proc/drbd