Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Medium V2

Medium V1

First start acd-store/lb and wait until it is up. Start acd-dbm and wait until it is up. Start acd-dbs/dbr and check the MySQL replication status after it is up. Start acd-jbN and acd-tel afterwards in no particular order.

Medium V2

First start acd-store and wait until it is up. Start acd-lb and wait until it is up. Start acd-dbm and wait until it is up. Start acd-dbs/dbr and wait until it is up. Start acd-jbN and acd-tel afterwards in no particular order.

Large V1

Sv translation
languageen

Introduction

A jtel ACD consists of a minimum 2 of virtual machines and may grow to sizes of 30 or more, in concurrence with for example redundancy or performance requirements. Based on a systems architecture, there are certain dependencies between the services, so a clean startup and shutdown is required and only done in a certain order to prevent problems during and after startup. The following page describes the various possible scenarios for proper shutdown and startup procedures, depending on the systems architecture. 

Info
titleOn Premise

Caution for on premise jtel ACD systems:

Shutting down and starting the virtual machines of your jtel ACD back is not part of the jtel service contract and may incur service fees as a result. For further information, or to book a service appointment, please contact us at service@jtel.de 

Page Layout

The pages are split into a parent and child pages. This parent page contains general information. The child pages each contain information about the specific procedure, depending on the sizing of the system. The sizing variants are Small, Medium and Large

General

Info

The Hostnames of an ACDs jtel virtual machines may not be in concurrence with the aliases displayed below.


AliasSignifiesShutdown PriorityStartup Priority
acd-dbmDatabase MasterThirdThird
acd-dbm1First Database MasterThirdThird
acd-dbm2Second Database MasterThirdThird
acd-dbsDatabase SlaveSecondFourth
acd-dbs1First Database SlaveSecondFourth
acd-dbs2Second Database SlaveSecondFourth
acd-dbrReporting DatabaseSecondFourth
acd-lbThe Load BalancerFourthSecond
acd-lb1First Load BalancerFourthSecond
acd-lb2Second Load BalancerFourthSecond
acd-storeThe File StorageLastFirst

acd-tel1
...
acd-telN

The Telephony Machine(s) Numbered from 1 ... NFirstLast

acd-jb1
...
acd-jbN

The Webserver Machine(s) Numbered from 1 ... NFirstLast
acd-apiThe REST-APIFirstLast
acd-chatChat and or WhatsAppFirstLast
acd-chatbotChatbot FirstLast

Dependencies

All services of the platform are dependent on two central entities:

  • The Storage(s)
  • The Database(s)

To boot the solution, the storage should always be booted first. If this is external to the solution, continue with the next step. Otherwise, the computer (or in case of redundancy - both computers) is started on which the storage is located. This is usually the load balancer, but for larger systems a separate storage machine could have been implemented. Afterwards, the master database is booted. As soon as its is finished, the slave can be started up. Finally, the telephony servers and web servers can be started up in any order, even simultaneously.

Startup Dependencies for Large Systems

Shutting down redundant clusters is generally easier than starting them back up.

If problems occur after starting back up, some components must be checked.

Database:

  • MySQL Replication between all replication partners

Load Balancing:

  • pcs cluster status

Storage:

  • pcs cluster status
  • drbd status

Checks after starting a system

To ensure proper system functionality after a boot, tests are required. Follow the guideline on this page: System Health Check

Medium V1

First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before starting the shutdown at acd-dbs/dbr. Wait until acd-dbs/dbr is shutdown before shutting acd-dbm down. Wait until acd-dbm is shutdown before shutting acd-store/lb down.

Medium V2

First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before shutting acd-dbs/dbr down. Wait until acd-dbs/dbr is shutdown before shutting acd-dbm down. Wait until acd-dbm is shutdown before shutting acd-lb down. Wait until acd-lb is shutdown before shutting acd-store down.

Large V1 - Redundant Databases

When shutting down a redundant database architecture, the inactive machines are shutdown first. Therefore, before a shutdown procedure can begin, the active machines must be known. The table at Large V1 assumes that acd-dbm2 is currently the active master database and displays the shutdown procedure according to that. Only start the next step if the previous shutdown was completed successfully:

First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before shutting acd-dbs1/dbr1 and acd-dbs2/dbr2 down simultaneously, but in correct order. Wait until acd-dbs1/dbr1 and acd-dbs2/dbr2 are down until shutting acd-dbm1 down. Wait until acd-dbm1 is down before shutting down acd-dbm2. Wait until acd-dbm2 is down before shutting down acd-lb. Wait until acd-lb is down before shutting acd-store down.

Large V2 - Redundant Databases + Load Balancing

When shutting down a redundant load balancing architecture, the inactive machines are shutdown first. Therefore, before a shutdown procedure can begin, the active machines must be known. The table at Large V2 assumes that acd-lb2 is currently the active load balancer and displays the shutdown procedure according to that. Only start the next step if the previous shutdown was completed successfully:

First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before shutting acd-dbs1/dbr1 and acd-dbs2/dbr2 down simultaneously, but in correct order. Wait until acd-dbs1/dbr1 and acd-dbs2/dbr2 are down until shutting acd-dbm1 down. Wait until acd-dbm1 is down before shutting down acd-dbm2. Wait until acd-dbm2 is down before shutting down acd-lb1. Wait until acd-lb1 is down before shutting down acd-lb2. Wait until acd-lb2 is down before shutting down acd-store.

Large V3 - Redundant Databases + Load Balancing + Storage

When shutting down a redundant storage architecture, the inactive machine is shutdown first. Therefore, before a shutdown procedure can begin, the active machine must be known. The table at Large V3 assumes that acd-store2 is currently the active storage and displays the shutdown procedure according to that. Only start the next step if the previous shutdown was completed successfully:

First shutdown acd-tel and acd-jbN in no particular order. You do not have to wait until acd-tel and acd-jbN are down before shutting acd-dbs1/dbr1 and acd-dbs2/dbr2 down simultaneously, but in correct order. Wait until acd-dbs1/dbr1 and acd-dbs2/dbr2 are down until shutting acd-dbm1 down. Wait until acd-dbm1 is down before shutting down acd-dbm2. Wait until acd-dbm2 is down before shutting down acd-lb1. Wait until acd-lb1 is down before shutting down acd-lb2. Wait until acd-lb2 is down before shutting down acd-store1. Wait until acd-store1 is down until shutting acd-store2 down.

Medium

Medium V1

Steps 1 to N

Shutdown

Startup

1

acd-tel

acd-store/lb

2

acd-jbN

acd-dbm

3

acd-dbs/dbr

acd-dbs/dbr

4

acd-dbm

acd-jbN

5

acd-store/lb

acd-tel

Steps 1 to N

Shutdown

Startup

1

acd-tel

acd-store

2

acd-jbN

acd-lb

3

acd-dbs/dbr

acd-dbm

4

acd-dbm

acd-dbs/dbr

5

acd-lb

acd-jbN

6

acd-store

acd-tel

Warning

When starting up a redundant database architecture, replication status must be checked on all replication partners of the system. Additional information can be found here: System Health Check

First start acd-store and wait until it is up. Start acd-lb and wait until it is up. Start acd-dbm2 and wait until it is up. Start acd-dbm1 and wait until it is up. start acd-dbs2/dbr2 and wait until it is up. Start acd-dbs1/dbr1 and wait until it is up. Start acd-jbN and acd-tel afterwards in no particular order.

Large V2

Warning

When starting up a redundant load balancing architecture, the pcs cluster status must be checked on all pcs cluster members. Additional information can be found here: System Health Check

First start acd-store and wait until it is up. Start acd-lb2 and wait until it is up. Start acd-lb1 and wait until it is up. Start acd-dbm2 and wait until it is up. Start acd-dbm1 and wait until it is up. start acd-dbs2/dbr2 and wait until it is up. Start acd-dbs1/dbr1 and wait until it is up. Start acd-jbN and acd-tel afterwards in no particular order.

Large V3

Warning

When starting up a redundant storage architecture, the pcs cluster and drbd replication status must be checked on all pcs cluster members. Additional information can be found here: System Health Check

First start acd-store2 and wait until it is up. Start acd-store1 and wait until it is up. Start acd-lb2 and wait until it is up. Start acd-lb1 and wait until it is up. Start acd-dbm2 and wait until it is up. Start acd-dbm1 and wait until it is up. start acd-dbs2/dbr2 and wait until it is up. Start acd-dbs1/dbr1 and wait until it is up. Start acd-jbN and acd-tel afterwards in no particular order.

Large

Large V1 - Redundant Databases

Steps 1 to NShutdownStartup1acd-telNacd-store2acd-jbNacd-lb3acd-dbs1/dbr1acd-dbm24acd-dbs2/dbr2acd-dbm15acd-dbm1acd-dbs2/dbr26acd-dbm2acd-dbs1/dbr17acd-lbacd-jbN8acd-storeacd-telN

Large V2 - Redundant Databases + Load Balancing

Steps 1 to NShutdownStartup1acd-telNacd-store2acd-jbNacd-lb23acd-dbs1/dbr1acd-lb14acd-dbs2/dbr2acd-dbm25acd-dbm1acd-dbm16acd-dbm2acd-dbs2/dbr27acd-lb1acd-dbs1/dbr18acd-lb2acd-jbN9acd-storeacd-telN

Large V3 - Redundant Databases + Load Balancing + Storage

Steps 1 to NShutdownStartup1acd-telNacd-store22acd-jbNacd-store13acd-dbs1/dbr1acd-lb24acd-dbs2/dbr2acd-lb15acd-dbm1acd-dbm26acd-dbm2acd-dbm17acd-lb1acd-dbs2/dbr28acd-lb2acd-dbs1/dbr19acd-store1acd-jbN10acd-store2acd-telN
Sv translation
languagede


Warning
titleAlt

Diese Seite ist veraltet und aktuell nur in englischer Version verfügbar.

Hoch- und Runterfahren

Alle Systeme sind so gestaltet, dass Sie hochfahren und alle Dienste selbstständig starten.

Dennoch ergeben sich gewisse Abhängigkeiten zwischen den Diensten, sodass ein sauberes Hoch- und Herunterfahren am besten geschieht, wenn eine gewisse Reihenfolge betrachtet wird. 

Hochfahren

Abhängigkeiten

Alle Dienste der Plattform sind von zwei zentrale Entitäten abhängig:

  • Das Storage
  • Die Datenbank(en)

Für das Hochfahren der Lösung, sollte immer zuerst das Storage hochgefahren werden. Wenn dies Extern zur Lösung ist, dann fährt man mit den nächsten Schritt fort. Ansonsten, wird der Rechner (oder bei Redundanz - beide Rechner) gestartet auf den das Storage ist. Dies ist in der Regel der Load-Balancer, bei größere Systeme kann jedoch eine separate Storage-Maschine implementiert worden sein.

Anschließend, wird die Master Datenbank hochgefahren. Sobald dieser oben ist, kann der Slave hochgefahren werden.

Zum Schluss können die Telefonieserver und Webserver in einer beliebigen Reihenfolge, auch gleichzeitig, hochgefahren werden.

Reihenfolge

Im Folgenden wird von einer nicht redundante Lösung ausgegangen. Der Betrieb einer redundanten Lösung bedarf eine gesonderte Schulung.

Somit ergibt sich folgende Startreihenfolge:

  1. Storage (Separates Storage acd-store oder Load-Balancer acd-lb). 
    Warten bis Storage oben ist.
  2. Bei separater Storage, nun den Load-Balancer acd-lb hochfahren.
    Hier muss nicht gewartet werden.
  3. acd-dbm - Datenbankmaster - hochfahren.
    Warten bis DB-Master oben ist.
  4. acd-dbs - Datenbankslave - hochfahren.
    Warten bis DB-Slave oben ist.
  5. acd-jb1 ... acd-jb(x) sowie acd-tel1 ... acd-tel(x) hochfahren.

Prüfungen

  1. Am Web über den Load-Balancer anmelden. 
    1. Anmeldung OK?
      Wenn nicht, /home/jtel/jboss-(version)/standalone/log/server.log prüfen.
    2. GANZ WICHTIG: Logo in der Webanwendung sichtbar?
      Wenn nicht, ist das Storage nicht oben. Mit mount auf den Webservern prüfen. Ggf. mit mount /home/jtel/shared Mount wiederherstellen.
  2. System anrufen.
    1. Anrufe werden durchgestellt?
      Wenn nicht, Telefonieserver prüfen.
      Nach rote Meldungen im Telefonieserver schauen und entsprechend handeln.
    2. Anrufe werden im Agent-Home bzw. Mini-Client signalisiert?
      Wenn nicht, Hazelcast-Cluster (PlatformListener auf Telefonie, sowie Webserver) prüfen.

Herunterfahren

Reihenfolge

Das Herunterfahren geschieht in der umgekehrten Reihenfolge:

  1. Alle Webserver acd-jb1 ... acd-jb(x) und Telefonieserver acd-tel1 ... acd-tel(x) herunterfahren.
    Schritt fertigstellen bzw. sicherstellen dass die Rechner wirklich am herunterfahren sind bevor man weitermacht.
  2. Datenbank Slave acd-dbs herunterfahren.
  3. Datenbank Master acd-dbm herunterfahren.
  4. Load Balancer acd-lb herunterfahren.
  5. Bei separater Storage, acd-store herunterfahren.

Zwischen den einzelnen Schritten ist es nicht zwingend notwendig zu warten, ledeglich Schritt 1 sollte insgesamt abgeschlossen sein bevor man fortfährt.

Hoch- und Runterfahren - Redundante Komponeten

Wenn das gesamte System heruntergefahren wird, kann es sein, dass manche Komponenten nicht sofort verfügbar sind, bzw. dass ein manueller Eingriff notwendig ist.

Storage (Redundant)

Runterfahren (nur einer)

Immer eins nacheinander herunterfahren.

Mit:

Translations Ignore


Code Block
pcs status 


feststellen welcher Node aktiv ist.

Auf den anderen node:

Translations Ignore


Code Block
drbdadm down jtelshared


Dann diesen Node herunterfahren.

Hochfahren (nur einer)

Wurde nur ein Node neu gestartet, dann kann der Betrieb wieder aufgenommen werden indem man auf den Secondary folgendes nach dem Neustart eingibt:

Translations Ignore


Code Block
drbdadm up jtelshared


Sync status prüfen mit:
Translations Ignore


Code Block
cat /proc/drbd


Nachdem der Sync ggf. erfolgt ist, und fertig ist:

Translations Ignore


Code Block
drbdadm primary jtelshared


Runterfahren (beide)

Mit:

Translations Ignore


Code Block
pcs status


feststellen welcher Node aktiv ist (der Node auf den das Samba sowie die virtuelle IP läuft). Dies ist der Primary Node. 

Mit:

Translations Ignore


Code Block
pcs cluster stop --all


den Cluster auf inaktiv setzen.

Auf auf den Secondary zuerst, dann Primary:

Translations Ignore


Code Block
drbdadm down jtelshared
shutdown now



Hochfahren (beide)

Beide nodes booten.

Auf beide Nodes (Primary zuerst):

Translations Ignore


Code Block
drbdadm up jtelshared


Prüfen mit:

Translations Ignore


Code Block
cat /proc/drbd


Beide Nodes sollten auf Secondary stehen, aber kein Sync sollte erfolgen.


Dann auf beide Nodes (Primary zuerst):

Translations Ignore


Code Block
drbdadm primary jtelshared


Prüfen mit: 
Translations Ignore


Code Block
cat /proc/drbd


Beide Nodes sollten auf Primary stehen, aber kein Sync sollte erfolgen.

Dann auf den Primary:

Translations Ignore


Code Block
pcs cluster start --all
pcs resource cleanup 



Dann prüfen ob alles läuft:

Translations Ignore


Code Block
pcs status




...