A normal operation of a jtel solution is considered among other variables if:

• The storage is mounted (available via acd-store) 
  
  • redundancy is also synchronised (DRBD is on primary / primary)
• The database replication is synchronised.
• The load balancer for the distribution of the web sessions is accessible, all web servers are available
• The telephony servers are up and processing calls

This can be examined in detail as follows.

Examination Storage

Redundant Storage

Testing that DRBD is synchronised

On one of the STORE, check if the DRBD is synchronised:

cat /proc/drbd

Expected output - DRBD is primary/primary

version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-04-26 12:10:42
 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
 ns:0 nr:28894328 dw:118174057 dr:74296 al:6116 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Determine which STORE is active.

pcs status

Check on the active STORE if /srv/jtel/shared is mounted

ls /srv/jtel/shared

Expected output:

Output with among others the following directories:

• Data
• JTEL
• JTELCarrierPortal

Alle Storage

Check if the smb service is running (with redundancy to the active STORE):

systemctl status smb
expected output - Active:

[root@test9-store2 ~]# systemctl status smb
● smb.service - Cluster Controlled smb
Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/smb.service.d
└─50-pacemaker.conf
Active: active (running) since Tue 2019-03-19 04:02:48 CET; 3 weeks 5 days ago

From any other machine, check access to the storage

From another machine (except STORE itself), either:

ls /home/jtel/shared

Or from the Windows machine, open an Explorer to the \\acd-store\shared directory.

When error:

• Start the SMB service
• Check if the file system is corrupt (see \var\log\messages) and take measures for repair - see man xfs_repair at xfs filesystems
• Recovery of DRBD replication and STORE cluster with redundancy - see DRBD - Maintenance and Resolve Split Brain or Node errors

Check database replication 

On all database slaves, with redundancy, also on both master databases:

Login to mysql

mysql -u root -p

Check slave status

mysql> SHOW SLAVE STATUS\G

The most important points are: 

Slave_IO_Running: Yes
 Slave_SQL_Running: Yes

 Example output:

*************************** 1. row ***************************
 Slave_IO_State: Waiting for master to send event
 Master_Host: acd-dbm2
 Master_User: repl
 Master_Port: 3306
 Connect_Retry: 60
 Master_Log_File: binlog.000014
 Read_Master_Log_Pos: 77769753
 Relay_Log_File: mysqld-relay-bin.000028
 Relay_Log_Pos: 2698
 Relay_Master_Log_File: binlog.000014
 Slave_IO_Running: Yes
 Slave_SQL_Running: Yes
 Replicate_Do_DB:
 Replicate_Ignore_DB:
 Replicate_Do_Table:
 Replicate_Ignore_Table:
 Replicate_Wild_Do_Table:
 Replicate_Wild_Ignore_Table:
 Last_Errno: 0
 Last_error:
 Skip_Counter: 0
 Exec_Master_Log_Pos: 77769753
 Relay_Log_Space: 2916
 Until_Condition: None
 Until_Log_File:
 Until_Log_Pos: 0
 Master_SSL_Allowed: No
 Master_SSL_CA_File:
 Master_SSL_CA_Path:
 Master_SSL_Cert:
 Master_SSL_Cipher:
 Master_SSL_Key:
 Seconds_Behind_Master: 0
 Master_SSL_Verify_Server_Cert: No
 Last_IO_Errno: 0
 Last_IO_error:
 Last_SQL_Errno: 0
 Last_SQL_error:
 Replicate_Ignore_Server_Ids:
 Master_Server_Id: 2
 Master_UUID: f5b276fa-bb92-11e8-a4a9-005056b98358
 Master_Info_File: /var/lib/mysql/master.info
 SQL_Delay: 0
 SQL_Remaining_Delay: NULL
 Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
 Master_Retry_Count: 86400
 Master_Bind:
 Last_IO_error_Timestamp:
 Last_SQL_error_Timestamp:
 Master_SSL_Crl:
 Master_SSL_Crlpath:
 Retrieved_Gtid_Set:
 Executed_Gtid_Set:
 Auto_Position: 0
1 row in set (0.35 sec)

When error

• Recovery of the Replication
• See Database Operations

Testing load balancers and web servers

Testing load balancer

The basic accessibility of the web server as well as the function of the load balancer can be checked via the haproxy Stats website.

The access takes place via:

http://acd-lb:7777

with indication of user name and password.

The screenshot shows a sample output for a system with: 

• Redundant master-master-slave-slave databases
• Two web servers

The services should be green except for the 2nd database master, which is shown in light blue, as this server is configured as a backup for the first database master.

Image Added

When error

Check if the haproxy service is started.

For redundancy: check cluster status with:

pcs status

and take appropriate measures.

Testing web server

The web servers can be checked individually for function by accessing and logging on to the jtel system, for example as sysadmin.

The following URL applies to https:

https://acd-lb/admin

After successful login the portal main page of the system administrator should be visible, here an example output:

Image Added

When error

Restart single web server with:

service jboss restart

Testing Telephony

The telephony services are either:

• In the autostart of the respective user
• Configured as a service to the system

In a standard case (with user login), the following applications are started:

• jtel 8-Server
• GI2

Note: depending on the installation, a PBX connector may also be started.

Normal readiness and processing is shown in the following screenshot. Note the light green active line and the system messages in yellow.

The following services are visible:

• jtel 8-Server
• GI2 Dienst
• Cluster Listener Dienst
• Innovaphone PBX Connector

Image Added

When error

Close all applications and restart them via the autostart group.

Or reboot the system.

Ein normaler Betrieb einer jtel Lösung liegt unter anderen vor wenn:

• Das Storage gemountet ist (über acd-store verfügbar), 
  • Bei Redundanz auch synchron ist (DRBD ist auf primary / primary)
• Die Datenbankreplikation ist syncrhon
• Der Load-Balancer für die Verteilung der Websessions ist erreichbar