A normal operation of a jtel solution is considered among other variables if:
- The storage is mounted (available via acd-store)
- redundancy is also synchronised (DRBD is on primary / primary)
- The database replication is synchronised.
- The load balancer for the distribution of the web sessions is accessible, all web servers are available
- The telephony servers are up and processing calls
This can be examined in detail as follows.
Examination Storage
Redundant Storage
Testing that DRBD is synchronised
On one of the STORE, check if the DRBD is synchronised:
cat /proc/drbd
Expected output - DRBD is primary/primary
version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-04-26 12:10:42 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:28894328 dw:118174057 dr:74296 al:6116 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Determine which STORE is active.
pcs status
Check on the active STORE if /srv/jtel/shared is mounted
ls /srv/jtel/shared
Expected output:
Output with among others the following directories:
- Data
- JTEL
- JTELCarrierPortal
Alle Storage
Check if the smb service is running (with redundancy to the active STORE):
systemctl status smb expected output - Active: [root@test9-store2 ~]# systemctl status smb ● smb.service - Cluster Controlled smb Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/smb.service.d └─50-pacemaker.conf Active: active (running) since Tue 2019-03-19 04:02:48 CET; 3 weeks 5 days ago
From any other machine, check access to the storage
From another machine (except STORE itself), either:
ls /home/jtel/shared
Or from the Windows machine, open an Explorer to the \\acd-store\shared
directory.
When error:
- Start the SMB service
- Check if the file system is corrupt (see \var\log\messages) and take measures for repair - see man xfs_repair at xfs filesystems
- Recovery of DRBD replication and STORE cluster with redundancy - see DRBD - Maintenance and Resolve Split Brain or Node errors
Check database replication
On all database slaves, with redundancy, also on both master databases:
Login to mysql
mysql -u root -p
Check slave status
mysql> SHOW SLAVE STATUS\G
The most important points are:
Slave_IO_Running: Yes Slave_SQL_Running: Yes
Example output:
*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: acd-dbm2 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: binlog.000014 Read_Master_Log_Pos: 77769753 Relay_Log_File: mysqld-relay-bin.000028 Relay_Log_Pos: 2698 Relay_Master_Log_File: binlog.000014 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_error: Skip_Counter: 0 Exec_Master_Log_Pos: 77769753 Relay_Log_Space: 2916 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_error: Last_SQL_Errno: 0 Last_SQL_error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2 Master_UUID: f5b276fa-bb92-11e8-a4a9-005056b98358 Master_Info_File: /var/lib/mysql/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_error_Timestamp: Last_SQL_error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 1 row in set (0.35 sec)
When error
- Recovery of the Replication
- See Database Operations
Testing load balancers and web servers
Testing load balancer
The basic accessibility of the web server as well as the function of the load balancer can be checked via the haproxy Stats website.
The access takes place via:
http://acd-lb:7777
with indication of user name and password.
The screenshot shows a sample output for a system with:
- Redundant master-master-slave-slave databases
- Two web servers
The services should be green except for the 2nd database master, which is shown in light blue, as this server is configured as a backup for the first database master.
When error
Check if the haproxy service is started.
For redundancy: check cluster status with:
pcs status
and take appropriate measures.
Testing web server
The web servers can be checked individually for function by accessing and logging on to the jtel system, for example as sysadmin.
The following URL applies to https:
https://acd-lb/admin
After successful login the portal main page of the system administrator should be visible, here an example output:
When error
Restart single web server with:
service jboss restart
Testing Telephony
The telephony services are either:
- In the autostart of the respective user
- Configured as a service to the system
In a standard case (with user login), the following applications are started:
- jtel 8-Server
- GI2
Note: depending on the installation, a PBX connector may also be started.
Normal readiness and processing is shown in the following screenshot. Note the light green active line and the system messages in yellow.
The following services are visible:
- jtel 8-Server
- GI2 Dienst
- Cluster Listener Dienst
- Innovaphone PBX Connector
When error
Close all applications and restart them via the autostart group.
Or reboot the system.