System Health Check

Make sure the VPN is running OK, and that all systems are accessible by SSH and RDP.

Connect via the IP address of the load balancer (if redundant, connect via the shared IP address).

Login as sysadmin, switch to a user with admin rights, and check a few views:

• Agent Home
• Mini Client
• Supervisor
• Execute a report

Make a test call.

Check that it is received by the system and that audio is heard.

If possible:

• Make a call to an ACD queue which is forwarded to an agent.
• Check the agent's Agent Home indicates the incoming call and that the status of the call changes from ringing to busy (this indicates, the webserver is communicating with the platform UDP listener cluster).
• Check that two way audio is heard.

Make a test chat.

Make sure it is distributed to an agent, and that the agent can reply.

No virus scanner should be installed on the telephony server. Virus scanners can have negative impact on live telephony systems and should not scan in and outgoing traffic in the network, or the filesystems used by jtel software. 

Use:

uptime

to check for how long the system has been up.

If it has been restarted, then check the following for why:

less /var/log/messages

You may need to go back to an older file than the current log file to see why.

Use:

df -h

to make sure the disk is not full.

Particularly on the database systems make sure this space will not run out soon.

Note: on some systems, /var/lib/mysql may be mounted separately.

Use: mount to verify how it is mounted or cat /etc/fstab

Use both of the following to identify CPU load problems:

top

sar (historical)

sar -u 1 (real-time)

Use the following commands to make sure the time sync is OK:

ntpq -p (older systems)

chronyc sources (newer systems)

timedatectl

Sometimes, if a file system error has occurred, linux will either:

• Not mount it at all (this is easy to see / find)
• Mount it read-only (this is not so obvious)

Try writing to a file:

cat << EOFF > /root/test.txt
Test
EOFF
cat /root/test.txt

Check the task manager to make sure no process is running very high CPU. 

Note: on a loaded system 50% or more CPU is perfectly fine, as long as it is either:

• 8-Server
• SIP and RTP Stack (Aculab)

DB Master-Master

DB Slaves

Make sure replication is running:

SHOW SLAVE STATUS\G

Make sure MySQL is not crashing:

less /var/log/mysqld.log

Check the status of the pcs cluster with:

pcs status

Make sure one of the load balancers has the shared IP address, and that all resources are started.

Open haproxy (on both load balancers in a redundant system) using the following url:

http://<load-balancer>:7777

Make sure the expected resources are present and all up:

• Expected database servers (in a redundant solution) are up and running.
• Only one load balancer is serving connections.
• Only one of the DB masters has active connections.

Check the status of DRBD with:

drbdadm status jtelshared

Make sure it is primary/primary (for older installations) or primary secondary (for newer installations).

Check the status of the pcs cluster with:

pcs status

Make sure all of the resources are started:

• Mount of /srv/jtel/shared is started on one of the STORE servers
• Samba is started on one of the STORE servers
• The shared IP is started on one of the STORE servers

Check the status of the samba service:

smbstatus

ps -ef | grep samba

Make sure it is running. 

Check the available space on the store with 

df -h

startup.cmd batch file window present?

giHal.exe batch file window present and running?

giAcu.exe (SIP Registrations) present and running (if required)?

8Server.exe present and running?

Check errors / warnings for excessive error messages, such as ODBC errors or .r5 files not found errors.

ODBC errors can indicate a database problem (for example, a mysql crash), or a syntax error in a sql procedure.

File not found errors (particularly .r5 files) can indicate a storage problem.

If 8-Server says "Initialising" on all lines, and does not exit this status for a long time, then there may be connectivity problems to the storage.

This is best mitigated by making sure the samba credentials are correct, and that all "jtel" users have the same credentials.

Start a wireshark trace for SIP using this capture filter:

port 5060 or ip[6:2] & 0x1fff != 0

If SIP messages are being received, then the problem might be a firewall rule injection by a domain policy. Check the firewall rules to make sure the required ports are opened.

Telephony connector started and running?

Messages being received from the PBX?

Check each webserver individually using

http://<server>:8080/CarrierPortal/sysadmin/login

Make sure it is possible to login.

Make sure the logo is visible (if not, this indicates a file share mount error to /home/jtel/shared)

Make sure the certificate is OK by accessing the system via the load-balancer (if necessary, using an alias in your hosts file).

This is mandatory, for example, for SalesForce integrations.

Make sure the chat application is running:

ps -ef | grep jtel-clientmessenger

Make sure the chat application is running:

ps -ef | grep jrest