IntroductionThis page describes the process of rebuilding a Master-Master DB & Replication. Since the MySQL Dump command locks the tables, there is no need to create it when there is no traffic on the machine. It can be done during operational hours. With --master-data the command mysqldump stores the correct position for inserting the replication on the slave server. ProcedureDetermine Good MasterIn a system with Master-Master replication, one of the masters is active and processing queries from the other jtel cluster members. This is the server from which the MySQL Dump is taken, since this master has the latest data and is up to date. Info |
---|
Further down in this page, the good master-database and server will be referred to as the GOOD MASTER GOOD SLAVE Further down in this page, the broken server will be referred to as the BROKEN MASTER BROKEN SLAVE |
Haproxy configurationDetermine good server Zuerst muss entschieden werden, welcher der "guter" Server ist. Wenn HAPROXY im Betrieb ist, dann ist der guter Master der auf den die Daten derzeit geschrieben werden. HAPROXY umstellenIf there is a HAPROXY, then remove the servers on the broken master BROKEN MASTER side from the distribution (also the slave BROKEN SLAVE on this side). On BOTH Master Server Translations Ignore |
---|
Make a backup of the good Master DB on the BROKEN Master Server Previous to Release 3.12: Translations Ignore |
---|
Code Block |
---|
mysqldump -h<GOOD_MASTER> -uroot -p<PASSWORD> --single-transaction --master-data=2 --databases JTELWeb JTELStats JTELLog --add-drop-database --add-drop-table --events --routines --triggers > master.sql |
STOP SLAVELogin to MySQL on both the GOOD MASTER and the BROKEN MASTER to stop the slave SQL. Leave MySQL again afterwards. Use the following commands for this: Code Block |
---|
mysql -uUSER -pPWD
STOP SLAVE;
QUIT; |
Phase 1 - MySQL DumpA MySQL Dump of the GOOD MASTER is now created. Perform the following steps to create a MySQL Dump and save it to the STORE:
Warning |
---|
| The mysqldump command is different, depending on the jtel portal release, as well as the MySQL software release installed on the databases. All different options and how to find out which one to choose is specified below. |
Warning |
---|
title | Master-Master Replication |
---|
| The mysqldump commands on this page can NOT be used to realign a master-slave replication. Visit the following page for that description Role DATA - Simple Master / Slave |
jtel Portal software releaseLog in to the Load Balancer of the cluster and execute the following commands as the jtel user Code Block |
---|
# Find out which software release is installed
cd /srv/jtel/shared/JTELCarrierPortal
git status
# If /srv/jtel/.. does not exist on the load balancer, attempt this
cd /home/jtel/shared/JTELCarrierPortal
git status
# Expected output
release-stable/3.XX |
Create Backup Directory Info |
---|
The following commands are designed to be executed on the load balancer as jtel user The following cd commands depend on the variable JT_DATE_TIME, which is set at the beginning of the next part. If the variable is not set, commands will fail. |
Code Block |
---|
# Create backup directory SLAVE MySQL Dump
JT_DATE_TIME=$(date +%F)
mkdir /srv/jtel/shared/backup/${JT_DATE_TIME}
# If /srv/jtel/.. does not exist on the load balancer, attempt this
JT_DATE_TIME=$(date +%F)
mkdir /home/jtel/shared/backup/${JT_DATE_TIME} |
Create MySQL Dump Warning |
---|
title | CAUTION - CREDENTIALS+IP-Adresses |
---|
| Credentials and IP-Addresses need to be changed before the following mysqldump commands can be executed |
Info |
---|
The following commands are designed to be executed on the load balancer as jtel user |
MySQL Dump - Until jtel Portal release 3.12 Code Block |
---|
# Change to backup directory
cd /srv/jtel/shared/backup/${JT_DATE_TIME}
# Create MysQL Dump
mysqldump -uUSER -pPWD -h<IP-Address-OR-Alias-GOOD-MASTER> |
As of release 3.12 please use the following command: Translations Ignore |
---|
Code Block | mysqldump -h<GOOD_MASTER> -uroot -p<PASSWORD> --single-transaction --master-data=2 --databases JTELWeb JTELStats JTELStats2 JTELLog --add-drop-database --add-drop-table --events --routines --triggers > master/srv/jtel/shared/backup/${JT_DATE_TIME}/acd-dbm_${JT_DATE_TIME}.sql |
Warning |
---|
| Versions versions 3.12, 3.14 and 3.15 |
| : | If someone logs on to the portal while the dump is being pulled, it will go wrong. Enclosed an a SQL query. If If the time changes after executing the query is executed, a login has taken place. If this happens, the dump has to be pulled again and in the meantime it has to be permanently checked if a login has taken place. Only Only if this is not the case, the dump can be replicated error-free to the slave. SELECT Max(dtAcdLoggedIn) FROM Users; In versions 3.11 and below and version 3.16 this problem does not exist. |
On the BROKEN master server, reset the slave and restore the backup translations-ignoreMySQL Dump - From jtel Portal release 3.12 until latest release Code Block |
---|
RESET# SLAVE;Change
SOURCE master.sql; |
On the BROKEN master server, determine the master position from the master.sql, and then reinitialize the slave Translations Ignore |
---|
Code Block |
---|
CHANGE MASTER TO MASTER_HOST = '<GOOD_MASTER>', MASTER_USER = 'repl', MASTER_PASSWORD = '<PASSWORD>', MASTER_LOG_FILE='<NAME_LOGFILE>', MASTER_LOG_POS=<POSITION_LOGFILE>;
START SLAVE; |
On the BROKEN master server Check the slave Translations Ignore |
---|
Code Block |
---|
SHOW SLAVE STATUS\G |
Only if everything is OK, and the replication is up to date, then continue. The status can be monitored with the following command: Translations Ignore |
---|
Code Block |
---|
watch 'mysql -u root -p<PASSWORD> -e "SHOW SLAVE STATUS\G" 2>/dev/null' |
On the BROKEN master server lock all tables and note master position Translations Ignore |
---|
Code Block |
---|
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS; |
The positions of SHOW MASTER STATUS are required in the following command. On the GOOD master server, reposition and start the replication. Translations Ignore |
---|
Code Block |
---|
CHANGE MASTER TO MASTER_HOST = '<SECOND_MASTER>', MASTER_USER = 'repl', MASTER_PASSWORD = '<PASSWORD>', MASTER_LOG_FILE='<NAME_LOGFILE>', MASTER_LOG_POS=<POSITION_LOGFILE>;
START SLAVE; |
Unlock the tables on the BROKEN master server Translations Ignore |
---|
Check Masters and Slaves On all servers now Translations Ignore |
---|
Code Block |
---|
SHOW SLAVE STATUS\G |
and check that everything is running smoothly It is usually not necessary to restore the slaves attached to both masters. If it is, they can be re-initialized with the normal slave recovery procedure. |