MySQL Source Compiling for debugging

I’ve been evaluating MySQL 8.0 for a while and thought to share quick steps to compile the source code for debugging. The steps I’ve shared are for Centos 7 x86_64 platform(or similar)

1. Download the latest MySQL Source


wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-8.0.3-0.1.rc.el7.src.rpm

2. You might need packages generally not found in by default in Linux Distributions

rpm-build.x86_64
devtoolset-6-runtime-6.1-1
devtoolset-6-binutils-2.27-12
devtoolset-6-gcc-6.3.1-3.1
devtoolset-6-libstdc++-devel-6.3.1-3.1
devtoolset-6-gcc-c++-6.3.1-3.1
cmake
perl(Env)
libaio-devel
ncurses-devel
openssl-devel
numactl-devel
cyrus-sasl-devel

3.  Open the rpm package in desired folder


rpm2cpio mysql-community-8.0.3-0.1.rc.el7.src.rpm | cpio -idmv
boost_1_64_0.tar.bz2
filter-provides.sh
filter-requires.sh
mysql-5.6.37.tar.gz
mysql-8.0.3-rc.tar.gz
mysql.spec

#Open the source zip/tar file

tar -xzvf mysql-8.0.3-rc.tar.gz

#Open boost zip/tar file

tar -xjvf boost_1_64_0.tar.bz2

#Change dir to mysql-8.0.3-rc

cd mysql-8.0.3-rc

4. Compile with debug


cmake -DWITH_DEBUG=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=mysql-8.0.3-rc/boost_1_64_0/boost
make
make install
ln -s /usr/local/mysql/bin/* /usr/bin/

5. Make symlink and create user


ln -s /usr/local/mysql/bin/* /usr/bin/
groupadd mysql
useradd -r -g mysql -s /bin/false mysql

It couldn’t be that simpler right…. Well it is. Have fun with MySQL ….

Advertisements

MySQL Group Replication Analyzed

This blog is to share salient features and brief steps for Installation/Configuration and Monitoring of MySQL Group Replication.

What is Group Replication?

Group Replication is a new shared-nothing multi-master HA which is based on Paxos algorithm. Group Replication would be the part of InnoDB Cluster(currently in Beta) MySQL 5.7.17  GA version comes with Group Replication plugin. Group replication provides HA with automatic failover when the primary crashes.

How it works?

On group replication, all nodes do a group communication with each other where every message/transaction has its own transaction ID, GTID (which was first introduced on MySQL 5.6).

group-clipart-small-group-discussion-clipart-1
Group Communication

Any read-write (RW) transaction ‘say T1’ is not committed on either of the node until all of the nodes certify that  T1 is safe to play (no conflicts) and hence T1 gets committed on all nodes. It should not be assumed similar to semi-synchronous replication where in semi-synchronous the slave node/s just acknowledges the transactions(that its been written on their relay-logs)  irrespective of any conflict or not. To maintain the quorum the group should have atleast 3 nodes.

g1

Starting the Group Replication:

Install client,server and MySQL lib packages of MySQL 5.7.17 available for the flavor of OS which you are going to use.

mkdir -p /data/server/d1
mkdir -p /data/server/d2
mkdir -p /data/server/d3
chown -R mysql. /data/server
Initialize the MySQL servers

mysqld --defaults-file=/etc/my.cnf11 --user=mysql --initialize-insecure
mysqld --defaults-file=/etc/my.cnf12 --user=mysql --initialize-insecure
mysqld --defaults-file=/etc/my.cnf13 --user=mysql --initialize-insecure

Example MySQL Configuration:
[mysqld]
user=mysql
datadir=/data/server/d1
port=3301
socket=/data/server/d1/mysql.sock
log_error=/data/server/d1/mysqld.err
log_error_verbosity=2
#######Load Group Replication Plugin#####
plugin-load=group_replication=group_replication.so
#######Replication Variables###
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
transaction_write_set_extraction=XXHASH64
group_replication_group_name="f792dee6-c286-11e6-bcaa-0800277caf98"
group_replication_start_on_boot=OFF
group_replication_local_address= "192.168.1.103:24901"
group_replication_group_seeds= "192.168.1.103:24901,192.168.1.103:24902,192.168.1.103:24903"
group_replication_bootstrap_group=OFF

 

Start all the nodes
mysqld --defaults-file=/etc/my.cnf11 --user=mysql --init-file=/data/server/group.init &
mysqld --defaults-file=/etc/my.cnf12 --user=mysql --init-file=/data/server/group.init &
mysqld --defaults-file=/etc/my.cnf13 --user=mysql --init-file=/data/server/group.init &

Where init file comprises of

[root@localhost103 server]# cat group.init
SET SQL_LOG_BIN=0;
CREATE USER rpl_user@'192.168.1.103';
GRANT REPLICATION SLAVE ON *.* TO rpl_user@'192.168.1.103' IDENTIFIED BY 'rpl123#';
FLUSH PRIVILEGES;
CHANGE MASTER TO MASTER_USER='rpl_user', MASTER_PASSWORD='rpl123#' FOR CHANNEL 'group_replication_recovery';
SET SQL_LOG_BIN=1;

Init file is not part of steps for group replication configuration. I added for my own convenience.

Bootstrapping the group:

Execute on the node, which you want to be the primary OR the initial node.


[root@localhost103 ~]# mysql -S/data/server/d1/mysql.sock
mysql> SET GLOBAL group_replication_bootstrap_group=ON;
mysql> START GROUP_REPLICATION;
mysql> SET GLOBAL group_replication_bootstrap_group=OFF;

Error logs for the primary should say:

Plugin group_replication reported: 'This server was declared online within the replication group'

Check on the initial/primary node:


[root@localhost103 ~]# mysql -S/data/server/d1/mysql.sock
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
| group_replication_applier | c68da19c-c439-11e6-b601-0800277caf98 | localhost103.nareshserver.com | 3301 | ONLINE |
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
1 rows in set (0.00 sec)

 

Adding secondary/other nodes:

Make sure you have all of the node’s hostnames and their listener ports in group_replication_group_seeds of master. If not we can add it as

[root@localhost103 ~]# mysql -S/data/server/d1/mysql.sock
mysql> set global group_replication_group_seeds= "192.168.1.103:24901,192.168.1.103:24902,192.168.1.103:24903"

While for nodes we need to add:

[root@localhost103 ~]# mysql -S/data/server/d2/mysql.sock
mysql>  START GROUP_REPLICATION;

[root@localhost103 ~]# mysql -S/data/server/d3/mysql.sock
mysql> START GROUP_REPLICATION;

Once added, for all nodes below status query should return as:

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST                   | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
| group_replication_applier | 7acae2a2-c44a-11e6-9365-0800277caf98 | localhost103.nareshserver.com |        3303 | ONLINE       |
| group_replication_applier | 9fd197a5-c43a-11e6-85a4-0800277caf98 | localhost103.nareshserver.com |        3302 | ONLINE       |
| group_replication_applier | c68da19c-c439-11e6-b601-0800277caf98 | localhost103.nareshserver.com |        3301 | ONLINE       |
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+

 

Monitoring:

We can check the current status for replication or committed transactions for any node:

mysql> select * from performance_schema.replication_group_member_stats\G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 14819730667727846:3
                         MEMBER_ID: 7acae2a2-c44a-11e6-9365-0800277caf98
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 30008
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 459
TRANSACTIONS_COMMITTED_ALL_MEMBERS: f792dee6-c286-11e6-bcaa-0800277caf98:1-29552
    LAST_CONFLICT_FREE_TRANSACTION: f792dee6-c286-11e6-bcaa-0800277caf98:30011
1 row in set (0.00 sec)

Here TRANSACTIONS_COMMITTED_ALL_MEMBERS is the last committed transaction ids on all nodes.
While LAST_CONFLICT_FREE_TRANSACTION is the last conflict free transaction.

When there are no writes happening in any nodes, we could see the stats as

mysql> select * from performance_schema.replication_group_member_stats\G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 14819730667727846:3
                         MEMBER_ID: 7acae2a2-c44a-11e6-9365-0800277caf98
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 30008
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: f792dee6-c286-11e6-bcaa-0800277caf98:1-30011
    LAST_CONFLICT_FREE_TRANSACTION: f792dee6-c286-11e6-bcaa-0800277caf98:30011
1 row in set (0.00 sec)

 

mysql> select * from performance_schema.replication_applier_configuration;
+----------------------------+---------------+
| CHANNEL_NAME               | DESIRED_DELAY |
+----------------------------+---------------+
| group_replication_applier  |             0 |
| group_replication_recovery |             0 |
+----------------------------+---------------+

If not configured for multi-source replication, by-default only 2 channels will be seen on group replication as see above, and default desired_delay as 0 if no delayed replication configured.

mysql> select * from performance_schema.replication_applier_status;
+----------------------------+---------------+-----------------+----------------------------+
| CHANNEL_NAME               | SERVICE_STATE | REMAINING_DELAY | COUNT_TRANSACTIONS_RETRIES |
+----------------------------+---------------+-----------------+----------------------------+
| group_replication_applier  | ON            |            NULL |                          0 |
| group_replication_recovery | OFF           |            NULL |                          0 |
+----------------------------+---------------+-----------------+----------------------------+
2 rows in set (0.00 sec)

If group_replication_applier is OFF that means the applier thread is not running and the node is not replicating.

mysql> select * from performance_schema.replication_connection_configuration\G
*************************** 1. row ***************************
                 CHANNEL_NAME: group_replication_applier
                         HOST: localhost103.nareshserver.com
                         PORT: 3301
                         USER: rpl_user
            NETWORK_INTERFACE:
                AUTO_POSITION: 1
                  SSL_ALLOWED: NO
                  SSL_CA_FILE:
                  SSL_CA_PATH:
              SSL_CERTIFICATE:
                   SSL_CIPHER:
                      SSL_KEY:
SSL_VERIFY_SERVER_CERTIFICATE: NO
                 SSL_CRL_FILE:
                 SSL_CRL_PATH:
    CONNECTION_RETRY_INTERVAL: 60
       CONNECTION_RETRY_COUNT: 86400
           HEARTBEAT_INTERVAL: 30.000
                  TLS_VERSION:
*************************** 2. row ***************************
                 CHANNEL_NAME: group_replication_recovery
                         HOST: NULL
                         PORT: 0
                         USER: rpl_user
            NETWORK_INTERFACE:
                AUTO_POSITION: 1
                  SSL_ALLOWED: NO
                  SSL_CA_FILE:
                  SSL_CA_PATH:
              SSL_CERTIFICATE:
                   SSL_CIPHER:
                      SSL_KEY:
SSL_VERIFY_SERVER_CERTIFICATE: NO
                 SSL_CRL_FILE:
                 SSL_CRL_PATH:
    CONNECTION_RETRY_INTERVAL: 60
       CONNECTION_RETRY_COUNT: 1
           HEARTBEAT_INTERVAL: 30.000
                  TLS_VERSION:
2 rows in set (0.00 sec)

replication_connection_configuration contains similar values as ‘SHOW SLAVE STATUS\G’

mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************
             CHANNEL_NAME: group_replication_applier
               GROUP_NAME: f792dee6-c286-11e6-bcaa-0800277caf98
              SOURCE_UUID: f792dee6-c286-11e6-bcaa-0800277caf98
                THREAD_ID: NULL
            SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
 RECEIVED_TRANSACTION_SET: f792dee6-c286-11e6-bcaa-0800277caf98:1-2338810
        LAST_ERROR_NUMBER: 0
       LAST_ERROR_MESSAGE:
     LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
             CHANNEL_NAME: group_replication_recovery
               GROUP_NAME:
              SOURCE_UUID:
                THREAD_ID: NULL
            SERVICE_STATE: OFF
COUNT_RECEIVED_HEARTBEATS: 0
 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
 RECEIVED_TRANSACTION_SET:
        LAST_ERROR_NUMBER: 0
       LAST_ERROR_MESSAGE:
     LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
2 rows in set (0.00 sec)

Here in RECEIVED_TRANSACTION_SET we have the UUID of the group name and the transaction ID received.

mysql> select * from performance_schema.replication_group_member_stats\G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 14868660117493416:10
                         MEMBER_ID: 9d7b6ef1-eeee-11e6-9857-0800277caf98
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 1
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: f792dee6-c286-11e6-bcaa-0800277caf98:1-2338810
    LAST_CONFLICT_FREE_TRANSACTION: f792dee6-c286-11e6-bcaa-0800277caf98:2338810
1 row in set (0.00 sec)

It shows the stats of group replication. Here VIEW_ID is the ID of the quorum, it shows how many times the quorum got modified after the bootsrtap of group replication.
replication_applier_status_by_coordinator will show no rows/info as we haven’t enabled parallel threads.

mysql> select * from performance_schema.replication_applier_status_by_coordinator \G
Empty set (0.00 sec)

After enabling parallel workers:

mysql> select * from performance_schema.replication_applier_status_by_coordinator \G
*************************** 1. row ***************************
        CHANNEL_NAME: group_replication_applier
           THREAD_ID: 33
       SERVICE_STATE: ON
   LAST_ERROR_NUMBER: 0
  LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
        CHANNEL_NAME: group_replication_recovery
           THREAD_ID: NULL
       SERVICE_STATE: OFF
   LAST_ERROR_NUMBER: 0
  LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
2 rows in set (0.00 sec)

 

Enabling Parallel Workers

By default( without parallel workers enabled) replication_applier_status_by_worker will always show 2 rows, one for channel group_replication_applier and another group_replication_recovery. If parallel workers are enabled, then the row count for this performance_schema.replication_applier_status_by_worker will be n*2 where n is number of parallel workers.

mysql> select * from performance_schema.replication_applier_status_by_worker \G
*************************** 1. row ***************************
         CHANNEL_NAME: group_replication_applier
            WORKER_ID: 0
            THREAD_ID: 33
        SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: f792dee6-c286-11e6-bcaa-0800277caf98:2338817
    LAST_ERROR_NUMBER: 0
   LAST_ERROR_MESSAGE:
 LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
         CHANNEL_NAME: group_replication_recovery
            WORKER_ID: 0
            THREAD_ID: NULL
        SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: f792dee6-c286-11e6-bcaa-0800277caf98:2338814
    LAST_ERROR_NUMBER: 0
   LAST_ERROR_MESSAGE:
 LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
2 rows in set (0.00 sec)

To enable parallel workers:


mysql> stop group_replication; set global slave_preserve_commit_order=1; set global slave_parallel_workers=10; set global slave_parallel_type='LOGICAL_CLOCK'; stop group_replication; start group_replication;

We need to enable slave_preserve_commit_order as well to start parallel workers.

 

 

Multi-Master/Multi-Primary

By default not all nodes will be ready to take writes. All nodes except the initial node, will have super_read_only=ON

mysql> show global variables like '%super%';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| super_read_only | ON    |
+-----------------+-------+
1 row in set (0.00 sec)

To have enable writes on any node, other than primary node we have to disable super_read_only.

SET GLOBAL super_read_only=OFF;

Its obvious we need not have to restart the MySQL server in this case.
To enable writes on all nodes, and to have a true multi-primary case. We need to set

group_replication_single_primary_mode=OFF
slave-preserve-commit-order=ON

on mysql config file on all nodes and restart and bootstrap the group.

In either of the case(more than 1 primary OR all primary) enabling group_replication_enforce_update_everywhere_checks is recommended.

Adding New Node/Crash Recovery:

The moment you add other node, it reads all the binary log events and replicates all changes.
Lets insert some data and start a node without any data, and check if its catches up without any hassle.

[root@localhost103 ~]# mkdir -p /data/server/d4
[root@localhost103 ~]# cat /etc/my.cnf14
[mysqld]
user=mysql
datadir=/data/server/d4
port=3304
socket=/data/server/d4/mysql.sock
log_error=/data/server/d4/mysqld.err
log_error_verbosity=3
#######Load Group Replication Plugin#####
plugin-load=group_replication=group_replication.so
#######Replication Variables###
server_id=4
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
transaction_write_set_extraction=XXHASH64
group_replication_group_name="f792dee6-c286-11e6-bcaa-0800277caf98"
group_replication_start_on_boot=OFF
group_replication_local_address= "192.168.1.103:24904"
group_replication_group_seeds= "192.168.1.103:24901,192.168.1.103:24902,192.168.1.103:24903,192.168.1.103:24904"
group_replication_bootstrap_group=OFF

[root@localhost103 ~]# mysqld --defaults-file=/etc/my.cnf14  --user=mysql --initialize-insecure
[root@localhost103 ~]# mysqld --defaults-file=/etc/my.cnf14 --user=mysql --init-file=/data/server/group.init &
[root@localhost103 ~]# mysql -S/data/server/d4/mysql.sock
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+-----------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+-----------+-------------+-------------+--------------+
| group_replication_applier |           |             |        NULL | OFFLINE      |
+---------------------------+-----------+-------------+-------------+--------------+
1 row in set (0.06 sec)

mysql> start group_replication;
Query OK, 0 rows affected (6.20 sec)

mysql>  SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST                   | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
| group_replication_applier | 7acae2a2-c44a-11e6-9365-0800277caf98 | localhost103.nareshserver.com |        3303 | RECOVERING   |
| group_replication_applier | 9fd197a5-c43a-11e6-85a4-0800277caf98 | localhost103.nareshserver.com |        3302 | ONLINE       |
| group_replication_applier | c68da19c-c439-11e6-b601-0800277caf98 | localhost103.nareshserver.com |        3301 | ONLINE       |
+---------------------------+--------------------------------------+-------------------------------+-------------+--------------+
4 rows in set (0.00 sec)

RECOVERING state means that the server has joined a group from which it is retrieving data.

mysql> select * from performance_schema.replication_group_member_stats\G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 14819730667727846:7
                         MEMBER_ID: 9fd197a5-c43a-11e6-85a4-0800277caf98
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 0
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: f792dee6-c286-11e6-bcaa-0800277caf98:1-30185
    LAST_CONFLICT_FREE_TRANSACTION:
1 row in set (0.01 sec)

If all nodes are in sync, their “TRANSACTIONS_COMMITTED_ALL_MEMBERS” must be same.

Things to consider while adding new node or starting crashed node of the group.

You cannot change group_replication_group_name while group replication is running.
group_replication_group_seeds shows the seeds/peer addresses from my.cnf of the instance but not all of the seeds which were later added OR even removed/crashed.

If you want to add a new node say node3, then either of the existing node/donor of group replication cluster must have all the binary logs from the inception of the group or else you may end up with error

 [ERROR] Error reading packet from server for channel 'group_replication_recovery': The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires. (server_errno=1236)

I know this is expected, as we have to first sync the data from an existing node to new node to be added. But it’s must to specify for the users who are using Galera flavored HA clusters(MariaDB Galera Cluster/Percona XtraDB Cluster) where with the help of SST (State Snapshot Transfer) either rsync or xtrabackup takes place from donor to joiner, which is entirely automated.

When starting the crashed node we might get the error like below:

[ERROR] Plugin group_replication reported: 'The member contains transactions not present in the group. The member will now exit the group.

The error logs might also contain the log as

 [Note] Plugin group_replication reported: 'To force this member into the group you can use the group_replication_allow_local_disjoint_gtids_join option'

But enabling group_replication_allow_local_disjoint_gtids_join is not a wise step to proceed as it will lead to data inconsistencies.

How to deal with such situation?
Since the data of the node with above error cannot be relied upon, the data is worth nothing. Re-sync the crashed node from one of the donor node of Replication Group.

 

Some important and unique attributes for group replication which we need to review:

1. All status monitoring views for group replication are on performance_schema.
2. Supports only row based replication. i.e. binlog_format=ROW
3. Supports only InnoDB tables.
4. DDL statements not considered safe for multi-primary environment.
5. Multi-primary mode do not support tables with multi-level foreign key dependencies,
6. specifically tables that have defined CASCADING foreign key constraints.
7. SERIALIZABLE isolation level is not supported in multi-primary groups.
8. Transaction savepoints are not supported.
9. All tables must have a primary key.
10. Auto_increment
*  The moment group replication is started auto_increment_increment changes as default 7 and auto_increment_offset changes to its own node number(if the node was added 3rd on group replication, the auto_increment_offset will change to 3)
* group_replication_auto_increment_increment stays as default 7.
11. Group replication with SELINUX
semanage port -a -t mysqld_port_t -p tcp 3306.
12. At max 9 nodes can participate in single group.
13. Need of a load balancer: Either HA Proxy or the new MySQL Router

 

 

I believe there are lot of things to cover and to analyze different behaviors(if not bugs) of Group Replication like, Group Replication with InnoDB Compressed Tables, SSL, parallel workers and complex topology with multi-source replication and traditional replication .

 

MySQL Router: Easy steps to install and setup

This blog is to provide ‘to the point’, to install and configure MySQL router. For in-depth information please check the Router Manual.
MySQL Router is a intelligent routing connection or a gateway for MySQL HA environment. Installing and configuring router is pretty simple and straight forward.

The installation and configuration is only focused on Linux platform for now.

We can download the MySQL Router from here.

Installation


[root@nrama12 nareshm]# wget https://dev.mysql.com/get/Downloads/MySQL-Router/mysql-router-2.0.3-1.el7.x86_64.rpm
--2016-05-01 19:17:46-- https://dev.mysql.com/get/Downloads/MySQL-Router/mysql-router-2.0.3-1.el7.x86_64.rpm
Resolving dev.mysql.com (dev.mysql.com)... 137.254.60.11
Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://cdn.mysql.com//Downloads/MySQL-Router/mysql-router-2.0.3-1.el7.x86_64.rpm [following]
--2016-05-01 19:17:49-- http://cdn.mysql.com//Downloads/MySQL-Router/mysql-router-2.0.3-1.el7.x86_64.rpm
Resolving cdn.mysql.com (cdn.mysql.com)... 23.211.0.73
Connecting to cdn.mysql.com (cdn.mysql.com)|23.211.0.73|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1118360 (1.1M) [application/x-redhat-package-manager]
Saving to: ‘mysql-router-2.0.3-1.el7.x86_64.rpm’

100%[=====================================================================>] 1,118,360 121KB/s in 8.3s

2016-05-01 19:17:58 (132 KB/s) - ‘mysql-router-2.0.3-1.el7.x86_64.rpm’ saved [1118360/1118360]

[root@nrama12 router]# rpm -ivh mysql-router-2.0.3-1.el7.x86_64.rpm
warning: mysql-router-2.0.3-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-router-2.0.3-1.1################################# [100%]

Lets check router help option


[root@nrama12 router]# mysqlrouter --help
MySQL Router v2.0.3 on Linux (64-bit) 
Copyright (c) 2015, 2016 Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Start MySQL Router.
Configuration read from the following files in the given order (enclosed
in parentheses means not available for reading):
/etc/mysqlrouter/mysqlrouter.ini
(/root/.mysqlrouter.ini)
Usage: mysqlrouter [-v|–version] [-h|–help]
[-c|–config=<path>]
[-a|–extra-config=<path>]
Options:
-v, –version
Display version information and exit.
-h, –help
Display this help and exit.
-c <path>, –config <path>
Only read configuration from given file.
-a <path>, –extra-config <path>
Read this file after configuration files are read from either
default locations or from files specified by the –config
option.

By default the router config file will look like below


[root@nrama12 router]# cat /etc/mysqlrouter/mysqlrouter.ini
# Copyright (c) 2015, Oracle and/or its affiliates. All

rights reserved.
# MySQL Router configuration file
#
# Documentation is available at
# http://dev.mysql.com/doc/mysql-router/en/

[DEFAULT]
logging_folder = /var/log/mysqlrouter/
plugin_folder = /usr/lib64/mysqlrouter
runtime_folder = /var/run/mysqlrouter
config_folder = /etc/mysqlrouter

[logger]
level = info

# If no plugin is configured which starts a service, keepalive
# will make sure MySQL Router will not immediately exit. It is
# safe to remove once Router is configured.
[keepalive]
interval = 60

For MySQLFabric

There are 6 MySQL instances running


[root@nrama12 nareshm]# ps -ef | grep mysqld
mysql 4356 1 0 00:51 ? 00:00:21 mysqld –defaults-file=/etc/mysql2.cnf –daemonize
mysql 4386 1 0 00:51 ? 00:00:24 mysqld –defaults-file=/etc/mysql3.cnf –daemonize
mysql 4416 1 0 00:51 ? 00:00:23 mysqld –defaults-file=/etc/mysql4.cnf –daemonize
mysql 4447 1 0 00:51 ? 00:00:23 mysqld –defaults-file=/etc/mysql5.cnf –daemonize
mysql 4477 1 0 00:51 ? 00:00:23 mysqld –defaults-file=/etc/mysql6.cnf –daemonize
mysql 6949 1 0 14:16 ? 00:00:38 mysqld –defaults-file=/etc/mysql1.cnf –daemonize

Where MySQL instance localhost:3311 is fabric repository/controller
And rest all localhost:3312, localhost:3313, localhost:3314, localhost:3315, localhost:3316 are part of fabric group local_grp.

fabricrouter

The lookup for fabric group local_grp is like below:


[root@nrama12]# mysqlfabric –config=/etc/mysql/fabric.cfg group lookup_servers local_grp 
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— ————– ——— ———- ——
3f02b679-bdad-11e5-9125-080027173a36 127.0.0.1:3313 SECONDARY READ_ONLY 1.0
86bb2f78-be45-11e5-9b02-080027173a36 127.0.0.1:3314 SECONDARY READ_ONLY 1.0
891fbac6-be45-11e5-9c23-080027173a36 127.0.0.1:3315 SECONDARY READ_ONLY 1.0
8b1b61cd-be45-11e5-9d1d-080027173a36 127.0.0.1:3316 SECONDARY READ_ONLY 1.0
f414740d-bdac-11e5-8b4d-080027173a36 127.0.0.1:3312 PRIMARY READ_WRITE 1.0

Changes in router config file


[root@nrama12 Downloads]# cat /etc/mysqlrouter/mysqlrouter.ini
# Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.
# MySQL Router configuration file
#
# Documentation is available at
# http://dev.mysql.com/doc/mysql-router/en/

[DEFAULT]
#Make sure all the folder exists and edit the plugin path
logging_folder = /var/log/mysqlrouter/ 
plugin_folder = /usr/lib64/mysqlrouter
runtime_folder = /var/run/mysqlrouter
config_folder = /etc/mysqlrouter

[logger]
#You can set it to ‘INFO’, if you don’t want debug logs
level = DEBUG

# If no plugin is configured which starts a service, keepalive
# will make sure MySQL Router will not immediately exit. It is
# safe to remove once Router is configured.
[keepalive]
interval = 60

#Below is for mysqlfabric connection
#address is hostname of fabric controller/repository
#user is user for fabric controller/repository
[fabric_cache:my_cache]
address = localhost
user = admin

#Below config parameters are for router
#bind_address is hostname for router, with unique port
#destination is connection URL for fabric
#mode is to set more either to read-write or read-only
[routing]
bind_address = 127.0.0.1:7002
destinations = fabric+cache://my_cache/group/local_grp
mode = read-write

Start router


[root@nrama12]# mysqlrouter –config=/etc/mysqlrouter/mysqlrouter.ini
Logging to /var/log/mysqlrouter/mysqlrouter.log
Password for [fabric_cache:my_cache], user admin:

^Z[2] Killed mysqlrouter –config=/etc/mysqlrouter/mysqlrouter.ini

[3]+ Stopped mysqlrouter –config=/etc/mysqlrouter/mysqlrouter.ini
[root@nrama12]# bg
[3]+ mysqlrouter –config=/etc/mysqlrouter/mysqlrouter.ini &

Check logs


[root@nrama12 Downloads]# tail -f /var/log/mysqlrouter/mysqlrouter.log
2016-05-01 15:08:42 INFO [7f48adbc6700] [routing] listening on 127.0.0.1:7002; read-write
2016-05-01 15:08:42 INFO [7f48aebc8700] Starting Fabric Cache ‘my_cache’ using MySQL Fabric running on localhost:32275
2016-05-01 15:08:42 INFO [7f48ae3c7700] keepalive started with interval 60
2016-05-01 15:08:42 INFO [7f48ae3c7700] keepalive
2016-05-01 15:08:42 INFO [7f48aebc8700] Connected with Fabric running on 127.0.0.1
2016-05-01 15:09:12 INFO [7f5a0ffe9700] [routing] listening on 127.0.0.1:7002; read-write
2016-05-01 15:09:12 INFO [7f5a10feb700] Starting Fabric Cache ‘my_cache’ using MySQL Fabric running on localhost:32275
2016-05-01 15:09:12 INFO [7f5a107ea700] keepalive started with interval 60
2016-05-01 15:09:12 INFO [7f5a107ea700] keepalive
2016-05-01 15:09:12 INFO [7f5a10feb700] Connected with Fabric running on 127.0.0.1

Try to connect to fabric environment


[root@nrama12 Downloads]# mysql -P7002 -h127.0.0.1
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1029
Server version: 5.7.10-community-advanced-log MySQL Community Server

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> show global variables like ‘hostname’;
+—————+—————————–+
| Variable_name | Value |
+—————+—————————–+
| hostname | nrama12.hsd1.ca.comcast.net |
+—————+—————————–+
1 row in set (0.01 sec)

mysql> show global variables like ‘port’;
+—————+——-+
| Variable_name | Value |
+—————+——-+
| port | 3312 |
+—————+——-+
1 row in set (0.00 sec)

For simple replication environment.

Number of instances running:


root@nrama12 nareshm]# ps -ef | grep mysqld
mysql 4356 1 0 00:51 ? 00:00:21 mysqld –defaults-file=/etc/mysql2.cnf –daemonize
mysql 4386 1 0 00:51 ? 00:00:24 mysqld –defaults-file=/etc/mysql3.cnf –daemonize
mysql 4416 1 0 00:51 ? 00:00:23 mysqld –defaults-file=/etc/mysql4.cnf –daemonize
mysql 4447 1 0 00:51 ? 00:00:23 mysqld –defaults-file=/etc/mysql5.cnf –daemonize
mysql 4477 1 0 00:51 ? 00:00:23 mysqld –defaults-file=/etc/mysql6.cnf –daemonize

The router config file will be:


[root@nrama12 Downloads]# cat /etc/mysqlrouter/mysqlrouter2.ini

# Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.

# MySQL Router configuration file
#
# Documentation is available at
# http://dev.mysql.com/doc/mysql-router/en/

[DEFAULT]
logging_folder = /var/log/mysqlrouter/
plugin_folder = /usr/lib64/mysqlrouter
runtime_folder = /var/run/mysqlrouter
config_folder = /etc/mysqlrouter

[logger]
level = DEBUG

# If no plugin is configured which starts a service, keepalive
# will make sure MySQL Router will not immediately exit. It is
# safe to remove once Router is configured.
[keepalive]
interval = 60

[routing]
bind_address = 127.0.0.1:7002
destinations = localhost:3312,localhost:3313,localhost:3314,localhost:3315,localhost:3316
mode = read-write

Start router


[root@nrama12 nareshm]# mysqlrouter –config=/etc/mysqlrouter/mysqlrouter2.ini
Logging to /var/log/mysqlrouter/mysqlrouter.log
^Z[1] Killed mysqlrouter –config=/etc/mysqlrouter/mysqlrouter2.ini

[2]+ Stopped mysqlrouter –config=/etc/mysqlrouter/mysqlrouter2.ini
[root@nrama12 nareshm]# bg
[2]+ mysqlrouter –config=/etc/mysqlrouter/mysqlrouter2.ini &

Let us check the logs


[root@nrama12 nareshm]# tail -f /var/log/mysqlrouter/mysqlrouter.log
2016-05-01 17:06:42 INFO [7f5bc236f700] keepalive
2016-05-01 17:06:42 INFO [7f5bc1b6e700] [routing] listening on 127.0.0.1:7002; read-write
2016-05-01 17:07:01 DEBUG [7f5bc0b6c700] Trying server localhost:3312 (index 0)
2016-05-01 17:07:01 DEBUG [7f5bc0b6c700] [routing] [127.0.0.1]:41553 – [::1]:3312
2016-05-01 17:07:17 DEBUG [7f5bc0b6c700] [routing] Routing stopped (up:3815b;down:962b)
2016-05-01 17:07:42 INFO [7f5bc236f700] keepalive
2016-05-01 17:08:42 INFO [7f5bc236f700] keepalive
2016-05-01 17:09:42 INFO [7f5bc236f700] keepalive
2016-05-01 17:10:42 INFO [7f5bc236f700] keepalive

Lets login and check:


[root@nrama12 nareshm]# mysql -P7002 -h127.0.0.1
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4885
Server version: 5.7.10-community-advanced-log MySQL Community Server

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> show global variables like ‘port’;
+—————+——-+
| Variable_name | Value |
+—————+——-+
| port | 3312 |
+—————+——-+
1 row in set (0.00 sec)

Lets kill the master and see the router behavior.


[root@nrama12 nareshm]# kill -9 4356
[root@nrama12 nareshm]# mysql -P7002 -h127.0.0.1
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5288
Server version: 5.7.10-community-advanced-log MySQL Community Server 
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> show global variables like ‘port’;
+—————+——-+
| Variable_name | Value |
+—————+——-+
| port | 3313 |
+—————+——-+
1 row in set (0.00 sec)

The router starts pointing to mysql instance localhost:3313 .
Note: When in simple replication setup, like above case, it will connect in round-robin method. The router does not trigger failover. We can make the best out of it if we use it with mysql HA environments like fabric and/or group replication.

MySQL 5.7 Multi-Source Replication

This post it to share a brief and precise information about MySQL 5.7 Multi Source Replication (configuration and monitoring) and not specifically for concepts of MySQL replication.

msr

There have been couple of third-party tools for multi-source replication in MySQL. But in MySQL57, its officially been included as one of the feature of MySQL.

A single source replication usually (with one master and 3 slaves) looks like the figure below:

ssr

 

For checking the complexity of multi-master, we configured 6 MySQL instances, with GTID enabled. (We can have multi-master feature with traditional file position based replication too.)

Below is the config file for a single instance. Rest 5 are all similar.

[mysqld]
datadir=/mysql/mysql6/data/
basedir=/usr
user=mysql
socket=/mysql/mysql6/data/mysql6.socket
binlog-format=MIXED
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
port=3316
server-id=16
log-bin=/mysql/mysql6/binlog/mysql6
log_error=/mysql/mysql6/log/error.log
slow_query_log=1
slow_query_log_file=/mysql/mysql6/log/slow.log
long_query_time=30

 

Configuring slave for multi-master.

On slave nrama12:3316

For every master we have to name a channel and hence the command “Change Master to

mysql> change master to master_host=’localhost’, master_port=3311, master_user=’repl’,master_password=’repl123#’ for channel ‘master1’;
mysql> change master to master_host=’localhost’, master_port=3312, master_user=’repl’,master_password=’repl123#’ for channel ‘master2’;
mysql> change master to master_host=’localhost’, master_port=3313, master_user=’repl’,master_password=’repl123#’ for channel ‘master3’;
mysql> change master to master_host=’localhost’, master_port=3314, master_user=’repl’,master_password=’repl123#’ for channel ‘master4’;
mysql> change master to master_host=’localhost’, master_port=3315, master_user=’repl’,master_password=’repl123#’ for channel ‘master5’;

Starting slave for multiple channels

mysql> start slave for channel ‘master1’;
mysql> start slave for channel ‘master2’;
mysql> start slave for channel ‘master3’;
mysql> start slave for channel ‘master4’;
mysql> start slave for channel ‘master5’;

Checking replication status

mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3311
Connect_Retry: 60
Master_Log_File: mysql1.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master1.000002
Relay_Log_Pos: 361
Relay_Master_Log_File: mysql1.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 578
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 11
Master_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master1
Master_TLS_Version:
*************************** 2. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql2.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master2.000002
Relay_Log_Pos: 361
Relay_Master_Log_File: mysql2.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 578
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 12
Master_UUID: f414740d-bdac-11e5-8b4d-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master2
Master_TLS_Version:
*************************** 3. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3313
Connect_Retry: 60
Master_Log_File: mysql3.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master3.000002
Relay_Log_Pos: 361
Relay_Master_Log_File: mysql3.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 578
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 13
Master_UUID: 3f02b679-bdad-11e5-9125-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master3
Master_TLS_Version:
*************************** 4. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3314
Connect_Retry: 60
Master_Log_File: mysql4.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master4.000002
Relay_Log_Pos: 361
Relay_Master_Log_File: mysql4.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 578
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 14
Master_UUID: 86bb2f78-be45-11e5-9b02-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master4
Master_TLS_Version:
*************************** 5. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3315
Connect_Retry: 60
Master_Log_File: mysql5.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master5.000002
Relay_Log_Pos: 361
Relay_Master_Log_File: mysql5.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 578
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 15
Master_UUID: 891fbac6-be45-11e5-9c23-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master5
Master_TLS_Version:
5 rows in set (0.00 sec)

5 rows were been returned. 1 row for 1 channel.

msr

For checking replication status for single channel

mysql> SHOW SLAVE STATUS FOR CHANNEL ‘master1’\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3311
Connect_Retry: 60
Master_Log_File: mysql1.000001
Read_Master_Log_Pos: 33715130
Relay_Log_File: nrama12-relay-bin-master1.000010
Relay_Log_Pos: 357
Relay_Master_Log_File: mysql1.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 33715130
Relay_Log_Space: 777
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 11
Master_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master1
Master_TLS_Version:
1 row in set (0.02 sec)

Be default the name convention for relay log would be
hostname-relay-bin-channel_name.00001

In our case its
nrama12-relay-bin-master1.000003
where nrama12 is hostname and master1 is one of the channel name.

Lets make some changes on one of the master and analyze the behavior of slave.

On master1 nrama12:3311

[root@nrama12 mysql]# mysql -u root -S/mysql/mysql1/data/mysql1.socket
mysql> create database test1 ;
Query OK, 1 row affected (0.01 sec)
mysql> use test1
Database changed
mysql> CREATE TABLE test1 (
Title int(4) NOT NULL AUTO_INCREMENT,
Col1 char(10) DEFAULT NULL,
Col2 varchar(20) DEFAULT NULL,
Date timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (Title)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
mysql> insert into test1(col1,col2) values (“raw”,”Raw123″);
Query OK, 1 row affected (0.00 sec)
mysql> insert into test1(col1,col2) values (“raw”,”Raw123″);
Query OK, 1 row affected (0.01 sec)
mysql> insert into test1(col1,col2) values (“raw”,”Raw123″);
Query OK, 1 row affected (0.00 sec)
mysql> insert into test1(col1,col2) values (“raw”,”Raw123″);
Query OK, 1 row affected (0.01 sec)
mysql> insert into test1(col1,col2) values (“raw”,”Raw123″);
Query OK, 1 row affected (0.01 sec)
mysql> insert into test1(col1,col2) values (“raw”,”Raw123″);
Query OK, 1 row affected (0.01 sec)
mysql> select * from test1 limit 3\G
*************************** 1. row ***************************
Title: 1
Col1: raw
Col2: Raw123
Date: 2016-03-20 03:30:00
*************************** 2. row ***************************
Title: 2
Col1: raw
Col2: Raw123
Date: 2016-03-20 03:30:04
*************************** 3. row ***************************
Title: 3
Col1: raw
Col2: Raw123
Date: 2016-03-20 03:30:05
3 rows in set (0.00 sec)
Checking slave on nrama12:3316
[root@nrama12 mysql]# mysql -u root -S/mysql/mysql6/data/mysql6.socket
mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test1 |
+——————–+
5 rows in set (0.00 sec)
mysql> use test1
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from test1 limit 3\G
*************************** 1. row ***************************
Title: 1
Col1: raw
Col2: Raw123
Date: 2016-03-20 03:30:00
*************************** 2. row ***************************
Title: 2
Col1: raw
Col2: Raw123
Date: 2016-03-20 03:30:04
*************************** 3. row ***************************
Title: 3
Col1: raw
Col2: Raw123
Date: 2016-03-20 03:30:05
3 rows in set (0.00 sec)

The commands for stopping and starting slave, thread sql_thread and io_thread will be the same if we want to do any changes for all channels. Below are few examples:

For stopping all channels at once

mysql> stop slave;

For stopping particular thread for all channel

[root@nrama12 mysql]# mysql -u root -S/mysql/mysql6/data/mysql6.socket
mysql> stop slave io_thread;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: localhost
Master_User: repl
Master_Port: 3311
Connect_Retry: 60
Master_Log_File: mysql1.000001
Read_Master_Log_Pos: 3335
Relay_Log_File: nrama12-relay-bin-master1.000003
Relay_Log_Pos: 3498
Relay_Master_Log_File: mysql1.000001
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 3335
Relay_Log_Space: 3922
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 11
Master_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master1
Master_TLS_Version:
*************************** 2. row ***************************
Slave_IO_State:
Master_Host: localhost
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql2.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master2.000003
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql2.000001
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 741
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 12
Master_UUID: f414740d-bdac-11e5-8b4d-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master2
Master_TLS_Version:
*************************** 3. row ***************************
Slave_IO_State:
Master_Host: localhost
Master_User: repl
Master_Port: 3313
Connect_Retry: 60
Master_Log_File: mysql3.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master3.000003
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql3.000001
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 741
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 13
Master_UUID: 3f02b679-bdad-11e5-9125-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master3
Master_TLS_Version:
*************************** 4. row ***************************
Slave_IO_State:
Master_Host: localhost
Master_User: repl
Master_Port: 3314
Connect_Retry: 60
Master_Log_File: mysql4.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master4.000003
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql4.000001
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 741
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 14
Master_UUID: 86bb2f78-be45-11e5-9b02-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master4
Master_TLS_Version:
*************************** 5. row ***************************
Slave_IO_State:
Master_Host: localhost
Master_User: repl
Master_Port: 3315
Connect_Retry: 60
Master_Log_File: mysql5.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master5.000003
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql5.000001
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 741
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 15
Master_UUID: 891fbac6-be45-11e5-9c23-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master5
Master_TLS_Version:
5 rows in set (0.00 sec)

For stopping particular thread for single channel

mysql> stop slave io_thread for channel ‘master1’;
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: localhost
Master_User: repl
Master_Port: 3311
Connect_Retry: 60
Master_Log_File: mysql1.000001
Read_Master_Log_Pos: 3335
Relay_Log_File: nrama12-relay-bin-master1.000004
Relay_Log_Pos: 357
Relay_Master_Log_File: mysql1.000001
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 3335
Relay_Log_Space: 3918
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 11
Master_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master1
Master_TLS_Version:
*************************** 2. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql2.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master2.000004
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql2.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 12
Master_UUID: f414740d-bdac-11e5-8b4d-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master2
Master_TLS_Version:
*************************** 3. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3313
Connect_Retry: 60
Master_Log_File: mysql3.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master3.000004
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql3.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 13
Master_UUID: 3f02b679-bdad-11e5-9125-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master3
Master_TLS_Version:
*************************** 4. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3314
Connect_Retry: 60
Master_Log_File: mysql4.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master4.000004
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql4.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 14
Master_UUID: 86bb2f78-be45-11e5-9b02-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master4
Master_TLS_Version:
*************************** 5. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3315
Connect_Retry: 60
Master_Log_File: mysql5.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master5.000004
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql5.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 15
Master_UUID: 891fbac6-be45-11e5-9c23-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master5
Master_TLS_Version:
5 rows in set (0.00 sec)

Similary for sql_thread

mysql> stop slave sql_thread for channel ‘master1’;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3311
Connect_Retry: 60
Master_Log_File: mysql1.000001
Read_Master_Log_Pos: 33715130
Relay_Log_File: nrama12-relay-bin-master1.000010
Relay_Log_Pos: 357
Relay_Master_Log_File: mysql1.000001
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 33715130
Relay_Log_Space: 777
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 11
Master_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master1
Master_TLS_Version:
*************************** 2. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql2.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master2.000007
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql2.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 12
Master_UUID: f414740d-bdac-11e5-8b4d-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master2
Master_TLS_Version:
*************************** 3. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3313
Connect_Retry: 60
Master_Log_File: mysql3.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master3.000007
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql3.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 13
Master_UUID: 3f02b679-bdad-11e5-9125-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master3
Master_TLS_Version:
*************************** 4. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3314
Connect_Retry: 60
Master_Log_File: mysql4.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master4.000007
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql4.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 14
Master_UUID: 86bb2f78-be45-11e5-9b02-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master4
Master_TLS_Version:
*************************** 5. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: localhost
Master_User: repl
Master_Port: 3315
Connect_Retry: 60
Master_Log_File: mysql5.000001
Read_Master_Log_Pos: 154
Relay_Log_File: nrama12-relay-bin-master5.000005
Relay_Log_Pos: 317
Relay_Master_Log_File: mysql5.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 697
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 15
Master_UUID: 891fbac6-be45-11e5-9c23-080027173a36
Master_Info_File: mysql.slave_master_info
SQL_Delay: 60
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 96720b8e-bda4-11e5-b8bc-080027173a36:1-100046
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master5
Master_TLS_Version:
5 rows in set (0.00 sec)

Since we moved to MySQL57, its good to explore other meta data tables too for checking replication status and monitoring:

Performance_schema: Replication Status and Monitoring

1. replication_connection_configuration
This table shows the replication parameters used by the slave server .

mysql> select * from performance_schema.replication_connection_configuration\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
HOST: localhost
PORT: 3311
USER: repl
NETWORK_INTERFACE:
AUTO_POSITION: 0
SSL_ALLOWED: NO
SSL_CA_FILE:
SSL_CA_PATH:
SSL_CERTIFICATE:
SSL_CIPHER:
SSL_KEY:
SSL_VERIFY_SERVER_CERTIFICATE: NO
SSL_CRL_FILE:
SSL_CRL_PATH:
CONNECTION_RETRY_INTERVAL: 60
CONNECTION_RETRY_COUNT: 86400
HEARTBEAT_INTERVAL: 30.000
TLS_VERSION:

2.replication_connection_status

This table shows the current status of the I/O thread of slave server.

mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
GROUP_NAME:
SOURCE_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
THREAD_ID: 67
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 122
LAST_HEARTBEAT_TIMESTAMP: 2016-03-20 04:17:50
RECEIVED_TRANSACTION_SET: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00

Lets stop I/O thread for one channel and see the difference in the above table

mysql> stop slave io_thread for channel ‘master1’;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
GROUP_NAME:
SOURCE_UUID: 96720b8e-bda4-11e5-b8bc-080027173a36
THREAD_ID: NULL
SERVICE_STATE: OFF
COUNT_RECEIVED_HEARTBEATS: 127
LAST_HEARTBEAT_TIMESTAMP: 2016-03-20 04:20:20
RECEIVED_TRANSACTION_SET: 96720b8e-bda4-11e5-b8bc-080027173a36:1-11
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00

3. replication_applier_configuration
This table shows the replication parameters that affect delay configured by the slave server.

mysql> select * from performance_schema.replication_applier_configuration\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
DESIRED_DELAY: 0
*************************** 2. row ***************************
CHANNEL_NAME: master2
DESIRED_DELAY: 0
*************************** 3. row ***************************
CHANNEL_NAME: master3
DESIRED_DELAY: 0
*************************** 4. row ***************************
CHANNEL_NAME: master4
DESIRED_DELAY: 0
*************************** 5. row ***************************
CHANNEL_NAME: master5
DESIRED_DELAY: 0
5 rows in set (0.00 sec)

Let do some changes on replication so that we may see any differnce in this table.

mysql> stop slave for channel ‘master5’;
Query OK, 0 rows affected (0.01 sec)
mysql> change master to master_host=’localhost’, master_port=3315, master_user=’repl’,master_password=’repl123#’, master_delay=60 for channel ‘master5’;
mysql> start slave for channel ‘master5’;
Query OK, 0 rows affected (0.07 sec)
mysql> select * from performance_schema.replication_applier_configuration\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
DESIRED_DELAY: 0
*************************** 2. row ***************************
CHANNEL_NAME: master2
DESIRED_DELAY: 0
*************************** 3. row ***************************
CHANNEL_NAME: master3
DESIRED_DELAY: 0
*************************** 4. row ***************************
CHANNEL_NAME: master4
DESIRED_DELAY: 0
*************************** 5. row ***************************
CHANNEL_NAME: master5
DESIRED_DELAY: 60
5 rows in set (0.00 sec)

4. replication_applier_status

This table shows the transaction execution status on the slave server.

mysql> select * from performance_schema.replication_applier_status\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
*************************** 2. row ***************************
CHANNEL_NAME: master2
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
*************************** 3. row ***************************
CHANNEL_NAME: master3
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
*************************** 4. row ***************************
CHANNEL_NAME: master4
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
*************************** 5. row ***************************
CHANNEL_NAME: master5
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
5 rows in set (0.01 sec)

Here remaining_delay is waiting for DESIRED_DELAY seconds to pass since the master applied an event, this field contains the number of delay seconds remaining. At other times, this field is NULL.
The service_state is effected only when sql_thread is stopped.

5. replication_applier_status_by_coordinator

For a multi-threaded slave, the slave uses multiple worker threads and a coordinator thread to manage them, and this table shows the status of the coordinator thread.
FYI: We have to enable slave_parallel_workers by increasing the number of parallel workers from 0 to our desired number of workers.

mysql> select * from performance_schema.replication_applier_status_by_coordinator\G
Empty set (0.00 sec)
Since default 0 parallel workers or no parallel workers are there the above table will be empty. Lets enable parallel workers.
mysql> set global slave_parallel_type=”LOGICAL_CLOCK”;
Query OK, 0 rows affected (0.00 sec)
mysql> SET GLOBAL slave_parallel_workers=20;
Query OK, 0 rows affected (0.01 sec)
mysql> stop slave;
Query OK, 0 rows affected (0.04 sec)
mysql> start slave;
Query OK, 0 rows affected (0.63 sec)

In our scenario we have 5 channels and have enabled 20 parallel workers. So total number of parallel workers or coordinater threads in a slave we will see as 20*5=100

mysql> select * from performance_schema.replication_applier_status_by_coordinator\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
THREAD_ID: 86
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME: master2
THREAD_ID: 108
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 3. row ***************************
CHANNEL_NAME: master3
THREAD_ID: 130
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 4. row ***************************
CHANNEL_NAME: master4
THREAD_ID: 152
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 5. row ***************************
CHANNEL_NAME: master5
THREAD_ID: 174
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
5 rows in set (0.00 sec)

6.replication_applier_status_by_worker
If the slave is not multi-threaded, this table shows the status of the applier thread.

mysql> select * from performance_schema.replication_applier_status_by_coordinator\G
*************************** 1. row ***************************
CHANNEL_NAME: master1
THREAD_ID: 438
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME: master2
THREAD_ID: 460
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 3. row ***************************
CHANNEL_NAME: master3
THREAD_ID: 482
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 4. row ***************************
CHANNEL_NAME: master4
THREAD_ID: 504
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 5. row ***************************
CHANNEL_NAME: master5
THREAD_ID: 526
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
5 rows in set (0.00 sec)