MySQL高可用之MHA切换测试

来源:http://www.smjxgs.com 作者:王中王高手论坛 人气:103 发布时间:2019-10-29
摘要:  Preface       I've installed MasterHA yesterday,Nowlet's test the master-slave switch and failover feature.   Framework   Hostname IP Port Identity OS Version MySQL Version zlm2 192.168.1.101 3306 master CentOS 7.0 5.7.21 zlm3 192.1

 

Preface

 

    I've installed MasterHA yesterday,Now let's test the master-slave switch and failover feature.

 

Framework

 

Hostname IP Port Identity OS Version MySQL Version
zlm2 192.168.1.101 3306 master CentOS 7.0 5.7.21
zlm3 192.168.1.102 3306 slave/mha-manager CentOS 7.0 5.7.21
null 192.168.1.200 null vip null null

Procedure

 

Test 1:Manual master switchover

 

Check state of MHA-manager on zlm3.

 1 [root@zlm3 07:35:00 ~]
 2 #masterha_check_ssh --conf=/etc/masterha/app1.conf
 3 Fri Aug  3 07:37:13 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
 4 Fri Aug  3 07:37:13 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
 5 Fri Aug  3 07:37:13 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
 6 Fri Aug  3 07:37:13 2018 - [info] Starting SSH connection tests..
 7 Fri Aug  3 07:37:13 2018 - [debug] 
 8 Fri Aug  3 07:37:13 2018 - [debug]  Connecting via SSH from root@192.168.1.101(192.168.1.101:22) to root@192.168.1.102(192.168.1.102:22)..
 9 Fri Aug  3 07:37:13 2018 - [debug]   ok.
10 Fri Aug  3 07:37:14 2018 - [debug] 
11 Fri Aug  3 07:37:13 2018 - [debug]  Connecting via SSH from root@192.168.1.102(192.168.1.102:22) to root@192.168.1.101(192.168.1.101:22)..
12 Fri Aug  3 07:37:13 2018 - [debug]   ok.
13 Fri Aug  3 07:37:14 2018 - [info] All SSH connection tests passed successfully.
14 
15 [root@zlm3 07:37:14 ~]
16 #masterha_check_repl --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf 
17 Fri Aug  3 07:37:37 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
18 Fri Aug  3 07:37:37 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
19 Fri Aug  3 07:37:37 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
20 Fri Aug  3 07:37:37 2018 - [info] MHA::MasterMonitor version 0.56.
21 Fri Aug  3 07:37:38 2018 - [info] GTID failover mode = 1
22 Fri Aug  3 07:37:38 2018 - [info] Dead Servers:
23 Fri Aug  3 07:37:38 2018 - [info] Alive Servers:
24 Fri Aug  3 07:37:38 2018 - [info]   192.168.1.101(192.168.1.101:3306)
25 Fri Aug  3 07:37:38 2018 - [info]   192.168.1.102(192.168.1.102:3306)
26 Fri Aug  3 07:37:38 2018 - [info] Alive Slaves:
27 Fri Aug  3 07:37:38 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
28 Fri Aug  3 07:37:38 2018 - [info]     GTID ON
29 Fri Aug  3 07:37:38 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
30 Fri Aug  3 07:37:38 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
31 Fri Aug  3 07:37:38 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306)
32 Fri Aug  3 07:37:38 2018 - [info] Checking slave configurations..
33 Fri Aug  3 07:37:38 2018 - [info]  read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306).
34 Fri Aug  3 07:37:38 2018 - [info] Checking replication filtering settings..
35 Fri Aug  3 07:37:38 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
36 Fri Aug  3 07:37:38 2018 - [info]  Replication filtering check ok.
37 Fri Aug  3 07:37:38 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
38 Fri Aug  3 07:37:38 2018 - [info] Checking SSH publickey authentication settings on the current master..
39 ssh_exchange_identification: Connection closed by remote host
40 Fri Aug  3 07:37:38 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
41 Fri Aug  3 07:37:38 2018 - [info] 
42 192.168.1.101(192.168.1.101:3306) (current master)
43   --192.168.1.102(192.168.1.102:3306)
44 
45 Fri Aug  3 07:37:38 2018 - [info] Checking replication health on 192.168.1.102..
46 Fri Aug  3 07:37:38 2018 - [info]  ok.
47 Fri Aug  3 07:37:38 2018 - [info] Checking master_ip_failover_script status:
48 Fri Aug  3 07:37:38 2018 - [info]   /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306  --orig_master_ssh_port=3306
49 Fri Aug  3 07:37:38 2018 - [info]  OK.
50 Fri Aug  3 07:37:38 2018 - [warning] shutdown_script is not defined.
51 Fri Aug  3 07:37:38 2018 - [info] Got exit code 0 (Not master dead).
52 
53 MySQL Replication Health is OK.
54 
55 [root@zlm3 07:40:03 ~]
56 #Fri Aug  3 07:40:03 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
57 Fri Aug  3 07:40:03 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
58 Fri Aug  3 07:40:03 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
59 ssh_exchange_identification: Connection closed by remote host
60 ^C
61 
62 [root@zlm3 07:40:11 ~]
63 #masterha_check_status --conf=/etc/masterha/app1.conf
64 app1 (pid:5628) is running(0:PING_OK), master:192.168.1.101

 

Switch master to slave and make it become a new slave of new master.

  1 [root@zlm3 08:21:27 ~]
  2 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=60
  3 Fri Aug  3 08:21:29 2018 - [info] MHA::MasterRotate version 0.56.
  4 Fri Aug  3 08:21:29 2018 - [info] Starting online master switch..
  5 Fri Aug  3 08:21:29 2018 - [info] 
  6 Fri Aug  3 08:21:29 2018 - [info] * Phase 1: Configuration Check Phase..
  7 Fri Aug  3 08:21:29 2018 - [info] 
  8 Fri Aug  3 08:21:29 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
  9 Fri Aug  3 08:21:29 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
 10 Fri Aug  3 08:21:29 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
 11 Fri Aug  3 08:21:30 2018 - [info] GTID failover mode = 1
 12 Fri Aug  3 08:21:30 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306)
 13 Fri Aug  3 08:21:30 2018 - [info] Alive Slaves:
 14 Fri Aug  3 08:21:30 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 15 Fri Aug  3 08:21:30 2018 - [info]     GTID ON
 16 Fri Aug  3 08:21:30 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 17 Fri Aug  3 08:21:30 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 18 
 19 It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:3306)? (YES/no): yes
 20 Fri Aug  3 08:21:33 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
 21 Fri Aug  3 08:21:33 2018 - [info]  ok.
 22 Fri Aug  3 08:21:33 2018 - [info] Checking MHA is not monitoring or doing failover..
 23 Fri Aug  3 08:21:33 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterRotate.pm, ln142] Getting advisory lock failed on the current master. MHA Monitor runs on the current master. Stop MHA Manager/Monitor and try again.
 24 Fri Aug  3 08:21:33 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR:  at /usr/bin/masterha_master_switch line 53.
 25 
 26 //It means that we should stop MHA-manager when donging switchover master.
 27 
 28 [root@zlm3 08:21:33 ~]
 29 #masterha_stop --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
 30 Stopped app1 successfully.
 31 [1]   Exit 1                  masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
 32 
 33 [root@zlm3 08:28:07 ~]
 34 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=60
 35 Fri Aug  3 08:28:21 2018 - [info] MHA::MasterRotate version 0.56.
 36 Fri Aug  3 08:28:21 2018 - [info] Starting online master switch..
 37 Fri Aug  3 08:28:21 2018 - [info] 
 38 Fri Aug  3 08:28:21 2018 - [info] * Phase 1: Configuration Check Phase..
 39 Fri Aug  3 08:28:21 2018 - [info] 
 40 Fri Aug  3 08:28:21 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
 41 Fri Aug  3 08:28:21 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
 42 Fri Aug  3 08:28:21 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
 43 Fri Aug  3 08:28:22 2018 - [info] GTID failover mode = 1
 44 Fri Aug  3 08:28:22 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306)
 45 Fri Aug  3 08:28:22 2018 - [info] Alive Slaves:
 46 Fri Aug  3 08:28:22 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 47 Fri Aug  3 08:28:22 2018 - [info]     GTID ON
 48 Fri Aug  3 08:28:22 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 49 Fri Aug  3 08:28:22 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 50 
 51 It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:3306)? (YES/no): yes
 52 Fri Aug  3 08:28:25 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
 53 Fri Aug  3 08:28:25 2018 - [info]  ok.
 54 Fri Aug  3 08:28:25 2018 - [info] Checking MHA is not monitoring or doing failover..
 55 Fri Aug  3 08:28:25 2018 - [info] Checking replication health on 192.168.1.102..
 56 Fri Aug  3 08:28:25 2018 - [info]  ok.
 57 Fri Aug  3 08:28:25 2018 - [info] 192.168.1.102 can be new master.
 58 Fri Aug  3 08:28:25 2018 - [info] 
 59 From:
 60 192.168.1.101(192.168.1.101:3306) (current master)
 61   --192.168.1.102(192.168.1.102:3306)
 62 
 63 To:
 64 192.168.1.102(192.168.1.102:3306) (new master)
 65   --192.168.1.101(192.168.1.101:3306)
 66 
 67 Starting master switch from 192.168.1.101(192.168.1.101:3306) to 192.168.1.102(192.168.1.102:3306)? (yes/NO): yes
 68 Fri Aug  3 08:28:31 2018 - [info] Checking whether 192.168.1.102(192.168.1.102:3306) is ok for the new master..
 69 Fri Aug  3 08:28:31 2018 - [info]  ok.
 70 Fri Aug  3 08:28:31 2018 - [info] 192.168.1.101(192.168.1.101:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
 71 Fri Aug  3 08:28:31 2018 - [info] 192.168.1.101(192.168.1.101:3306): Resetting slave pointing to the dummy host.
 72 Fri Aug  3 08:28:31 2018 - [info] ** Phase 1: Configuration Check Phase completed.
 73 Fri Aug  3 08:28:31 2018 - [info] 
 74 Fri Aug  3 08:28:31 2018 - [info] * Phase 2: Rejecting updates Phase..
 75 Fri Aug  3 08:28:31 2018 - [info] 
 76 Fri Aug  3 08:28:31 2018 - [info] Executing master ip online change script to disable write on the current master:
 77 Fri Aug  3 08:28:31 2018 - [info]   /etc/masterha/master_ip_online_change --command=stop --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root  --orig_master_ssh_port=3306  --new_master_ssh_port=3306 --orig_master_is_new_slave
 78 Unknown option: new_master_ssh_port
 79 Fri Aug  3 08:28:32 2018 116409 Set read_only on the new master.. ok.
 80 Fri Aug  3 08:28:32 2018 125643 drop vip 10.33.101.239..
 81 ssh_exchange_identification: Connection closed by remote host
 82 Fri Aug  3 08:28:32 2018 142948 Waiting all running 1 threads are disconnected.. (max 1500 milliseconds)
 83 {'Time' => '13435','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
 84 Fri Aug  3 08:28:32 2018 646769 Waiting all running 1 threads are disconnected.. (max 1000 milliseconds)
 85 {'Time' => '13435','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
 86 Fri Aug  3 08:28:33 2018 149221 Waiting all running 1 threads are disconnected.. (max 500 milliseconds)
 87 {'Time' => '13436','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
 88 Fri Aug  3 08:28:33 2018 650816 Set read_only=1 on the orig master.. ok.
 89 Fri Aug  3 08:28:33 2018 653323 Waiting all running 1 queries are disconnected.. (max 500 milliseconds)
 90 {'Time' => '13436','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
 91 Fri Aug  3 08:28:34 2018 154965 Killing all application threads..
 92 Fri Aug  3 08:28:34 2018 167919 done.
 93 Fri Aug  3 08:28:34 2018 - [info]  ok.
 94 Fri Aug  3 08:28:34 2018 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
 95 Fri Aug  3 08:28:34 2018 - [info] Executing FLUSH TABLES WITH READ LOCK..
 96 Fri Aug  3 08:28:34 2018 - [info]  ok.
 97 Fri Aug  3 08:28:34 2018 - [info] Orig master binlog:pos is mysql-bin.000050:2361.
 98 Fri Aug  3 08:28:34 2018 - [info]  Waiting to execute all relay logs on 192.168.1.102(192.168.1.102:3306)..
 99 Fri Aug  3 08:28:34 2018 - [info]  master_pos_wait(mysql-bin.000050:2361) completed on 192.168.1.102(192.168.1.102:3306). Executed 0 events.
100 Fri Aug  3 08:28:34 2018 - [info]   done.
101 Fri Aug  3 08:28:34 2018 - [info] Getting new master's binlog name and position..
102 Fri Aug  3 08:28:34 2018 - [info]  mysql-bin.000003:2321
103 Fri Aug  3 08:28:34 2018 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
104 Fri Aug  3 08:28:34 2018 - [info] Executing master ip online change script to allow write on the new master:
105 Fri Aug  3 08:28:34 2018 - [info]   /etc/masterha/master_ip_online_change --command=start --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root  --orig_master_ssh_port=3306  --new_master_ssh_port=3306 --orig_master_is_new_slave
106 Unknown option: new_master_ssh_port
107 Fri Aug  3 08:28:34 2018 327146 Set read_only=0 on the new master.
108 Fri Aug  3 08:28:34 2018 328259Add vip 10.33.101.239 on p3p1..
109 ssh_exchange_identification: Connection closed by remote host
110 Fri Aug  3 08:28:34 2018 - [info]  ok.
111 Fri Aug  3 08:28:34 2018 - [info] 
112 Fri Aug  3 08:28:34 2018 - [info] * Switching slaves in parallel..
113 Fri Aug  3 08:28:34 2018 - [info] 
114 Fri Aug  3 08:28:34 2018 - [info] Unlocking all tables on the orig master:
115 Fri Aug  3 08:28:34 2018 - [info] Executing UNLOCK TABLES..
116 Fri Aug  3 08:28:34 2018 - [info]  ok.
117 Fri Aug  3 08:28:34 2018 - [info] Starting orig master as a new slave..
118 Fri Aug  3 08:28:34 2018 - [info]  Resetting slave 192.168.1.101(192.168.1.101:3306) and starting replication from the new master 192.168.1.102(192.168.1.102:3306)..
119 Fri Aug  3 08:28:34 2018 - [info]  Executed CHANGE MASTER.
120 Fri Aug  3 08:28:35 2018 - [info]  Slave started.
121 Fri Aug  3 08:28:35 2018 - [info] All new slave servers switched successfully.
122 Fri Aug  3 08:28:35 2018 - [info] 
123 Fri Aug  3 08:28:35 2018 - [info] * Phase 5: New master cleanup phase..
124 Fri Aug  3 08:28:35 2018 - [info] 
125 Fri Aug  3 08:28:35 2018 - [info]  192.168.1.102: Resetting slave info succeeded.
126 Fri Aug  3 08:28:35 2018 - [info] Switching master to 192.168.1.102(192.168.1.102:3306) completed successfully.
127 
128 [root@zlm3 08:28:35 ~]
129 #

 

Check the master-slave replication status.

 1 //New master(original slave)
 2 (zlm@192.168.1.102 3306)[(none)]>show master status;
 3  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
 4 | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                              |
 5  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
 6 | mysql-bin.000003 |     2321 |              |                  | 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 |
 7  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
 8 1 row in set (0.00 sec)
 9 
10 (zlm@192.168.1.102 3306)[(none)]>show slave statusG
11 Empty set (0.00 sec)
12 
13 //New slave(original master)
14 (zlm@192.168.1.101 3306)[(none)]>show master status;
15  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
16 | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                              |
17  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
18 | mysql-bin.000050 |     2361 |              |                  | 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 |
19  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
20 1 row in set (0.01 sec)
21 
22 (zlm@192.168.1.101 3306)[(none)]>show slave statusG
23 *************************** 1. row ***************************
24                Slave_IO_State: Waiting for master to send event
25                   Master_Host: 192.168.1.102
26                   Master_User: repl
27                   Master_Port: 3306
28                 Connect_Retry: 60
29               Master_Log_File: mysql-bin.000003
30           Read_Master_Log_Pos: 2321
31                Relay_Log_File: relay-bin.000002
32                 Relay_Log_Pos: 398
33         Relay_Master_Log_File: mysql-bin.000003
34              Slave_IO_Running: Yes
35             Slave_SQL_Running: Yes
36               Replicate_Do_DB: 
37           Replicate_Ignore_DB: 
38            Replicate_Do_Table: 
39        Replicate_Ignore_Table: 
40       Replicate_Wild_Do_Table: 
41   Replicate_Wild_Ignore_Table: 
42                    Last_Errno: 0
43                    Last_Error: 
44                  Skip_Counter: 0
45           Exec_Master_Log_Pos: 2321
46               Relay_Log_Space: 591
47               Until_Condition: None
48                Until_Log_File: 
49                 Until_Log_Pos: 0
50            Master_SSL_Allowed: No
51            Master_SSL_CA_File: 
52            Master_SSL_CA_Path: 
53               Master_SSL_Cert: 
54             Master_SSL_Cipher: 
55                Master_SSL_Key: 
56         Seconds_Behind_Master: 0
57 Master_SSL_Verify_Server_Cert: No
58                 Last_IO_Errno: 0
59                 Last_IO_Error: 
60                Last_SQL_Errno: 0
61                Last_SQL_Error: 
62   Replicate_Ignore_Server_Ids: 
63              Master_Server_Id: 1023306
64                   Master_UUID: 842ea497-9551-11e8-83ca-080027de0e0e
65              Master_Info_File: mysql.slave_master_info
66                     SQL_Delay: 0
67           SQL_Remaining_Delay: NULL
68       Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
69            Master_Retry_Count: 86400
70                   Master_Bind: 
71       Last_IO_Error_Timestamp: 
72      Last_SQL_Error_Timestamp: 
73                Master_SSL_Crl: 
74            Master_SSL_Crlpath: 
75            Retrieved_Gtid_Set: 
76             Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259
77                 Auto_Position: 1
78          Replicate_Rewrite_DB: 
79                  Channel_Name: 
80            Master_TLS_Version: 
81 1 row in set (0.00 sec)

 

Check the log of MasterHA on zlm3.

 1 [root@zlm3 08:28:35 ~]
 2 #cd /var/log/masterha/app1
 3 
 4 [root@zlm3 08:29:12 /var/log/masterha/app1]
 5 #cat app1.log 
 6 Fri Aug  3 07:39:13 2018 - [info] MHA::MasterMonitor version 0.56.
 7 Fri Aug  3 07:39:14 2018 - [info] GTID failover mode = 1
 8 Fri Aug  3 07:39:14 2018 - [info] Dead Servers:
 9 Fri Aug  3 07:39:14 2018 - [info] Alive Servers:
10 Fri Aug  3 07:39:14 2018 - [info]   192.168.1.101(192.168.1.101:3306)
11 Fri Aug  3 07:39:14 2018 - [info]   192.168.1.102(192.168.1.102:3306)
12 Fri Aug  3 07:39:14 2018 - [info] Alive Slaves:
13 Fri Aug  3 07:39:14 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
14 Fri Aug  3 07:39:14 2018 - [info]     GTID ON
15 Fri Aug  3 07:39:14 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
16 Fri Aug  3 07:39:14 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
17 Fri Aug  3 07:39:14 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306)
18 Fri Aug  3 07:39:14 2018 - [info] Checking slave configurations..
19 Fri Aug  3 07:39:14 2018 - [info]  read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306).
20 Fri Aug  3 07:39:14 2018 - [info] Checking replication filtering settings..
21 Fri Aug  3 07:39:14 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
22 Fri Aug  3 07:39:14 2018 - [info]  Replication filtering check ok.
23 Fri Aug  3 07:39:14 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
24 Fri Aug  3 07:39:14 2018 - [info] Checking SSH publickey authentication settings on the current master..
25 Fri Aug  3 07:39:14 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
26 Fri Aug  3 07:39:14 2018 - [info] 
27 192.168.1.101(192.168.1.101:3306) (current master)
28   --192.168.1.102(192.168.1.102:3306)
29 
30 Fri Aug  3 07:39:14 2018 - [info] Checking master_ip_failover_script status:
31 Fri Aug  3 07:39:14 2018 - [info]   /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306  --orig_master_ssh_port=3306
32 Fri Aug  3 07:39:14 2018 - [info]  OK.
33 Fri Aug  3 07:39:14 2018 - [warning] shutdown_script is not defined.
34 Fri Aug  3 07:39:14 2018 - [info] Set master ping interval 1 seconds.
35 Fri Aug  3 07:39:14 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
36 Fri Aug  3 07:39:14 2018 - [info] Starting ping health check on 192.168.1.101(192.168.1.101:3306)..
37 Fri Aug  3 07:39:14 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
38 Fri Aug  3 07:39:27 2018 - [info] Got terminate signal. Exit.
39 Fri Aug  3 07:40:03 2018 - [info] MHA::MasterMonitor version 0.56.
40 Fri Aug  3 07:40:04 2018 - [info] GTID failover mode = 1
41 Fri Aug  3 07:40:04 2018 - [info] Dead Servers:
42 Fri Aug  3 07:40:04 2018 - [info] Alive Servers:
43 Fri Aug  3 07:40:04 2018 - [info]   192.168.1.101(192.168.1.101:3306)
44 Fri Aug  3 07:40:04 2018 - [info]   192.168.1.102(192.168.1.102:3306)
45 Fri Aug  3 07:40:04 2018 - [info] Alive Slaves:
46 Fri Aug  3 07:40:04 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
47 Fri Aug  3 07:40:04 2018 - [info]     GTID ON
48 Fri Aug  3 07:40:04 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
49 Fri Aug  3 07:40:04 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
50 Fri Aug  3 07:40:04 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306)
51 Fri Aug  3 07:40:04 2018 - [info] Checking slave configurations..
52 Fri Aug  3 07:40:04 2018 - [info]  read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306).
53 Fri Aug  3 07:40:04 2018 - [info] Checking replication filtering settings..
54 Fri Aug  3 07:40:04 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
55 Fri Aug  3 07:40:04 2018 - [info]  Replication filtering check ok.
56 Fri Aug  3 07:40:04 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
57 Fri Aug  3 07:40:04 2018 - [info] Checking SSH publickey authentication settings on the current master..
58 Fri Aug  3 07:40:04 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
59 Fri Aug  3 07:40:04 2018 - [info] 
60 192.168.1.101(192.168.1.101:3306) (current master)
61   --192.168.1.102(192.168.1.102:3306)
62 
63 Fri Aug  3 07:40:04 2018 - [info] Checking master_ip_failover_script status:
64 Fri Aug  3 07:40:04 2018 - [info]   /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306  --orig_master_ssh_port=3306
65 Fri Aug  3 07:40:04 2018 - [info]  OK.
66 Fri Aug  3 07:40:04 2018 - [warning] shutdown_script is not defined.
67 Fri Aug  3 07:40:04 2018 - [info] Set master ping interval 1 seconds.
68 Fri Aug  3 07:40:04 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
69 Fri Aug  3 07:40:04 2018 - [info] Starting ping health check on 192.168.1.101(192.168.1.101:3306)..
70 Fri Aug  3 07:40:04 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
71 Fri Aug  3 08:28:07 2018 - [info] Got terminate signal. Exit.

 

Test 2:Manual master failover

 

Execute "masterha_master_switch" script again to generate a failover on zlm2.

  1 [root@zlm2 10:11:10 ~]
  2 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover 
  3 --dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102.
  4 --dead_master_port=<dead_master_port> is not set. Using 3306.
  5 Fri Aug  3 10:11:50 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
  6 Fri Aug  3 10:11:50 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
  7 Fri Aug  3 10:11:50 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
  8 Fri Aug  3 10:11:50 2018 - [info] MHA::MasterFailover version 0.56.
  9 Fri Aug  3 10:11:50 2018 - [info] Starting master failover.
 10 Fri Aug  3 10:11:50 2018 - [info] 
 11 Fri Aug  3 10:11:50 2018 - [info] * Phase 1: Configuration Check Phase..
 12 Fri Aug  3 10:11:50 2018 - [info] 
 13 Fri Aug  3 10:11:51 2018 - [info] GTID failover mode = 1
 14 Fri Aug  3 10:11:51 2018 - [info] Dead Servers:
 15 Fri Aug  3 10:11:51 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln187] None of server is dead. Stop failover.
 16 Fri Aug  3 10:11:51 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR:  at /usr/bin/masterha_master_switch line 53.
 17 
 18 //Stop mysqld of master on zlm3.
 19 [root@zlm3 10:13:56 ~]
 20 #mysqladmin shutdown
 21 
 22 [root@zlm3 10:14:04 ~]
 23 #ps aux|grep mysqld
 24 mysql     5368  0.0 19.6 1110812 200292 pts/0  Sl   04:44   0:09 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
 25 root      8827  0.0  0.0 112640   960 pts/0    R    10:14   0:00 grep --color=auto mysqld
 26 
 27 [root@zlm3 10:14:08 ~]
 28 #ps aux|grep mysqld
 29 mysql     5368  0.0 19.6 1110812 200292 pts/0  Sl   04:44   0:09 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
 30 root      8833  0.0  0.0 112640   960 pts/0    R    10:14   0:00 grep --color=auto mysqld
 31 
 32 [root@zlm3 10:14:12 ~]
 33 #ps aux|grep mysqld
 34 mysql     5368  0.0 19.1 995088 194692 pts/0   Sl   04:44   0:09 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
 35 root      8839  0.0  0.0 112640   960 pts/0    R    10:14   0:00 grep --color=auto mysqld
 36 
 37 [root@zlm3 10:14:23 ~]
 38 #ps aux|grep mysqld
 39 root      8854  0.0  0.0 112640   960 pts/0    R    10:14   0:00 grep --color=auto mysqld
 40 
 41 //Execute the above command again on zlm2.
 42 [root@zlm2 10:15:43 ~]
 43 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover 
 44 --dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102.
 45 --dead_master_port=<dead_master_port> is not set. Using 3306.
 46 Fri Aug  3 10:15:43 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
 47 Fri Aug  3 10:15:43 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
 48 Fri Aug  3 10:15:43 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
 49 Fri Aug  3 10:15:43 2018 - [info] MHA::MasterFailover version 0.56.
 50 Fri Aug  3 10:15:43 2018 - [info] Starting master failover.
 51 Fri Aug  3 10:15:43 2018 - [info] 
 52 Fri Aug  3 10:15:43 2018 - [info] * Phase 1: Configuration Check Phase..
 53 Fri Aug  3 10:15:43 2018 - [info] 
 54 Fri Aug  3 10:15:44 2018 - [info] GTID failover mode = 1
 55 Fri Aug  3 10:15:44 2018 - [info] Dead Servers:
 56 Fri Aug  3 10:15:44 2018 - [info]   192.168.1.102(192.168.1.102:3306)
 57 Fri Aug  3 10:15:44 2018 - [info] Checking master reachability via MySQL(double check)...
 58 Fri Aug  3 10:15:44 2018 - [info]  ok.
 59 Fri Aug  3 10:15:44 2018 - [info] Alive Servers:
 60 Fri Aug  3 10:15:44 2018 - [info]   192.168.1.101(192.168.1.101:3306)
 61 Fri Aug  3 10:15:44 2018 - [info] Alive Slaves:
 62 Fri Aug  3 10:15:44 2018 - [info]   192.168.1.101(192.168.1.101:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 63 Fri Aug  3 10:15:44 2018 - [info]     GTID ON
 64 Fri Aug  3 10:15:44 2018 - [info]     Replicating from 192.168.1.102(192.168.1.102:3306)
 65 Fri Aug  3 10:15:44 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 66 Master 192.168.1.102(192.168.1.102:3306) is dead. Proceed? (yes/NO): yes
 67 Fri Aug  3 10:15:46 2018 - [info] Starting GTID based failover.
 68 Fri Aug  3 10:15:46 2018 - [info] 
 69 Fri Aug  3 10:15:46 2018 - [info] ** Phase 1: Configuration Check Phase completed.
 70 Fri Aug  3 10:15:46 2018 - [info] 
 71 Fri Aug  3 10:15:46 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
 72 Fri Aug  3 10:15:46 2018 - [info] 
 73 ssh: connect to host 192.168.1.102 port 3306: Connection refused
 74 Fri Aug  3 10:15:46 2018 - [warning] HealthCheck: SSH to 192.168.1.102 is NOT reachable.
 75 Fri Aug  3 10:15:46 2018 - [info] Forcing shutdown so that applications never connect to the current master..
 76 Fri Aug  3 10:15:46 2018 - [info] Executing master IP deactivation script:
 77 Fri Aug  3 10:15:46 2018 - [info]   /etc/masterha/master_ip_failover --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port=3306 --command=stop  --orig_master_ssh_port=3306
 78 ssh: connect to host 192.168.1.102 port 3306: Connection refused
 79 Fri Aug  3 10:15:48 2018 - [info]  done.
 80 Fri Aug  3 10:15:48 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
 81 Fri Aug  3 10:15:48 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
 82 Fri Aug  3 10:15:48 2018 - [info] 
 83 Fri Aug  3 10:15:48 2018 - [info] * Phase 3: Master Recovery Phase..
 84 Fri Aug  3 10:15:48 2018 - [info] 
 85 Fri Aug  3 10:15:48 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
 86 Fri Aug  3 10:15:48 2018 - [info] 
 87 Fri Aug  3 10:15:48 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000003:2321
 88 Fri Aug  3 10:15:48 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
 89 Fri Aug  3 10:15:48 2018 - [info]   192.168.1.101(192.168.1.101:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 90 Fri Aug  3 10:15:48 2018 - [info]     GTID ON
 91 Fri Aug  3 10:15:48 2018 - [info]     Replicating from 192.168.1.102(192.168.1.102:3306)
 92 Fri Aug  3 10:15:48 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 93 Fri Aug  3 10:15:48 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000003:2321
 94 Fri Aug  3 10:15:48 2018 - [info] Oldest slaves:
 95 Fri Aug  3 10:15:48 2018 - [info]   192.168.1.101(192.168.1.101:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 96 Fri Aug  3 10:15:48 2018 - [info]     GTID ON
 97 Fri Aug  3 10:15:48 2018 - [info]     Replicating from 192.168.1.102(192.168.1.102:3306)
 98 Fri Aug  3 10:15:48 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 99 Fri Aug  3 10:15:48 2018 - [info] 
100 Fri Aug  3 10:15:48 2018 - [info] * Phase 3.3: Determining New Master Phase..
101 Fri Aug  3 10:15:48 2018 - [info] 
102 Fri Aug  3 10:15:48 2018 - [info] 192.168.1.101 can be new master.
103 Fri Aug  3 10:15:48 2018 - [info] New master is 192.168.1.101(192.168.1.101:3306)
104 Fri Aug  3 10:15:48 2018 - [info] Starting master failover..
105 Fri Aug  3 10:15:48 2018 - [info] 
106 From:
107 192.168.1.102(192.168.1.102:3306) (current master)
108   --192.168.1.101(192.168.1.101:3306)
109 
110 To:
111 192.168.1.101(192.168.1.101:3306) (new master)
112 
113 Starting master switch from 192.168.1.102(192.168.1.102:3306) to 192.168.1.101(192.168.1.101:3306)? (yes/NO): yes
114 Fri Aug  3 10:15:56 2018 - [info] New master decided manually is 192.168.1.101(192.168.1.101:3306)
115 Fri Aug  3 10:15:56 2018 - [info] 
116 Fri Aug  3 10:15:56 2018 - [info] * Phase 3.3: New Master Recovery Phase..
117 Fri Aug  3 10:15:56 2018 - [info] 
118 Fri Aug  3 10:15:56 2018 - [info]  Waiting all logs to be applied.. 
119 Fri Aug  3 10:15:56 2018 - [info]   done.
120 Fri Aug  3 10:15:56 2018 - [info] Getting new master's binlog name and position..
121 Fri Aug  3 10:15:56 2018 - [info]  mysql-bin.000051:190
122 Fri Aug  3 10:15:56 2018 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.101', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
123 Fri Aug  3 10:15:56 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000051, 190, 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259
124 Fri Aug  3 10:15:56 2018 - [info] Executing master IP activate script:
125 Fri Aug  3 10:15:56 2018 - [info]   /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port=3306 --new_master_host=192.168.1.101 --new_master_ip=192.168.1.101 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm'  --orig_master_ssh_port=3306  --new_master_ssh_port=3306
126 Unknown option: new_master_ssh_port
127 Set read_only=0 on the new master.
128 ssh_exchange_identification: Connection closed by remote host
129 Fri Aug  3 10:15:56 2018 - [info]  OK.
130 Fri Aug  3 10:15:56 2018 - [info] ** Finished master recovery successfully.
131 Fri Aug  3 10:15:56 2018 - [info] * Phase 3: Master Recovery Phase completed.
132 Fri Aug  3 10:15:56 2018 - [info] 
133 Fri Aug  3 10:15:56 2018 - [info] * Phase 4: Slaves Recovery Phase..
134 Fri Aug  3 10:15:56 2018 - [info] 
135 Fri Aug  3 10:15:56 2018 - [info] 
136 Fri Aug  3 10:15:56 2018 - [info] * Phase 4.1: Starting Slaves in parallel..
137 Fri Aug  3 10:15:56 2018 - [info] 
138 Fri Aug  3 10:15:56 2018 - [info] All new slave servers recovered successfully.
139 Fri Aug  3 10:15:56 2018 - [info] 
140 Fri Aug  3 10:15:56 2018 - [info] * Phase 5: New master cleanup phase..
141 Fri Aug  3 10:15:56 2018 - [info] 
142 Fri Aug  3 10:15:56 2018 - [info] Resetting slave info on the new master..
143 Fri Aug  3 10:15:56 2018 - [info]  192.168.1.101: Resetting slave info succeeded.
144 Fri Aug  3 10:15:56 2018 - [info] Master failover to 192.168.1.101(192.168.1.101:3306) completed successfully.
145 Fri Aug  3 10:15:56 2018 - [info] 
146 
147 ----- Failover Report -----
148 
149 app1: MySQL Master failover 192.168.1.102(192.168.1.102:3306) to 192.168.1.101(192.168.1.101:3306) succeeded
150 
151 Master 192.168.1.102(192.168.1.102:3306) is down!
152 
153 Check MHA Manager logs at zlm2 for details.
154 
155 Started manual(interactive) failover.
156 Invalidated master IP address on 192.168.1.102(192.168.1.102:3306)
157 Selected 192.168.1.101(192.168.1.101:3306) as a new master.
158 192.168.1.101(192.168.1.101:3306): OK: Applying all logs succeeded.
159 192.168.1.101(192.168.1.101:3306): OK: Activated master IP address.
160 192.168.1.101(192.168.1.101:3306): Resetting slave info succeeded.
161 Master failover to 192.168.1.101(192.168.1.101:3306) completed successfully.

 

Check the status of new master on zlm2.

 1 (zlm@192.168.1.101 3306)[(none)]>show master status;
 2  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
 3 | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                              |
 4  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
 5 | mysql-bin.000051 |      190 |              |                  | 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 |
 6  ------------------ ---------- -------------- ------------------ ------------------------------------------------ 
 7 1 row in set (0.00 sec)
 8 
 9 (zlm@192.168.1.101 3306)[(none)]>show slave statusG
10 Empty set (0.00 sec)

4887王中王鉄算盘奖结果, 

Check the file and log on MasterHA.

 1 [root@zlm2 10:15:56 ~]
 2 #cd /var/log/masterha/app1
 3 
 4 [root@zlm2 10:20:04 /var/log/masterha/app1]
 5 #ls -l
 6 total 4
 7 -rw-r--r-- 1 root root    0 Aug  3 10:15 app1.failover.complete 
 8 -rw-r--r-- 1 root root 3883 Aug  2 11:29 app1.log
 9 
10 //The option of "--ignore_last_failover" can neglect the influence of existence of "app1.failover.complete".Otherwise,the failover operation will be terminated by error.
11 //This file will be created after a failover operation and it will be created only on the original slave who wants to become a new master.
12 
13 [root@zlm2 10:20:05 /var/log/masterha/app1]
14 #cat app1.log
15 Thu Aug  2 11:12:03 2018 - [info] MHA::MasterMonitor version 0.56.
16 Thu Aug  2 11:12:04 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.101(192.168.1.101:3306) :1045:Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings.
17  at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297.
18 Thu Aug  2 11:12:04 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.102(192.168.1.102:3306) :1045:Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings.
19  at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297.
20 Thu Aug  2 11:12:05 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations
21 Thu Aug  2 11:12:05 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations.  at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line 326.
22 Thu Aug  2 11:12:05 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.
23 Thu Aug  2 11:12:05 2018 - [info] Got exit code 1 (Not master dead).
24 Thu Aug  2 11:13:56 2018 - [info] MHA::MasterMonitor version 0.56.
25 Thu Aug  2 11:13:57 2018 - [info] GTID failover mode = 1
26 Thu Aug  2 11:13:57 2018 - [info] Dead Servers:
27 Thu Aug  2 11:13:57 2018 - [info] Alive Servers:
28 Thu Aug  2 11:13:57 2018 - [info]   192.168.1.101(192.168.1.101:3306)
29 Thu Aug  2 11:13:57 2018 - [info]   192.168.1.102(192.168.1.102:3306)
30 Thu Aug  2 11:13:57 2018 - [info] Alive Slaves:
31 Thu Aug  2 11:13:57 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
32 Thu Aug  2 11:13:57 2018 - [info]     GTID ON
33 Thu Aug  2 11:13:57 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
34 Thu Aug  2 11:13:57 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
35 Thu Aug  2 11:13:57 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306)
36 Thu Aug  2 11:13:57 2018 - [info] Checking slave configurations..
37 Thu Aug  2 11:13:57 2018 - [info]  read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306).
38 Thu Aug  2 11:13:57 2018 - [info] Checking replication filtering settings..
39 Thu Aug  2 11:13:57 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
40 Thu Aug  2 11:13:57 2018 - [info]  Replication filtering check ok.
41 Thu Aug  2 11:13:57 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
42 Thu Aug  2 11:13:57 2018 - [info] Checking SSH publickey authentication settings on the current master..
43 Thu Aug  2 11:13:57 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
44 Thu Aug  2 11:13:57 2018 - [info] 
45 192.168.1.101(192.168.1.101:3306) (current master)
46   --192.168.1.102(192.168.1.102:3306)
47 
48 Thu Aug  2 11:13:57 2018 - [info] Checking master_ip_failover_script status:
49 Thu Aug  2 11:13:57 2018 - [info]   /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306  --orig_master_ssh_port=3306
50 Thu Aug  2 11:13:57 2018 - [info]  OK.
51 Thu Aug  2 11:13:57 2018 - [warning] shutdown_script is not defined.
52 Thu Aug  2 11:13:57 2018 - [info] Set master ping interval 1 seconds.
53 Thu Aug  2 11:13:57 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
54 Thu Aug  2 11:13:57 2018 - [info] Starting ping health check on 192.168.1.101(192.168.1.101:3306)..
55 Thu Aug  2 11:13:57 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
56 Thu Aug  2 11:29:51 2018 - [info] Got terminate signal. Exit.

 

Test3: Automation of master switchover.

 

Repair the salve replication on zlm3.

 1 [root@zlm3 10:14:24 ~]
 2 #mysql
 3 Welcome to the MySQL monitor.  Commands end with ; or g.
 4 Your MySQL connection id is 3
 5 Server version: 5.7.21-log MySQL Community Server (GPL)
 6 
 7 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
 8 
 9 Oracle is a registered trademark of Oracle Corporation and/or its
10 affiliates. Other names may be trademarks of their respective
11 owners.
12 
13 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
14 
15 (zlm@192.168.1.102 3306)[(none)]>change master to 
16     -> master_host='192.168.1.101',
17     -> master_port=3306,
18     -> master_user='repl',
19     -> master_password='repl4slave',
20     -> master_auto_position=1;
21 Query OK, 0 rows affected, 2 warnings (0.02 sec)
22 
23 (zlm@192.168.1.102 3306)[(none)]>show slave statusG
24 *************************** 1. row ***************************
25                Slave_IO_State: Waiting for master to send event
26                   Master_Host: 192.168.1.101
27                   Master_User: repl
28                   Master_Port: 3306
29                 Connect_Retry: 60
30               Master_Log_File: mysql-bin.000051
31           Read_Master_Log_Pos: 190
32                Relay_Log_File: relay-bin.000002
33                 Relay_Log_Pos: 355
34         Relay_Master_Log_File: mysql-bin.000051
35              Slave_IO_Running: Yes
36             Slave_SQL_Running: Yes
37               Replicate_Do_DB: 
38           Replicate_Ignore_DB: 
39            Replicate_Do_Table: 
40        Replicate_Ignore_Table: 
41       Replicate_Wild_Do_Table: 
42   Replicate_Wild_Ignore_Table: 
43                    Last_Errno: 0
44                    Last_Error: 
45                  Skip_Counter: 0
46           Exec_Master_Log_Pos: 190
47               Relay_Log_Space: 548
48               Until_Condition: None
49                Until_Log_File: 
50                 Until_Log_Pos: 0
51            Master_SSL_Allowed: No
52            Master_SSL_CA_File: 
53            Master_SSL_CA_Path: 
54               Master_SSL_Cert: 
55             Master_SSL_Cipher: 
56                Master_SSL_Key: 
57         Seconds_Behind_Master: 0
58 Master_SSL_Verify_Server_Cert: No
59                 Last_IO_Errno: 0
60                 Last_IO_Error: 
61                Last_SQL_Errno: 0
62                Last_SQL_Error: 
63   Replicate_Ignore_Server_Ids: 
64              Master_Server_Id: 1013306
65                   Master_UUID: 1b7181ee-6eaf-11e8-998e-080027de0e0e
66              Master_Info_File: mysql.slave_master_info
67                     SQL_Delay: 0
68           SQL_Remaining_Delay: NULL
69       Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
70            Master_Retry_Count: 86400
71                   Master_Bind: 
72       Last_IO_Error_Timestamp: 
73      Last_SQL_Error_Timestamp: 
74                Master_SSL_Crl: 
75            Master_SSL_Crlpath: 
76            Retrieved_Gtid_Set: 
77             Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259
78                 Auto_Position: 1
79          Replicate_Rewrite_DB: 
80                  Channel_Name: 
81            Master_TLS_Version: 
82 1 row in set (0.00 sec)

 

Start MasterHA-manager.

 1 [root@zlm3 10:48:00 /var/log/masterha/app1]
 2 #nohup masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf &
 3 [1] 9265
 4 nohup: ignoring input and appending output to ‘nohup.out’
 5 
 6 [root@zlm3 10:48:12 /var/log/masterha/app1]
 7 #ls -l
 8 total 24
 9 -rw-r--r-- 1 root root 16370 Aug  3 10:48 app1.log
10 -rw-r--r-- 1 root root    35 Aug  3 10:48 app1.master_status.health //This file is created only when MasterHA-manager is running.It will continuously record the health status between slave and master.
11 -rw------- 1 root root   371 Aug  3 10:48 nohup.out
12 
13 [root@zlm3 10:48:14 /var/log/masterha/app1]
14 #cat nohup.out
15 Fri Aug  3 10:48:12 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
16 Fri Aug  3 10:48:12 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
17 Fri Aug  3 10:48:12 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
18 ssh_exchange_identification: Connection closed by remote host
19 
20 [root@zlm3 10:48:26 /var/log/masterha/app1]
21 #cat app1.master_status.health 
22 9265    0:PING_OK    master:192.168.1.101
23 [root@zlm3 10:48:31 /var/log/masterha/app1]
24 #ps aux|grep manager
25 root      9265  0.6  2.1 299172 21516 pts/1    S    10:48   0:00 perl /usr/bin/masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
26 root      9332  0.0  0.0 112640   960 pts/1    R    10:48   0:00 grep --color=auto manager

 

Kill mysqld on zlm2 to pretend the master is dead.

1 [root@zlm2 10:54:31 /var/log/masterha/app1]
2 #pkill mysqld
3 
4 [root@zlm2 10:55:28 /var/log/masterha/app1]
5 #ps aux|grep mysqld
6 root      6067  0.0  0.0 112640   960 pts/1    R    10:55   0:00 grep --color=auto mysqld

 

Observe the app1.log on zlm3.

  1 [root@zlm3 10:54:59 /var/log/masterha/app1]
  2 #echo ''> app1.log
  3 
  4 [root@zlm3 10:55:17 /var/log/masterha/app1]
  5 #tail -f app1.log
  6 
  7 Fri Aug  3 10:55:29 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
  8 Fri Aug  3 10:55:29 2018 - [info] Executing SSH check script: exit 0
  9 Fri Aug  3 10:55:29 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
 10 Fri Aug  3 10:55:30 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.1.101' (111))
 11 Fri Aug  3 10:55:30 2018 - [warning] Connection failed 2 time(s)..
 12 Fri Aug  3 10:55:31 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.1.101' (111))
 13 Fri Aug  3 10:55:31 2018 - [warning] Connection failed 3 time(s)..
 14 Fri Aug  3 10:55:32 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.1.101' (111))
 15 Fri Aug  3 10:55:32 2018 - [warning] Connection failed 4 time(s)..
 16 Fri Aug  3 10:55:32 2018 - [warning] Master is not reachable from health checker!
 17 Fri Aug  3 10:55:32 2018 - [warning] Master 192.168.1.101(192.168.1.101:3306) is not reachable!
 18 Fri Aug  3 10:55:32 2018 - [warning] SSH is NOT reachable.
 19 Fri Aug  3 10:55:32 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha/masterha_default.conf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status..
 20 Fri Aug  3 10:55:32 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
 21 Fri Aug  3 10:55:32 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf..
 22 Fri Aug  3 10:55:32 2018 - [info] Reading server configuration from /etc/masterha/app1.conf..
 23 Fri Aug  3 10:55:33 2018 - [info] GTID failover mode = 1
 24 Fri Aug  3 10:55:33 2018 - [info] Dead Servers:
 25 Fri Aug  3 10:55:33 2018 - [info]   192.168.1.101(192.168.1.101:3306)
 26 Fri Aug  3 10:55:33 2018 - [info] Alive Servers:
 27 Fri Aug  3 10:55:33 2018 - [info]   192.168.1.102(192.168.1.102:3306)
 28 Fri Aug  3 10:55:33 2018 - [info] Alive Slaves:
 29 Fri Aug  3 10:55:33 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 30 Fri Aug  3 10:55:33 2018 - [info]     GTID ON
 31 Fri Aug  3 10:55:33 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 32 Fri Aug  3 10:55:33 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 33 Fri Aug  3 10:55:33 2018 - [info] Checking slave configurations..
 34 Fri Aug  3 10:55:33 2018 - [info]  read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306).
 35 Fri Aug  3 10:55:33 2018 - [info] Checking replication filtering settings..
 36 Fri Aug  3 10:55:33 2018 - [info]  Replication filtering check ok.
 37 Fri Aug  3 10:55:33 2018 - [info] Master is down!
 38 Fri Aug  3 10:55:33 2018 - [info] Terminating monitoring script.
 39 Fri Aug  3 10:55:33 2018 - [info] Got exit code 20 (Master dead).
 40 Fri Aug  3 10:55:33 2018 - [info] MHA::MasterFailover version 0.56.
 41 Fri Aug  3 10:55:33 2018 - [info] Starting master failover.
 42 Fri Aug  3 10:55:33 2018 - [info] 
 43 Fri Aug  3 10:55:33 2018 - [info] * Phase 1: Configuration Check Phase..
 44 Fri Aug  3 10:55:33 2018 - [info] 
 45 Fri Aug  3 10:55:34 2018 - [info] GTID failover mode = 1
 46 Fri Aug  3 10:55:34 2018 - [info] Dead Servers:
 47 Fri Aug  3 10:55:34 2018 - [info]   192.168.1.101(192.168.1.101:3306)
 48 Fri Aug  3 10:55:34 2018 - [info] Checking master reachability via MySQL(double check)...
 49 Fri Aug  3 10:55:34 2018 - [info]  ok.
 50 Fri Aug  3 10:55:34 2018 - [info] Alive Servers:
 51 Fri Aug  3 10:55:34 2018 - [info]   192.168.1.102(192.168.1.102:3306)
 52 Fri Aug  3 10:55:34 2018 - [info] Alive Slaves:
 53 Fri Aug  3 10:55:34 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 54 Fri Aug  3 10:55:34 2018 - [info]     GTID ON
 55 Fri Aug  3 10:55:34 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 56 Fri Aug  3 10:55:34 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 57 Fri Aug  3 10:55:34 2018 - [info] Starting GTID based failover.
 58 Fri Aug  3 10:55:34 2018 - [info] 
 59 Fri Aug  3 10:55:34 2018 - [info] ** Phase 1: Configuration Check Phase completed.
 60 Fri Aug  3 10:55:34 2018 - [info] 
 61 Fri Aug  3 10:55:34 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
 62 Fri Aug  3 10:55:34 2018 - [info] 
 63 Fri Aug  3 10:55:34 2018 - [info] Forcing shutdown so that applications never connect to the current master..
 64 Fri Aug  3 10:55:34 2018 - [info] Executing master IP deactivation script:
 65 Fri Aug  3 10:55:34 2018 - [info]   /etc/masterha/master_ip_failover --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --command=stop  --orig_master_ssh_port=3306
 66 ssh: connect to host 192.168.1.101 port 3306: Connection refused
 67 Fri Aug  3 10:55:37 2018 - [info]  done.
 68 Fri Aug  3 10:55:37 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
 69 Fri Aug  3 10:55:37 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
 70 Fri Aug  3 10:55:37 2018 - [info] 
 71 Fri Aug  3 10:55:37 2018 - [info] * Phase 3: Master Recovery Phase..
 72 Fri Aug  3 10:55:37 2018 - [info] 
 73 Fri Aug  3 10:55:37 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
 74 Fri Aug  3 10:55:37 2018 - [info] 
 75 Fri Aug  3 10:55:37 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000051:190
 76 Fri Aug  3 10:55:37 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
 77 Fri Aug  3 10:55:37 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 78 Fri Aug  3 10:55:37 2018 - [info]     GTID ON
 79 Fri Aug  3 10:55:37 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 80 Fri Aug  3 10:55:37 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 81 Fri Aug  3 10:55:37 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000051:190
 82 Fri Aug  3 10:55:37 2018 - [info] Oldest slaves:
 83 Fri Aug  3 10:55:37 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 84 Fri Aug  3 10:55:37 2018 - [info]     GTID ON
 85 Fri Aug  3 10:55:37 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 86 Fri Aug  3 10:55:37 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 87 Fri Aug  3 10:55:37 2018 - [info] 
 88 Fri Aug  3 10:55:37 2018 - [info] * Phase 3.3: Determining New Master Phase..
 89 Fri Aug  3 10:55:37 2018 - [info] 
 90 Fri Aug  3 10:55:37 2018 - [info] Searching new master from slaves..
 91 Fri Aug  3 10:55:37 2018 - [info]  Candidate masters from the configuration file:
 92 Fri Aug  3 10:55:37 2018 - [info]   192.168.1.102(192.168.1.102:3306)  Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
 93 Fri Aug  3 10:55:37 2018 - [info]     GTID ON
 94 Fri Aug  3 10:55:37 2018 - [info]     Replicating from 192.168.1.101(192.168.1.101:3306)
 95 Fri Aug  3 10:55:37 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
 96 Fri Aug  3 10:55:37 2018 - [info]  Non-candidate masters:
 97 Fri Aug  3 10:55:37 2018 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
 98 Fri Aug  3 10:55:37 2018 - [info] New master is 192.168.1.102(192.168.1.102:3306)
 99 Fri Aug  3 10:55:37 2018 - [info] Starting master failover..
100 Fri Aug  3 10:55:37 2018 - [info] 
101 From:
102 192.168.1.101(192.168.1.101:3306) (current master)
103   --192.168.1.102(192.168.1.102:3306)
104 
105 To:
106 192.168.1.102(192.168.1.102:3306) (new master)
107 Fri Aug  3 10:55:37 2018 - [info] 
108 Fri Aug  3 10:55:37 2018 - [info] * Phase 3.3: New Master Recovery Phase..
109 Fri Aug  3 10:55:37 2018 - [info] 
110 Fri Aug  3 10:55:37 2018 - [info]  Waiting all logs to be applied.. 
111 Fri Aug  3 10:55:37 2018 - [info]   done.
112 Fri Aug  3 10:55:37 2018 - [info] Getting new master's binlog name and position..
113 Fri Aug  3 10:55:37 2018 - [info]  mysql-bin.000004:190
114 Fri Aug  3 10:55:37 2018 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
115 Fri Aug  3 10:55:37 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000004, 190, 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259
116 Fri Aug  3 10:55:37 2018 - [info] Executing master IP activate script:
117 Fri Aug  3 10:55:37 2018 - [info]   /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm'  --orig_master_ssh_port=3306  --new_master_ssh_port=3306
118 Unknown option: new_master_ssh_port
119 Set read_only=0 on the new master.
120 ssh_exchange_identification: Connection closed by remote host
121 Fri Aug  3 10:55:37 2018 - [info]  OK.
122 Fri Aug  3 10:55:37 2018 - [info] ** Finished master recovery successfully.
123 Fri Aug  3 10:55:37 2018 - [info] * Phase 3: Master Recovery Phase completed.
124 Fri Aug  3 10:55:37 2018 - [info] 
125 Fri Aug  3 10:55:37 2018 - [info] * Phase 4: Slaves Recovery Phase..
126 Fri Aug  3 10:55:37 2018 - [info] 
127 Fri Aug  3 10:55:37 2018 - [info] 
128 Fri Aug  3 10:55:37 2018 - [info] * Phase 4.1: Starting Slaves in parallel..
129 Fri Aug  3 10:55:37 2018 - [info] 
130 Fri Aug  3 10:55:37 2018 - [info] All new slave servers recovered successfully.
131 Fri Aug  3 10:55:37 2018 - [info] 
132 Fri Aug  3 10:55:37 2018 - [info] * Phase 5: New master cleanup phase..
133 Fri Aug  3 10:55:37 2018 - [info] 
134 Fri Aug  3 10:55:37 2018 - [info] Resetting slave info on the new master..
135 Fri Aug  3 10:55:37 2018 - [info]  192.168.1.102: Resetting slave info succeeded.
136 Fri Aug  3 10:55:37 2018 - [info] Master failover to 192.168.1.102(192.168.1.102:3306) completed successfully.
137 Fri Aug  3 10:55:37 2018 - [info] 
138 
139 ----- Failover Report -----
140 
141 app1: MySQL Master failover 192.168.1.101(192.168.1.101:3306) to 192.168.1.102(192.168.1.102:3306) succeeded
142 
143 Master 192.168.1.101(192.168.1.101:3306) is down!
144 
145 Check MHA Manager logs at zlm3:/var/log/masterha/app1/app1.log for details.
146 
147 Started automated(non-interactive) failover.
148 Invalidated master IP address on 192.168.1.101(192.168.1.101:3306)
149 Selected 192.168.1.102(192.168.1.102:3306) as a new master.
150 192.168.1.102(192.168.1.102:3306): OK: Applying all logs succeeded.
151 192.168.1.102(192.168.1.102:3306): OK: Activated master IP address.
152 192.168.1.102(192.168.1.102:3306): Resetting slave info succeeded.
153 Master failover to 192.168.1.102(192.168.1.102:3306) completed successfully.
154 
155 //Above failover report shows all the evidence and results of automation master switchover.All of the steps are executed successfully.

 

本文由4887王中王鉄算盘奖结果发布于王中王高手论坛,转载请注明出处:MySQL高可用之MHA切换测试

关键词:

上一篇:mysql数据库相关整理,常见问题

下一篇:没有了

最火资讯