Cassandra Cluster Management-Node abnormal restart
Cassandra Cluster Management-Node abnormal restart
Log in to a cluster node, restart the server directly (172.20.101.166), and set cassandra to boot.
Note:
This document is only part of the system documentation. For more information on the previous documentation, please see:
Test preparation + offline normal node: https://blog.51cto.com/michaelkang/2419518
Node abnormal restart: https://blog.51cto.com/michaelkang/2419524
Add a new node: https://blog.51cto.com/michaelkang/2419521
Delete exception node: https://blog.51cto.com/michaelkang/2419525
Scene:
The node is restarted abnormally and reacts to the cluster.
Cassandra.log basically does not output tailf / var/log/cassandra/cassandra.log system.log
There is an obvious log report of 172.20.101.166 DOWN!
172.20.101.165 Node: [root@kubm-03 lib] # tailf / var/log/cassandra/system.log INFO [GossipStage:1] 2019-07-11 18VAND 19var/log/cassandra/system.log INFO 23372 Gossiper.java:1026-InetAddress / 172.20.101.166 is now DOWN View the exception node [root@kubm-01] # nodetool describeclusterCluster Information: Name: pttest Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch DynamicEndPointSnitch: enabled Partitioner: org.apache. Cassandra.dht.Murmur3Partitioner Schema versions: cfce5a85-19c8-327a-ab19-e1faae2358f7: [172.20.101.164 172.20.101.165, 172.20.101.167, 172.20.101.160, 172.20.101.157] UNREACHABLE: [172.20.101.166] debug.log
A large number of reports cannot be connected to 172.20.101.166
172.20.101.164 Node: tailf / var/log/cassandra/debug.logDEBUG [GossipStage:1] 2019-07-11 18V 19V 23374 OutboundTcpConnection.java:205-Enqueuing socket close for / 172.20.101.166DEBUG [MessagingService-Outgoing-/172.20.101.166-Small] 2019-07-11 18V 19V 23374 OutboundTcpConnection.java:411-Socket to / 172.20.101.166 closedDEBUG [GossipStage:1] 2019-07-11 1818VOR 23374 OutboundTcpConnection.java:205-Enqueuing socket Close for / 172.20.101.166DEBUG [MessagingService-Outgoing-/172.20.101.166-Gossip] 2019-07-11 18MessagingService-Outgoing-/172.20.101.166-Gossip 1923374 OutboundTcpConnection.java:411-Socket to / 172.20.101.166 closedDEBUG [GossipStage:1] 2019-07-11 18VO 1915 23374 FailureDetector.java:313-Forcing conviction of / 172.20.101.166DEBUG [MessagingService-Outgoing-/172.20.101.166-Gossip] 2019-07-11 18VOG 19VOU 24740 OutboundTcpConnection.java:425-Attempting to connect To / 172.20.101.166INFO [HANDSHAKE-/172.20.101.166] 2019-07-11 18 OutboundTcpConnection.java:561 1941 OutboundTcpConnection.java:561-Handshaking version with / 172.20.101.166DEBUG [MessagingService-Outgoing-/172.20.101.166-Gossip] 2019-07-11 18 18 Fringe 24742 OutboundTcpConnection.java:533-Done connecting to / 172.20.101.166 Authentication query
After the system starts, the service starts naturally and can join the cluster normally.
Cassandra@cqlsh > SELECT * from kevin_test.t_users User_id | emails | first_name | last_name-+--+-+- 6 | {'k6-6roomgmail.com' 'k6roompt.com'} | kevin6 | kang 7 | {'K7-7roomgmail.composts, 'k7roompt.com'} | kevin7 | kang 9 | {' k9-9roomgmail.compositions, 'k9roompt.com'} | kevin9 | kang 4 | {' k4-4roomgmail.compositions, 'k4roompt.com'} | kevin4 | kang 3 | {' k3-3roomgmail.com' 'k3roompt.com'} | kevin3 | kang 5 | {'k5-5roomgmail.composts, 'k5roompt.com'} | kevin5 | kang 0 | {' k0-0roomgmail.compositions, 'k0roompt.com'} | kevin0 | kang 8 | {' k8-8roomgmail.compositions, 'k8roompt.com'} | kevin8 | kang 2 | {' K2-2roomgmail.com' 'k2roompt.com'} | kevin2 | kang 1 | {'K1-1roomgmail.compose, 'k1roompt.com'} | kevin1 | kang test result:
Restart the node repeatedly, and the query table content is normal.
