Mongodb cluster shard status exception: RECOVERING
Mongodb cluster shard status exception: RECOVERING
Error message 2018-11-28T06:46:55.783+0000 I REPL [replication-0] We are too stale to use 172.19.9.12 replication-0 27003 as a sync source. Blacklisting this sync source because our last fetched timestamp: Timestamp (1542344943, 1) is before their earliest timestamp: Timestamp (1543387334, 5197) for 1min until: 2018-11-28T06:47:55.783+00002018-11-28T06:46:55.783+0000 I REPL [replication-0] sync source candidate: 172.19.9.11 REPL 270032018-11-28T06:46:55.783+0000 I REPL [replication-0] We are too stale to use 172.19.9.11 REPL 27003 as a sync source. Blacklisting this sync source because our last fetched timestamp: Timestamp (1542344943, 1) is before their earliest timestamp: Timestamp (1543387334, 5953) for 1min until: 2018-11-28T06:47:55.783+0000 error cause analysis:
The error node data is too "obsolete: stale"; network exception or node exception, there is no synchronous data operation for too long, resulting in the data operation log of other nodes has been overwritten, so this node is considered stale and cannot synchronize data from other nodes.
Recovery plan
1: stop the database, delete the local data of the exception node (shard) directly, and then start the mongo database. After starting, there is a synchronization process, depending on the amount of data, network, disk performance and other factors.
2: stop the database, directly copy the data on the master node, and then start mongo, so that there is no process of data synchronization. The problem is that the data is changing all the time, and it is inevitable that some data will be left out in the copying process.
How to handle it:
Our mongodb cluster is pulled up using docker, using scenario 1
First determine the exception sharding node = "and then determine the mapping directory =" delete the exception sharding instance data directory = "the docker service will automatically pull up the service =" the cluster starts data recovery
Confirm the exception message: / mongo localhost:27017/admin:PRIMARY > rs.status () {"_ id": 2, "name": "172.19.9.13 health 27003", "= = node information" health ": 1," state ": 5," stateStr ":" RECOVERING " "= = abnormal state" uptime ": 64," optime ": {" ts ": Timestamp (0,0)," t ": NumberLong (- 1)} Specific operation: data directory: / data/shard3 ssh $HOSTNAME cd / data/ rm-rf shard3
After deleting the data directory, the container is abnormal, and the cluster will automatically pull up a new docker instance to run the shard 3 instance.
View the recovery status:
STARTUP2 indicates that the data is being initialized and synchronized, and you will see that the data directory files are constantly adding files.
/ mongo localhost:27017/admin:PRIMARY > rs.status () {"_ id": 2, "name": "172.19.9.13 health 27003", "health": 1, "state": 5, "stateStr": "STARTUP2" "= = indicates that data is being initialized and synchronized. "uptime": 64, "optime": {"ts": Timestamp (0,0), "t": NumberLong (- 1)}, view the recovery result / mongo localhost:27017/admin:PRIMARY > rs.status () "_ id": 2, "name": "172.19.9.13 health 27003", "health": 1, "state": 2, "stateStr": "SECONDARY"
