Etcd cluster backup and recovery

I. backup

This backup method is based on the snapshot function of etcd. If needed, the etcd cluster can be rolled back to a specific backup point. This backup is based on etcd api 3 version. If the data is written in api 2 version, the data written in api 2 version cannot be recovered.

1. Backup

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem snapshot save / chj/backup/etcd3/snapshot.db

2. Backup script

#! / bin/bashDATA= `date +% y% m% d`BACUPDIR = / chj/backup/etcdENDPOINTS=' https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379'timestamp=`date +% Y% m% d% H% M% S`if [!-d "$BACUPDIR"] Then echo "making dir $BACUPDIR" mkdir-p $BACUPDIRfiETCDCTL_API=3 etcdctl-endpoints=$ENDPOINTS-- cert=/etc/etcd/ssl/etcd.pem-- key=/etc/etcd/ssl/etcd-key.pem-- cacert=/etc/etcd/ssl/ca.pem snapshot save $BACUPDIR/snapshot_$timestamp.dbfind $BACUPDIR-name * .db-mtime + 5-exec rm-rf {}\

II. Recovery

1. Stop the service of all nodes

Systemctl stop etcd

2. Back up the current data directory

Mv / var/lib/etcd / var/lib/etcd_bak_20190523

3. Confirm the snapshot files that need to be rolled back, and restore the distribution to each node in the cluster.

# restore 192.168.9.87 nodes

ETCDCTL_API=3 etcdctl-name=k8stest-master-1.chj.cloud-endpoints= "https://192.168.9.87:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem-initial-cluster-token=etcd-cluster-0-initial-advertise-peer-urls= https://192.168.9.87:2380-initial-cluster=k8stest-master-1.chj.cloud=https : / / 192.168.9.87:2380 K8stestMusure master2.chj.cloud = https://192.168.9.88:2380,k8stest-master-3.chj.cloud=https://192.168.9.89:2380-- data-dir=/var/lib/etcd snapshot restore / chj/backup/etcd3/snapshot.db

# restore 192.168.9.88 nodes

ETCDCTL_API=3 etcdctl-name=k8stest-master-2.chj.cloud-endpoints= "https://192.168.9.88:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem-initial-cluster-token=etcd-cluster-0-initial-advertise-peer-urls= https://192.168.9.88:2380-initial-cluster=k8stest-master-1.chj.cloud=https : / / 192.168.9.87:2380 K8stestMusure master2.chj.cloud = https://192.168.9.88:2380,k8stest-master-3.chj.cloud=https://192.168.9.89:2380-- data-dir=/var/lib/etcd snapshot restore / chj/backup/etcd3/snapshot.db

# restore 192.168.9.89 nodes

ETCDCTL_API=3 etcdctl-name=k8stest-master-3.chj.cloud-endpoints= "https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem-initial-cluster-token=etcd-cluster-0-initial-advertise-peer-urls= https://192.168.9.89:2380-initial-cluster=k8stest-master-1.chj.cloud=https : / / 192.168.9.87:2380 K8stestMusure master2.chj.cloud = https://192.168.9.88:2380,k8stest-master-3.chj.cloud=https://192.168.9.89:2380-- data-dir=/var/lib/etcd snapshot restore / chj/backup/etcd3/snapshot.db

III. Common commands

Write data using API 3

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem put / chj/3 api3

Use API 3 to read data

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem get / chj/3

Use API 3 to view all key

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-- cert=/etc/etcd/ssl/etcd.pem-- key=/etc/etcd/ssl/etcd-key.pem-- cacert=/etc/etcd/ssl/ca.pem get /-- prefix-- keys-only

Delete all key using API 3

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem del"-prefix

View cluster members

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem member list-write-out=table

View cluster health status

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem endpoint health

View node status

ETCDCTL_API=3 etcdctl-endpoints= "https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379"-cert=/etc/etcd/ssl/etcd.pem-key=/etc/etcd/ssl/etcd-key.pem-cacert=/etc/etcd/ssl/ca.pem endpoint status-write-out=table