|
1、集群环境
IP分配：
192.168.10.110 k8s-deploy-harbor 2c2g
192.168.10.111 k8s-master1-etcd1-haproxy1 2c4g
192.168.10.112 k8s-master2-etcd2-haproxy2 2c4g
192.168.10.113 k8s-master3-etcd3-haproxy3 2c4g
192.168.10.114 k8s-node1 2c3g
192.168.10.115 k8s-node2 2c3g
192.168.10.116 k8s-node3 2c3g
VIP：
192.168.10.118
2、etcd的介绍
etcd是CoreOS团队于2013年6月发起的开源项目，它的目标是构建一个高可用的分布式键值(key-value)数据库。etcd内部采用raft协议作为一致性算法，etcd基于Go语言实现。
3、etcd的特点
简单：安装配置简单，而且提供了HTTP API进行交互，使用也很简单
安全：支持SSL证书验证
快速：根据官方提供的benchmark数据，单实例支持每秒2k+读操作
可靠：采用raft算法，实现分布式系统数据的可用性和一致性
完全复制：集群中的每个节点都可以使⽤完整的存档
高可用性：Etcd可用于避免硬件的单点故障或网络络问题
一致性：每次读取都会返回跨多主机的最新写入
4、etcd的客户端操作
etcd有多个不同的API访问版本，v1版本已经废弃etcdv2和v3本质上是共享同一套raft 协议代码的两个独立的应用，接口不一样，
存储不一样，数据互相隔离。也就是说如果从Etcd v2升级到Etcd v3，原来v2的数据还是只能用v2的接口访问，v3的接口创建的数据也只能访问通过v3的接口访问。
etcd命令用法：
root@k8s-deploy:~/shell# etcdctl --help
4.1查看etcd集群成员客户端信息
etcd集群成员的心跳信息
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# export NODE_IPS="192.168.10.111 192.168.10.112 192.168.10.113" && for ip in ${NODE_IPS};do /usr/local/bin/etcdctl_AP1=3 /usr/local/bin//usr/local/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; done
2023-01-07 09:39:39.946578 W | pkg/flags: unrecognized environment variable /usr/local/bin/etcdctl_AP1=3
https://192.168.10.111:2379 is healthy: successfully committed proposal: took = 44.530175ms
2023-01-07 09:39:40.006976 W | pkg/flags: unrecognized environment variable /usr/local/bin/etcdctl_AP1=3
https://192.168.10.112:2379 is healthy: successfully committed proposal: took = 10.252366ms
2023-01-07 09:39:40.031803 W | pkg/flags: unrecognized environment variable /usr/local/bin/etcdctl_AP1=3
https://192.168.10.113:2379 is healthy: successfully committed proposal: took = 10.760717ms
etcd集群的成员信息
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# export NODE_IPS="192.168.10.111 192.168.10.112 192.168.10.113" && for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.111:2379 | 1463ac8ab5f4a708 | 3.4.13 | 3.4 MB | false | false | 5 | 843466 | 843466 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.112:2379 | 6d195d5665446f3c | 3.4.13 | 3.4 MB | true | false | 5 | 843466 | 843466 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.113:2379 | cce14a1a3ff3b166 | 3.4.13 | 3.3 MB | false | false | 5 | 843466 | 843466 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
显示etcd群集的详细信息
root@k8s-master1-etcd1-haproxy1:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.111:2379 | 1463ac8ab5f4a708 | 3.4.13 | 3.6 MB | false | false | 2 | 1313 | 1313 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.112:2379 | 6d195d5665446f3c | 3.4.13 | 3.6 MB | true | false | 2 | 1313 | 1313 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.113:2379 | cce14a1a3ff3b166 | 3.4.13 | 3.6 MB | false | false | 2 | 1313 | 1313 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
192.168.10.112为etcd集群的leader
人为的关闭etcd的leader:192.168.10.112,查看etcd leader是否会跳转到其它的etcd
#1、关闭etcd leader
root@k8s-master2-etcd2-haproxy2:~# systemctl stop etcd
#2、在其他的etcd
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.111:2379 | 1463ac8ab5f4a708 | 3.4.13 | 3.4 MB | false | false | 6 | 844734 | 844734 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
{"level":"warn","ts":"2023-01-07T09:50:18.198+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.10.112:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.10.112:2379: connect: connection refused\""}
Failed to get the status of endpoint https://192.168.10.112:2379 (context deadline exceeded)
+----------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+----------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.113:2379 | cce14a1a3ff3b166 | 3.4.13 | 3.3 MB | true | false | 6 | 844750 | 844750 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
可以看到现在的LEADER变成了113
4.2查看etcd集群的数据信息
4.2.1查看所有的key
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get / --prefix --keys-onl #以路径的⽅式所有key信息
...................................
/registry/pods/default/net-test1
/registry/pods/default/net-test3
/registry/flowschemas/workload-leader-election
/registry/leases/kube-node-lease/192.168.10.111
/registry/leases/kube-node-lease/192.168.10.112
/registry/leases/kube-node-lease/192.168.10.113
/registry/leases/kube-node-lease/192.168.10.114
/registry/leases/kube-node-lease/192.168.10.115
/registry/leases/kube-node-lease/192.168.10.116
4.2.2查看kubernetes中所有pod的信息
root@k8s-master1-etcd1-haproxy1:~#
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl get / --prefix --keys-only | grep pods
/registry/pods/default/net-test1
/registry/pods/default/net-test2
/registry/pods/default/net-test3
/registry/pods/kube-system/calico-kube-controllers-78bc9689fb-dld4v
/registry/pods/kube-system/calico-node-5f6cq
/registry/pods/kube-system/calico-node-6rkmq
/registry/pods/kube-system/calico-node-759hl
/registry/pods/kube-system/calico-node-hpxd2
/registry/pods/kube-system/calico-node-vmzlj
/registry/pods/kube-system/calico-node-zbbct
/registry/pods/kube-system/coredns-f97dc456d-tghvb
#在kubernetes中查看pod信息
root@k8s-master1-etcd1-haproxy1:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default net-test1 1/1 Running 0 79m
default net-test2 1/1 Running 0 79m
default net-test3 1/1 Running 0 76m
kube-system calico-kube-controllers-78bc9689fb-dld4v 1/1 Running 0 80m
kube-system calico-node-5f6cq 1/1 Running 0 80m
kube-system calico-node-6rkmq 1/1 Running 0 80m
kube-system calico-node-759hl 1/1 Running 0 80m
kube-system calico-node-hpxd2 1/1 Running 0 80m
kube-system calico-node-vmzlj 1/1 Running 0 80m
kube-system calico-node-zbbct 1/1 Running 0 80m
kube-system coredns-f97dc456d-tghvb 1/1 Running 0 32
4.2.3查看kubernetes中所有namespace的信息
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get / --prefix --keys-only | grep namespaces
/registry/namespaces/default
/registry/namespaces/kube-node-lease
/registry/namespaces/kube-public
/registry/namespaces/kube-system
/registry/namespaces/kubernetes-dashboard
#在kubernetes中查看namespaces信息
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# kubectl get namespaces
NAME STATUS AGE
default Active 116s
kube-node-lease Active 112s
kube-public Active 112s
kube-system Active 117s
4.2.4查看kubernetes中所有deployments的信息
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep deployments
/registry/deployments/kube-system/calico-kube-controllers
/registry/deployments/kube-system/coredns
#在kubernetes中查看deployments
root@k8s-master1-etcd1-haproxy1:~# kubectl get deployments -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system calico-kube-controllers 1/1 1 1 82m
kube-system coredns 1/1 1 1 34m
4.2.5查看calico网络组件信息
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get / --prefix --keys-only | grep calico
.................................................
/calico/resources/v3/projectcalico.org/profiles/ksa.kube-system.calico-node
/registry/clusterrolebindings/calico-node
/registry/clusterroles/calico-node
/registry/controllerrevisions/kube-system/calico-node-69bd9bc66c
/registry/controllerrevisions/kube-system/calico-node-fbb95bfdf
/registry/daemonsets/kube-system/calico-node
/registry/pods/kube-system/calico-node-4nxsv
/registry/pods/kube-system/calico-node-bzccb
/registry/pods/kube-system/calico-node-msvg7
/registry/pods/kube-system/calico-node-wfv28
/registry/pods/kube-system/calico-node-xr2df
/registry/pods/kube-system/calico-node-zbtc8
/registry/secrets/kube-system/calico-node-token-9vrg6
/registry/serviceaccounts/kube-system/calico-node
4.2.6查看指定的key
#查看namespaces中default的key
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 etcdctl get /registry/namespaces/default
/registry/namespaces/default
k8s
v1 Namespace
default"*$60c1aa10-122b-41e2-a80d-24aead782cda2°䛆Z&
ubernetes.io/metadata.namedefaultz{
kube-apiserverUpdatev°䛆FieldsV1:I
G{"f:metadata":{"f:labels":{".":{},"f:kubernetes.io/metadata.name":{}}}}
kubernetes
Active"
#查看calico的key
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get /calico/ipam/v2/assignment/ipv4/block/10.200.107.192-26
/calico/ipam/v2/assignment/ipv4/block/10.200.107.192-26
{"cidr":"10.200.107.192/26","affinity":"host:k8s-node3","allocations":[0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"unallocated":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],"attributes":[{"handle_id":"ipip-tunnel-addr-k8s-node3","secondary":{"node":"k8s-node3","type":"ipipTunnelAddress"}}],"deleted":false}
4.2.7查看所有calico的数据
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 etcdctl get --keys-only --prefix /calico
/calico/ipam/v2/assignment/ipv4/block/10.200.107.192-26
/calico/ipam/v2/assignment/ipv4/block/10.200.159.128-26
/calico/ipam/v2/assignment/ipv4/block/10.200.169.128-26
/calico/ipam/v2/assignment/ipv4/block/10.200.219.0-26
/calico/ipam/v2/assignment/ipv4/block/10.200.36.64-26
/calico/ipam/v2/assignment/ipv4/block/10.200.97.0-26
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master1-etcd1-haproxy1
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master2-etcd2-haproxy2
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master3-etcd3-haproxy3
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node1
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node2
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node3
....................................................................................
4.3 etcd增删改查数据
4.3.1添加数据
#etcd添加数据
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl put /magedu/n56 linux
OK
#验证添加的数据
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get /magedu/n56
/magedu/n56
linux
4.3.2改动数据
#修改数据,把name的值改为test
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl put /magedu/n56 test
OK
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get /magedu/n56
/magedu/n56
test
4.3.3删除数据
#删除name
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl del /magedu/n56 test
348
root@k8s-master1-etcd1-haproxy1:/usr/local/bin# ETCDCTL_API=3 /usr/local/bin/etcdctl get /magedu/n56
4.4 etcd数据watch机制
基于不断监看数据，发生变化就主动触发通知客户端，Etcd v3的watch机制支持watch某个固定的key，也支持watch一个范围。
相比Etcd v2, Etcd v3的一些主要变化：
1)接口通过grpc提供rpc接口，放弃了v2的http接口，优势是长连接效率提升明显，缺点是使用不如以前方便，尤其对不方便维护长连接的场景。
2)废弃了原来的目录结构，变成了纯粹的kv，用户可以通过前缀匹配模式模拟目录。
3)内存中不再保存value，同样的内存可以支持存储更多的key。
4)watch机制更稳定，基本上可以通过watch机制实现数据的完全同步。
5)提供了批量操作以及事务机制，用户可以通过批量事务请求来实现Etcd v2的CAS机制（批量事务支持if条件判断）。
在etcd02上监控一个name的key
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl watch /name
在etcd01上修改name的key，在etcd02上验证
#添加一个name的值
root@k8s-master2-etcd2-haproxy2:~# ETCDCTL_API=3 /usr/local/bin/etcdctl put /name zhai
OK
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl watch /name
PUT
/name
zhai
#修改name的值
root@k8s-master2-etcd2-haproxy2:~# ETCDCTL_API=3 /usr/local/bin/etcdctl put /name aaa
OK
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl watch /name
PUT
/name
zhai
PUT
/name
aaa
#删除name
root@k8s-master2-etcd2-haproxy2:~# ETCDCTL_API=3 /usr/local/bin/etcdctl del /name
1
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl watch /name
PUT
/name
zhai
PUT
/name
aaa
DELETE
/name
4.5 etcd数据备份与恢复机制
WAL是write ahead log的缩写，顾名思义，也就是在执行真正的写操作之前先写一个日志，预写日志。 wal:存放预写式日志,最大的作用是记录了整个数据变化的全部历程。在etcd中，所有数据的修改在提交前，都要先写入到WAL中。
4.5.1 etcd 集群v3版本数据⼿动备份与恢复
数据备份
root@k8s-master1-etcd1-haproxy1:/tools# ETCDCTL_API=3 /usr/local/bin/etcdctl snapshot save etcd-20230107-bak.db
{"level":"info","ts":1673060607.7006812,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"etcd-20230107-bak.db.part"}
{"level":"info","ts":"2023-01-07T11:03:27.701+0800","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1673060607.7012792,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"127.0.0.1:2379"}
{"level":"info","ts":"2023-01-07T11:03:27.735+0800","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1673060607.7730155,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"127.0.0.1:2379","size":"3.4 MB","took":0.072251129}
{"level":"info","ts":1673060607.7731135,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"etcd-20230107-bak.db"}
Snapshot saved at etcd-20230107-bak.db
数据恢复
数据恢复时需要对整个etcd集群的所有成员做恢复
#第一步先停止业务及etcd
root@k8s-master1-etcd1-haproxy1:~# systemctl stop etcd
root@k8s-master2-etcd2-haproxy2:~# systemctl stop etcd
root@k8s-master3-etcd3-haproxy3:~# systemctl stop etcd
#第二步删除etcd的原数据目录
root@k8s-master1-etcd1-haproxy1:~# rm -rf /var/lib/etcd
root@k8s-master2-etcd2-haproxy2:~# rm -rf /var/lib/etcd
root@k8s-master3-etcd3-haproxy3:~# rm -rf /var/lib/etcd
#恢复命令
ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
--name=etcd-192.168.10.111 \
--initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-advertise-peer-urls=https://192.168.10.111:2380 \
--data-dir=/var/lib/etcd
#注意
#--name=etcd-192.168.10.111 \
#--initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
#--initial-cluster-token=etcd-cluster-0 \
#--initial-advertise-peer-urls=https://192.168.10.111:2380 \
#--data-dir=/var/lib/etcd
上面的信息可以在下面的文件/etc/systemd/system/etcd.service中查找
#第三步在所有etcd成员上恢复数据
ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db
#恢复etcd01上的数据
ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
--name=etcd-192.168.10.111 \
--initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-advertise-peer-urls=https://192.168.10.111:2380 \
--data-dir=/var/lib/etcd
----------------------------------------------------------------------------------------------
root@k8s-master1-etcd1-haproxy1:/var/lib# ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
> --name=etcd-192.168.10.111 \
> --initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
> --initial-cluster-token=etcd-cluster-0 \
> --initial-advertise-peer-urls=https://192.168.10.111:2380 \
> --data-dir=/var/lib/etcd
{"level":"info","ts":1673061700.2319326,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/tools/etcd-20230107-bak.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
{"level":"info","ts":1673061700.2965689,"caller":"mvcc/kvstore.go:380","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":677669}
{"level":"info","ts":1673061700.302237,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"1463ac8ab5f4a708","added-peer-peer-urls":["https://192.168.10.111:2380"]}
{"level":"info","ts":1673061700.30231,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"6d195d5665446f3c","added-peer-peer-urls":["https://192.168.10.112:2380"]}
{"level":"info","ts":1673061700.30234,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"cce14a1a3ff3b166","added-peer-peer-urls":["https://192.168.10.113:2380"]}
{"level":"info","ts":1673061700.309144,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/tools/etcd-20230107-bak.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
#只能将数据恢复到一个新的不存在的目录中，如果需要恢复到原目录，需要将原目录删除
#恢复etcd02上的数据
ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
--name=etcd-192.168.10.112 \
--initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-advertise-peer-urls=https://192.168.10.112:2380 \
--data-dir=/var/lib/etcd
----------------------------------------------------------------------------------------------
root@k8s-master2-etcd2-haproxy2:~# ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
> --name=etcd-192.168.10.111 \
> --initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
> --initial-cluster-token=etcd-cluster-0 \
> --initial-advertise-peer-urls=https://192.168.10.111:2380 \
> --data-dir=/var/lib/etcd
{"level":"info","ts":1673061838.0671895,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/tools/etcd-20230107-bak.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
{"level":"info","ts":1673061838.1501634,"caller":"mvcc/kvstore.go:380","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":677669}
{"level":"info","ts":1673061838.1551597,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"1463ac8ab5f4a708","added-peer-peer-urls":["https://192.168.10.111:2380"]}
{"level":"info","ts":1673061838.1553109,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"6d195d5665446f3c","added-peer-peer-urls":["https://192.168.10.112:2380"]}
{"level":"info","ts":1673061838.1553912,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"cce14a1a3ff3b166","added-peer-peer-urls":["https://192.168.10.113:2380"]}
{"level":"info","ts":1673061838.169916,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/tools/etcd-20230107-bak.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
#只能将数据恢复到一个新的不存在的目录中，如果需要恢复到原目录，需要将原目录删除
#恢复etcd03上的数据
ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
--name=etcd-192.168.10.113 \
--initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-advertise-peer-urls=https://192.168.10.113:2380 \
--data-dir=/var/lib/etcd
----------------------------------------------------------------------------------------------
root@k8s-master3-etcd3-haproxy3:~# ETCDCTL_API=3 etcdctl snapshot restore /tools/etcd-20230107-bak.db \
> --name=etcd-192.168.10.111 \
> --initial-cluster=etcd-192.168.10.111=https://192.168.10.111:2380,etcd-192.168.10.112=https://192.168.10.112:2380,etcd-192.168.10.113=https://192.168.10.113:2380 \
> --initial-cluster-token=etcd-cluster-0 \
> --initial-advertise-peer-urls=https://192.168.10.111:2380 \
> --data-dir=/var/lib/etcd
{"level":"info","ts":1673061839.653366,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/tools/etcd-20230107-bak.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
{"level":"info","ts":1673061839.8076127,"caller":"mvcc/kvstore.go:380","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":677669}
{"level":"info","ts":1673061839.8133101,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"1463ac8ab5f4a708","added-peer-peer-urls":["https://192.168.10.111:2380"]}
{"level":"info","ts":1673061839.8133879,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"6d195d5665446f3c","added-peer-peer-urls":["https://192.168.10.112:2380"]}
{"level":"info","ts":1673061839.8134158,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"ff9cf838088b5a51","local-member-id":"0","added-peer-id":"cce14a1a3ff3b166","added-peer-peer-urls":["https://192.168.10.113:2380"]}
{"level":"info","ts":1673061839.819993,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/tools/etcd-20230107-bak.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
#只能将数据恢复到一个新的不存在的目录中，如果需要恢复到原目录，需要将原目录删除
#第四步重启etcd
systemctl daemon-reload
systemctl restart etcd
#etcd1
root@k8s-master1-etcd1-haproxy1:~# systemctl daemon-reload
root@k8s-master1-etcd1-haproxy1:~# systemctl restart etcd
#etcd2
root@k8s-master2-etcd2-haproxy2:~# systemctl daemon-reload
root@k8s-master2-etcd2-haproxy2:~# systemctl restart etcd
#etcd3
root@k8s-master3-etcd3-haproxy3:~# systemctl daemon-reload
root@k8s-master3-etcd3-haproxy3:~# systemctl restart etcd
第五步验证
root@k8s-master1-etcd1-haproxy1:/var/lib/etcd# export NODE_IPS="192.168.10.111 192.168.10.112 192.168.10.113" && for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.111:2379 | 1463ac8ab5f4a708 | 3.4.13 | 3.4 MB | true | false | 443 | 11 | 11 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.112:2379 | 6d195d5665446f3c | 3.4.13 | 3.4 MB | false | false | 443 | 11 | 11 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.113:2379 | cce14a1a3ff3b166 | 3.4.13 | 3.4 MB | false | false | 443 | 11 | 11 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
4.5.3 使用kubeasz来备份和恢复etcd
备份，用ezctl备份也是连接到一台etcd执行备份，然后打个快照，在把快照考到操作的机器
#备份
./ezctl backup k8s-01
#恢复
./ezctl restore k8s-01
#备份
root@k8s-harbor-deploy:/etc/kubeasz# ./ezctl backup k8s-01
#刚刚做的快照文件
root@k8s-harbor-deploy:/etc/kubeasz# ls /etc/kubeasz/clusters/k8s-01/backup/
snapshot.db snapshot_202301071605.db
#查看本机的pod
root@k8s-master1-etcd1-haproxy1:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default net-test1 1/1 Running 0 60m
default net-test2 1/1 Running 0 60m
default net-test3 1/1 Running 0 57m
default net-test4 1/1 Running 0 57m
kube-system calico-kube-controllers-78bc9689fb-dld4v 1/1 Running 0 61m
kube-system calico-node-5f6cq 1/1 Running 0 61m
kube-system calico-node-6rkmq 1/1 Running 0 61m
kube-system calico-node-759hl 1/1 Running 0 61m
kube-system calico-node-hpxd2 1/1 Running 0 61m
kube-system calico-node-vmzlj 1/1 Running 0 61m
kube-system calico-node-zbbct 1/1 Running 0 61m
kube-system coredns-f97dc456d-tghvb 1/1 Running 0 13m
#删除pod
root@k8s-master1-etcd1-haproxy1:~# kubectl delete pod net-test1 net-test2 net-test3
pod "net-test1" deleted
pod "net-test2" deleted
pod "net-test3" deleted
#现有pod
root@k8s-master1-etcd1-haproxy1:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default net-test4 1/1 Running 0 5s
kube-system calico-kube-controllers-78bc9689fb-dld4v 1/1 Running 0 65m
kube-system calico-node-5f6cq 1/1 Running 0 65m
kube-system calico-node-6rkmq 1/1 Running 0 65m
kube-system calico-node-759hl 1/1 Running 0 65m
kube-system calico-node-hpxd2 1/1 Running 0 65m
kube-system calico-node-vmzlj 1/1 Running 0 65m
kube-system calico-node-zbbct 1/1 Running 0 65m
kube-system coredns-f97dc456d-tghvb 1/1 Running 0 17m
#在etcd删除net-test4
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl get / --prefix --keys-only |grep net-test4
/registry/events/default/net-test4.1737f87a712e63a9
/registry/events/default/net-test4.1737f87a9b8149c3
/registry/events/default/net-test4.1737f87a9ed434ec
/registry/events/default/net-test4.1737f87aa2f33577
/registry/events/default/net-test4.1737f87aab51831f
/registry/pods/default/net-test4
#在etcd执行删除pod
ETCDCTL_API=3 /usr/local/bin/etcdctl del /registry/pods/default/net-test4
root@k8s-master1-etcd1-haproxy1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl del /registry/pods/default/net-test4
#检查pod
root@k8s-master1-etcd1-haproxy1:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-78bc9689fb-dld4v 1/1 Running 0 67m
kube-system calico-node-5f6cq 1/1 Running 0 67m
kube-system calico-node-6rkmq 1/1 Running 0 67m
kube-system calico-node-759hl 1/1 Running 0 67m
kube-system calico-node-hpxd2 1/1 Running 0 67m
kube-system calico-node-vmzlj 1/1 Running 0 67m
kube-system calico-node-zbbct 1/1 Running 0 67m
kube-system coredns-f97dc456d-tghvb 1/1 Running 0 19m
#通过etcd备份恢复，通过ezctl恢复会停止apiserver和etcd、kube-controll-manager，三台master都会停止,也会停止node节点上的kube-proxy和kubetel
./ezctl restore k8s-01
root@k8s-harbor-deploy:/etc/kubeasz# ./ezctl restore k8s-01
ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/95.restore.yml
2023-01-07 16:12:51 INFO cluster:k8s-01 restore begins in 5s, press any key to abort:
#通过etcd的备份成功恢复回pod
root@k8s-master1-etcd1-haproxy1:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default net-test1 1/1 Running 0 69m
default net-test2 1/1 Running 0 69m
default net-test3 1/1 Running 0 66m
default net-test4 1/1 Running 0 66m
kube-system calico-kube-controllers-78bc9689fb-dld4v 1/1 Running 0 70m
kube-system calico-node-5f6cq 1/1 Running 0 70m
kube-system calico-node-6rkmq 1/1 Running 0 70m
kube-system calico-node-759hl 1/1 Running 0 70m
kube-system calico-node-hpxd2 1/1 Running 0 70m
kube-system calico-node-vmzlj 1/1 Running 0 70m
kube-system calico-node-zbbct 1/1 Running 0 70m
kube-system coredns-f97dc456d-tghvb 1/1 Running 0 22m
#验证etcd是否是一主2从，一个LEADER
export NODE_IPS="192.168.10.111 192.168.10.112 192.168.10.113" && for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
root@k8s-master2-etcd2-haproxy2:~# export NODE_IPS="192.168.10.111 192.168.10.112 192.168.10.113" && for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.111:2379 | 1463ac8ab5f4a708 | 3.4.13 | 3.6 MB | false | false | 2 | 148 | 148 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.112:2379 | 6d195d5665446f3c | 3.4.13 | 3.6 MB | true | false | 2 | 148 | 148 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.113:2379 | cce14a1a3ff3b166 | 3.4.13 | 3.6 MB | false | false | 2 | 148 | 148 | |
4.5.4 使用kubeasz来添加和删除etcd节点
root@k8s-deploy:/etc/kubeasz# ./ezctl --help
Usage: ezctl COMMAND [args]
-------------------------------------------------------------------------------------
Cluster setups:
list to list all of the managed clusters
checkout to switch default kubeconfig of the cluster
new to start a new k8s deploy with name 'cluster'
setup to setup a cluster, also supporting a step-by-step way
start to start all of the k8s services stopped by 'ezctl stop'
stop to stop all of the k8s services temporarily
upgrade to upgrade the k8s cluster
destroy to destroy the k8s cluster
backup to backup the cluster state (etcd snapshot)
restore to restore the cluster state from backups
start-aio to quickly setup an all-in-one cluster with 'default' settings
Cluster ops:
add-etcd to add a etcd-node to the etcd cluster
add-master to add a master node to the k8s cluster
add-node to add a work node to the k8s cluster
del-etcd to delete a etcd-node from the etcd cluster
del-master to delete a master node from the k8s cluster
del-node to delete a work node from the k8s cluster
Extra operation:
kcfg-adm to manage client kubeconfig of the k8s cluster
Use "ezctl help " for more information about a given command.
#添加新的etcd节点
root@k8s-harbor-deploy:/etc/kubeasz# ./ezctl add-etcd k8s-01 192.168.10.114
#删除etcd节点
root@k8s-harbor-deploy:/etc/kubeasz# ./ezctl add-del k8s-01 192.168.10.114
4.6 ETCD数据恢复流程
1、恢复服务器系统
2、重新部署ETCD集群
3、停止kube-apiserver/controller-manager/scheduler/kubelet/kube-proxy
4、停止ETCD集群
5、各ETCD节点恢复统一份备份数据
6、启动各节点并验证ETCD集群
7、启动kube-apiserver/controller-manager/scheduler/kubelet/kube-proxy
8、验证k8s master状态及pod数据
原文地址：https://www.cnblogs.com/yeyouqing/p/17033084.html
|