服务器资源变更,kubelet报错
背景
服务器资源由8c32g升级为24c48g,服务器重启kubelet报错
[root@k8s-node2 ~]# journalctl -u kubelet.service
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.377270 6134 reflector.go:175] Starting reflector *v1beta1.RuntimeClass (0s) from k8s.io/client-go/informers/factory.go:135
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.387753 6134 factory.go:356] Registering Docker factory
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.387775 6134 factory.go:54] Registering systemd factory
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.387931 6134 factory.go:101] Registering Raw factory
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.388082 6134 manager.go:1158] Started watching for new ooms in manager
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.391096 6134 manager.go:272] Starting recovery of all containers
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.398831 6134 manager.go:277] Recovery completed
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449657 6134 cpu_manager.go:184] [cpumanager] starting with static policy
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449673 6134 cpu_manager.go:185] [cpumanager] reconciling every 10s
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449691 6134 state_mem.go:36] [cpumanager] initializing new in-memory state store
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449861 6134 state_mem.go:88] [cpumanager] updated default cpuset: "0-7"
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449878 6134 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]"
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449889 6134 state_checkpoint.go:136] [cpumanager] state checkpoint: restored state from checkpoint
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449897 6134 state_checkpoint.go:137] [cpumanager] state checkpoint: defaultCPUSet: 0-7
Nov 27 09:27:39 k8s-node2 kubelet[6134]: E1127 09:27:39.449930 6134 policy_static.go:122] [cpumanager] static policy invalid state: current set of available CPUs "0-23" doesn't match with CPUs in state "0-7", plea
Nov 27 09:27:39 k8s-node2 kubelet[6134]: E1127 09:27:39.449938 6134 cpu_manager.go:201] [cpumanager] policy start error: current set of available CPUs "0-23" doesn't match with CPUs in state "0-7"
Nov 27 09:27:39 k8s-node2 kubelet[6134]: F1127 09:27:39.449944 6134 kubelet.go:1383] Failed to start ContainerManager start cpu manager error: current set of available CPUs "0-23" doesn't match with CPUs in state
Nov 27 09:27:39 k8s-node2 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 27 09:27:39 k8s-node2 systemd[1]: Unit kubelet.service entered failed state.
Nov 27 09:27:39 k8s-node2 systemd[1]: kubelet.service failed.
Nov 27 09:27:39 k8s-node2 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 27 09:27:39 k8s-node2 systemd[1]: Stopped Kubernetes Kubelet.
解决:
[root@k8s-node2 ~]# find / -name "cpu_manager_state"
[root@k8s-node2 ~]# cat /var/lib/kubelet/cpu_manager_state
{"policyName":"static","defaultCpuSet":"0-7","checksum":14413152}
[root@k8s-node2 ~]# rm -rf /var/lib/kubelet/cpu_manager_state
[root@k8s-node2 ~]# systemctl restart kubelet
[root@k8s-node2 ~]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2020-11-27 09:42:34 CST; 4s ago
Main PID: 7765 (kubelet)
Tasks: 29
Memory: 51.1M
CGroup: /system.slice/kubelet.service
└─7765 /usr/local/bin/kubelet --logtostderr=true --v=2 --hostname-override=k8s-node2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --feature...
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.610917 7765 clientconn.go:933] ClientConn switching balancer to "pick_first"
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.610936 7765 server.go:1071] Using root directory: /var/lib/kubelet
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.610949 7765 kubelet.go:317] Watching apiserver
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611031 7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b770, {CONNECTING <nil>}
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611050 7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b620, {CONNECTING <nil>}
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611079 7765 reflector.go:175] Starting reflector *v1.Pod (0s) from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611104 7765 reflector.go:175] Starting reflector *v1.Service (0s) from k8s.io/kubernetes/pkg/kubelet/kubelet.go:517
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611151 7765 reflector.go:175] Starting reflector *v1.Node (0s) from k8s.io/kubernetes/pkg/kubelet/kubelet.go:526
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611269 7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b620, {READY <nil>}
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611281 7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b770, {READY <nil>}##