服务器资源变更，kubelet报错

背景

服务器资源由8c32g升级为24c48g,服务器重启kubelet报错

[root@k8s-node2 ~]# journalctl -u kubelet.service
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.377270    6134 reflector.go:175] Starting reflector *v1beta1.RuntimeClass (0s) from k8s.io/client-go/informers/factory.go:135
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.387753    6134 factory.go:356] Registering Docker factory
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.387775    6134 factory.go:54] Registering systemd factory
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.387931    6134 factory.go:101] Registering Raw factory
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.388082    6134 manager.go:1158] Started watching for new ooms in manager
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.391096    6134 manager.go:272] Starting recovery of all containers
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.398831    6134 manager.go:277] Recovery completed
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449657    6134 cpu_manager.go:184] [cpumanager] starting with static policy
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449673    6134 cpu_manager.go:185] [cpumanager] reconciling every 10s
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449691    6134 state_mem.go:36] [cpumanager] initializing new in-memory state store
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449861    6134 state_mem.go:88] [cpumanager] updated default cpuset: "0-7"
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449878    6134 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]"
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449889    6134 state_checkpoint.go:136] [cpumanager] state checkpoint: restored state from checkpoint
Nov 27 09:27:39 k8s-node2 kubelet[6134]: I1127 09:27:39.449897    6134 state_checkpoint.go:137] [cpumanager] state checkpoint: defaultCPUSet: 0-7
Nov 27 09:27:39 k8s-node2 kubelet[6134]: E1127 09:27:39.449930    6134 policy_static.go:122] [cpumanager] static policy invalid state: current set of available CPUs "0-23" doesn't match with CPUs in state "0-7", plea
Nov 27 09:27:39 k8s-node2 kubelet[6134]: E1127 09:27:39.449938    6134 cpu_manager.go:201] [cpumanager] policy start error: current set of available CPUs "0-23" doesn't match with CPUs in state "0-7"
Nov 27 09:27:39 k8s-node2 kubelet[6134]: F1127 09:27:39.449944    6134 kubelet.go:1383] Failed to start ContainerManager start cpu manager error: current set of available CPUs "0-23" doesn't match with CPUs in state
Nov 27 09:27:39 k8s-node2 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 27 09:27:39 k8s-node2 systemd[1]: Unit kubelet.service entered failed state.
Nov 27 09:27:39 k8s-node2 systemd[1]: kubelet.service failed.
Nov 27 09:27:39 k8s-node2 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 27 09:27:39 k8s-node2 systemd[1]: Stopped Kubernetes Kubelet.

解决：

[root@k8s-node2 ~]#  find / -name "cpu_manager_state"
[root@k8s-node2 ~]# cat /var/lib/kubelet/cpu_manager_state
{"policyName":"static","defaultCpuSet":"0-7","checksum":14413152}
[root@k8s-node2 ~]# rm -rf /var/lib/kubelet/cpu_manager_state
[root@k8s-node2 ~]# systemctl restart kubelet
[root@k8s-node2 ~]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-11-27 09:42:34 CST; 4s ago
 Main PID: 7765 (kubelet)
    Tasks: 29
   Memory: 51.1M
   CGroup: /system.slice/kubelet.service
           └─7765 /usr/local/bin/kubelet --logtostderr=true --v=2 --hostname-override=k8s-node2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --feature...

Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.610917    7765 clientconn.go:933] ClientConn switching balancer to "pick_first"
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.610936    7765 server.go:1071] Using root directory: /var/lib/kubelet
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.610949    7765 kubelet.go:317] Watching apiserver
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611031    7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b770, {CONNECTING <nil>}
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611050    7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b620, {CONNECTING <nil>}
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611079    7765 reflector.go:175] Starting reflector *v1.Pod (0s) from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611104    7765 reflector.go:175] Starting reflector *v1.Service (0s) from k8s.io/kubernetes/pkg/kubelet/kubelet.go:517
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611151    7765 reflector.go:175] Starting reflector *v1.Node (0s) from k8s.io/kubernetes/pkg/kubelet/kubelet.go:526
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611269    7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b620, {READY <nil>}
Nov 27 09:42:34 k8s-node2 kubelet[7765]: I1127 09:42:34.611281    7765 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0x400021b770, {READY <nil>}##