What if k8s cpu limit node node is abnormal?

This article shows you how to do abnormal k8s cpu limit node nodes. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Log information:

OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused\" process_linux.go:367: setting cgroup config for procHooks process caused\\ "failed to write 100000 to cpu.cfs_period_us: write / sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod0f117614-9bae-11e9-b422-525400530fa1/fa79711ac936f4650b00bab66e21f5efb64fac642aa38e9c314385ab4023b484/cpu.cfs_period_us: invalid argument\": unknown

From the description of log information: cgroup cput parameter: cpu.cfs_period_us: invalid argument\\ "\": there is a problem with unknown!

Parameter file directory: / sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod0f117614-9bae-11e9-b422-525400530fa1/fa79711ac936f4650b00bab66e21f5efb64fac642aa38e9c314385ab4023b484/cpu.cfs_period_us

The specific reason is unknown!

Symptoms that occur:

The newly dispatched pod has started and reported the above error. There seems to be no problem with the previous pod.

Https://my.oschina.net/xiaominmin/blog/3068378

Https://kubernetes.io/zh/docs/tasks/administer-cluster/cpu-management-policies/

Https://qingwave.github.io/2019/01/09/%E6%B7%B1%E5%85%A5%E7%90%86%E8%A7%A3K8s%E8%B5%84%E6%BA%90%E9%99%90%E5%88%B6/#%E5%86%85%E5%AD%98%E9%99%90%E5%88%B6

Https://www.yangcs.net/posts/understanding-resource-limits-in-kubernetes-cpu-time/

Cause: kernel issu

Solution: node executes script

# / usr/bin/python2 "" Fixs a CFS bug in Container-optimized OS (COS) nodes.The bug is described at: https://docs.google.com/document/d/13KLD__6A935igLXpTFFomqfclATC89nAkhPOxsuKA0I/edit#Examples:To check script sanity in COS VM: sudo python fix-cfs.py-dry_runTo fix cfs bug in COS VM: sudo python fix-cfs.pyWhen running in a container/DaemonSet: sudo python fix-cfs.py-sys=/host/sys-dry_run sudo python fix-cfs.py -- sys=/host/sys sudo python fix-cfs.py-- sys=/host/sys-- interval=10 "" from _ _ future__ import print_functionimport argparseimport mathimport osimport timedef _ ParseCommandLine (): parser = argparse.ArgumentParser (description='Fix cfs bug.') Parser.add_argument ('- dry_run', dest='dry_run', action='store_true', help='Whether or not to run in dry_run mode') parser.set_defaults (dry_run=False) parser.add_argument ('- sys', type=str, default='/sys', help='The root directory of the / sys fs') parser.add_argument ('- interval', type=int) Help='Seconds to wait between invocations') return parser.parse_args () def ReadFile (directory, filename): with open (os.path.join (directory, filename),'r') as f: return f.read () def WriteFile (directory, filename, number): with open (os.path.join (directory, filename) 'w') as f: f.write ('% d'% number) def ListSubdirs (directory): return [os.path.join (directory, f) for f in os.listdir (directory) if os.path.isdir (os.path.join (directory, f))] def CalculateNewQuotaPeriodForPod (quota Period): scaled_times = math.ceil ((math.log (period)-math.log (100000)) / (math.log-math.log (100000)) new_period = 100000 new_quota = math.floor (quota * math.pow (128.0 / 147, scaled_times)) + 1 + scaled_times return new_quota, new_perioddef CalculateNewQuotaPeriodForContainer (quota Period): scaled_times = math.ceil ((math.log (period)-math.log (100000)) / (math.log (147147-math.log)) new_period = 100000 new_quota = math.floor (quota * math.pow (128.0 / 147147, scaled_times)) new_quota = max (new_quota, 1000) return new_quota, new_perioddef FixPodIfAffected (pod_dir, dry_run): try: quota = long (ReadFile (pod_dir) 'cpu.cfs_quota_us')) period = long (ReadFile (pod_dir,' cpu.cfs_period_us')) except IOError: print ('Skipping pod_dir% s. The pod may have disappeared from cfs before',' it could be examined'% (pod_dir)) return if quota