Combing the expression of Prometheus Monitoring K8s Index

Calculation of Pod memory utilization

Metrics monitoring initiated by docker-cadvisor uses container_memory_rss

Metrics monitoring obtained by starting prometheus in k8s uses container_memory_usage_bytes or container_memory_working_set_bytes

The calculation of Pod memory usage is much simpler. You can simply divide the actual memory usage by the memory usage limit:

Cadvisor container memory utilization is greater than 90%

Sum by (name, id, job, node) (container_memory_rss {imagination = "", job= "ali-prod-executor-cadvisor"}) / sum by (name, id, job, node) (container_spec_memory_limit_bytes {imagination = "", job= "ali-prod-executor-cadvisor"}) * 100! = + Inf > 80

K8s method 1

Avg by (pod_name) (container_memory_usage_bytes {podroomnameplates = "", imageframes ~ ". * pause-amd64:1031 |. * pause-amd64:3.0"} / container_spec_memory_limit_bytes {podroomnameplates = "", imageframes ~ ". * pause-amd64:1031 |. * pause-amd64:3.0"}) * 100 > 90

Method 2

Sum by (pod_name, namespace, job) (container_memory_working_set_bytes {imagewings = "", imageframes ~ "xxxxx.com/xxs/pause.+", joblies = "xxxd-executor-cadvisor"}) / sum by (pod_name, namespace, job) (container_spec_memory_limit_bytes {imageframes = "", imageframes ~ "reg.linkdoc-inc.com/ops/pause.+", jobstones = "ali-prod-executor-cadvisor"}) * 100! = + Inf > 90

CPU utilization of the container:

Sum by (pod_name, namespace, job) (rate (container_cpu_usage_seconds_total {imagehorse = ""} [1m])) / (sum by (pod_name, namespace, job) (container_spec_cpu_quota {imagehorse = ""} / 100000)) * 100 > 90

Container inbound bandwidth is greater than 50m

Sum by (namespace,job,pod_name) (irate (container_network_receive_bytes_total {imagination = ""} [3m])) / 1024 / 1024 > 50

Container outbound bandwidth is greater than 50m

Sum by (namespace,job,pod_name) (irate (container_network_transmit_bytes_total {imagination = ""} [1m])) / 1024 / 1024 > 50