Docker container disk occupies 100%

Question:

Root@anxxxx-prod1:~# df-hFilesystem Size Used Avail Use% Mounted onudev 3.9G 0 3.9G 0% / devtmpfs 799M 3.4M 795m 1% / run/dev/vda1 99G 96G 0% / tmpfs 3.9G 548K 3.9G 1% / dev/shmtmpfs 5.0M 05.0M 0% / run/locktmpfs 3.9G 0 3.9G 0% / sys/fs/cgroup3f5be49b80-fuf99.cn-shanghai.nas.aliyuncs.com:/ 10p 18G 10P 1% / mntnone 99G 96G 0100% / var/lib/docker/aufs/mnt/da4441ba96f3a657e90e289da6f59090bb41baba95355b1b62e239e0000a17afnone 99G 96G 0100% / var/lib/ Docker/aufs/mnt/aaf7527a3a42fc6d3695dc95d6ddb6bc6473337bf14d5fa3debf4ba70c5be70eshm 64M 0 64m 0% / var/lib/docker/containers/019052bc4d2c1c0367abdca83222ca2dc4a9bc88a264aa5d8e1f4575f4f59d45/shmshm 64M 0 64m 0% / var/lib/docker/containers/f0fbd839ff52f77a9f1d4a13772194553c2a3d52f8cd76547da696c6e683973a/shmtmpfs 799m 0 799m 0% / run/user/0none 99G 96G 0% / var/lib/docker/aufs/mnt/3ca913d1e6aad8c7908c949b4f2472fde89d1c6f10d2fb29779287fa4c9c3efbshm 64M 0 64m 0% / var/lib/docker/containers/bc9fe7700f8c8f5d249ff2e710515218969b0f09bd39d82d90e6622f71f8152d/shm

After removing the mirror, there is only a slight reduction in used usage (reduced 3G space)

Then clean up the virtual overhang mirror.

Docker system prune-a

Find large files in the system

Find /-type f-size + 100m-print0 | xargs-0 du-h | sort-nr # finds all files larger than 100m in the "/" directory. 175m / root/log/monolith/monolith.2019-11-23.09.log175M / root/log/monolith/monolith.2019-11-23.07.log175M / mnt/backup/mysql/backup/2019-08-01-22:28:56/prod-mysql.mysql.rds. Aliyuncs.com/data/db_ankobot.sql.gz125M / root/log/monolith/monolith.2019-12-10.11.log123M / mnt/test.ankobot.com/uploads/facesets/facesets.zip121M / root/log/monolith/monolith.2019-11-23.05.log69G / var/lib/docker/containers/bc9fe7700f8c8f5d249ff2e710515218969b0f09bd39d82d90e6622f71f8152d/bc9fe7700f8c8f5d249ff2e710515218969b0f09bd39d82d90e6622f71f8152d-json.log1.7G / var/lib/docker/containers/019052bc4d2c1c0367abdca83222ca2dc4a9bc88a264aa5d8e1f4575f4f59d45/019052bc4d2c1c0367abdca83222ca2dc4a9bc88a264aa5d8e1f4575f4f59d45-json.log0 / proc/kcore

It is found that there is a *-json.log log file with large data under / var/lib/docker/containers/ {container_id} /. Baidu found that this is the standard input log run by the docker container and deleted it. Project output log files have been mounted in the way of-v in the project, so there is no need for container running logs. After the study, it is found that the size and number of standard input logs can be limited when building parameters. In order to reduce the storage space occupied by log files, the following configuration is the maximum capacity of log files and the maximum number of log files.

Cleared the log file

Root@anxxxx-prod1:~# echo'> / var/lib/docker/containers/bc9fe7700f8c8f5d249ff2e710515218969b0f09bd39d82d90e6622f71f8152d/bc9fe7700f8c8f5d249ff2e710515218969b0f09bd39d82d90e6622f71f8152d-json.logdocker run. -log-opt max-size=10m-log-opt max-file=1

You can also make global modifications in the configuration file of docker: create or modify / etc/docker/daemon.json, and add log-dirver and log-opts parameters (daemon.json parameter description:

{"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "1"}}