failed to collect filesystem stats - du command failed on /rootfs/

[zouyee]

E0728 10:24:15.958564 1 fsHandler.go:106] failed to collect filesystem stats - du command failed on /rootfs/var/lib/docker/overlay/0a56cad0b8709b76aefe3cf4dc43202016fce897e8010eb3b4b348a22ad3c00a with output stdout: 49780 /rootfs/var/lib/docker/overlay/0a56cad0b8709b76aefe3cf4dc43202016fce897e8010eb3b4b348a22ad3c00a
, stderr: du: /rootfs/var/lib/docker/overlay/0a56cad0b8709b76aefe3cf4dc43202016fce897e8010eb3b4b348a22ad3c00a/merged/proc/1/task/10/fdinfo/10: No such file or directory

[wallnerryan]

same

[dashpole]

What os version, docker version, and cadvisor version are you using?

[wallnerryan]

centos 7, docker 1.12.6, google/cadvisor:latest

[otakup0pe]

seeing similar on a host with a lot of short lived containers... Ubuntu 16.04.2 LTS, Docker 17.04.0-ce, google/cadvisor:latest.

[steinbrueckri]

same ...

• CentOS Linux release 7.3.1611 (Core)
• Docker version 1.12.6, build 3a094bd/1.12.6
• google/cadvisor:latest (cAdvisor version v0.25.0 (17543be))

[dashpole]

I just realized, this error occurs when a container is deleted while cAdvisor is traversing its filesystem with du. If you have a lot of short lived containers, it isn't surprising to see this. Is anyone seeing any other adverse effects, or just this error? Is anyone seeing seeing it multiple times for the same container? How "short lived" are these containers?

du on containers with a moderately large number of files can take ~seconds, so if containers last less than a couple seconds, it may not be possible to get disk statistics on them. However, theoretically, this shouldnt cause any problems, other than possibly no disk stats on short lived containers.

[steinbrueckri]

In my case it's a "long running" container... but yes i have also some "short lived" containers.

[narcoticfresh]

Incredible that this issue it still open.. it is quite prominent.. AFAIK, everybody in a swarm cluster and their nodes using overlayfs, has this issue.. obviously, the proc structures/folders in those folders are pseudo files and du is having problems with that..

--> if only we could configure some exclude pattern (--exclude= option from du) on cadvisor..