Docker containing consuming disk space

I did install an instance of gitlab using docker compose, as explained in Install GitLab in a Docker container | GitLab

The docker-compose.yml file:
version: ‘3.6’
services:
gitlab:
image: ‘gitlab/gitlab-ce:17.1.4-ce.0’
restart: always
hostname: ‘gitlab.local’
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url ‘https://my.url’
ports:
- ‘8001:80’
- ‘9443:443’
- ‘22:22’
volumes:
- ‘/home/git/gitlab/config:/etc/gitlab’
- ‘/home/git/gitlab/logs:/var/log/gitlab’
- ‘/home/git/gitlab/data:/var/opt/gitlab’
shm_size: ‘256m’

Although I have set the volume paths, it seems that there are other files being written in the container as it slowly grows until it fills up the full disk space (40GB).

For now, my workaround is to recreate the container about once a month, taking advantage of pulling updated docker images, but I would like to solve this permanently.

Do you know what could be causing the container image to grow?

It’s unclear exactly what would be causing the container image to grow.

Are you perhaps creating GitLab backups? (these would be stored in /var/opt/gitlab/backups (or /home/git/gitlab/data/backups in your case) by default.

One thing you can do to investigate an unexplained increase in disk consumption would be to regularly run a command that determines disk usage on the directories which GitLab uses, and monitor this over for any signifigant changes in disk consumption.

For example,

du -h --max-depth=1 --threshold=100M | sort -hr /home/git/gitlab/data 
du -h --max-depth=1 --threshold=100M | sort -hr /home/git/gitlab/logs

I have the same issue here. And i’ve had it for a while, but so long whereas I’ve been running this container for years, meaning something must have changed, say in the last year.

Mounted volumes are as documented:

• ‘/home/git/gitlab/config:/etc/gitlab’
• ‘/home/git/gitlab/logs:/var/log/gitlab’
• ‘/home/git/gitlab/data:/var/opt/gitlab’

This containerized instance does very little, it’s used as as POC with other services integration.

Nevertheless gitlab/gitlab-ce:17.4.0-ce.0 has grown to 19Gb in 5weeks.

19Gb is as measured from the host:
jean@host:~/dockerhost$ sudo du -hxd1 /var/lib/docker/containers |sort -h
40K /var/lib/docker/containers/c5e764e5371e5fd1f57e135e3e0773818eeff85f372f62b4a83c2df7d267f6ba
[…]
/var/lib/docker/containers/04913295f19bb9116b609fcefaaecb0f9a2f1c00f64271d2e82def6f0bf6c7db
507M /var/lib/docker/containers/6006e89e7816a087b8dc0e1020f21050ba653ecd8b45f8244db50029d46d900e
19G /var/lib/docker/containers/705cc4dd81a7bf27f2efda04b190fb62d1dc118d57eb742602cb0b0febc280ea
20G /var/lib/docker/containers

But as measured from within the container, I can’t find this amount of data:

root@705cc4dd81a7:/# du -xhd1 / |sort -h
4.0K /boot
4.0K /home
4.0K /media
4.0K /mnt
4.0K /srv
8.0K /tmp
16K /root
28K /run
32K /assets
1.9M /etc
27M /var
145M /usr
3.0G /opt
3.2G /

NB: du -x is to skip mounted FS

But
docker rm 705cc4dd81a7
docker compose start gitlab

just solves my issue.
the new container just uses less than 1MB (though growing quickly) and my FS usage size was reduced by 19Gb!
And the Gitlab instance is fully functional with its data.

There must be data accumulating in the container somehow.

Too late now, but I should simply have been looking into the /var/lib/containers/<contianer_id> folder.

I’d bet the <container_id>_json.log is the culprit.

I’m trying this: <google dockerdocs “Configure logging drivers”> (I’m not allowed to post the URL here for some reason ?!?!)
which could well be the cause of this growing log file with my “logging-unconfigured” docker instance.

Obviously, setting /etc/docker/daemon.json with the log config

“log-driver”: “local”

doesn’t solve the issue.

The <container_id>_json.log file within /var/lib/docker/containers/<container_id> has grown to 2Gb since my last post.

Looking for new way to get a proper log rotation…

Thanks for the information. I realised that my problem is also about the _json.log files

I just added to the compose file the lines:

logging:
  driver: "json-file"
  options:
    max-size: "512m"

EDIT: This has solved the issue for me. Since gitlab logs are kept in a different place, I don’t think that it’s an issue to add this limit.

For those not using compose you can add this to your docker command

docker run
–name gitlab
–log-driver=json-file
–log-opt max-size=512m
–log-opt max-file=3

gitlab/gitlab-ce:latest

I solved this setting the log options in /etc/docker.daemon.json:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m"
  }
}