Gitlab Rails production_json.log

Infrastructure as Code & Cloud Native
1

Hi,

Recently I found Gitlab Logs taking too many storage at my Gitlab CE Server. Is it safe for me to truncate the files and manually delete specific archived logs?

Can you share the best config for Logs Management, assuming I only have 500GB allocation for storage. Currently I haven’t change anything within my gitlab.rb files.

################################################################################
## GitLab Logging
##! Docs: https://docs.gitlab.com/omnibus/settings/logs.html
################################################################################

# logging['svlogd_size'] = 200 * 1024 * 1024 # rotate after 200 MB of log data
# logging['svlogd_num'] = 30 # keep 30 rotated log files
# logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours
# logging['svlogd_filter'] = "gzip" # compress logs with gzip
# logging['svlogd_udp'] = nil # transmit log messages via UDP
# logging['svlogd_prefix'] = nil # custom prefix for log messages
# logging['logrotate_frequency'] = "daily" # rotate logs daily
# logging['logrotate_maxsize'] = nil # rotate logs when they grow bigger than size ...
# logging['logrotate_size'] = nil # do not rotate by size by default
# logging['logrotate_rotate'] = 30 # keep 30 rotated logs
# logging['logrotate_compress'] = "compress" # see 'man logrotate'
# logging['logrotate_method'] = "copytruncate" # see 'man logrotate'
# logging['logrotate_postrotate'] = nil # no postrotate command by default
# logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers ...

Thanks in advance

2

Hi @erdeha :wave:

Log files should not take up any significant amount of your 500Gb storage allocation. If logs are taking up more than ~3% of available storage, I suspect something happened to prevent the logs from successfully rotating.

If you run du -h --threshold=50M /var/log/gitlab/* | sort -hr, do you see log directories consuming an abnormally large amount of storage?

If so, can you share the output from

# return 10 largest files in <large-log-directory>
ls -lhS /var/log/gitlab/<large-directory>/* | head -n 10
# run logrotate in debug mode 
/opt/gitlab/embedded/sbin/logrotate -d -v -s /var/opt/gitlab/logrotate/logrotate.status /var/opt/gitlab/logrotate/logrotate.conf
1 Like
3

Hi Greg, thanks for the reply,

Brief: I have my Gitlab running at Docker, current version 16.5.1-ce. Mounting the log files at main/logs folder

# du -h --threshold=50M main/logs/* | sort -hr
90G     main/logs/gitlab-rails
27G     main/logs/nginx
1.6G    main/logs/gitaly
479M    main/logs/gitlab-workhorse
430M    main/logs/sidekiq
196M    main/logs/puma

# ls -lhS main/logs/gitlab-rails/* | head -n 10
-rwxrwxrwx 1 lxd  docker  58G Nov 24 14:55 main/logs/gitlab-rails/production_json.log
-rwxrwxrwx 1 lxd  docker  13G Nov 24 14:55 main/logs/gitlab-rails/api_json.log
-rwxrwxrwx 1 lxd  docker 8.6G Nov 24 14:55 main/logs/gitlab-rails/production.log
-rwxrwxrwx 1 lxd  docker 8.4G Nov 24 14:55 main/logs/gitlab-rails/graphql_json.log
-rwxrwxrwx 1 lxd  docker 1.9G Nov 24 14:54 main/logs/gitlab-rails/application_json.log
-rwxrwxrwx 1 lxd  docker 103M Apr  1  2023 main/logs/gitlab-rails/application.log
-rwxrwxrwx 1 lxd  docker 102M Nov 24 10:40 main/logs/gitlab-rails/auth.log
-rwxrwxrwx 1 lxd  docker  86M Nov 24 14:54 main/logs/gitlab-rails/sidekiq_client.log
-rwxrwxrwx 1 lxd  docker  54M Nov 24 11:39 main/logs/gitlab-rails/exceptions_json.log
-rwxrwxrwx 1 lxd  docker 3.8M Nov 24 14:45 main/logs/gitlab-rails/audit_json.log

I think Logrotate working fine, we can see 30 archived log files within the folder

# ncdu main/logs/gitlab-rails 
   57.7 GiB [##########]  production_json.log
   12.6 GiB [##        ]  api_json.log
    8.6 GiB [#         ]  production.log
    8.3 GiB [#         ]  graphql_json.log
    1.9 GiB [          ]  application_json.log
  102.4 MiB [          ]  application.log
  101.3 MiB [          ]  auth.log   
   85.0 MiB [          ]  sidekiq_client.log
   53.5 MiB [          ]  exceptions_json.log
    3.7 MiB [          ]  audit_json.log
    3.4 MiB [          ]  production_json.log.26.gz
    3.1 MiB [          ]  production_json.log.30.gz
    3.1 MiB [          ]  production_json.log.9.gz
    3.0 MiB [          ]  production_json.log.6.gz
    3.0 MiB [          ]  production_json.log.15.gz
    3.0 MiB [          ]  production_json.log.14.gz
    2.9 MiB [          ]  production_json.log.1.gz
    2.8 MiB [          ]  production_json.log.21.gz
    ...
4

Hi Greg,

Yay, solved the problem. You were right, logrotate doesn’t work properly due to permission problem at /var/opt/gitlab/logrotate/logrotate.d/xxx files.

Found it with script you gave earlier:
/opt/gitlab/embedded/sbin/logrotate -d -v -s /var/opt/gitlab/logrotate/logrotate.status /var/opt/gitlab/logrotate/logrotate.conf

I have changed the permission to 600, then run logrotate manually. It drastically reduce the disk usage.

Thank you very much

2 Likes