Cannot delete old job logs on Privately managed gitlab server

Problem to solve

I have read (Job logs | GitLab) that explains how to delete old job logs using the following command:

find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete

I have direct access to VM where the Gitlab service is running. I have ssh into the VM and executed the command with sudo privilleges, however, the old job logs have not been deleted.

Steps to reproduce

  1. Have some old logs (older than 60 days) in your Gitlab repository
  2. SSH into the VM where your privately managed Gitlab service is running
  3. Execute command:
find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete

Configuration

  • Privately Managed Gitlab server

Versions

  • [v] Self-managed
System information
System:         Ubuntu 20.04
Current User:   git
Using RVM:      no
Ruby Version:   3.1.4p223
Gem Version:    3.5.5
Bundler Version:2.5.5
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.1.6
Go Version:     unknown

GitLab information
Version:        16.9.2
Revision:       8443896c867
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.13
URL:            https://xxx-test.test.lt
HTTP Clone URL: https://xxx-test.test.lt/some-group/some-project.git
SSH Clone URL:  git@xxx-test.test.lt:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.33.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      16.9.2
- default Git Version:  2.43.0

Some additional information

I have cd’ed into the /var/opt/gitlab/gitlab-rails/shared/artifacts directory to confirm that there is something stored and I can positvely confirm. See logs below:

root@xxx-test:/var/opt/gitlab/gitlab-rails/shared/artifacts# ls
2a  89  a8  b1  tmp

I have tried to enter one of the directories to see what is inside. See the example below:

root@xxx-test:/var/opt/gitlab/gitlab-rails/shared/artifacts/a8/8a/a88a7902cb4ef697ba0b6759c50e8c10297ff58f942243de19b984841bfe1f73# ls
2023_12_20  2024_01_02  2024_01_09  2024_01_16  2024_01_18  2024_01_22  2024_01_25  2024_02_12  2024_02_14  2024_03_14  2024_03_18  2024_03_20  2024_03_25
2023_12_22  2024_01_08  2024_01_15  2024_01_17  2024_01_19  2024_01_23  2024_01_29  2024_02_13  2024_02_21  2024_03_15  2024_03_19  2024_03_21

As you can see from above, there are bunch of logs from longer than 60 days. I have tried to execute the command:

find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete

After executing the command, I can still see that there are old log for example (2023/12/20) and etc…

What could be the issue? How can I properly delete old job logs?