Artifacts stored/searched in a wrong date directory

Hi!

We’re running version 13.8.1-ce in a docker container using this image - gitlab/gitlab-ce:13.8.1-ce.0

And we expirience weird behavior of job’s artifacts and traces storage.
Artifacts of jobs that run in period of 0:00AM - 3:00AM UTC cannot be found by gitlab. These artifacts do exist on disk, but gitlab is looking for them in a wrong date directory.

For example there is a job with id 4996 that was run on 16.04.2020 0:00UTC, artifacts of this job are located in /gitlab-rails/shared/artifacts/../../../2021_04_16/4996/
But in gitlab ui there is no traces (and artifacts) of that job. And if requested through api it responds with 500 error and following error in logs:
Errno::ENOENT (No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/../../../2021_04_15/4996/470/job.log): and long stack trace.

So gitlab is looking for this job’s traces in previous day’s directory. Host machine’s local time is GMT+3, so this 3 hour different might be relevant to that.
We tried changing gitlab_rails[‘time_zone’] in omnibus config to different values and mounting host machine /etc/localtome to container but none of that made any difference, traces in this time window (0:00AM - 3:00AM UTC) are still beign ‘lost’ by gitlab.

We are honestly out of ideas at this point, asking for advice.

So the problem was in timezone mismatch in postgresql. We are running it in a container and were mounting host’s /etc/localtime inside of it.
Because of that timestamps in db were saved in UTC time but with GMT+3 timezone, so that resulted in -3 hour difference throught gitlab.
Not mounting /etc/localtime/ in the postgresql container solved the issue.