Gitlab Runner fails when getting source from Git repository – fatal: loose object

GitLab CI/CD
, ,
1

I’m hosting GitLab CE and a GitLab Runner on Docker. Since a server crash the Runner fails when getting source from Git repository.

Since the crash every time the CI is triggered, the pipeline fails with the following error message:

Fetching changes with git depth set to 1...
Reinitialized existing Git repository in /builds/myproject/myrepo/.git/
error: object file .git/objects/35/830ac815ebe92a2c5e614396d429ad75a3658c is empty
fatal: loose object 35830ac815ebe92a2c5e614396d429ad75a3658c (stored in .git/objects/35/830ac815ebe92a2c5e614396d429ad75a3658c) is corrupt

I’m running GitLab 13.3.7 and GitLab runner 13.3.2 on Docker 19.03.13 and I setup everything up using docker-compose:

version: '3'

services:
    gitlab:
        image: 'gitlab/gitlab-ce:13.3.7-ce.0'
        container_name: 'gitlab'
        restart: unless-stopped
        hostname: 'gitlab.kontextlab.com'
        environment:
            - GIT_SSL_CAINFO=/etc/gitlab/trusted-certs/ca-certificates.crt
        ports:
            - '127.0.0.1:8080:80'
            - '127.0.0.1:9090:9090'
            - '127.0.0.1:9093:9093'
            - '0.0.0.0:2222:22'
        volumes:
            - '/var/lib/gitlab/config:/etc/gitlab'
            - '/var/lib/gitlab/logs:/var/log/gitlab'
            - '/var/lib/gitlab//data:/var/opt/gitlab'
        networks:
            - gitlab_net

    gitlab-runner-1:
        image: 'gitlab/gitlab-runner:v13.3.2'
        container_name: 'gitlab-runner-1'
        restart: unless-stopped
        volumes:
            - '/var/lib/gitlab-runners/1/config-runner:/etc/gitlab-runner'
            - '/var/lib/gitlab-runners/1/config/ssl/ca-certificates.crt:/etc/gitlab-runner/certs/ca.crt'
            - '/run/docker.sock:/var/run/docker.sock'
        depends_on:
            - gitlab
        networks:
            - gitlab_net

networks:
    gitlab_net:
        driver: bridge

I recreated all the containers with docker-compose. I cloned the repository on my local machine and on the server (with and without --depth=1) and I could not reproduce any loose objects. I also ran git fsck on the repository on the server and it didn’t report any errors. I also cleared the caches and pruned dangling docker images and containers.

I’m running out of ideas on how to solve or debug the issue. Any ideas or hints are welcome.

2

Today the problem seems to have disappeared. All I did is to change the logging verbosity of the runner to "debug".

3

And since today it’s happening again with the same loose objects :frowning: