Under GitLab v16.8 this worked fine. In v16.9 it appears to be broken.
Our versions and stuff
System information
System: Ubuntu 22.04
Current User: git
Using RVM: no
Ruby Version: 3.1.4p223
Gem Version: 3.3.26
Bundler Version:2.5.5
Rake Version: 13.0.6
Redis Version: 6.2.14
Sidekiq Version:7.1.6
Go Version: go1.20.8 linux/amd64
GitLab information
Version: 16.9.9-rc42
Revision: b4dfa475b45
Directory: /home/git/gitlab
DB Adapter: PostgreSQL
DB Version: 14.12
URL: https://gitlab.domain.com
HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git
SSH Clone URL: git@gitlab.domain.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: <hidden>
GitLab Shell
Version: 14.33.0
Repository storages:
- default: unix:/home/git/gitlab/tmp/sockets/private/gitaly.socket
GitLab Shell path: /home/git/gitlab-shell
Gitaly
- default Address: unix:/home/git/gitlab/tmp/sockets/private/gitaly.socket
- default Version: 14.8.0-rc1-7359-g9f0fa9cdb5
- default Git Version: 2.43.0
Extended check also passes, along with all the repositories checks etc.
What we are doing
We push mirror https://gitlab.domain.com/group/project-docs → https://gitlab.domain.com/group/project.wiki.git, using a token with scopes api, read_api, read_user, read_repository, write_repository.
What happened
When we upgraded from v16.8 to v16.9, under Settings → Repository → Mirror repositories: We now see:
(Error: The remote mirror URL is invalid.)
Oddly, we also see under Last successful update: 17 minutes ago. Or some time that updates every hour or so. However, the repository is not updating.
These two errors from gitaly.log I believe to be related:
time="2024-06-12T14:44:51.596Z"
level=info
msg="finished streaming call with code NotFound"
catfile.duration_ms=0
catfile.flush_count=1
catfile.flush_ms=0
catfile.read_object_count=1
catfile.read_object_ms=0
catfile.request_object_count=1
catfile.request_object_ms=0
command.count=2
command.cpu_time_ms=1
command.inblock=0
command.majflt=0
command.maxrss=447340
command.minflt=199
command.oublock=0
command.real_time_ms=1
command.spawn_token_fork_ms=0
command.spawn_token_wait_ms=0
command.system_time_ms=0
command.user_time_ms=1
component=gitaly.StreamServerInterceptor
correlation_id=01J06EH93ZV6VD3GR1V51HY20M
error="tree entry not found"
error_metadata="map[path:.gitlab-ci.yml]"
grpc.code=NotFound
grpc.meta.auth_version=v2
grpc.meta.client_name=gitlab-web
grpc.meta.deadline_type=regular
grpc.meta.method_operation=accessor
grpc.meta.method_scope=repository
grpc.meta.method_type=server_stream
grpc.method=TreeEntry
grpc.request.deadline="2024-06-12T14:45:21.393"
grpc.request.fullMethod=/gitaly.CommitService/TreeEntry
grpc.request.glProjectPath=Group/project-docs
grpc.request.glRepository=project-719
grpc.request.payload_bytes=169
grpc.request.repoPath=@hashed/02/83/02837c1944876b4fa860432c13f2d9b11a7fd94dae707c4143d1217dee66fc43.git
grpc.request.repoStorage=default
grpc.response.payload_bytes=0
grpc.service=gitaly.CommitService
grpc.start_time="2024-06-12T14:44:51.593"
grpc.time_ms=3.386
pid=2434282
remote_ip=<removed>
span.kind=server
system=grpc
user_id=9
username=<removed>
time="2024-06-12T14:44:51.983Z"
level=error
msg="fatal: Not a valid object name main\n"
args="[/usr/local/bin/git --git-dir /home/git/repositories/@hashed/02/83/02837c1944876b4fa860432c13f2d9b11a7fd94dae707c4143d1217dee66fc43.wiki.git -c gc auto=0 -c maintenance.auto=0 -c core.autocrlf=input -c core.useReplaceRefs=false -c core.fsync=objects,derived-metadata,reference -c core.fsyncMethod=fsync -c core.packedRefsTimeout=10000 -c core.filesRefLockTimeout=1000 -c core.bigFileThreshold=50m ls-tree --full-tree --name-status -r -z --end-of-options main -- .]"
command.cpu_time_ms=1
command.exitCode=128
command.inblock=0
command.maxrss=447340
command.oublock=0
command.real_time_ms=1
command.system_time_ms=0
command.user_time_ms=1
component=gitaly.StreamServerInterceptor
correlation_id=01J06EH93ZV6VD3GR1V51HY20M
grpc.meta.auth_version=v2
grpc.meta.client_name=gitlab-web
grpc.meta.deadline_type=regular
grpc.meta.method_operation=accessor
grpc.meta.method_scope=repository
grpc.meta.method_type=server_stream
grpc.method=SearchFilesByName
grpc.request.deadline="2024-06-12T14:45:01.010"
grpc.request.fullMethod=/gitaly.RepositoryService/SearchFilesByName
grpc.request.glProjectPath=Group/projects-docs.wiki
grpc.request.glRepository=wiki-719
grpc.request.repoPath=@hashed/02/83/02837c1944876b4fa860432c13f2d9b11a7fd94dae707c4143d1217dee66fc43.wiki.git
grpc.request.repoStorage=default
grpc.service=gitaly.RepositoryService
grpc.start_time="2024-06-12T14:44:51.980"
path=/usr/local/bin/git
pid=2434282
remote_ip=<removed>
span.kind=server
system=grpc
user_id=9
username=<removed>
What we’ve tried:
- created a new push mirror set up with a valid personal token with the same result
- verified that we can clone the repository without issue, so we know it exists
- verified that both the source and recipient repositories are using the same branch (master)
- tried push with and without protected branches only
Questions
- anyone else experiencing this?
- any obvious error I am missing?
- any suggestions on what to look for or try?
Thanks.