[FIXED] Git clone 403 for runner after 16.6.0 upgrade (wrong permissions?)

Greetings.

After upgrading to 16.6.0, some of my pipelines broke due to an inability to clone a parent project.
Weirdly, this happened to only a single project, of which i’m owner.

Running on runner-6zuem4c3-project-86-concurrent-0 via ca2b20704239...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /my/new/project.git/
Checking out 4fd9c480 as detached HEAD (ref is main)...
Removing .m2/
Skipping Git submodules setup
Restoring cache 00:01
Checking cache for default-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script 00:01
Using docker image sha256:a0a0daf50c07d743a7bfdc6cf8965e5d3fe89562bc3088011dd596ed2eb3533f for docker.mycompany.com/jobimage:1.0.0 with digest docker.mycompany.com/jobimage@sha256:42d0f537091acae3518dd5b19e1cfae1e690e8ce8d7e40d3b234d98c2a2d4fd0 ...
$ rm -rf ../path || true
 git clone --branch ${CI_COMMIT_REF_NAME} https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab/my/project/path.git ../path
Cloning into '../path'...
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab/my/project/path.git/': The requested URL returned error: 403

proof of ownership:

Relevant Omnibus logs :

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.42.144.247 - - [28/Nov/2023:09:18:07 +0000] "GET /my/new/project/-/jobs/4203/trace.json HTTP/1.1" 200 843 "https://gitlab/my/new/project/-/jobs/4203" "Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0" 2.28

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/my/project/path.git/info/refs","format":"*/*","controller":"Repositories::GitHttpController","action":"info_refs","status":401,"time":"2023-11-28T09:18:08.265Z","params":[{"key":"service","value":"git-upload-pack"},{"key":"repository_path","value":"my/project/path.git"}],"correlation_id":"01HGAKHEBY62PPWMW13YEX63C3","repository_storage":"default","remote_ip":"10.42.144.247","ua":"git/2.20.1","request_urgency":"default","target_duration_s":1,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":3,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.001,"db_main_duration_s":0.001,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.009689,"mem_objects":6735,"mem_bytes":865040,"mem_mallocs":2251,"mem_total_bytes":1134440,"pid":580,"worker_id":"puma_3","rate_limiting_gates":[],"db_duration_s":0.00083,"view_duration_s":0.00023,"duration_s":0.00563}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/plain; charset=utf-8","correlation_id":"01HGAKHEBY62PPWMW13YEX63C3","duration_ms":12,"host":"gitlab","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"myip:0","remote_ip":"myip","route":"^/.+\\.git/info/refs\\z","status":401,"system":"http","time":"2023-11-28T09:18:08Z","ttfb_ms":12,"uri":"/my/project/path.git/info/refs?service=git-upload-pack","user_agent":"git/2.20.1","written_bytes":281}

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.42.144.247 - - [28/Nov/2023:09:18:08 +0000] "GET /my/project/path.git/info/refs?service=git-upload-pack HTTP/1.1" 401 281 "" "git/2.20.1" -

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/my/project/path.git/info/refs","format":"*/*","controller":"Repositories::GitHttpController","action":"info_refs","status":403,"time":"2023-11-28T09:18:08.353Z","params":[{"key":"service","value":"git-upload-pack"},{"key":"repository_path","value":"my/project/path.git"}],"correlation_id":"01HGAKHEE9NWT40EB99GYB4SEQ","meta.caller_id":"Repositories::GitHttpController#info_refs","meta.remote_ip":"10.42.144.247","meta.feature_category":"source_code_management","meta.user":"brajaut","meta.user_id":16,"meta.project":"my/project/path","meta.root_namespace":"best","meta.client_id":"user/16","repository_storage":"default","remote_ip":"10.42.144.247","user_id":16,"username":"brajaut","ua":"git/2.20.1","request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000172,"redis_read_bytes":102,"redis_write_bytes":60,"redis_repository_cache_calls":1,"redis_repository_cache_duration_s":0.000172,"redis_repository_cache_read_bytes":102,"redis_repository_cache_write_bytes":60,"db_count":13,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_main_count":11,"db_ci_count":2,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"db_main_duration_s":0.003,"db_ci_duration_s":0.001,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.02049,"mem_objects":15505,"mem_bytes":1818560,"mem_mallocs":4905,"mem_total_bytes":2438760,"pid":580,"worker_id":"puma_3","rate_limiting_gates":[],"db_duration_s":0.00328,"view_duration_s":0.00019,"duration_s":0.0185}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/plain; charset=utf-8","correlation_id":"01HGAKHEE9NWT40EB99GYB4SEQ","duration_ms":25,"host":"gitlab","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"myip:0","remote_ip":"myip","route":"^/.+\\.git/info/refs\\z","status":403,"system":"http","time":"2023-11-28T09:18:08Z","ttfb_ms":25,"uri":"/my/project/path.git/info/refs?service=git-upload-pack","user_agent":"git/2.20.1","written_bytes":55}

Permissions of that parent project :

I tried to remove myself from group, invite myself back and all.
I’m the administrator.

I believe it’s something simple but i cant figure what.

EDIT : stumbled upon it, it’s the token access control / limit access to this project.
leaving this topic for reference for other users :


This was somehow checked by default on the non-working clone, but unchecked where it works.

Many thanks

1 Like

Hi,

first: There is runner 16.6.1, try this

best greetings

Thanks for answering. The runner was running the latest docker image available.

This affected us as well and the fix worked. Why were we affected in version 16.6.2 when this was released in 15.9 and 15.10?