GitLab 18 — LFS over SSH fails, internal API 404 (/api/v4/internal/authorized_keys)

How to Use GitLab Self-managed
, , ,
1

Problem to solve

I am running GitLab 18 (Omnibus, Docker) and attempting to use LFS over pure SSH with:

lfs:
  pure_ssh_protocol: true

GitLab-Shell connects via the internal UNIX socket to GitLab Rails as expected, but SSH operations (including LFS) trigger repeated 404 responses from /api/v4/internal/authorized_keys.

LFS authentication succeeds, but when the transfer starts, GitLab Shell cannot retrieve key information and the workflow fails.

Steps to reproduce

  1. Create docker-compose.yml with following configuration:
services:
  gitlab:
    image: gitlab/gitlab-ce:nightly
    mem_limit: 7g
    container_name: gitlab
    restart: always
    hostname: 'gitlab.example.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.com'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        gitlab_rails['gitlab_ssh_host'] = 'git.example.com'
        gitlab_shell['lfs_pure_ssh_protocol'] = true
    ports:
      - '8100:80'
      - '2424:22'
    volumes:
      - './config:/etc/gitlab'
      - './logs:/var/log/gitlab'
      - './data:/var/opt/gitlab'
    shm_size: '512m'
  1. Run docker compose up -d
  2. Verify Gitlab Workhorse exists:
docker exec -it gitlab bash
netstat -lx | grep gitlab-workhorse
# Example output:
# unix  2  [ ACC ]  STREAM  LISTENING  78281334  /var/opt/gitlab/gitlab-workhorse/sockets/socket
  1. Create and push a test repository with LFS:
git clone git@gitlab.example.com:username/test-lfs.git
cd test-lfs
git lfs install
git lfs track "largefile.bin"
git add .gitattributes largefile.bin
git commit -m "Add large LFS file"
git push origin main
  1. Observe the issue:
  • Push fails during the LFS upload stage.
  • gitlab-shell logs show:
"msg":"Internal API error","status":404,"url":"http://unix/api/v4/internal/authorized_keys"
  • The LFS transfer command falls back to HTTPS, failing with:
POST https://gitlab.example.com/user/repo.git/info/lfs/objects/batch giving up after 4 attempt(s):
dial tcp 172.18.0.2:443: connect: connection refused

Versions

  • Self-managed Community Edition 18.1.0-pre

Versions

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	3.2.5
Gem Version:	3.6.8
Bundler Version:2.6.5
Rake Version:	13.0.6
Redis Version:	7.2.7
Sidekiq Version:7.3.9
Go Version:	unknown

GitLab information
Version:	18.1.0-pre
Revision:	2b2746757ed
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	16.8
URL:		https://gitlab.example.com
HTTP Clone URL:	https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:	git@git.example.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.42.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	17.10.0-rc1-452-gb820245b9
- default Git Version: 	2.49.0.gl2

Other notes

  • Normal SSH Git pushes/pulls work.
  • Manual socket connectivity works
  • No external network issue — TCP dumps confirm correct packet routing.
  • Internal API calls via http+unix:// are partially successful (some 200, some 404).

What I suspect

  • The 404 responses from /api/v4/internal/authorized_keys indicate Rails receives the request but cannot resolve the SSH key.