"remote: Nil JSON web token" While cloning gitlab repo

peachyhead · April 13, 2025, 5:16am

Problem to solve

Hello, I found out an issue recently with runner.
Generally, all my jobs runs an error:

Reinitialized existing Git repository in /home/acid_admin/builds/t1_PpxTcq/0/dodge/myclassroom/.git/
[10](https://git.dodgeguild.ru/dodge/myclassroom/-/jobs/40#L10)
remote: Nil JSON web token
[11](https://git.dodgeguild.ru/dodge/myclassroom/-/jobs/40#L11)
fatal: unable to access 'https://git.dodgeguild.ru/dodge/myclassroom.git/': The requested URL returned error: 403
[12](https://git.dodgeguild.ru/dodge/myclassroom/-/jobs/40#L12)Cleaning up project directory and file based variables00:00
[13](https://git.dodgeguild.ru/dodge/myclassroom/-/jobs/40#L13)ERROR: Job failed: exit status 1

The error says the runner couldn’t clone the gitlab repo through https.
Note: the site successfully establish an https connection in browser.
Note 2: the user has an access to gitlab repo and the access token is correct with all required roles granted
Note 3: I don’t know much about setting up self-hosted services so the chat gpt was my best friend here (some of what I applied here may seem weird so i’m more than happy to receive your helpful commentaries)

Steps to reproduce

I tried cloning the repo via ssh and https through the terminal.
The ssh works fine, however https always returns 403 error code.
git clone log:

acid_admin@bellaciao-dev:~$ GIT_TRACE=1 GIT_CURL_VERBOSE=1 git clone https://git.dodgeguild.ru/dodge/myclassroom.git
04:34:59.195131 git.c:455               trace: built-in: git clone https://git.dodgeguild.ru/dodge/myclassroom.git
Cloning into 'myclassroom'...
04:34:59.197962 run-command.c:668       trace: run_command: git remote-https origin https://git.dodgeguild.ru/dodge/myclassroom.git
04:34:59.199155 git.c:742               trace: exec: git-remote-https origin https://git.dodgeguild.ru/dodge/myclassroom.git
04:34:59.199185 run-command.c:668       trace: run_command: git-remote-https origin https://git.dodgeguild.ru/dodge/myclassroom.git
04:34:59.203251 http.c:664              == Info: Couldn't find host git.dodgeguild.ru in the (nil) file; using defaults
04:34:59.241195 http.c:664              == Info:   Trying 176.109.108.111:443...
04:34:59.241313 http.c:664              == Info: Connected to git.dodgeguild.ru (176.109.108.111) port 443 (#0)
04:34:59.275728 http.c:664              == Info: found 440 certificates in /etc/ssl/certs
04:34:59.275825 http.c:664              == Info: GnuTLS ciphers: NORMAL:-ARCFOUR-128:-CTYPE-ALL:+CTYPE-X509:-VERS-SSL3.0
04:34:59.275852 http.c:664              == Info: ALPN, offering h2
04:34:59.275857 http.c:664              == Info: ALPN, offering http/1.1
04:34:59.277642 http.c:664              == Info: SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
04:34:59.278753 http.c:664              == Info:   server certificate verification OK
04:34:59.278773 http.c:664              == Info:   server certificate status verification SKIPPED
04:34:59.278841 http.c:664              == Info:   common name: git.dodgeguild.ru (matched)
04:34:59.278854 http.c:664              == Info:   server certificate expiration date OK
04:34:59.278861 http.c:664              == Info:   server certificate activation date OK
04:34:59.278870 http.c:664              == Info:   certificate public key: EC/ECDSA
04:34:59.278876 http.c:664              == Info:   certificate version: #3
04:34:59.278888 http.c:664              == Info:   subject: CN=git.dodgeguild.ru
04:34:59.278897 http.c:664              == Info:   start date: Wed, 09 Apr 2025 15:22:12 GMT
04:34:59.278903 http.c:664              == Info:   expire date: Tue, 08 Jul 2025 15:22:11 GMT
04:34:59.278919 http.c:664              == Info:   issuer: C=US,O=Let's Encrypt,CN=E6
04:34:59.278933 http.c:664              == Info: ALPN, server accepted to use http/1.1
04:34:59.278998 http.c:611              => Send header, 0000000249 bytes (0x000000f9)
04:34:59.279011 http.c:623              => Send header: GET /dodge/myclassroom.git/info/refs?service=git-upload-pack HTTP/1.1
04:34:59.279017 http.c:623              => Send header: Host: git.dodgeguild.ru
04:34:59.279022 http.c:623              => Send header: User-Agent: git/2.34.1
04:34:59.279027 http.c:623              => Send header: Accept: */*
04:34:59.279041 http.c:623              => Send header: Accept-Encoding: deflate, gzip, br, zstd
04:34:59.279050 http.c:623              => Send header: Accept-Language: C, *;q=0.9
04:34:59.279055 http.c:623              => Send header: Pragma: no-cache
04:34:59.279060 http.c:623              => Send header: Git-Protocol: version=2
04:34:59.279067 http.c:623              => Send header:
04:34:59.299247 http.c:664              == Info: Mark bundle as not supporting multiuse
04:34:59.299270 http.c:611              <= Recv header, 0000000027 bytes (0x0000001b)
04:34:59.299278 http.c:623              <= Recv header: HTTP/1.1 401 Unauthorized
04:34:59.299284 http.c:611              <= Recv header, 0000000031 bytes (0x0000001f)
04:34:59.299290 http.c:623              <= Recv header: Server: nginx/1.18.0 (Ubuntu)
04:34:59.299296 http.c:611              <= Recv header, 0000000037 bytes (0x00000025)
04:34:59.299301 http.c:623              <= Recv header: Date: Sun, 13 Apr 2025 04:34:59 GMT
04:34:59.299307 http.c:611              <= Recv header, 0000000041 bytes (0x00000029)
04:34:59.299312 http.c:623              <= Recv header: Content-Type: text/plain; charset=utf-8
04:34:59.299317 http.c:611              <= Recv header, 0000000028 bytes (0x0000001c)
04:34:59.299345 http.c:623              <= Recv header: Transfer-Encoding: chunked
04:34:59.299351 http.c:611              <= Recv header, 0000000024 bytes (0x00000018)
04:34:59.299358 http.c:623              <= Recv header: Connection: keep-alive
04:34:59.299363 http.c:611              <= Recv header, 0000000029 bytes (0x0000001d)
04:34:59.299369 http.c:623              <= Recv header: X-Frame-Options: SAMEORIGIN
04:34:59.299378 http.c:611              <= Recv header, 0000000021 bytes (0x00000015)
04:34:59.299383 http.c:623              <= Recv header: X-XSS-Protection: 0
04:34:59.299389 http.c:611              <= Recv header, 0000000033 bytes (0x00000021)
04:34:59.299395 http.c:623              <= Recv header: X-Content-Type-Options: nosniff
04:34:59.299399 http.c:611              <= Recv header, 0000000028 bytes (0x0000001c)
04:34:59.299405 http.c:623              <= Recv header: X-Download-Options: noopen
04:34:59.299411 http.c:611              <= Recv header, 0000000041 bytes (0x00000029)
04:34:59.299416 http.c:623              <= Recv header: X-Permitted-Cross-Domain-Policies: none
04:34:59.299422 http.c:611              <= Recv header, 0000000050 bytes (0x00000032)
04:34:59.299427 http.c:623              <= Recv header: Referrer-Policy: strict-origin-when-cross-origin
04:34:59.299435 http.c:611              <= Recv header, 0000000040 bytes (0x00000028)
04:34:59.299439 http.c:623              <= Recv header: Www-Authenticate: Basic realm="GitLab"
04:34:59.299442 http.c:611              <= Recv header, 0000000014 bytes (0x0000000e)
04:34:59.299446 http.c:623              <= Recv header: Vary: Accept
04:34:59.299450 http.c:611              <= Recv header, 0000000025 bytes (0x00000019)
04:34:59.299454 http.c:623              <= Recv header: Cache-Control: no-cache
04:34:59.299460 http.c:611              <= Recv header, 0000000021 bytes (0x00000015)
04:34:59.299465 http.c:623              <= Recv header: X-Runtime: 0.018077
04:34:59.299471 http.c:611              <= Recv header, 0000000088 bytes (0x00000058)
04:34:59.299479 http.c:623              <= Recv header: X-Gitlab-Meta: {"correlation_id":"c5d86a75-a8f0-45e4-8f25-af97e5d239e5","version":"1"}
04:34:59.299487 http.c:611              <= Recv header, 0000000052 bytes (0x00000034)
04:34:59.299493 http.c:623              <= Recv header: X-Request-Id: c5d86a75-a8f0-45e4-8f25-af97e5d239e5
04:34:59.299502 http.c:611              <= Recv header, 0000000002 bytes (0x00000002)
04:34:59.299508 http.c:623              <= Recv header:
04:34:59.299523 http.c:664              == Info: Connection #0 to host git.dodgeguild.ru left intact
Username for 'https://git.dodgeguild.ru': v.chalov
Password for 'https://v.chalov@git.dodgeguild.ru':
04:35:05.825818 http.c:664              == Info: Found bundle for host git.dodgeguild.ru: 0x564b1003daf0 [serially]
04:35:05.825832 http.c:664              == Info: Can not multiplex, even if we wanted to!
04:35:05.825842 http.c:664              == Info: Re-using existing connection! (#0) with host git.dodgeguild.ru
04:35:05.825850 http.c:664              == Info: Connected to git.dodgeguild.ru (176.109.108.111) port 443 (#0)
04:35:05.825864 http.c:664              == Info: Server auth using Basic with user 'v.chalov'
04:35:05.825931 http.c:611              => Send header, 0000000320 bytes (0x00000140)
04:35:05.825944 http.c:623              => Send header: GET /dodge/myclassroom.git/info/refs?service=git-upload-pack HTTP/1.1
04:35:05.825948 http.c:623              => Send header: Host: git.dodgeguild.ru
04:35:05.825950 http.c:623              => Send header: Authorization: Basic <redacted>
04:35:05.825951 http.c:623              => Send header: User-Agent: git/2.34.1
04:35:05.825955 http.c:623              => Send header: Accept: */*
04:35:05.825957 http.c:623              => Send header: Accept-Encoding: deflate, gzip, br, zstd
04:35:05.825959 http.c:623              => Send header: Accept-Language: C, *;q=0.9
04:35:05.825961 http.c:623              => Send header: Pragma: no-cache
04:35:05.825971 http.c:623              => Send header: Git-Protocol: version=2
04:35:05.825975 http.c:623              => Send header:
04:35:05.869137 http.c:664              == Info: Mark bundle as not supporting multiuse
04:35:05.869162 http.c:611              <= Recv header, 0000000024 bytes (0x00000018)
04:35:05.869168 http.c:623              <= Recv header: HTTP/1.1 403 Forbidden
04:35:05.869174 http.c:611              <= Recv header, 0000000031 bytes (0x0000001f)
04:35:05.869176 http.c:623              <= Recv header: Server: nginx/1.18.0 (Ubuntu)
04:35:05.869178 http.c:611              <= Recv header, 0000000037 bytes (0x00000025)
04:35:05.869180 http.c:623              <= Recv header: Date: Sun, 13 Apr 2025 04:35:05 GMT
04:35:05.869183 http.c:611              <= Recv header, 0000000041 bytes (0x00000029)
04:35:05.869185 http.c:623              <= Recv header: Content-Type: text/plain; charset=utf-8
04:35:05.869187 http.c:611              <= Recv header, 0000000028 bytes (0x0000001c)
04:35:05.869189 http.c:623              <= Recv header: Transfer-Encoding: chunked
04:35:05.869191 http.c:611              <= Recv header, 0000000024 bytes (0x00000018)
04:35:05.869193 http.c:623              <= Recv header: Connection: keep-alive
04:35:05.869195 http.c:611              <= Recv header, 0000000029 bytes (0x0000001d)
04:35:05.869196 http.c:623              <= Recv header: X-Frame-Options: SAMEORIGIN
04:35:05.869198 http.c:611              <= Recv header, 0000000033 bytes (0x00000021)
04:35:05.869200 http.c:623              <= Recv header: X-XSS-Protection: 1; mode=block
04:35:05.869202 http.c:611              <= Recv header, 0000000033 bytes (0x00000021)
04:35:05.869204 http.c:623              <= Recv header: X-Content-Type-Options: nosniff
04:35:05.869206 http.c:611              <= Recv header, 0000000028 bytes (0x0000001c)
04:35:05.869210 http.c:623              <= Recv header: X-Download-Options: noopen
04:35:05.869212 http.c:611              <= Recv header, 0000000041 bytes (0x00000029)
04:35:05.869213 http.c:623              <= Recv header: X-Permitted-Cross-Domain-Policies: none
04:35:05.869215 http.c:611              <= Recv header, 0000000050 bytes (0x00000032)
04:35:05.869217 http.c:623              <= Recv header: Referrer-Policy: strict-origin-when-cross-origin
04:35:05.869219 http.c:611              <= Recv header, 0000000040 bytes (0x00000028)
04:35:05.869221 http.c:623              <= Recv header: Permissions-Policy: interest-cohort=()
04:35:05.869223 http.c:611              <= Recv header, 0000000026 bytes (0x0000001a)
04:35:05.869224 http.c:623              <= Recv header: X-UA-Compatible: IE=edge
04:35:05.869226 http.c:611              <= Recv header, 0000000014 bytes (0x0000000e)
04:35:05.869228 http.c:623              <= Recv header: Vary: Accept
04:35:05.869229 http.c:611              <= Recv header, 0000000025 bytes (0x00000019)
04:35:05.869231 http.c:623              <= Recv header: Cache-Control: no-cache
04:35:05.869233 http.c:611              <= Recv header, 0000000027 bytes (0x0000001b)
04:35:05.869234 http.c:623              <= Recv header: Content-Security-Policy:
04:35:05.869236 http.c:611              <= Recv header, 0000000021 bytes (0x00000015)
04:35:05.869238 http.c:623              <= Recv header: X-Runtime: 0.041060
04:35:05.869239 http.c:611              <= Recv header, 0000000088 bytes (0x00000058)
04:35:05.869242 http.c:623              <= Recv header: X-Gitlab-Meta: {"correlation_id":"28009483-632a-4fbf-8962-e182402759a1","version":"1"}
04:35:05.869245 http.c:611              <= Recv header, 0000000052 bytes (0x00000034)
04:35:05.869249 http.c:623              <= Recv header: X-Request-Id: 28009483-632a-4fbf-8962-e182402759a1
04:35:05.869257 http.c:611              <= Recv header, 0000000002 bytes (0x00000002)
04:35:05.869259 http.c:623              <= Recv header:
04:35:05.869270 http.c:664              == Info: Connection #0 to host git.dodgeguild.ru left intact
remote: Nil JSON web token
fatal: unable to access 'https://git.dodgeguild.ru/dodge/myclassroom.git/': The requested URL returned error: 403

Configuration

nginx config:

server {
    listen 80;
    server_name git.dodgeguild.ru;

    location /.well-known/acme-challenge/ {
        root <the crt path is good>;
    }

    location / {
        return 301 https://$host$request_uri;
    }

    location ~* \.(css|js|svg|eot|woff|woff2|ttf|otf|ico|png|jpg|jpeg)$ {
        root /opt/gitlab/embedded/service/gitlab-rails/public;
        expires max;
        add_header Cache-Control public;
    }
}

server {
    listen 443 ssl;
    server_name git.dodgeguild.ru;
    root /opt/gitlab/embedded/service/gitlab-rails/public;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    ssl_certificate     <the crts path is good>
    ssl_certificate_key <the crts path is good>

    location / {
        proxy_pass http://127.0.0.1:8181;
        client_max_body_size 0;
        gzip off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Ssl on;
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_redirect off;
    }

    location ~* \.(css|js|svg|eot|woff|woff2|ttf|otf|ico|png|jpg|jpeg)$ {
        root /opt/gitlab/embedded/service/gitlab-rails/public;
        expires max;
        add_header Cache-Control public;
    }
}

gitlab.rb:

external_url 'https://git.dodgeguild.ru'
nginx['enable'] = false
puma['enable'] = true
puma['listen'] = '127.0.0.1'
puma['port'] = 8181

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "127.0.0.1"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = "dodgeguild.ru"
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['gitlab_email_from'] = 'gitlab@dodgeguild.ru'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@dodgeguild.ru'

I tried setting up remote connection in runner through ssh. However it won’t work and didn’t even try using it.
pipeline (idk maybe it will help)

stages:
  - build
  - deploy

variables:
  FLUTTER_VERSION: "3.19.0"
  DEPLOY_PATH: "/var/www/myclassroom.ru"

before_script:
  - echo "Using Flutter version $FLUTTER_VERSION"
  - git remote set-url origin git@git.dodgeguild.ru:dodge/myclassroom.git

build_flutter_web:
  stage: build
  image: cirrusci/flutter:latest
  script:
    - flutter pub get
    - flutter build web --release
  artifacts:
    paths:
      - build/web
    expire_in: 1 hour
  only:
    - develop

deploy_to_server:
  stage: deploy
  script:
    - sudo mkdir -p "$DEPLOY_PATH"
    - sudo cp -r build/web/* "$DEPLOY_PATH/"
    - sudo systemctl reload nginx
  only:
    - develop
  dependencies:
    - build_flutter_web
  tags:
    - self-hosted

I tried modifying runner config:

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "bellaciao-dev"
  url = "https://git.dodgeguild.ru"
  id = 6
  token = <all good here>
  token_obtained_at = 2025-04-12T16:40:51Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  clone_url = "git@git.dodgeguild.ru:dodge/myclassroom.git"
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.ssh]
    user = "git"
    identity_file = <all good here>
  [runners.git]
    clone_url = "git@git.dodgeguild.ru:dodge/myclassroom.git"

Versions

Self-managed
GitLab.com SaaS
Dedicated

Versions

GitLab 17.10

What I need

Help setting up this https monster, either help setting up runner connection through ssh

Helpful resources

I saw some people facing the same issue and they using nginx proxy. But seems the issue was resolved with magic

peachyhead · April 17, 2025, 2:08pm

Update: switching to bundled nginx resolves the issue and https cloning runs well
However this is not a resolve for me since i need working nginx on this server as well

Topic		Replies	Views
Can not use ssh and https to clone/push on my Gitlab 10.0.3 How to Use GitLab	0	3534	October 31, 2017
Gitlab runner can no longer clone How to Use GitLab	0	1251	April 5, 2020
Git push => fatal: unable to access 'http://gitlab-ci-token.....': The requested URL returned error: 400 GitLab CI/CD runner	3	13400	November 25, 2022
Gitlab runner connection refused, unable to access GitLab CI/CD	12	39985	May 13, 2019
Clone button: only HTTP no HTTPS How to Use GitLab	3	5093	October 5, 2020

"remote: Nil JSON web token" While cloning gitlab repo

Problem to solve

Steps to reproduce

Configuration

Versions

What I need

Helpful resources

Related topics