Gitlab-runner on macos - 429 Too Many Requests

kamilko · April 25, 2019, 6:16pm

I have a problem with finding why there is error 429 “Too many requests” from gitlab-runner running on Mojave while trying to run simple job.
So I have question, what could I do to dive deeper into debug this?
Maybe somebody had similar issue and solved in some way?

Basic information:

I have tested everything on

Giltab 11.10.1 and 11.9.9
gitlab-ce is on Debian 9.8 (4 CPU 8GB RAM NET 300/300)
gitlab-runner 11.10.1 and 11.9.3~beta.1.g5d4271af (installation from homebrew or direct)

The setup with ElCapitan on MBP and older versions of Gitlab (something around version 11.5) was working without problem.
The same gitlab-ce instance with runners on other debian/ubuntu/armbian working without problem.
For now in .gitlab-ci.yaml is very simple: script: pwd && ls -la
I cleared out build directory and I know that git clone is working, but first stage is pending indefinitely.

Debug log from runner:

$ gitlab-runner --debug --log-format text -l debug run
INFO[0000] Runtime platform                              arch=amd64 os=darwin pid=7095 revision=1f513601 version=11.10.1
INFO[0000] Starting multi-runner from /Volumes/users/Users/builduser/.gitlab-runner/config.toml ...  builds=0
DEBU[0000] Checking runtime mode                         GOOS=darwin uid=501
WARN[0000] Running in user-mode.
WARN[0000] Use sudo for system-mode:
WARN[0000] $ sudo gitlab-runner...
INFO[0000]
INFO[0000] Configuration loaded                          builds=0
DEBU[0000] listenaddress: ""
sessionserver:
  listenaddress: ""
  advertiseaddress: ""
  sessiontimeout: 1800
metricsserveraddress: ""
concurrent: 1
checkinterval: 10
loglevel: null
logformat: null
user: ""
runners:
- name: shell@buildmacos
  limit: 0
  outputlimit: 0
  requestconcurrency: 0
  runnercredentials:
    url: https://gitlab.example.com
    token: 68a495878bf9174be76c0818510a0c
    tlscafile: ""
    tlscertfile: ""
    tlskeyfile: ""
  runnersettings:
    executor: shell
    buildsdir: /Users/builduser/tmp/volumes/.builds
    cachedir: /Users/builduser/tmp/volumes/.cache
    cloneurl: ""
    environment: []
    preclonescript: ""
    prebuildscript: ""
    postbuildscript: ""
    debugtracedisabled: false
    shell: ""
    custombuilddir: null
    ssh: null
    docker: null
    parallels: null
    virtualbox: null
    cache:
      type: ""
      path: ""
      shared: false
      s3: null
      gcs: null
      s3cachepath: ""
      cacheshared: false
      serveraddress: ""
      accesskey: ""
      secretkey: ""
      bucketname: ""
      bucketlocation: ""
      insecure: false
    machine: null
    kubernetes: null
sentrydsn: null
modtime: 2019-04-25T16:00:45.293941004+02:00
loaded: true  builds=0
DEBU[0000] Waiting for stop signal                       builds=0
INFO[0000] listen_address not defined, metrics & debug endpoints disabled  builds=0
INFO[0000] [session_server].listen_address not defined, session endpoints disabled  builds=0
DEBU[0000] Feeding runners to channel                    builds=0
DEBU[0000] Starting worker                               builds=0 worker=0
DEBU[0000] Dialing: tcp gitlab.example.com:443 ...
DEBU[0000] Checking for jobs... nothing                  runner=68f49587
DEBU[0010] Feeding runners to channel                    builds=0
DEBU[0010] Checking for jobs... nothing                  runner=68f49587
DEBU[0020] Feeding runners to channel                    builds=0
INFO[0020] Checking for jobs... received                 job=553 repo_url="https://gitlab.example.com/games/cmdgame.git" runner=68f49587
DEBU[0020] Failed to requeue the runner:                 builds=1 runner=68f49587
DEBU[0020] Running with gitlab-runner 11.10.1 (1f513601)  job=553 project=23 runner=68f49587
DEBU[0020]   on shell@buildmacos 68f49587                 job=553 project=23 runner=68f49587
DEBU[0020] Shell configuration: environment: []
dockercommand:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash --login\nelif [
  -x /usr/bin/bash ]; then\n\texec /usr/bin/bash --login\nelif [ -x /bin/bash ]; then\n\texec
  /bin/bash --login\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh
  --login\nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh --login\nelif [ -x /bin/sh
  ]; then\n\texec /bin/sh --login\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh
  --login\nelse\n\techo shell not found\n\texit 1\nfi\n\n"
command: bash
arguments:
- --login
passfile: false
extension: ""  job=553 project=23 runner=68f49587
DEBU[0020] Using Shell executor...                       job=553 project=23 runner=68f49587
DEBU[0020] Waiting for signals...                        job=553 project=23 runner=68f49587
DEBU[0020] Executing build stage                         build_stage=prepare_script job=553 project=23 runner=68f49587
DEBU[0020] Executing build stage                         build_stage=get_sources job=553 project=23 runner=68f49587
DEBU[0022] Executing build stage                         build_stage=restore_cache job=553 project=23 runner=68f49587
DEBU[0022] Executing build stage                         build_stage=download_artifacts job=553 project=23 runner=68f49587
DEBU[0022] Executing build stage                         build_stage=build_script job=553 project=23 runner=68f49587
DEBU[0022] Executing build stage                         build_stage=after_script job=553 project=23 runner=68f49587
DEBU[0022] Executing build stage                         build_stage=archive_cache job=553 project=23 runner=68f49587
DEBU[0022] Executing build stage                         build_stage=upload_artifacts_on_success job=553 project=23 runner=68f49587
INFO[0022] Job succeeded                                 duration=2.355494s job=553 project=23 runner=68f49587
WARN[0022] Appending trace to coordinator... failed      code=502 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="502 Bad Gateway"
WARN[0022] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0026] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
WARN[0026] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0029] Appending trace to coordinator... failed      code=502 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="502 Bad Gateway"
WARN[0029] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
DEBU[0030] Feeding runners to channel                    builds=1
WARN[0033] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
WARN[0033] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0037] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
WARN[0038] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0042] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
WARN[0045] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0051] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
WARN[0054] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0067] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
WARN[0069] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
WARN[0091] Appending trace to coordinator... failed      code=429 job=553 job-log= job-status= runner=68f49587 sent-log=0-2110 status="429 Too Many Requests"
^CWARN[0103] Requested service stop: interrupt             builds=1
WARN[0107] Submitting job to coordinator... failed       code=429 job=553 job-status= runner=68f49587 status="429 Too Many Requests"
FATA[0133] shutdown timed out

kamilko · April 26, 2019, 6:01pm

[Solved]

For some reason, my mac’s gitlab-runner is treated as unauthenticated request.
I did two things:

I’ve changed settings in Administration->Settings->Network->User and IP Rate Limits and uncheck Enable unauthenticated request rate limit (as I’m filtering all the crawlers/bots on my firewall).
I’ve add my IP’s for rake_attack into config in /etc/gitlab/gitlab.rb like this:

gitlab_rails['rack_attack_git_basic_auth'] = {
  'ip_whitelist' => [
        "127.0.0.1",
        "<my other IPs>"
    ],
}

I don’t know if I missing something and only one thing should do the job, but for me, only when both are setup then no more 429 errors in gitlab-runner logs.

Topic		Replies	Views
Error 429 - Too many requests received Self-managed api , ci , gitlab-pages	7	6765	September 7, 2022
Gitlab-runner Windows "409 Conflict" Error GitLab CI/CD	3	3909	May 21, 2020
How to view self-hosted runner logs in macOS? GitLab CI/CD runner , macos , docker	6	5195	August 7, 2023
Error 500 on job submission GitLab CI/CD	3	2496	August 25, 2017
Gitlab runner online but not picking up jobs, when debug : 503 Service Temporarily Unavailable GitLab CI/CD ci , runner , docker , pipelines	2	2946	September 20, 2022

Gitlab-runner on macos - 429 Too Many Requests

Related Topics