Error 429 - Too many requests received

Hi,

Since 2 days we (as a company) are encountering the error Too many requests received (429) when performing any API call during CI jobs execution, but also while simply going to gitlab.com page or any other sections. This problem is quite blocking for us.
Anything has changed in rate limits? How can we solve this?

Thank you

2 Likes

We had the same problems last 5 days. Can someone explain how we can find out what limit periodically blocks us from accessing to gitlab.com?

1 Like

This link might help: GitLab.com settings | GitLab

Pipeline API request are 25 per minute, which would be related to CI.

But i received Too many requests received (429) when only 11 jobs has been active.

1 Like

Maybe your pipelines made too many requests. You will have to check it. One pipeline doesn’t necessarily mean 1 API request. A single pipeline could make lots of API requests depending on what it is doing. So you need to check and provide far more information and explain your entire process and what exactly you are doing. As for now, all you mentioned is you had the same problem, but without any specific details that can help someone diagnose your problem.

Thank you for your help. I can show you my gitlab-ci.yml here or in private message if it helps you.

Post on the forum. By DM nobody else can see it which means less people being able to help. I don’t use CI as such, but once you explain step-by-step what you are doing, what you are using, and also the showing the CI file, could help people see where the issue is and why you might be hitting limits.


variables: &pipeline_variables
  FF_USE_FASTZIP: "true"
  FF_SCRIPT_SECTIONS: "true"
  NODE_OPTIONS: "--max-old-space-size=8192"
  COMPOSE_FILE: "docker-compose.yml:docker-compose.ci.yml"
  REPOSITORY_NAME: ${CI_PROJECT_NAME}
  SOURCE_BRANCH_NAME: ${CI_COMMIT_BRANCH}
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1
  DOCKER_CLIENT_TIMEOUT: "300"
  COMPOSE_HTTP_TIMEOUT: "300"
  COMPOSE_PROJECT_NAME: ${CI_JOB_ID}
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_PROJECT_NAME/$CI_JOB_ID

stages:
  - test

.cypress_variables:
  variables: &cypress_variables
    CYPRESS_API_URL: "http://cypress.co:1234"
    CYPRESS_MONGO_URI: "mongodb://cypress.co:27017"

# -------------------------------------------------
# workflow
# -------------------------------------------------

before_script:
  - mkdir -p $HOME/.docker && echo $DOCKER_AUTH_CONFIG > $HOME/.docker/config.json

# -------------------------------------------------
# extends
# -------------------------------------------------
.default_rules_mr: &default_rules_mr
  if: $CI_PIPELINE_SOURCE == 'merge_request_event' && ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'master')

.default_rules_push_master: &default_rules_push_master
  if: $CI_PIPELINE_SOURCE == 'push' && ($CI_COMMIT_BRANCH == 'master')

.stage_test: &stage_test
  variables:
    <<: *pipeline_variables
  stage: test
  interruptible: true
  tags:
    - docker
  retry:
    max: 2
    when: runner_system_failure

.stage_test_highload: &stage_test_highload
  variables:
    <<: *pipeline_variables
  stage: test
  interruptible: true
  tags:
    - high_load_job
  retry:
    max: 2
    when: runner_system_failure

.changed_back: &changed_back
  rules:
    - <<: *default_rules_mr
      changes:
        - Dockerfile
        - composer.json
        - composer.lock
        - "**/*.php"
        - "**/*.sql"
        - "*.xml"
        - "*.yml"
        - "**/*.sh"
      when: on_success
    - <<: *default_rules_push_master
      when: on_success
    - when: never

.changed_front: &changed_front
  rules:
    - <<: *default_rules_mr
      changes:
        - package.json
        - yarn.lock
        - "**/*.js"
        - "**/*.ts"
        - "**/*.html"
        - "**/*.css"
        - "**/*.scss"
        - "*.yml"
        - "**/*.json"
        - "**/*.sh"
      when: on_success
    - <<: *default_rules_push_master
      when: on_success
    - when: never

.changed_e2e: &changed_e2e
  rules:
    - <<: *default_rules_mr
      changes:
        - package.json
        - yarn.lock
        - "tests/Protractor/**/*.js"
        - "tests/Cypress/**/*.ts"
        - "tests/Cypress/**/*.json"
        - "tests/Wdio/**/*.ts"
        - "tests/Wdio/**/*.json"
      when: on_success
    - when: never

.changed_all: &changed_all
  rules:
    - <<: *default_rules_mr
      changes:
        - Dockerfile
        - composer.json
        - composer.lock
        - package.json
        - yarn.lock
        - "**/*.php"
        - "**/*.sql"
        - "*.xml"
        - "*.yml"
        - "**/*.sh"
        - "**/*.js"
        - "**/*.ts"
        - "**/*.html"
        - "**/*.css"
        - "**/*.scss"
        - "**/*.json"
      when: on_success
    - when: never

.cache-composer:
  cache: &cache-composer
    key:
      files:
        - composer.lock
      prefix: composer-php81-docker
    paths:
      - vendor
    policy: pull-push

.cache-yarn:
  cache: &cache-yarn
    key:
      files:
        - yarn.lock
        - resources/assets/angular/yarn.lock
      prefix: yarn-node14-hybrid-docker
    paths:
      - node_modules
      - resources/assets/angular/node_modules
    policy: pull-push

.cache-cypress:
  cache: &cache-cypress
    key:
      files:
        - tests/Cypress/yarn.lock
      prefix: cypress-node14-docker
    paths:
      - tests/Cypress/.cache/Cypress
      - tests/Cypress/node_modules
    policy: pull-push

.cache-wdio:
  cache: &cache-wdio
    key:
      files:
        - tests/Wdio/yarn.lock
      prefix: wdio-node14-docker
    paths:
      - tests/Wdio/node_modules
    policy: pull-push

# -------------------------------------------------
# test
# -------------------------------------------------
backend_test:
  <<: *stage_test
  extends:
    - .changed_back
  cache:
    - *cache-composer
    - key: coverage-backend
      paths:
        - .coverage-backend-cache
      policy: pull-push
  artifacts:
    name: backend_test
    expire_in: "1 day"
    paths:
      - storage/logs/
      - storage/coverage/backend/
      - storage/app/public/api-docs/
      - storage/framework/views/
    reports:
      junit:
        - storage/coverage/backend/junit.xml
    when: always
  script:
    - docker-compose exec -T app /app/sh/pipeline/backend_test.sh

backend_code_style:
  <<: *stage_test
  extends:
    - .changed_back
  cache:
    - *cache-composer
    - key: php_cs_fixer_staging
      paths:
        - .php_cs.cache
      policy: pull-push
  script:
    - sh/pipeline/backend_code_style.sh

backend_larastan:
  <<: *stage_test
  extends:
    - .changed_back
  cache:
    - *cache-composer
    - key: "larastan_cache"
      paths:
        - storage/tmp/phpstan/current
      policy: pull-push
  artifacts:
    when: always
    reports:
      codequality: storage/tmp/phpstan/current/report.json
  script:
    - sh/pipeline/backend_larastan.sh

backend_larastan_highest:
  <<: *stage_test
  extends:
    - .changed_back
  cache:
    - *cache-composer
    - key: "larastan_highest_cache"
      paths:
        - storage/tmp/phpstan/highest
      policy: pull-push
  script:
    - sh/pipeline/backend_larastan_highest.sh

code_quality:
  <<: *stage_test
  extends:
    - .changed_back
  cache:
    - *cache-composer
    - key: "code_quality_cache"
      paths:
        - storage/tmp/code_quality
      policy: pull-push
  allow_failure:
    exit_codes:
      - 1
  artifacts:
    when: always
    reports:
      codequality: storage/tmp/code_quality/report.json
  script:
    - sh/pipeline/code_quality.sh

frontend_build:
  <<: *stage_test
  extends:
    - .cache-yarn
    - .changed_all
  artifacts:
    name: frontend_build_$CI_PIPELINE_ID
    expire_in: "3 day"
    paths:
      - resources/web
    when: on_success
  script:
    - yarn ci
    - yarn production

frontend_test:
  <<: *stage_test
  extends:
    - .cache-yarn
    - .changed_front
  variables:
    CHROME_BIN: /usr/bin/chromium-browser
  artifacts:
    name: "frontend"
    expire_in: "1 day"
    paths:
      - storage/coverage/ts
      - storage/coverage/angular2
    reports:
      junit:
        - storage/tmp/karma/junit.xml
    when: always
  script:
    - sh/pipeline/frontend_test.sh

frontend_code_style:
  <<: *stage_test
  extends:
    - .cache-yarn
    - .changed_front
  script:
    - sh/pipeline/frontend_code_style.sh

.e2e_cypress_general: &e2e_cypress_general
  artifacts:
    name: "e2e_cypress"
    expire_in: "1 day"
    paths:
      - storage/logs
    reports:
      junit:
        - tests/Cypress/results/junit.xml
    when: always
  timeout: "3 hours"

.cypress_prepare_env: &cypress_prepare_env
  - env | grep '^DOCKER_APP_NAME\|^COMPOSE_FILE\|^HOST=\|^DEBUG=\|^BROWSER' > .env

e2e_cypress:
  <<: *stage_test_highload
  variables:
    <<: *pipeline_variables
    <<: *cypress_variables
    CYPRESS_ENV: "configFile=local"
    CYPRESS_ENV_MOBILE: "configFile=local,isMobile=true,grepTags=@mobile"
  extends:
    - .changed_all
    - .e2e_cypress_general
  needs:
    - job: frontend_build
  cache:
    - *cache-composer
    - *cache-yarn
    - *cache-cypress
  script:
    - *cypress_prepare_env
    - docker-compose up -d
    - docker-compose exec -T cypress /app/sh/pipeline/e2e_cypress.sh

cypress_code_style:
  <<: *stage_test
  extends:
    - .changed_e2e
  cache:
    - *cache-cypress
  script:
    - docker-compose exec -T cypress /app/sh/pipeline/e2e_code_style.sh

.e2e_wdio_general: &e2e_wdio_general
  artifacts:
    name: 'e2e_mobile_reports'
    expire_in: '1 day'
    paths:
     - storage/tmp/e2e_mobile_report
    reports:
      junit:
        - tests/Wdio/results/junit.xml
    when: always
  timeout: "3 hours"

build_mobile_app:
  stage: test
  extends:
    - .changed_all
  needs:
    - job: frontend_build
      artifacts: true
  variables:
    MULTI_PIPELINE: "true"
  trigger:
    project: mobile-app
    branch: $MOBILE_MULTIPIPELINE_BRAHCN
    strategy: depend
  resource_group: build_mobile_app

e2e_wdio:
  <<: *stage_test
  tags:
    - wdio
  extends:
    - .changed_all
    - .e2e_wdio_general
  needs:
    - job: build_mobile_app
      artifacts: true
    - job: frontend_build
      artifacts: true
  cache:
    - *cache-composer
    - *cache-yarn
    - *cache-wdio
  script:
    - docker-compose up -d
    - docker-compose exec -T e2e sh/pipeline/e2e_wdio.sh

In build_mobile_app job we trigger multi pipeline where we get frontend_build job ID and download artifacts. But build_mobile_app has resource_group and don’t run in parallel. So, we don’t invoke Gitlab API to get job ID or download artifacts in parallel pipelines.

if [[ "$MULTI_PIPELINE" == "true" ]]; then
        FRONTEND_BUILD_JOB_ID=$(curl --silent \
          --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$PIPELINE_ID/jobs" \
          | jq -r '.[]' \
          | jq -r 'select(.name | endswith("frontend_build"))' \
          | jq -r '.id')
        curl -L -o artifacts.zip \
          -H "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$FRONTEND_BUILD_JOB_ID/artifacts"

We has been used this config last 6 month and din’t have problems with 429 errors. Only last 5 days, like author this thread. Is there any real way to use server resources to find out which of these limits we’re breaking, which is causing the lockout, so we can fix our problem?