Hi,
Since 2 days we (as a company) are encountering the error Too many requests received (429) when performing any API call during CI jobs execution, but also while simply going to gitlab.com page or any other sections. This problem is quite blocking for us.
Anything has changed in rate limits? How can we solve this?
Thank you
We had the same problems last 5 days. Can someone explain how we can find out what limit periodically blocks us from accessing to gitlab.com?
This link might help: GitLab.com settings | GitLab
Pipeline API request are 25 per minute, which would be related to CI.
But i received Too many requests received (429) when only 11 jobs has been active.
Maybe your pipelines made too many requests. You will have to check it. One pipeline doesn’t necessarily mean 1 API request. A single pipeline could make lots of API requests depending on what it is doing. So you need to check and provide far more information and explain your entire process and what exactly you are doing. As for now, all you mentioned is you had the same problem, but without any specific details that can help someone diagnose your problem.
Thank you for your help. I can show you my gitlab-ci.yml here or in private message if it helps you.
Post on the forum. By DM nobody else can see it which means less people being able to help. I don’t use CI as such, but once you explain step-by-step what you are doing, what you are using, and also the showing the CI file, could help people see where the issue is and why you might be hitting limits.
variables: &pipeline_variables
FF_USE_FASTZIP: "true"
FF_SCRIPT_SECTIONS: "true"
NODE_OPTIONS: "--max-old-space-size=8192"
COMPOSE_FILE: "docker-compose.yml:docker-compose.ci.yml"
REPOSITORY_NAME: ${CI_PROJECT_NAME}
SOURCE_BRANCH_NAME: ${CI_COMMIT_BRANCH}
DOCKER_BUILDKIT: 1
COMPOSE_DOCKER_CLI_BUILD: 1
DOCKER_CLIENT_TIMEOUT: "300"
COMPOSE_HTTP_TIMEOUT: "300"
COMPOSE_PROJECT_NAME: ${CI_JOB_ID}
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_PROJECT_NAME/$CI_JOB_ID
stages:
- test
.cypress_variables:
variables: &cypress_variables
CYPRESS_API_URL: "http://cypress.co:1234"
CYPRESS_MONGO_URI: "mongodb://cypress.co:27017"
# -------------------------------------------------
# workflow
# -------------------------------------------------
before_script:
- mkdir -p $HOME/.docker && echo $DOCKER_AUTH_CONFIG > $HOME/.docker/config.json
# -------------------------------------------------
# extends
# -------------------------------------------------
.default_rules_mr: &default_rules_mr
if: $CI_PIPELINE_SOURCE == 'merge_request_event' && ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'master')
.default_rules_push_master: &default_rules_push_master
if: $CI_PIPELINE_SOURCE == 'push' && ($CI_COMMIT_BRANCH == 'master')
.stage_test: &stage_test
variables:
<<: *pipeline_variables
stage: test
interruptible: true
tags:
- docker
retry:
max: 2
when: runner_system_failure
.stage_test_highload: &stage_test_highload
variables:
<<: *pipeline_variables
stage: test
interruptible: true
tags:
- high_load_job
retry:
max: 2
when: runner_system_failure
.changed_back: &changed_back
rules:
- <<: *default_rules_mr
changes:
- Dockerfile
- composer.json
- composer.lock
- "**/*.php"
- "**/*.sql"
- "*.xml"
- "*.yml"
- "**/*.sh"
when: on_success
- <<: *default_rules_push_master
when: on_success
- when: never
.changed_front: &changed_front
rules:
- <<: *default_rules_mr
changes:
- package.json
- yarn.lock
- "**/*.js"
- "**/*.ts"
- "**/*.html"
- "**/*.css"
- "**/*.scss"
- "*.yml"
- "**/*.json"
- "**/*.sh"
when: on_success
- <<: *default_rules_push_master
when: on_success
- when: never
.changed_e2e: &changed_e2e
rules:
- <<: *default_rules_mr
changes:
- package.json
- yarn.lock
- "tests/Protractor/**/*.js"
- "tests/Cypress/**/*.ts"
- "tests/Cypress/**/*.json"
- "tests/Wdio/**/*.ts"
- "tests/Wdio/**/*.json"
when: on_success
- when: never
.changed_all: &changed_all
rules:
- <<: *default_rules_mr
changes:
- Dockerfile
- composer.json
- composer.lock
- package.json
- yarn.lock
- "**/*.php"
- "**/*.sql"
- "*.xml"
- "*.yml"
- "**/*.sh"
- "**/*.js"
- "**/*.ts"
- "**/*.html"
- "**/*.css"
- "**/*.scss"
- "**/*.json"
when: on_success
- when: never
.cache-composer:
cache: &cache-composer
key:
files:
- composer.lock
prefix: composer-php81-docker
paths:
- vendor
policy: pull-push
.cache-yarn:
cache: &cache-yarn
key:
files:
- yarn.lock
- resources/assets/angular/yarn.lock
prefix: yarn-node14-hybrid-docker
paths:
- node_modules
- resources/assets/angular/node_modules
policy: pull-push
.cache-cypress:
cache: &cache-cypress
key:
files:
- tests/Cypress/yarn.lock
prefix: cypress-node14-docker
paths:
- tests/Cypress/.cache/Cypress
- tests/Cypress/node_modules
policy: pull-push
.cache-wdio:
cache: &cache-wdio
key:
files:
- tests/Wdio/yarn.lock
prefix: wdio-node14-docker
paths:
- tests/Wdio/node_modules
policy: pull-push
# -------------------------------------------------
# test
# -------------------------------------------------
backend_test:
<<: *stage_test
extends:
- .changed_back
cache:
- *cache-composer
- key: coverage-backend
paths:
- .coverage-backend-cache
policy: pull-push
artifacts:
name: backend_test
expire_in: "1 day"
paths:
- storage/logs/
- storage/coverage/backend/
- storage/app/public/api-docs/
- storage/framework/views/
reports:
junit:
- storage/coverage/backend/junit.xml
when: always
script:
- docker-compose exec -T app /app/sh/pipeline/backend_test.sh
backend_code_style:
<<: *stage_test
extends:
- .changed_back
cache:
- *cache-composer
- key: php_cs_fixer_staging
paths:
- .php_cs.cache
policy: pull-push
script:
- sh/pipeline/backend_code_style.sh
backend_larastan:
<<: *stage_test
extends:
- .changed_back
cache:
- *cache-composer
- key: "larastan_cache"
paths:
- storage/tmp/phpstan/current
policy: pull-push
artifacts:
when: always
reports:
codequality: storage/tmp/phpstan/current/report.json
script:
- sh/pipeline/backend_larastan.sh
backend_larastan_highest:
<<: *stage_test
extends:
- .changed_back
cache:
- *cache-composer
- key: "larastan_highest_cache"
paths:
- storage/tmp/phpstan/highest
policy: pull-push
script:
- sh/pipeline/backend_larastan_highest.sh
code_quality:
<<: *stage_test
extends:
- .changed_back
cache:
- *cache-composer
- key: "code_quality_cache"
paths:
- storage/tmp/code_quality
policy: pull-push
allow_failure:
exit_codes:
- 1
artifacts:
when: always
reports:
codequality: storage/tmp/code_quality/report.json
script:
- sh/pipeline/code_quality.sh
frontend_build:
<<: *stage_test
extends:
- .cache-yarn
- .changed_all
artifacts:
name: frontend_build_$CI_PIPELINE_ID
expire_in: "3 day"
paths:
- resources/web
when: on_success
script:
- yarn ci
- yarn production
frontend_test:
<<: *stage_test
extends:
- .cache-yarn
- .changed_front
variables:
CHROME_BIN: /usr/bin/chromium-browser
artifacts:
name: "frontend"
expire_in: "1 day"
paths:
- storage/coverage/ts
- storage/coverage/angular2
reports:
junit:
- storage/tmp/karma/junit.xml
when: always
script:
- sh/pipeline/frontend_test.sh
frontend_code_style:
<<: *stage_test
extends:
- .cache-yarn
- .changed_front
script:
- sh/pipeline/frontend_code_style.sh
.e2e_cypress_general: &e2e_cypress_general
artifacts:
name: "e2e_cypress"
expire_in: "1 day"
paths:
- storage/logs
reports:
junit:
- tests/Cypress/results/junit.xml
when: always
timeout: "3 hours"
.cypress_prepare_env: &cypress_prepare_env
- env | grep '^DOCKER_APP_NAME\|^COMPOSE_FILE\|^HOST=\|^DEBUG=\|^BROWSER' > .env
e2e_cypress:
<<: *stage_test_highload
variables:
<<: *pipeline_variables
<<: *cypress_variables
CYPRESS_ENV: "configFile=local"
CYPRESS_ENV_MOBILE: "configFile=local,isMobile=true,grepTags=@mobile"
extends:
- .changed_all
- .e2e_cypress_general
needs:
- job: frontend_build
cache:
- *cache-composer
- *cache-yarn
- *cache-cypress
script:
- *cypress_prepare_env
- docker-compose up -d
- docker-compose exec -T cypress /app/sh/pipeline/e2e_cypress.sh
cypress_code_style:
<<: *stage_test
extends:
- .changed_e2e
cache:
- *cache-cypress
script:
- docker-compose exec -T cypress /app/sh/pipeline/e2e_code_style.sh
.e2e_wdio_general: &e2e_wdio_general
artifacts:
name: 'e2e_mobile_reports'
expire_in: '1 day'
paths:
- storage/tmp/e2e_mobile_report
reports:
junit:
- tests/Wdio/results/junit.xml
when: always
timeout: "3 hours"
build_mobile_app:
stage: test
extends:
- .changed_all
needs:
- job: frontend_build
artifacts: true
variables:
MULTI_PIPELINE: "true"
trigger:
project: mobile-app
branch: $MOBILE_MULTIPIPELINE_BRAHCN
strategy: depend
resource_group: build_mobile_app
e2e_wdio:
<<: *stage_test
tags:
- wdio
extends:
- .changed_all
- .e2e_wdio_general
needs:
- job: build_mobile_app
artifacts: true
- job: frontend_build
artifacts: true
cache:
- *cache-composer
- *cache-yarn
- *cache-wdio
script:
- docker-compose up -d
- docker-compose exec -T e2e sh/pipeline/e2e_wdio.sh
In build_mobile_app job we trigger multi pipeline where we get frontend_build job ID and download artifacts. But build_mobile_app has resource_group and don’t run in parallel. So, we don’t invoke Gitlab API to get job ID or download artifacts in parallel pipelines.
if [[ "$MULTI_PIPELINE" == "true" ]]; then
FRONTEND_BUILD_JOB_ID=$(curl --silent \
--header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$PIPELINE_ID/jobs" \
| jq -r '.[]' \
| jq -r 'select(.name | endswith("frontend_build"))' \
| jq -r '.id')
curl -L -o artifacts.zip \
-H "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$FRONTEND_BUILD_JOB_ID/artifacts"
We has been used this config last 6 month and din’t have problems with 429 errors. Only last 5 days, like author this thread. Is there any real way to use server resources to find out which of these limits we’re breaking, which is causing the lockout, so we can fix our problem?
