Specific Gitlab-Runner fails with 403 Forbidden during deploy stage

Specific Gitlab-Runner fails with 403 Forbidden

What I’m trying to do

  • I have my own server on Linode that has gitlab-runner installed (latest version). This then is setup as a specific runner in my project and is used for various tasks where it builds and deployed applications. The server is whitelisted to deploy and up until early December was working perfectly. Love the setup!
  • My application has a Build and Deploy stage, with both stages happening on the gitlab-runner Linode server. The application is a Laravel app and gets deployed to EBS.
  • The applications are all hosted on Gitlab.com and I’m on the free tier at the moment. So Gitlab is Gitlab.com, but the gitlab-runner is self-hosted.

My gitlab-ci.yml

stages:
  - build
  - deploy

cache:
  key: $CI_COMMIT_REF_SLUG
  # paths:
  #   - vendor/

######################
##   BUILD STAGE    ##
######################

Build:
  stage: build
  script:
    # - echo "Running composer"
    # - composer install
    - zip -r build.zip . --exclude '.git/*' .gitignore '.elasticbeanstalk/*'
  artifacts:
    paths:
      - build.zip
    # expire_in: 1 day

# ######################
# ##   DEPLOY STAGE   ##
# ######################

Deploy to Test:
  stage: deploy
  script:
    - aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
    - aws configure set region eu-central-1
    - aws s3 cp build.zip s3://$S3_BUCKET/test/Client-API-test-$CI_PIPELINE_ID.zip
    - aws elasticbeanstalk create-application-version --application-name Client-API --version-label Client-API-test-$CI_PIPELINE_ID --source-bundle S3Bucket=$S3_BUCKET,S3Key=test/Client-API-test-$CI_PIPELINE_ID.zip
    - aws elasticbeanstalk update-environment --application-name Client-API --environment-name client-test --version-label Client-API-test-$CI_PIPELINE_ID
  dependencies:
    - Build
  environment:
    name: testing
    url: https:/api.client-test.com
  only:
    - develop

Gitlab-runner version

Version:      14.6.0
Git revision: 5316d4ac
Git branch:   14-6-stable
GO version:   go1.13.8
Built:        2021-12-17T17:36:04+0000
OS/Arch:      linux/amd64

What is going wrong?
Early December, all of a sudden I started getting a 403 Forbidden error at the deploy stage. It started with a view deploys giving me this error, then soon all deploys started failing with this error.

eRunning with gitlab-runner 13.8.0 (775dd39d)
  on donor-dashboard H2ib2tWD
Preparing the "shell" executore
Using Shell executor...
Preparing environmente
Running on localhost...
Getting source from Git repositorye
Fetching changes with git depth set to 50...e
Initialized empty Git repository in /home/gitlab-runner/builds/H2ib2tWD/0/company/client-api-laraval/.git/
Created fresh repository.e
Checking out 7ec427c5 as develop...e
Skipping Git submodules setupe
Downloading artifactse
Downloading artifacts for Build (1912251578)...e
Runtime platform      earche=amd64 ose=linux pide=18313 revisione=775dd39d versione=13.8.0
eERROR: Downloading artifacts from coordinator... forbiddene  eide=1912251578 responseStatuse=403 Forbidden statuse=403 Forbidden tokene=Uqi4oun9
eFATAL: permission denied                          e
Cleaning up file based variablese
ERROR: Job failed: exit status 1

What did I find out so far?
The problem seems to be that gitlab-runner during the Deploy to Test stage is trying to download the latest artifact from the Build stage and gets a 403 Forbidden from Gitlab.com. Pretty much everything the logs tells us.

  • Debug
    • Running the runner in debug mode unfortunately isn’t giving me more information, just the 403 response
  • Debug
    • Running gitlab-runner artifact-download gives me the exact same result
  • Theory “My IP is blocked.”
    • The runner is successfully uploading the artifact (which is a POST request I believe) and downloading the repo from gitlab.com, so it seems not to be blocked there.
  • Theory "You need Premium to use CI_JOB_TOKEN"
    • Maybe, but it was working without a problem before December for over a year. So unless someone confirms that this is it, I’ll wait on trying this (also because I would have to set up a “group” first etc. etc.)
  • Theory “Have you tried turning it on and off?”
    • Yes, I upgraded gitlab-runner to the latest version to see if that solved it. Remove old gitlab runner registers and added them new. Rebooted the machine. Uninstalled gitlab-runner completely and reinstalled everything.

THANKS!
I do appreciate if anyone has had this happen to them and would help me to push me in the right direction. For now I’ve written my own .sh deploy script that can run locally, but I do want to fix this for all my projects so that I don’t rely on local machines to do deploys.

Thanks everyone!