Gitlab.com CI job 15x slower since two weeks

GitLab CI/CD
, ,
1

Hello,

I have a testing job that used to take less than 15 minutes to run in CI on Gitlab.com (here’s a working example), but since around two weeks, it takes now one full hour to run less than 30% of the test suite and then it fails after reaching the one hour timeout (here’s a failing example). The test suite still runs quite fast on my local machine, so I’m puzzled why it takes so long on Gitlab.com CI.

  • Version:
    • GitLab: GitLab Enterprise Edition 15.1.0-pre 2980e0208ac
    • Runners:
      • #12270807 (j1aLDqxS) 1-blue.shared.runners-manager.gitlab.com/default ← working
      • #12270837 (J2nyww-s) 4-blue.shared.runners-manager.gitlab.com/default ← failing

Below’s my .gitlab-ci.yml config:

---
.python:
  image: python:3.9-slim
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  cache:
    paths:
      - .cache/pip
      - .venv/
  before_script:
    - apt-get update && apt-get upgrade -y &&
      apt-get install -y libhdf5-dev make pkg-config gcc git
    - python -m venv .venv/
    - source .venv/bin/activate
    - python --version
    - pip install --upgrade pip
    - pip install -e .[dev]

stages:
  - Static Analysis
  - Test
  - Install

big_files:
  stage: Static Analysis
  image: debian:stable-slim
  script:
    - >
      big_files=$(find . -size +160M -not -path "./.git/*");
      if [[ $big_files ]];
      then printf "Found some huge files 😱\n$big_files"; exit 1;
      else printf "No huge files found ✔️"; exit 0; fi;

lint:
  stage: Static Analysis
  extends: .python
  script:
    - make lint-python

prettier:
  stage: Static Analysis
  image: node:lts-slim
  before_script:
    - apt-get update && apt-get upgrade -y && apt-get install -y make
    - npm install --global prettier
  script:
    - make prettier-check

# This is the failing job
test:
  stage: Test
  extends: .python
  script:
    - calimag --help
    - make test
  coverage: "/TOTAL.+ ([0-9]{1,3}%)/"

windows:
  stage: Install
  tags:
    - windows
  before_script:
    - Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
    - choco install miniconda3 -y
    - $env:PATH = "C:\tools\miniconda3;C:\tools\miniconda3\Library\usr\bin;C:\tools\miniconda3\Scripts;C:\tools\miniconda3\bin;C:\tools\miniconda3\Library\bin" + $env:PATH
    - cp C:\tools\miniconda3\Library\bin\libcrypto-1_1-x64.dll C:\tools\miniconda3\DLLs
    - cp C:\tools\miniconda3\Library\bin\libssl-1_1-x64.dll C:\tools\miniconda3\DLLs
    - conda init powershell
  script:
    - conda env create -f environment.yml
    - cmd /c "activate calimag && calimag --help"

Thanks in advance for any help you could provide!