How to use cache in gitlab ci pipeline

Hi all,
I have Gitlab CE on premise and a php8 app.
This is intermediate image that include php8 , composer , mssql driver for php8 , I build and push this image as “username/repo:buildtool”

#Dockerfile.buildtool
FROM alpine
# install php8
RUN apk --update add some tools , php8
# install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer
##### install PHP drivers for Microsoft SQL Server
RUN curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/msodbcsql18_18.1.1.1-1_amd64.apk
RUN curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/mssql-tools18_18.1.1.1-1_amd64.apk
RUN apk add --allow-untrusted msodbcsql18_18.1.1.1-1_amd64.apk mssql-tools18_18.1.1.1-1_amd64.apk
RUN pecl install sqlsrv pdo_sqlsrv
RUN mkdir -p /app
WORKDIR /app
CMD ["/bin/sh"]
ENTRYPOINT ["/bin/sh", "-c"]

This is image of my php8 app , I build and push it as “username/repo:myapp”

#Dockerfile
FROM username/repo:buildtool AS composer
COPY . ./
RUN composer update \
  --optimize-autoloader \
  --no-interaction \
  --no-progress
# Use an image with nginx+php8 to run my app
FROM trafex/alpine-nginx-php7
USER root
RUN apk add some tools and php8 && \
	curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/msodbcsql18_18.1.1.1-1_amd64.apk && \
	curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/mssql-tools18_18.1.1.1-1_amd64.apk && \
	apk add --allow-untrusted msodbcsql18_18.1.1.1-1_amd64.apk mssql-tools18_18.1.1.1-1_amd64.apk && \
	pecl8 install sqlsrv pdo_sqlsrv && \
	echo extension=pdo_sqlsrv > /etc/php8/conf.d/10_pdo_sqlsrv.ini && \
	echo extension=sqlsrv > /etc/php8/conf.d/00_sqlsrv.ini
USER nobody
COPY --chown=nginx --from=composer /app /var/www/html

This is .gitlab-ci.yml

build_image:
  stage: build
  image: docker:20.10.9
  services:
    - docker:20.10.9-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build -t username/repo:myapp .
    - docker push username/repo:myapp

Everything works fine.
Now I want to use cache to speed up gitlab ci process (run php composer at seprated job in ci pipeline, cache “vendor” folder and reuse cache at build image job), but I’m new to gitlab ci/cd and I want to ask for your advice.
What I attempt to do:

#.gitlab-ci.yml
# this job run composer , prepare "vendor" folder and cache it for other jobs/pipelines
build_php:
  stage: build
  image: alpine
  script:
    - apk --update add wget curl git gcc g++ musl-dev make unixodbc unixodbc-dev
      php8 php8-curl php8-openssl php8-iconv php8-json php8-mbstring php8-phar php8-ldap php8-dev php8-xml
      php8-session php8-pear php8-pdo php8-tokenizer php8-xmlwriter php8-fileinfo php8-dom
    - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer
    - curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/msodbcsql18_18.1.1.1-1_amd64.apk
    - curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/mssql-tools18_18.1.1.1-1_amd64.apk
    - apk add --allow-untrusted msodbcsql18_18.1.1.1-1_amd64.apk mssql-tools18_18.1.1.1-1_amd64.apk
    - pecl install sqlsrv pdo_sqlsrv
    - composer update --optimize-autoloader --no-interaction --no-progress
  extends: .php_cache
# this job build image for app using cache from above
build_image:
  stage: build
  image: docker:20.10.9
  services:
    - docker:20.10.9-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build -f Dockerfile-new -t username/repo:myapp .
    - docker push username/repo:myapp
  extends: .php_cache

This is Dockerfile-new for my app

FROM trafex/alpine-nginx-php7
USER root
RUN apk add wget curl gcc g++ musl-dev make unixodbc unixodbc-dev php8-ldap php8-dev php8-xml php8-session php8-pear php8-pdo php8-tokenizer php8-xmlwriter php8-fileinfo  && \
	curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/msodbcsql18_18.1.1.1-1_amd64.apk && \
	curl -O https://download.microsoft.com/download/b/9/f/b9f3cce4-3925-46d4-9f46-da08869c6486/mssql-tools18_18.1.1.1-1_amd64.apk && \
	apk add --allow-untrusted msodbcsql18_18.1.1.1-1_amd64.apk mssql-tools18_18.1.1.1-1_amd64.apk && \
	pecl8 install sqlsrv pdo_sqlsrv && \
	echo extension=pdo_sqlsrv > /etc/php8/conf.d/10_pdo_sqlsrv.ini && \
	echo extension=sqlsrv > /etc/php8/conf.d/00_sqlsrv.ini
USER nobody
COPY --chown=nginx ./ /var/www/html

When I run job “build_php” from pipeline I get error:

Running with gitlab-runner 14.3.2 (e0218c92)
  on centos8-shared Cmtq_DVz
Preparing the "docker" executor
Using Docker executor with image alpine ...
Pulling docker image alpine ...
Using docker image sha256:49176f190c7e9cdb51ac85ab6c6d5e4512352218190cd69b08e6fd803ffbf3da for alpine with digest alpine@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 ...
Preparing environment
Running on runner-cmtqdvz-project-290-concurrent-0 via f92b96ddfffd...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /myapp/.git/
Checking out 8866069c as dev...

Skipping Git submodules setup
Restoring cache
Checking cache for default...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
Using docker image sha256:49176f190c7e9cdb51ac85ab6c6d5e4512352218190cd69b08e6fd803ffbf3da for alpine with digest alpine@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 ...
$ apk --update add wget curl git gcc g++ musl-dev make unixodbc unixodbc-dev php8 php8-curl php8-openssl php8-iconv php8-json php8-mbstring php8-phar php8-ldap php8-dev php8-xml php8-session php8-pear php8-pdo php8-tokenizer php8-xmlwriter php8-fileinfo php8-dom
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
  php8 (no such package):
    required by: world[php8]
  php8-curl (no such package):
    required by: world[php8-curl]
  php8-dev (no such package):
    required by: world[php8-dev]
  php8-dom (no such package):
    required by: world[php8-dom]
  php8-fileinfo (no such package):
    required by: world[php8-fileinfo]
  php8-iconv (no such package):
    required by: world[php8-iconv]
  php8-json (no such package):
    required by: world[php8-json]
  php8-ldap (no such package):
    required by: world[php8-ldap]
  php8-mbstring (no such package):
    required by: world[php8-mbstring]
  php8-openssl (no such package):
    required by: world[php8-openssl]
  php8-pdo (no such package):
    required by: world[php8-pdo]
  php8-pear (no such package):
    required by: world[php8-pear]
  php8-phar (no such package):
    required by: world[php8-phar]
  php8-session (no such package):
    required by: world[php8-session]
  php8-tokenizer (no such package):
    required by: world[php8-tokenizer]
  php8-xml (no such package)ERROR: unable to select packages:
:
    required by: world[php8-xml]
  php8-xmlwriter (no such package):
    required by: world[php8-xmlwriter]
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 17

After reading this article GitLab CI/CD: GitLab vs. Docker caching – how to get insanely fast CI/CD pipelines
I can use caching to speed up CI process, here my implementations :

Use Gitlab caching:
.gitlab-ci.yml

cache: &global_cache
  key: $CI_COMMIT_REF_SLUG
  paths:
    - source/vendor/
  policy: pull-push

build_php:
  stage: build
  image: username/repo:buildtool
  script:
    - cd source
    - composer update --optimize-autoloader --no-interaction --no-progress
  cache:
    <<: *global_cache
  
build_image:
  stage: build
  image: docker:20.10.9
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build name:tag ./source
    - docker push name:tag
  cache:
    <<: *global_cache
    # override the policy , this job only pull/download folder "vendor" from cache, not push/upload
    policy: pull

Dockerfile

FROM trafex/alpine-nginx-php7
COPY --chown=nginx ./ /var/www/html

Job “build_php” will run faster from second time.

Use Docker caching
.gitlab-ci.yml

variables:
  IMAGE_VENDOR_NAME: username/repo:vendor
  IMAGE_FINAL_NAME: username/repo:myapp
  DOCKER_BUILDKIT: 1


default:
  image: docker:20.10.9

.docker-build:
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
# this job only build dependencies to an intermediate image , stage "composer" in Dockerfile only
Install Dependencies:
  stage: build
  extends: .docker-build
  script:
    - docker build --pull --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from $IMAGE_VENDOR_NAME -f Dockerfile --target composer -t $IMAGE_VENDOR_NAME ./source
    - docker push $IMAGE_VENDOR_NAME

# this job build app to image docker reuse intermediate image above
Build_image_final:
  stage: build
  extends: .docker-build
  script:
    - docker build --pull --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from $IMAGE_VENDOR_NAME --cache-from $IMAGE_FINAL_NAME -f Dockerfile --target buildimage -t $IMAGE_FINAL_NAME ./source
    - docker push $IMAGE_FINAL_NAME

Dockerfile

FROM username/repo:buildtool AS composer
WORKDIR /app
# copying the source directory and install the dependencies with composer
COPY . ./
# run composer install to install the dependencies
# cache folder vendor for the first time , reuse cache if exists
RUN --mount=type=cache,target=./vendor composer update \
  --optimize-autoloader \
  --no-interaction \
  --no-progress

FROM trafex/alpine-nginx-php7 as buildimage
COPY --chown=nginx --from=composer /app /var/www/html

Job “Install Dependencies” run faster from second time.

1 Like