Auto DevOps - Customize .gitlab-ci.yml to Auto build using individual components of Auto DevOps

amhg · October 23, 2023, 8:50am

Currently, my pipeline is able to build during the build phase my project and execute SAST and Dependency scans during the test phase.

During the build phase, it is generated two artifacts:

.m2 (maven repository) and
war file.

.m2 is used by spotbugs to avoid compiling the project.

You can see here the pipeline:

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml

variables:
  CI_DEBUG_TRACE: "true"
  SECURE_LOG_LEVEL: "debug"
  DS_JAVA_VERSION: 8
  SAST_SCANNER_ALLOWED_CLI_OPTS: "-effort more"
  JAVA_OPTS: "-Xmx8g"
  
stages:
  - build
  - test

java-build:
  image: maven:3.8-openjdk-8
  stage: build
  script:
    - mvn clean package -DskipTests=true -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository
  artifacts:
    expire_in: 1 day
    when: always
    paths:
      - .m2/
      - "modules/target/"

spotbugs-sast:
  dependencies:
    - java-build
  variables:
    MAVEN_REPO_PATH: $CI_PROJECT_DIR/.m2/repository
    COMPILE: "false"
  timeout: 10h 0m

The idea is to use individual components of Auto DevOps (Auto Build) instead of using my build job (java-build).

I guess it is necessary to keep the build stage and to include the following in my pipeline.

stages:
  - build

include:
  - template: Jobs/Build.gitlab-ci.yml

My question is:

How should I customize Auto Build in my pipeline to get the same behavior as my my current build job (java-build)?

Thanks for taking the time to be thorough in my request, it really helps and will make a difference in my project!

balonik · October 23, 2023, 3:51pm

You can see all the templates here lib/gitlab/ci/templates · master · GitLab.org / GitLab · GitLab

You can browse through the job definitions and see how to customize (override) particular jobs to your need.

amhg · October 25, 2023, 9:20am

Hello,

Thanks a lot for you feedback

I have updated the pipeline as follows to overwrite job build-artifact:

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Jobs/Build.gitlab-ci.yml

variables:
  CI_DEBUG_TRACE: "true"
  SECURE_LOG_LEVEL: "debug"
  DS_JAVA_VERSION: 8
  # SAST_EXCLUDED_ANALYZERS: "brakeman,flawfinder,nodejs-scan,spotbugs"
  # DS_EXCLUDED_ANALYZERS: "gemnasium-python,gemnasium"
  # Effort levels (min, less, more, max) for spotbug scan https://docs.gitlab.com/ee/user/application_security/sast/#security-scanner-configuration
  SAST_SCANNER_ALLOWED_CLI_OPTS: "-effort more"
  # Memory for spotbugs to avoid out of memory java heap space. Configure based on runner capacities
  JAVA_OPTS: "-Xmx8g"
  
stages:
  - build
  - test

build_artifact:
  #image: maven:3.8-openjdk-8
  stage: build
  script:
    - printf "Auto build"
    - mvn clean package -DskipTests=true -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository
  artifacts:
    expire_in: 1 day
    when: always
    paths:
      - .m2/
      - "modules/web/target/"

spotbugs-sast:
  dependencies:
    - build_artifact
  variables:
    MAVEN_REPO_PATH: $CI_PROJECT_DIR/.m2/repository
    COMPILE: "false"
  timeout: 10h 0m

Now I get the following error

Running with gitlab-runner 15.11.0 (436955cb)
  on BTW_runner_2 2By6xFn9, system ID: r_p3gqSHMzkB5H
Resolving secrets
Preparing the "docker" executor
Using Docker executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.38.1 ...
Starting service docker:20.10.12-dind ...
Pulling docker image docker:20.10.12-dind ...
Using docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:20.10.12-dind with digest docker@sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-2by6xfn9-project-85-concurrent-0-5dbddec6432fab6e-docker-0 probably didn't start properly.
Health check error:
start service container: Error response from daemon: Cannot link to a non running container: /runner-2by6xfn9-project-85-concurrent-0-5dbddec6432fab6e-docker-0 AS /runner-2by6xfn9-project-85-concurrent-0-5dbddec6432fab6e-docker-0-wait-for-service/service (services.go:189:0s)
Service container logs:
2023-10-25T09:16:47.692690448Z ip_tables              36864  0 
2023-10-25T09:16:47.692774199Z x_tables               65536  5 xt_conntrack,xt_MASQUERADE,xt_addrtype,nft_compat,ip_tables
2023-10-25T09:16:47.692811550Z ip: can't find device 'ip_tables'
2023-10-25T09:16:47.692826360Z modprobe: can't change directory to '/lib/modules': No such file or directory
2023-10-25T09:16:47.694421417Z mount: permission denied (are you root?)
2023-10-25T09:16:47.694629764Z Could not mount /sys/kernel/security.
2023-10-25T09:16:47.694718682Z AppArmor detection and --privileged mode might break.
2023-10-25T09:16:47.696172987Z mount: permission denied (are you root?)
*********
Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.38.1 ...
Using docker image sha256:9c1af7758ccc7e9c2beb26bf009e529d4b8da22a1d5dceafea551170fb9b7379 for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.38.1 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:b6279edf717f2b8def58f44411c0e680e0f2936ee4fd365237fae1532f0a3a4e ...
Preparing environment
Running on runner-2by6xfn9-project-85-concurrent-0 via 90563ade74ab...
Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/BTW/core/.git/
Checking out bef31be7 as detached HEAD (ref is 1186-update-pipeline-to-use-auto-build)...
Removing .m2/
Removing modules/config/target/
Removing modules/domain/target/
Removing modules/i18n/target/
Removing modules/integration/target/
Removing modules/js/target/
Removing modules/repository/target/
Removing modules/security-preauth/target/
Removing modules/security-saml/target/
Removing modules/security/target/
Removing modules/services/target/
Removing modules/target/
Removing modules/testSupport/target/
Removing modules/ui-angular/target/
Removing modules/ui/src/main/assets/css/
Removing modules/ui/src/main/resources/VAADIN/themes/BTW/styles.css
Removing modules/ui/target/
Removing modules/usecases/target/
Removing target/
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:9c1af7758ccc7e9c2beb26bf009e529d4b8da22a1d5dceafea551170fb9b7379 for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.38.1 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:b6279edf717f2b8def58f44411c0e680e0f2936ee4fd365237fae1532f0a3a4e ...
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
$ /build/build.sh
Logging in to GitLab Container Registry with CI credentials...
error during connect: Post "http://docker:2375/v1.24/auth": dial tcp: lookup docker on 10.69.43.161:53: server misbehaving
Uploading artifacts for failed job
Uploading artifacts...
WARNING: gl-auto-build-variables.env: no matching files. Ensure that the artifact path is relative to the working directory (/builds/BTW/core) 
ERROR: No files to upload                          
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

Any hint about this issue?

balonik · October 25, 2023, 9:27am

If you want to use DinD on your local Runner, that runner must have privileged = true in it’s config.toml.

I do not recommend using DinD, tho. There are plenty of k8s native container image build tools you can use instead.

amhg · October 30, 2023, 1:04pm

@balonik Thanks a lot for your feedback

For the moment we are working on PoC (proof of concept) to test gitlab features and it is not possible to use k8s, we have set privileged to true.

Now, build process is able to download all dependencies and run maven properly but we are experiencing a new issue:

ERROR: failed to export: failed to write image to the following tags: [tmp-cnb-image-294510: loading image "tmp-cnb-image-294510". first error: embedded daemon response: Error processing tar file(exit status 1): write /e43e631ae071c3ea5f544a195e92ebd3178c9e1af60c22f41f107dbf2c4decf4.tar: no space left on device]
ERROR: failed to build: executing lifecycle: failed with status code: 62

Please see stack trace after running maven

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.496 s
[INFO] Finished at: 2023-10-29T06:34:51Z
[INFO] ------------------------------------------------------------------------
Timer: Builder ran for 7m24.694027152s and ended at 2023-10-29T06:34:51Z
===> EXPORTING
Timer: Exporter started at 2023-10-29T06:34:51Z
Adding layer 'heroku/jvm:heroku_metrics_agent'
Adding layer 'heroku/jvm:openjdk'
Adding layer 'heroku/jvm:runtime'
Adding layer 'heroku/maven:maven'
Adding layer 'buildpacksio/lifecycle:launch.sbom'
Adding 1/1 app layer(s)
Adding layer 'buildpacksio/lifecycle:launcher'
Adding layer 'buildpacksio/lifecycle:config'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
no default process type
Timer: Saving tmp-cnb-image-294510... started at 2023-10-29T06:35:29Z
*** Images (8188de344e5f):
      tmp-cnb-image-294510 - loading image "tmp-cnb-image-294510". first error: embedded daemon response: Error processing tar file(exit status 1): write /e43e631ae071c3ea5f544a195e92ebd3178c9e1af60c22f41f107dbf2c4decf4.tar: no space left on device
Timer: Saving tmp-cnb-image-294510... ran for 55.568816256s and ended at 2023-10-29T06:36:24Z
Timer: Exporter ran for 1m32.972675334s and ended at 2023-10-29T06:36:24Z
ERROR: failed to export: failed to write image to the following tags: [tmp-cnb-image-294510: loading image "tmp-cnb-image-294510". first error: embedded daemon response: Error processing tar file(exit status 1): write /e43e631ae071c3ea5f544a195e92ebd3178c9e1af60c22f41f107dbf2c4decf4.tar: no space left on device]
ERROR: failed to build: executing lifecycle: failed with status code: 62
Uploading artifacts for failed job
Uploading artifacts...
WARNING: gl-auto-build-variables.env: no matching files. Ensure that the artifact path is relative to the working directory (/builds/poc/core) 
ERROR: No files to upload                          
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1
1mERROR: Job failed: exit code 1

Any hint about this issue?

balonik · November 2, 2023, 7:51am

I see no space left on device which means your server run out of disk space

amhg · November 10, 2023, 8:36am

Hello @balonik,

Thanks a lot for your feedback

We have already checked the capacity of the server

and after running the command “docker container prune”

but we are still having issues with disk space

ERROR: failed to export: exporting app layers: caching layer (sha256:696a3a6b6c1712392fc0bbbc7e19dfd1cc253424987541b7d2aaaea9e2b38144): write /launch-cache/staging/sha256:696a3a6b6c1712392fc0bbbc7e19dfd1cc253424987541b7d2aaaea9e2b38144.tar: no space left on device
ERROR: failed to build: executing lifecycle: failed with status code: 62

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.343 s
[INFO] Finished at: 2023-11-06T09:51:03Z
[INFO] ------------------------------------------------------------------------
#######################################################################
WARNING: This builder image (heroku/buildpacks:20) is deprecated,
since it uses legacy shimmed classic Heroku buildpacks, rather than
Heroku's next-generation Cloud Native Buildpacks.
As such, this image is no longer supported and will soon stop receiving
security updates.
Please switch to one of our newer 'heroku/builder:*' builder images,
such as 'heroku/builder:22':
https://github.com/heroku/cnb-builder-images#heroku-cnb-builder-images
If you are using the Pack CLI, you will need to adjust your '--builder'
CLI argument, or else change the default builder configuration:
https://buildpacks.io/docs/tools/pack/cli/pack_config_default-builder/
If you are using a third-party platform to deploy your app, check their
documentation for how to adjust the builder image used for your build.
#######################################################################
Timer: Builder ran for 7m23.580829008s and ended at 2023-11-06T09:51:03Z
===> EXPORTING
Timer: Exporter started at 2023-11-06T09:51:03Z
Adding layer 'heroku/jvm:heroku_metrics_agent'
Adding layer 'heroku/jvm:openjdk'
Adding layer 'heroku/jvm:runtime'
Adding layer 'heroku/maven:maven'
Adding layer 'buildpacksio/lifecycle:launch.sbom'
Timer: Exporter ran for 42.279586281s and ended at 2023-11-06T09:51:45Z
ERROR: failed to export: exporting app layers: caching layer (sha256:696a3a6b6c1712392fc0bbbc7e19dfd1cc253424987541b7d2aaaea9e2b38144): write /launch-cache/staging/sha256:696a3a6b6c1712392fc0bbbc7e19dfd1cc253424987541b7d2aaaea9e2b38144.tar: no space left on device
ERROR: failed to build: executing lifecycle: failed with status code: 62
Uploading artifacts for failed job
Uploading artifacts...
WARNING: gl-auto-build-variables.env: no matching files. Ensure that the artifact path is relative to the working directory (/builds/leos/core) 
ERROR: No files to upload                          
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1
1mERROR: Job failed: exit code 1

Any hint for this issue?

balonik · November 10, 2023, 8:50am

Just 13GB of free space might not be enough. After the job is finished the space is cleaned up so you won’t see it. I suggest to monitor free space during the job execution.

It is also possible that you limit max storage somewhere. What storage driver do you use in Docker? You can check Docker storage config using docker info.