How to convert circleci pipeline to gitlab

Hi everyone! I need help with migrating circleci pipeline to gitlab. I read GitLab Docs , but it’s not enough for migration. Below is a copy of my script. Very appreciated for any help. Thank you

version: 2
jobs:
build:
working_directory: ~/sti
docker:
- image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
steps:
- checkout:
path: ~/sti
- restore_cache:
keys:
- gradle-v2-{{ checksum “build.gradle” }}
- gradle-v2-
- run:
name: Gradle build process.
command: ./gradlew build --refresh-dependencies
- save_cache:
key: gradle-v2-{{ checksum “build.gradle” }}
paths:
- .gradle
- /root/.gradle
- save_cache:
key: build-{{ .Branch }}-{{ .Revision }}
paths:
- ~/sti

deploy-release:
working_directory: /
docker:
- image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:aws-cli
environment:
AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_TAGGING
AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_TAGGING
steps:
- run:
name: Retag stage image for prod
command: |
IMAGE_LIST=$(aws ecr --region $AWS_DEFAULT_REGION list-images --registry-id $AWS_ACCOUNT_ID --repository-name sti/$CIRCLE_PROJECT_REPONAME --filter tagStatus=TAGGED | jq .imageIds | jq . | jq .imageTag | grep rc-${CIRCLE_TAG} | sort -n)
LATEST_SG_TAG=echo $IMAGE_LIST | awk '{print $NF}'
MANIFEST=$(aws ecr --region $AWS_DEFAULT_REGION batch-get-image --registry-id $AWS_ACCOUNT_ID --repository-name sti/$CIRCLE_PROJECT_REPONAME --image-ids imageTag=$LATEST_SG_TAG --query ‘images.imageManifest’ --output text)
PROD_TAG=${CIRCLE_TAG////-}.${CIRCLE_BUILD_NUM}
aws ecr --region $AWS_DEFAULT_REGION put-image --registry-id $AWS_ACCOUNT_ID --repository-name sti/$CIRCLE_PROJECT_REPONAME --image-tag $PROD_TAG --image-manifest “$MANIFEST”

deploy-release-candidate:
working_directory: ~/sti/src
docker:
- image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
steps:
- add_ssh_keys
- restore_cache:
keys:
- build-{{ .Branch }}-{{ .Revision }}
- build-{{ .Branch }}-
- build-
- run: pip3 install awscli
- run:
name: Install Docker client ( we don’t want docker in the container )
command: |
set -x
VER=“17.09.1-ce”
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
mv /tmp/docker/* /usr/bin
chmod +x /usr/bin/dockerd
- run: dockerd &
- setup-docker-engine:
version: 17.11.0-ce
- run: eval $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
- run:
name: Build and deploy latest image to ECR
command: |
IMAGE_TAG=${CIRCLE_BRANCH/release//rc-}.${CIRCLE_BUILD_NUM}
docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/$CIRCLE_PROJECT_REPONAME:$IMAGE_TAG …
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/$CIRCLE_PROJECT_REPONAME:$IMAGE_TAG

deploy-branch:
working_directory: ~/sti/src
docker:
- image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
steps:
- add_ssh_keys
- restore_cache:
keys:
- build-{{ .Branch }}-{{ .Revision }}
- build-{{ .Branch }}-
- build-
- run: pip3 install awscli
- run:
name: Install Docker client ( we don’t want docker in the container )
command: |
set -x
VER=“17.09.1-ce”
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
mv /tmp/docker/* /usr/bin
chmod +x /usr/bin/dockerd
- run: dockerd &
- setup-docker-engine:
version: 17.11.0-ce
- run: eval $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
- run:
name: Build and deploy latest image to ECR
command: |
IMAGE_TAG=${CIRCLE_BRANCH////-}
docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/$CIRCLE_PROJECT_REPONAME:$IMAGE_TAG …
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/$CIRCLE_PROJECT_REPONAME:$IMAGE_TAG

deploy-latest:
working_directory: ~/sti/src
docker:
- image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
steps:
- add_ssh_keys
- restore_cache:
keys:
- build-{{ .Branch }}-{{ .Revision }}
- build-{{ .Branch }}-
- build-
- run: pip3 install awscli
- run:
name: Install Docker client ( we don’t want docker in the container )
command: |
set -x
VER=“17.09.1-ce”
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
mv /tmp/docker/* /usr/bin
chmod +x /usr/bin/dockerd
- run: dockerd &
- setup-docker-engine:
version: 17.11.0-ce
- run: eval $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
- run:
name: Build and deploy latest image to ECR
command: |
IMAGE_TAG=latest
docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/$CIRCLE_PROJECT_REPONAME:$IMAGE_TAG …
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/$CIRCLE_PROJECT_REPONAME:$IMAGE_TAG

workflows:
version: 2
build-deploy:
jobs:
- build
- deploy-release:
filters:
tags:
only: /^v./
branches:
ignore: /.
/
- deploy-release-candidate:
requires:
- build
filters:
tags:
ignore: /./
branches:
only: /^release.
/
- deploy-branch:
requires:
- build
filters:
branches:
ignore:
- /^master./
- /^release.
/
- deploy-latest:
requires:
- build
filters:
branches:
only: master

Hi,

I’d suggest starting small and select one job you’d like to migrate, and then iterate from there. Please also edit your post to use code blocks encapsulated with 3 backticks in Markdown and keep the indent in the config.yaml file - without indent, it is hard to guess which syntax applies in a YAML file.

```
config line 1
config line 2
```

Analyse the first job

Looking at the first job, let’s start analysing while reading the migration docs in parallel. Migrating from CircleCI | GitLab Below steps are marked with # 1) and so on.

version: 2
jobs:
build: # 1) job name in CircleCI, follow https://docs.gitlab.com/ee/ci/migration/circleci.html#jobs 
working_directory: ~/sti
docker:
- image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11 # 2) image path built using cI/CD variables. They need to be specified in the project settings > CI/CD > Variables https://docs.gitlab.com/ee/ci/migration/circleci.html#docker-image-definition 
steps:
- checkout:
path: ~/sti
- restore_cache: # 3) caching - ignore that for now, you can add that later once the pipeline works. https://docs.gitlab.com/ee/ci/caching/ + https://docs.gitlab.com/ee/ci/migration/circleci.html#caching
keys:
- gradle-v2-{{ checksum “build.gradle” }}
- gradle-v2-
- run:
name: Gradle build process. # 4) Job names don't exist in GitLab but you can copy it over into a comment 
command: ./gradlew build --refresh-dependencies # 5) This command will be executed, same as `script` in GitLab https://docs.gitlab.com/ee/ci/migration/circleci.html#jobs
- save_cache:
key: gradle-v2-{{ checksum “build.gradle” }}
paths:
- .gradle
- /root/.gradle
- save_cache:
key: build-{{ .Branch }}-{{ .Revision }}
paths:
- ~/sti

Let’s follow the marked steps above from 1) to 5)

I highly recommend using the pipeline editor to live lint the GitLab CI/CD configuration. Pipeline Editor | GitLab

Job migration, step by step

# 1) Job name
build:
  # 2) Image
  image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
  # 3) cache (later) https://docs.gitlab.com/ee/ci/migration/circleci.html#caching
  # 4) job name as comment or echo in the job output (later)
  # 5) command/script execution
  script:
    - ./gradlew build --refresh-dependencies

Refining the iteration with the job name inline as echo print

# 1) Job name
build:
  # 2) Image
  image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
  # 3) cache (later) https://docs.gitlab.com/ee/ci/migration/circleci.html#caching
  # 4) job name as comment or echo in the job output
  # 5) command/script execution
  script:
    - echo "Starting Gradle build process"
    - ./gradlew build --refresh-dependencies

Adding caching

# global gradle cache between jobs 
cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - .gradle

# 1) Job name
build:
  # 2) Image
  image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/sti/base:jdk-11
  # 3) cache https://docs.gitlab.com/ee/ci/migration/circleci.html#caching 

  # 4) job name as comment or echo in the job output
  # 5) command/script execution
  script:
    - echo "Starting Gradle build process"
    - ./gradlew build --refresh-dependencies

More migration steps

The procedure is the same for all existing jobs - identify the keywords, and migrate them step by step, commit and run from the pipeline editor view.

Note that you don’t need checkout working directories, GitLab runner clones the repository from the server and everything is there upon script command execution. before_script can be helpful to install additional dependencies, if any.

Cheers,
Michael

Thank you

Thank you Michael!

Hi Michael! Sorry bothering you, how to migrate “command” option from circleci to gitlab pipeline? Do I need to use script?


Thank you

Yes, suggest following the documentation steps for specific keywords, in the above example I’ve also added docs URLs next to the 1) to 5). notes.

Sorry, missed it.

Hi everyone! I migrating from circleci to gitlab ci pipeline and have question about artifacts. In circleci I have script for artifacts, but it’s not located in the main script.


Currently I’m getting this error and possibly it related to artifacts which is missing:

This is a Dockerfile

How to add artifacts to gitlab pipeline script globally, so it will use by any deploy stage in the script. Thank you