Hi community,
first thing … everything works as expected and I have Debian packages in the end on the right host … but … the yaml looks not nice, and I think there is much room for optimize it.
What we have: The devops working mostly on the develop branch and for a release, they push the changes to the master branch. The pipelines starts and on the end, we have the new packages on our test systems. If the test is OK, we copy the packages on the Aptly repo from testing to release and starting our Rundeck jobs for installing them on the production servers.
Because of a new project, I want to create the Debian packages in a “better” way and instead of using plain dpkg-buildpackage and use dirty commands for the package version and debian/changelog, I’ve switched to gbp to get the changelog filled and also the version number.
What I wanted: my initial idea was, to also build packages from the develop branch, with snapshot option on dch, which looks like “mypackage_1.0.152~5.gbpff44a5_all.deb” and on a real release, it should just like “mypackage_1.0.152_all.deb”. So I had two gitlab-ci-yml files, one in master and one in develop branch, but it failed, because of merge conflics debian/changelog. So I dropped that idea for now.
If I have a look on my gitlab-ci.-yml file, I’ve tested a lot, copied from here and there but I think, there are many steps, which can be better or dropped, because they are needless.
# Defines stages which are to be executed
variables:
TOOL_ARGS: apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends --yes --allow-unauthenticated -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"
stages:
- build
- publish
- deploy
# Is performed before the scripts in the stages step
# We need a special Docker imnage for faster deploy
build:stretch: &build
stage: build
tags:
- docker
image: git.example.com:5555/inatec_internal/container/stretch/build_mypackage21:v1
before_script:
- apt update
- apt install git-buildpackage ca-certificates
- git reset --hard
- git clean -fd
- git checkout $CI_COMMIT_REF_NAME
- git pull origin $CI_COMMIT_REF_NAME
- git config user.name "git"
- git config user.email git@example.com
- mkdir -p ~/.ssh ; chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- eval $(ssh-agent -s)
- ssh-add <(echo "${RUNNER_PRIV_SSH_KEY}")
- ssh-keyscan git.example.com >> ~/.ssh/known_hosts ; chmod 0644 ~/.ssh/known_hosts
- git remote set-url origin git@git.example.com:$CI_PROJECT_PATH.git
script:
- mk-build-deps --install --remove --tool="$TOOL_ARGS" debian/control
- /usr/bin/gbp dch -a --commit --snapshot --commit-msg='Update changelog for %(version)s release [skip ci]'
- DEB_VERSION=$(dpkg-parsechangelog --count 1 -S version | awk -F '+' '{print $1}')
- echo "DEB_ARCHITECTURE=$(grep Architecture debian/control | awk '{print $2}')" >> .vars
- echo "DEB_PACKAGE_NAME=$(grep 'Package:' debian/control | awk '{print $2}')" >> .vars
- echo "DEB_FILENAME=${DEB_PACKAGE_NAME}_${DEB_VERSION}_${DEB_ARCHITECTURE}.deb" >> .vars
- /usr/bin/gbp tag --debian-tag='v%(version)s' --ignore-new
- /usr/bin/gbp buildpackage --git-upstream-tag='v%(version)s' --git-ignore-branch --git-ignore-new --build=all -us -uc -rfakeroot
- /usr/bin/gbp push
- echo "DEB_VERSION=${DEB_VERSION}" >> .vars
- git push -u origin master --follow-tags
- source .vars
- mkdir -p binaries
- install -d -m 777 /publish/${CI_COMMIT_SHORT_SHA}
- for suffix in deb changes; do cp ../${DEB_PACKAGE_NAME}_${DEB_VERSION}_${DEB_ARCHITECTURE}.${suffix} /publish/${CI_COMMIT_SHORT_SHA} ; chmod 666 /publish/${CI_COMMIT_SHORT_SHA}/* ; done
- for suffix in deb changes; do cp ../${DEB_PACKAGE_NAME}_${DEB_VERSION}_${DEB_ARCHITECTURE}.${suffix} binaries/ ; done
artifacts:
expire_in: 1 day
paths:
- build/
- binaries/
- .vars
only:
- master
publish:stretch: &publish
tags:
- shared
stage: publish
dependencies:
- build:stretch
script:
- sudo -u aptly /usr/bin/aptly repo -force-replace -remove-files add $repo /opt/aptly/incoming/${CI_COMMIT_SHORT_SHA}
- sudo -u aptly /usr/bin/aptly publish update stretch
- rm -rf /publish/${CI_COMMIT_SHORT_SHA}
variables:
repo: stretch-testing
only:
- master
# Deploy via SSH and get the key out of Vault
deploy:stretch: &deploy
tags:
- shared
stage: deploy
artifacts:
paths:
- binaries/
- .vars
dependencies:
- publish:stretch
- build:stretch
before_script:
- get-vault-secrets-by-approle > ${VAULT_VAR_FILE}
- source ${VAULT_VAR_FILE} && rm ${VAULT_VAR_FILE}
- eval $(ssh-agent -s)
- ssh-add <(echo "${DEPLOY_SSH_PRIV_KEY}")
- source .vars
script:
- echo "============= Deploy on fra-test-pasint-03 ============="
- export DEBIAN_FRONTEND=noninteractive
- ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no rundeck@fra-test-pasint-03.example.local sudo su -c \""/usr/local/bin/deploy_mypackage\""
only:
- master
I use the .vars file as helper, to get the package name, version etc. pp. I was thinking, to put most commands in a shell script, inside the project to keep the YAML more clean. This project has later a 2nd repo inside this one, which is not a submodule (I dropped all projects with git submodules, because it was never working correctly on Gitlab and I had very often old commits, but not the latest one)
The project itself has only plain files, so no Makefile / configure etc. pp. just putting the files inside the Debian package, appending the debian/changelog out of the Git log and push the version tag.
I’ve changed also the CI/CD option from fetch to clone, because of merge problems … but maybe, that was not needed, because the problem was somewhere else …
Looking around the other Gitlab Debian pipelines … many of them are too complicated (mainly for bigger projects and builds for many plattforms) or they don’t use the pipeline at all.
So, I’m asking here … how can I optimize my ci/cd config and make it beautiful, like my inital idea to have builds for develop branch too and drop the snapshot option on the master branch, without having problems with debian/changelog in master / develop branch.
For the old system - Jenkins - the Devop (release manager) had to change the branch on the Debian buildpackage plugin, which was checked out, build the Debian package and this plugin also changes the version number on debian/changelog and commit it to git back.
One of my devop is asking: Can Gitlab searches for a “keyword” on a commit message. Maybe than we can use this message to get a version number and build the package.