Problem to solve
- The CI/CD pipeline fails because due to missing git tags
- We have embedded projects that execute a python script to extract git information. This script starts failing when the latest tag is far in the “history”.
- The git information contains the version of the firmware and is needed to track the software currently running on the different devices.
- Blow is a screenshot of the error message that we receive.
The full log can be found in the resources section at the bottom of this message.
Steps to reproduce
- I have attached a slimmed down copy of the
gitlab-ci.ymland the original python script used to extract the git information. - When the script is ran locally, we do not get any errors
- After a number of commits (about 50), the CI starts failing
- In order to solve the problem, we can create a more “recent” tag
Configuration
- In our CI, the python script is started from a Makefile, but the python script can be executed directly in the script. *
Versions
Please select whether options apply, and add the version information.
- GitLab.com SaaS
- Self-hosted Runners (we have 2 versions of runners:14.2.0 and 16.0.2)
Helpful resources
- We found a similar issue on the GitLab forums (not sure if it’s the same problem or not). I tried adding
git fetch --tagsin thebefore_scriptbut it hasn’t helped: Missing git tags in CI jobs (#2221) · Issues · GitLab.org / gitaly · GitLab - Below is a slimmed down version of our .gitlab-ci.yml file (it executes only the python script)
stages:
- build
build-debug:
stage: build
before_script:
# The CI fails at this stage if we uncomment the `git fetch` below
#- git fetch --tags
script:
- echo "Building DEBUG application"
# Original make command that contains the python script
#- make -j all DEBUG=1
# Execution of the python script
- python generate_git_version.py git_version.h C
tags:
- compile
- Below is the full python script
#
######################################################################################
# This script generates the FW/SS version based on git tags
# Date created : 2023-09-06
######################################################################################
import re
import sys
from datetime import date
import subprocess
VERSION_ITEM_COUNT = 3
MAJ_INDEX = 1
MIN_INDEX = 2
PATCH_INDEX = 3
COMMIT_INDEX = VERSION_ITEM_COUNT + 1
COMMIT_INFO_SIZE = 2
COMMIT_N_DIRTY_SIZE = VERSION_ITEM_COUNT + COMMIT_INFO_SIZE
DIRTY_N_COMMIT_INDEX = VERSION_ITEM_COUNT + 1
DIRTY_W_COMMIT_INDEX = COMMIT_INDEX + COMMIT_INFO_SIZE
def print_usage():
print("Invalid number of arguments given")
print("Usage:")
print("py version_check.py <output_file_with_path> <output_type>")
print("")
print("<output_type> can be one of:")
print(" C for C/C++ header file")
print(" PY for Python file")
def generate_c_version_file(file, ver_info):
file.write("/***************************************************************************************\n")
file.write(" * This is an automatically generated file. DO NOT MODIFY IT AS IT WILL BE OVERWRITTEN *\n")
file.write(f" * Date generated: {date.today()} *\n")
file.write(" ***************************************************************************************/\n")
file.write("\n")
file.write("#ifndef _GIT_VERSION_H_\n")
file.write("#define _GIT_VERSION_H_\n")
file.write("\n")
file.write(f"#define _GIT_VERSION_HASH_ \"{ver_info['hash']}\"\n")
file.write(f"#define _GIT_VERSION_TAG_ \"{ver_info['tag']}\"\n")
file.write("\n")
file.write(f"#define _GIT_FIRMWARE_VERSION_MAJOR {ver_info['major']}\n")
file.write(f"#define _GIT_FIRMWARE_VERSION_MINOR {ver_info['minor']}\n")
file.write(f"#define _GIT_FIRMWARE_VERSION_PATCH {ver_info['patch']}\n")
file.write(f"#define _GIT_FIRMWARE_VERSION_COMMIT {ver_info['commit']}\n")
file.write("\n")
file.write("#endif /* _GIT_VERSION_H_ */\n")
def generate_python_version_file(file, ver_info):
file.write("#######################################################################################\n")
file.write("# This is an automatically generated file. DO NOT MODIFY IT AS IT WILL BE OVERWRITTEN #\n")
file.write(f"# Date generated: {date.today()} #\n")
file.write("#######################################################################################\n")
file.write("\n")
file.write(f"_GIT_VERSION_HASH_ = \"{ver_info['hash']}\"\n")
file.write(f"_GIT_VERSION_TAG_ = \"{ver_info['tag']}\"\n")
file.write("\n")
file.write(f"_GIT_VERSION_MAJOR = {ver_info['major']}\n")
file.write(f"_GIT_VERSION_MINOR = {ver_info['minor']}\n")
file.write(f"_GIT_VERSION_PATCH = {ver_info['patch']}\n")
file.write(f"_GIT_VERSION_COMMIT = {ver_info['commit']}\n")
file.write("\n")
if len(sys.argv) != 3:
print_usage()
sys.exit(-1)
hash_cmd = subprocess.run(['git', 'describe', '--match=NeVeRmAtCh', '--always', '--dirty', '--abbrev=0'], stdout=subprocess.PIPE)
if hash_cmd.stderr is not None:
print(f"An error occurred while executing the git command to get the Hash: {hash_cmd.stderr}")
exit(-1)
hash_str = hash_cmd.stdout.decode('utf-8').strip()
if hash_str == "":
print(f"An error occurred while fetching the Hash")
exit(-1)
print(f"Hash: {hash_str}")
tag_cmd = subprocess.run(['git', 'describe', '--tags', '--dirty'], stdout=subprocess.PIPE)
if tag_cmd.stderr is not None:
print(f"An error occurred while executing the git command to get the latest tag: {tag_cmd.stderr}")
exit(-1)
tag_str = tag_cmd.stdout.decode('utf-8').strip()
if tag_str == "":
print(f"An error occurred while fetching the Hash")
exit(-1)
print(f"Tag: {tag_str}")
print("")
# Start with any number of any character
regex = '.*'
# Version then contains the MAJOR, MINOR and PATCH separated by a '.'
regex = regex + ('(\\d+)\\.' * (VERSION_ITEM_COUNT-1))
# Finally, the version can contain the number of commits, its hash and the -dirty notation
regex = regex + '(\\d+)(?:-(\\d+)-g((?:[a-fA-F]|\\d){7}))?(?:-(dirty))?'
# Extract the regex groups from the git tag
tag_version = re.search(regex, tag_str)
git_version_major = tag_version.group(MAJ_INDEX)
git_version_minor = tag_version.group(MIN_INDEX)
git_version_patch = tag_version.group(PATCH_INDEX)
tag_item_count = len(tag_version.groups())
is_dirty = False
if tag_item_count == DIRTY_N_COMMIT_INDEX or tag_item_count == DIRTY_W_COMMIT_INDEX:
is_dirty = True
if tag_item_count == COMMIT_N_DIRTY_SIZE or tag_item_count == DIRTY_W_COMMIT_INDEX:
commit_count = tag_version.group(COMMIT_INDEX)
if commit_count is None:
commit_count = 0
print(f"version {git_version_major}.{git_version_minor}.{git_version_patch} dirty: {is_dirty} commit: {commit_count}")
version_info = {"hash": hash_str,
"tag": tag_str,
"major": git_version_major,
"minor": git_version_minor,
"patch": git_version_patch,
"commit": commit_count,
"dirty": is_dirty}
if sys.argv[2] == "c" or sys.argv[2] == "C":
generator_func = generate_c_version_file
elif sys.argv[2] == "py" or sys.argv[2] == "PY":
generator_func = generate_python_version_file
else:
print(f"Unrecognized output type received: {sys.argv[2]}")
print_usage()
sys.exit(-1)
f = open(f"{sys.argv[1]}", "w")
generator_func(f, version_info)
f.close()
- Below is the full log of a pipeline output when the script fails (the python script is executed from a makefile in the given output, but the result is the same).
Running with gitlab-runner 14.2.0 (58ba2b95)
on compile with container w1zzsG2X
Resolving secrets
00:00
Preparing the "docker" executor
00:04
Using Docker executor with image registry.gitlab.com/zzzzzz/cichecker/default-brio:latest ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/zzzzzz/briolab/cichecker/default-brio:latest ...
Using docker image sha256:299e1628c5ce7aee9fce4c0d66654b986ea54e27efc8e66b6a67c4895edaef13 for registry.gitlab.com/zzzzzz/cichecker/default-brio:latest with digest registry.gitlab.com/zzzzzz/cichecker/default-brio@sha256:f995698ac8e88448b4a442f2a094319d428a46b9aa9b4b09c00eee094e08a932 ...
Preparing environment
00:02
Running on runner-w1zzsg2x-project-58960237-concurrent-0 via 39d53cd3aafd...
Getting source from Git repository
00:02
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/zzzzzz/project/repo_name/.git/
Checking out c3153fd3 as bugfix/fix-ci-fail-at-git-version...
Removing build/
Removing ci.txt
Skipping Git submodules setup
Downloading artifacts
00:01
Downloading artifacts for prepare files (7656492798)...
Downloading artifacts from coordinator... ok id=7656492798 responseStatus=200 OK token=glcbt-66
Executing "step_script" stage of the job script
00:04
Using docker image sha256:299e1628c5ce7aee9fce4c0d66654b986ea54e27efc8e66b6a67c4895edaef13 for registry.gitlab.com/zzzzzz/cichecker/default-brio:latest with digest registry.gitlab.com/zzzzzz/cichecker/default-brio@sha256:f995698ac8e88448b4a442f2a094319d428a46b9aa9b4b09c00eee094e08a932 ...
$ echo "Building RELEASE application"
Building RELEASE application
$ make -j all
mkdir -p build/release
PRE-BUILD STEP
"python3" ci/scripts/generate_git_version.py build/release/git_version.h C
fatal: No tags can describe 'c3153fd32c1787272567c5b408644d117da2724a'.
Try --always, or create some tags.
Hash: c3153fd32c1787272567c5b408644d117da2724a
An error occurred while fetching the Hash
make: *** [makefile:169: pre-build] Error 255
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1
- Below is the error message we get in the pipeline when we execute
git fetch --tagsin thebefore_scriptsection
$ git fetch --tags
From https://gitlab.com/zzzzzz/project
+ 3d1b1d7...11d2d53 feature/FixUpdateResumeAndRestart -> origin/feature/FixUpdateResumeAndRestart (forced update)
+ 60f9b1a...e8a6b6e feature/WaitRxCompleteNotification -> origin/feature/WaitRxCompleteNotification (forced update)
+ c1b5ff0...f914da2 feature/fixVersion -> origin/feature/fixVersion (forced update)
+ ef62215...ab43b30 feature/synchronize-linker-for-app-and-boot -> origin/feature/synchronize-linker-for-app-and-boot (forced update)
! [rejected] FW-0.0.0 -> FW-0.0.0 (would clobber existing tag)
