Running jobs for a certain tag pattern

GitLab CI/CD
1

Hello everybody!

I’m currently struggling to implement something my team would like to have for our CI:
We would like to run a certain job only for a tag that equals a certain variable in our CI.yml.

For example and to be precise, we want to run a deploy job only for the tag v1.0.0 when a CI variable VERSION is 1.0.0. Afterwards we would like to change it so the tag v1.0.0-rc1 or v1.0.0-rc2, etc. also runs the deploy step.

I’ve been running into a lot of problems even trying to get the basic setup working:

  • I can’t seem to be able to have the rules keyword even check for a variable with a leading “v”
  • Trying to combat that, i was trying to define a pattern to see how that would go and it can’t even get that to work

Has anybody been doing something similar? Any pointers on how we might be able to achieve what we’re aiming at?

What version are you on?

I’m using gitlab.com - i can link my test repository, if that is any helpful.

Add the CI configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml)

This is my .gitlab-ci.yml:

variables:
    VERSION: "1.0.0"
    VERSION_REGEX: /v1\.0\.0/


stages:
    - prepare
    - deploy


prepare:
    stage: prepare
    script:
        - echo "preparing"

deploy (snapshot):
    stage: deploy
    extends:
        - .snapshot-job
    script:
        - echo "deploying snapshot"

deploy (release):
    stage: deploy
    extends:
        - .release-job
    script:
        - echo "deploying release"


.release-job:
    tags:
        - windows
    rules:
        - if: $CI_COMMIT_TAG && $CI_COMMIT_TAG =~ $VERSION_REGEX
          when: always

.snapshot-job:
    rules:
        - if: $CI_COMMIT_TAG && $CI_COMMIT_TAG =~ $VERSION_REGEX
          when: never
        - when: always

What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

I’ve mostly read through the official page Keyword reference for the .gitlab-ci.yml file | GitLab - which did not help me much for now.

2

Hi @IloSophiep
as stated in docs storing expressions in variable is currently limited. To be hones only basic stuff works.

You can either use VERSION_REGEX: '/v1.0.0/' or you can store the entire condition in variable which is working as well

variables:
    VERSION: 'v1.0.0-rc1'
    VERSION_REGEX: '/v1.0.0/'
    VERSION_REGEX2: '$VERSION =~ /v1\.0\.0/'

stages:
    - deploy

dep:one:
    stage: deploy
    rules:
        - if: $VERSION =~ $VERSION_REGEX
          when: always
    script:
        - echo 'deploy'

dep:two:
    stage: deploy
    rules:
        - if: $VERSION_REGEX2
          when: always
    script:
        - echo 'deploy'
1 Like
3

Hello @balonik and thanks for the response!

It seems like i missed that part of the docs, sorry…!

Still, trying to implement your solution does not seem to work for me. I adapted your variables and your rule to this CI.yml:

variables:
    VERSION: '1.0.0'
    IS_VERSION_TAGGED: '$CI_COMMIT_TAG =~ /v1\.0\.0/'


stages:
    - prepare
    - deploy


prepare:
    stage: prepare
    script:
        - echo "preparing"


deploy (snapshot):
    stage: deploy
    extends:
        - .snapshot-job
    script:
        - echo "deploying snapshot"


deploy (release):
    stage: deploy
    extends:
        - .release-job
    script:
        - echo "deploying release"




.release-job:
    tags:
        - windows
    rules:
        - if: $IS_VERSION_TAGGED
          when: always

.snapshot-job:
    tags:
        - windows
    rules:
        - if: $IS_VERSION_TAGGED
          when: never
        - when: always

Things i changed:

  • our version variable needs to be the in the format x.y.z without the leading “v”. Is that any important? It’s not currently being used in this example anyways. We would like to have the regex use that variable, but that seems impossible?
  • i changed the variable name of the pattern to $IS_VERSION_TAGGED
  • i changed what is being compared in the regex: We want to check the tag of the current commit, so i am using the gitlab CI variable for the tag

It results in the “deploy (release)” job always running, even on master without any tag.

Anything obvious i missed? And thanks a lot again!

4

To be honest I don’t know how it evaluates if the variable CI_COMMIT_TAG is not defined. Try to put another rule that checks if it is defined

# don't run if its not tag
- if: $CI_COMMIT_TAG == null
  when: never

I don’t think it matters if the variable is leading with ‘v’ or not, but always put it in quotes in that case so it isn’t evaluated somewhere like integer instead of string.

5

I’m not sure that’s going to work either. I’ve changed the variables and rules to the following:

variables:
    VERSION_MAJOR: '1'
    VERSION_MINOR: '0'
    VERSION_PATCH: '0'
    VERSION: '$VERSION_MAJOR.$VERSION_MINOR.$VERSION_PATCH'
    IS_VERSION_TAGGED: '$CI_COMMIT_TAG =~ /v$VERSION_MAJOR\.$VERSION_MINOR\.$VERSION_PATCH(-rc\d+)?/'

...

.release-job:
    tags:
        - windows
    rules:
        - if: $CI_COMMIT_TAG != null && $IS_VERSION_TAGGED
          when: always

.snapshot-job:
    tags:
        - windows
    rules:
        - if: $CI_COMMIT_TAG != null && $IS_VERSION_TAGGED
          when: never
        - when: always

Now it runs the snapshot job for the master branch (good) and
it runs the deploy job for the tag v1.0.0 (good).
But it also runs the deploy job for the tag v0.1.0 (bad) and
it even runs the deploy job for the tag peter (bad) which makes me think it does not care what exactly the tag contains.

Any more ideas?

Edit: To me it honestly feels like a bug in GitLab CI at this point…?

6

@IloSophiep can you add this to the script or before_script in one of your jobs and see what gets printed?

script:
    - echo $VERSION
    - echo $IS_VERSION_TAGGED
7

Sure thing! (and thanks for trying to help out)
I changed my prepare job to include those two echo’s:

prepare:
    stage: prepare
    script:
        - echo "preparing ${VERSION}"
        - echo $VERSION
        - echo $IS_VERSION_TAGGED

Results for:

  • master-Branch:
$ echo "preparing ${VERSION}"
preparing 1.0.0
$ echo $VERSION
1.0.0
$ echo $IS_VERSION_TAGGED
=~ /v1\.0\.0(-rc\d+)?/
  • for tag peter:
$ echo "preparing ${VERSION}"
preparing 1.0.0
$ echo $VERSION
1.0.0
$ echo $IS_VERSION_TAGGED
peter =~ /v1\.0\.0(-rc\d+)?/
  • for tag v1.0.0:
$ echo "preparing ${VERSION}"
preparing 1.0.0
$ echo $VERSION
1.0.0
$ echo $IS_VERSION_TAGGED
v1.0.0 =~ /v1\.0\.0(-rc\d+)?/
  • for tag v0.1.0:
$ echo "preparing ${VERSION}"
preparing 1.0.0
$ echo $VERSION
1.0.0
$ echo $IS_VERSION_TAGGED
v0.1.0 =~ /v1\.0\.0(-rc\d+)?/

Looks to me as if GitLab CI doesn’t look at the pattern inside of the variable but only checks if the variable has any value?

Thanks again!

8

Right, and I think there’s an epic up for this sort of interpolation somewhere.

The way I would approach this is:

.release-job:
    tags:
        - windows
    rules:
        - if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ $VERSION'
          when: always

.snapshot-job:
    tags:
        - windows
    rules:
        - if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ $VERSION'
          when: never
        - when: always
9

This is the epic for variable substitution, and this issue is probably the most relevant one to up-vote.

10

I tried that out and while it’s working in a way, it’s not really what i was going for: It seems like then the tag can only be the exact version - so it’s basically just ==? I tried the tag 1.0.0 which results in a deploy job. I also tried the tag 1a0.0 expecting the =~ operator would do a regex matching - but that did only trigger a snapshot job. Would make sense, because $VERSION does not include the / for it to become a regex?

But what we would like to have is a regex match so we can include the release candidate tagging. So i tried to extend your version:

  • I added a variable VSN_REGEX: '/v1\.0\.0/' because of your response saying i cannot use nested variables
  • I use that variable in the rule like this:
    if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ $VSN_REGEX'

I was hoping that would get it to work, but now it nevers runs the release job.
I tried changing my regex variable to be this, thinking i might do a mistake in matching the literal dot:
VSN_REGEX: '/v1\\.0\\.0/' which does not help either.

It made me wonder if the problem is due to having the regex in a variable, like @balonik said earlier:

but even placing the regex - as in if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /v1\\.0\\.0/' - in the rules does not make it work…

I think i screwed up my regex and will start investigating that first of all. If i can’t even get my hardcoded regex to work then there is no use trying to see what issues there are with trying to refactor variables out and so on.

11

So there was a root problem with my regex. With the rule:

if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /^v1\.0\.0(-rc.+)?$/'

i got all my tags to create the desired release or snapshot jobs.

And now i seem to be stuck at known issues:

Seem to me like there is not much for me to do in regards to my CI for now.

Can someboy tell me: Do i go and comment in those issues with my (shortened) use case? Or does that add unnecessary noise and i should only upvote the issues via the “thumbsup”-emoji?

Thanks again for your help, much appreciated! Although it was mostly my own stupidity with the regex pattern not being valid :frowning:

12

I use regexps in rules too, they can work, but I wouldn’t put them in a variable. Does something like this get you a bit closer (you might want to tweak the reg exp here, I haven’t tested it):

if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/'`
13

I didn’t mean to say that regex’s don’t work in rules. Sorry if i made it sound like that!

What i meant is that my regex in the rule works perfectly fine now. It’s just that i now have to repeat the “version information” three times in different places in the CI.yml. That’s a lot worse than having one version variable at the top of the CI.yml, like we had before.

Also I think the rule i had

if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /^v1\.0\.0(-rc.+)?$/'

has some advantages that are important to us:

  • The “tagged version number” equals the expected version number. With your change we would lose that confidence and might accidently publish a build with $VERSION: 1.0.0 via tag v2.0.0
  • The missing release-candidate recognition is important to us and manly the reason we even change our rule from a simple $CI_COMMIT_TAG == $VERSION - but maybe you left that part out only for easier reading.

I will have to discuss with my team if we feel like the added maintenance / error chance of having our current version three times in our CI.yml is worth it to be able to recognize the release candidate tags. Maybe we will simply stick with the version from before until GitLab fixes the issues i linked earlier, because it’s just not worth it for us for now.

14

Ah, I think you probably want YAML aliases. I haven’t tested this, but I think it’s close to what you are looking for:

.release-job:
    tags:
        - windows
    rules:
        - if: &version '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /^v1\.0\.0(-rc.+)?$/' 
          when: always

.snapshot-job:
    tags:
        - windows
    rules:
        - if: *version
          when: never
        - when: always

Once you have that, you might also want to use anchors to move these rules to the top of the file. Some of my configs look a bit like this:

.rules-review: &rules-review
    rules:
        - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
          when: never
        - if: '$CI_COMMIT_BRANCH =~ "/^v1\.0\.0(-rc.+)?$/"'
          when: never
        - when: always

.rules-production: &rules-production
    rules:
        - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
          when: always
        - when: never

...

.deploy: &deploy
    stage: deploy
    script:
        - ...
    artifacts:
        paths:
            - ...

deploy:review:
    <<: *deploy
    <<: *rules-review
    environment:
        name: review/$CI_COMMIT_REF_NAME
        url: ...
        auto_stop_in: ...

deploy:production:
    <<: *deploy
    <<: *rules-production
    script:
        - ...
    environment:
        name: production
        url: ...
15

Thanks, i marked that last post as the solution!

I knew about anchors but didn’t think far enough to use it in this situation. Very cool!

I am now using an anchor to not duplicate the rule with the pattern, just like you already showed:

.release-job:
    tags:
        - windows
    rules:
        - if: &version '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /^v1\.0\.0(-rc.+)?$/'
          when: always

.snapshot-job:
    tags:
        - windows
    rules:
        - if: *version
          when: never
        - when: always

It still has the disadvantage, that i have the version duplicated once though: At the top in the variable and in the line with the rule. I don’t think i can use anchors to “fix” that either, but i haven’t looked into that (yet).

The rest of your example was interesting to look at, but i don’t see any advantages over the extend-approach i’m currently using where i extend the “release” or “snapshot” job template, as necessary. Feel free to correct me if i’m missing something though :slight_smile: !

Thanks again for the help, much appreciated! “As soon” as GitLab supports the two linked functionalities this approach will be a breeze to work with anyways.

16

So, taking your last question first "is there a difference between anchors and extends", the main difference is just that extends is specific to GtiLab and anchors are a YAML thing. In the example I gave there isn’t a meaningful difference. Once you start extending jobs and overriding variables etc. then it’s worth understanding how definitions are merged, but I doubt there’s much else.

For your other query, about duplicating the regex definition, it’s not clear to me why you need that other variable; but it may be that you can use an alias there too.

17

Thanks for that link, will have a look!

We use that $VERSION variable in the .gitlab-ci.yml to “set” the version of our build. So that variable is used for a few things like the file version of the build artifact, also in some Maven POM’s, while building an installer executable and maybe more i’m currently forgetting.

We like having that version only defined in one place for the whole project and bumping only that value up when working on a new version. That’s why it will be nice for us, when GitLab supports to refactor our rule to use the $VERSION variable that already “controls” all other pieces in our CI.

1 Like