CI variables missing when triggering build based on Tag [SOLVED]

I’m trying to build and push tcx2gpx to PyPI automatically and wish to do so only when a tag is applied to a commit so that only versions I want to release get built and uploaded.

Currently I’m testing against Test PyPI using twine and tokens and so I’ve added variabes to the CI/CD for $TWINE_USERNAME (as __token__) and $TWINE_PASSWORD (as the token value itself). The options for both of these variables are set as…

IMPORTANT - I have protected both $TWINE_USERNAME and $TWINE_PASSWORD in their settings by enabling “Protect variable : Export variable to pipelines running on protected branches and tags only”.

I started with the following in my .gitlab-ci.yaml

pypi:
    stage: pypi
    rules:
        - if: $CI_COMMIT_BRANCH == "master"
        # - if: $CI_COMMIT_TAG
    script:
        - pip install .[pypi]
        - pip install build
        - python -m build
        - twine upload --repository testpypi dist/*

…and this works. I’m using setuptools_scm to automatically version and I get packages with post### uploaded (e.g. 0.1.4.post140 and 0.1.5rc3.post3).

This is great it works, however, as mentioned I don’t want everything merged into master to trigger a build and upload so instead I switch the rules to use $CI_COMMIT_TAG and not $CI_COMMIT_BRANCH == "master"…

pypi:
    stage: pypi
    rules:
        # - if: $CI_COMMIT_BRANCH == "master"
        - if: $CI_COMMIT_TAG
    script:
        - pip install .[pypi]
        - pip install build
        - python -m build
        - twine upload --repository testpypi dist/*

This results in the job only being triggered if I commit a tag, which is what I want, however uploading fails with…

Successfully built tcx2gpx-0.1.5rc5.tar.gz and tcx2gpx-0.1.5rc5-py3-none-any.whl
$ twine upload --repository testpypi dist/*
Uploading distributions to https://test.pypi.org/legacy/
Enter your username: Traceback (most recent call last):
  File "/usr/local/bin/twine", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/__main__.py", line 33, in main
    error = cli.dispatch(sys.argv[1:])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/cli.py", line 123, in dispatch
    return main(args.args)
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/commands/upload.py", line 198, in main
    return upload(upload_settings, parsed_args.dists)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/commands/upload.py", line 127, in upload
    repository = upload_settings.create_repository()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/settings.py", line 329, in create_repository
    self.username,
    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/settings.py", line 131, in username
    return cast(Optional[str], self.auth.username)
                               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/auth.py", line 34, in username
    return utils.get_userpass_value(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/utils.py", line 248, in get_userpass_value
    value = prompt_strategy()
            ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/auth.py", line 85, in username_from_keyring_or_prompt
    return self.prompt("username", input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twine/auth.py", line 96, in prompt
    return how(f"Enter your {what}: ")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: EOF when reading a line
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

To investigate this I added a short debug job and allowed these variables to be shown in the logs…

print-all-env-vars-job:
    stage: debug
    script:
        - echo "GitLab CI/CD | Print all environment variables"
        - env

…and re-ran the pipelines using both rules (i.e. one to only build on master and one to run only on tags). Saved the env output and diffed them and it transpires that the $TWINE_USERNAME and $TWINE_PASSWORD variables are not present when running the pipeline on $CI_COMMIT_TAG and so twine is not getting the value of $TWINE_USERNAME passed to it (nor would it get $TWINE_PASSWORD if it got that far) which is I think the cause of the failure.

I’m confused by this because as I noted above I have protected these variables and they should only be _Exported to pipelines running on protected branches and tags only_ (i.e. master or commits with tags) and would therefore expect them to be exported on pipelines that are triggered by tags.

Any suggestions on what I’ve done wrong would be gratefully received.

Solved

I was unaware that you could protect tags on GitLab, having done this (Settings > Repository > Protected Tags and setting an appropriate wildcard) the pipeline/job runs as expected.

Thanks to users on Mastodon who saw my toot about this and provided the solution.

Thank you this was very helpful!