Upload package to registry using twine fails: certificate verify failed: unable to get local issuer certificate

aviv.quantum-art · June 28, 2023, 6:58am

Hi,

I am using a local GitLab instance (ver. 16.1). I have installed a certificate and everything works well: Git operations using git clients (command-line, Sourcetree) and WEB access as well.
Lately I tried to use CI/CD jobs to upload a python package to project package registry using twine. I followed the documentation and performed:
$ TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

The fob failed with the following error:
File “/usr/lib/python3.10/ssl.py”, line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

Here is the job definition:
create-dist:
stage: deploy
script:
- pip install build twine
- python -m build
- echo TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

I have tried the same on my machine (Mac) with my own credentials instead of the job token, but I get the same error.

What may be the problem? Is it in the client or server? Appreciate your help.

Thanks,
Aviv

paula.kokic · June 28, 2023, 8:10am

Hi Aviv,

You need to make sure that your CA certificate is successfully installed not just on the GitLab Runner, but also inside a docker image that performs the job (in this case some version of Python I believe, as I don’t see the image definition?) Please have a look at official documentation for details.

However, from my experience (as I spent so much time on custom certificate topic), this is how I solve the problem of needing the CA inside any docker job:

I define a CA_CERT variable (as a file variable) in a top level group (or instance level) with full certificate chain of CA that signed certs for GitLab Server

I use that variable whenever some of my scripts need it. E.g. my pypi upload with twine looks like this:

upload package:
  stage: package
  image: python:3.10-alpine
  needs:
    - "build package"
  variables:
    TWINE_USERNAME: 'gitlab-ci-token'
    TWINE_PASSWORD: '${CI_JOB_TOKEN}'
    TWINE_REPOSITORY_URL: '${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi'
  before_script:
    - pip install twine
  script:
    - python -m twine upload --cert $CA_CERT dist/*
  only:
    - main

Of course, I am aware there also exists predefined variable CI_SERVER_TLS_CA_FILE which you could also use for this case. But in my case with some tools this didn’t work as they needed the full certificate chain, which apparently was not provided with this variable… Maybe I did something wrong though, so you may try with this variable first instead of custom CA_CERT.

I’m not sure why it wouldn’t work on your Mac though, but perhaps also the full chain might be missing.

Hope this helps!

aviv.quantum-art · June 28, 2023, 11:04am

Hi Paula,

Thanks for your answer! It looks like the right approach, but I have a few questions:

Not sure I use a docker image. Is it the default for CI jobs? I didn’t define any image
How do I get the “full certificate chain of CA that signed certs for GitLab Server”? Is it the same CRT file I use for GitLab?
nginx[‘ssl_certificate’] = “/etc/gitlab/ssl/gitlab.quantum-art.tech1.crt”
As I get the same behavior when I run on my Mac, I assume you are right and I am missing the full chain

To complete the picture, here is the full output when I run it locally on my mac:
Avivs-MacBook-Pro:$ python -m twine upload --repository-url https://gitlab.quantum-art.tech:4434/projects/5/packages/pypi --cert cert.crt dist/*
Uploading distributions to https://gitlab.quantum-art.tech:4434/projects/5/packages/pypi
Enter your username: user
Enter your password:
Uploading pulse_generator-0.0.1-py3-none-any.whl
WARNING Retrying (Retry(total=9, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=8, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=7, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=6, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=5, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=4, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=3, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=2, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=1, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
WARNING Retrying (Retry(total=0, connect=5, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’))’: /projects/5/packages/pypi
0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/25.9 kB • --:-- • ?
Traceback (most recent call last):
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 467, in _make_request
self._validate_conn(conn)
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 1092, in _validate_conn
conn.connect()
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connection.py”, line 635, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connection.py”, line 774, in ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/urllib3/util/ssl.py", line 459, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/urllib3/util/ssl.py", line 503, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py”, line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py”, line 1075, in _create
self.do_handshake()
File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py”, line 1346, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 491, in _make_request
raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/opt/homebrew/lib/python3.11/site-packages/requests/adapters.py”, line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 874, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 874, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 874, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
[Previous line repeated 7 more times]
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 844, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/urllib3/util/retry.py”, line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘gitlab.quantum-art.tech’, port=4434): Max retries exceeded with url: /projects/5/packages/pypi (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “”, line 198, in _run_module_as_main
File “”, line 88, in _run_code
File “/opt/homebrew/lib/python3.11/site-packages/twine/main.py”, line 51, in
sys.exit(main())
^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/twine/main.py”, line 33, in main
error = cli.dispatch(sys.argv[1:])
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/twine/cli.py”, line 123, in dispatch
return main(args.args)
^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/twine/commands/upload.py”, line 198, in main
return upload(upload_settings, parsed_args.dists)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/twine/commands/upload.py”, line 142, in upload
resp = repository.upload(package)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/twine/repository.py”, line 186, in upload
resp = self._upload(package)
^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/twine/repository.py”, line 172, in _upload
resp = self.session.post(
^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/requests/sessions.py”, line 637, in post
return self.request(“POST”, url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/requests/sessions.py”, line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/requests/sessions.py”, line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/homebrew/lib/python3.11/site-packages/requests/adapters.py”, line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host=‘gitlab.quantum-art.tech’, port=4434): Max retries exceeded with url: /projects/5/packages/pypi (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)’)))

paula.kokic · June 28, 2023, 11:44am

Hi Aviv,

Okay, I was a bit too fast.

Actually I assumed you use GitLab Runner with docker executor (because otherwise you probably wouldn’t have this issue, if certs on Runner are missing, it wouldn’t even be able to do git clone). But let’s check the following things:

you are using your own self-hosted GitLab instance → if yes, this means you have your own GitLab Runners as well → What executor are you using (docker, shell, etc)?

No. This is your GitLab certificate. This certificate was signed by a CA, which also has a certificate - well, this certificate you need. So, now again I assume you have a self-signed CA, otherwise you probably would not have this problem, but the best would be to check with your GitLab administrators or whoever provided you the .crt in the first place - figure out who signed you this certificate. If this is your company system admins, I’m sure they will understand the problem and provide you CA certificate in full chain. GitLab docs writes about it here
The same CA certificates should be installed on your Mac as well.

Bottom line: what is happening here is that your clients (Runner/docker image/Mac) do not have a full chain of trust for the CA that signed your GitLab’s certificate. This is why they don’t trust it. You fix it by installing full chain for your own CA on every client that needs to communicate with GitLab.

Hope this helps!

aviv.quantum-art · June 28, 2023, 12:15pm

Hi Paula,

It is a shell executor:
Running with gitlab-runner 15.10.1 (dcfb4b66)

on gitlab-server WCjhXdbL, system ID: s_348dda0d4a30*
Preparing the “shell” executor
00:00
Using Shell (bash) executor…

The certificate was issued by GoDaddy, and you are probably right, it doesn’t contain the full chain. I will contact our sys admin to get the the rest
Once I get the full chain I will try that

Just curious: Why all other git operations (e.g. pull, push) succeed, and only this requires the full CA chain?

Thank you very much for your help!
Best,
Aviv

paula.kokic · June 28, 2023, 12:59pm

I’ve just checked, seems like twine does not care about certificates installed on your system, but instead uses only certificates he gets via Requests / Certifi, so if your CA is not part of that list, you need to provide it extra to twine. That might explain why your git clone/fetch works, but not twine commands.

You could still try to just pass in CI_SERVER_TLS_CA_FILE to your twine command - perhaps it does the trick, as that would probably be the easiest solution.

aviv.quantum-art · July 4, 2023, 10:16am

Hi Paula,

Sorry for the long pause. I was busy with other things…
Just to conclude this: The issue was that the certificate installed on the GitLab server was partial and not full-chain. It was good enough for browsing and git commands but not for connections based on openssl like twine, both on Linux and Mac.
It took some time to get hold of a compatible format full-chain certificate, but once we configured in as the gitlab crt file, the issue was resolved.

Thanks for your help!
Aviv