Registry.gitlab.com certificate not recognized

Hello
I have a problem with registry.gitlab.com certificate it’s not recognized and not trusted so I can’t pull any image form registry.gitlab.com other images from docker hub for example or my own registry works fine

my pipeline STDOUT

Running with gitlab-runner 14.5.2 (e91107dd)
on Shared Docker Runner V2fXbG8p
Preparing the “docker” executor
00:10
Using Docker executor with image registry.gitlab.com/pipeline-components/yamllint:latest
Pulling docker image registry.gitlab.com/pipeline-components/yamllint:latest
WARNING: Failed to pull image with policy “always”: error pulling image configuration: download failed after attempts=6: x509: certificate signed by unknown authority (manager.go:203:7s)
ERROR: Job failed: failed to pull image “registry.gitlab.com/pipeline-components/yamllint:latest” with specified policies [always]: error pulling image configuration: download failed after attempts=6: x509: certificate signed by unknown authority (manager.go:203:7s)

samething if I do docker pull registry.gitlab.com/pipeline-components/yamllint:latest

docker pull registry.gitlab.com/pipeline-components/yamllint
Using default tag: latest
latest: Pulling from pipeline-components/yamllint
339de151aab4: Retrying in 1 second
f10fb83f8d87: Retrying in 1 second
f6e0acdc69e4: Retrying in 1 second
70bbe8425d57: Waiting
16dfeab63973: Waiting
b5861c94c899: Waiting
353d9ab46cbe: Waiting
4f4fb700ef54: Waiting
db027001410d: Waiting
943155464b8c: Waiting
error pulling image configuration: download failed after attempts=6: x509: certificate signed by unknown authority

it’s like the ca certificate is not recognized by the system which is weird I don’t have the problem on my desktop machine which runs on debian last version.

My gitlab server is installed on an Ubuntu 20.04 LTS with last updates this problem began since last week something like that.

I imported registry.gitlab.com added to certificates list in /etc/ssl/certs update ca-certificates packages even added in /etc/docker/daemon.json

“untrusted-registries” : [“registry.gitlab.com:443”]

But getting the same error

Any Idea ?
Thank you

Hi @tba77

it’s not clear from your post, but I suppose you are using self-signed or custom CA for your TLS certificate.

Whats the output of openssl s_client -connect registry.yourdomain.com:443 on the affected system?

Hi @balonik
No I am not using a self signed certificate my gitlab server works fine and everything is OK it’s just when I connect to registry.gitlab.com I get unknown certificate which means that the certificate chain is probably not installed on the machine in which gitlab-runner is installed which is an ubuntu 20.04 I tried to update certificates remove and reinstall ca-certificates without success I have issue only with registry.gitlab.com

registry.gitlab.com is domain for GitLab SaaS gitlab.com registry. If you are really trying to connect to domain registry.gitlab.com you are trying to connect to gitlab.com registry and not registry of your own GitLab server.

If that is the case and you have issue with the official registry.gitlab.com domain I would suggest to run command I have asked above to see how your OS handles that connection.

Yes that’s why I don’t understand connection to registry.gitlab.com shoudn’t have this kind of behavior.

I will check DNS first because I haven’t that kind of problem before and I am thinking that the connection to public DNS has been shutdown

 openssl s_client -connect registry.gitlab.com:443
CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
 1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFgzCCBSigAwIBAgIQBiPZw4be7paOmVGMBVHSLDAKBggqhkjOPQQDAjBKMQsw
CQYDVQQGEwJVUzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEgMB4GA1UEAxMX
Q2xvdWRmbGFyZSBJbmMgRUNDIENBLTMwHhcNMjIwNzA0MDAwMDAwWhcNMjIxMDAy
MjM1OTU5WjBqMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjET
MBEGA1UEAxMKZ2l0bGFiLmNvbTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABACu
ir0FpY/Td6QwLGHdBMssRJ9jEvqFqYj2y6K9ZQkiKbgzITxLPzUX5it8/ctNGiqn
+KOhQQefGKkRWYTzs5qjggPOMIIDyjAfBgNVHSMEGDAWgBSlzjfq67B1DpRniLRF
+tkkEIeWHzAdBgNVHQ4EFgQUp7B0Ubbe4FBSgpW+ucHTDarKhd4wgZQGA1UdEQSB
jDCBiYITcGFja2FnZXMuZ2l0bGFiLmNvbYIUY3VzdG9tZXJzLmdpdGxhYi5jb22C
CmdpdGxhYi5jb22CD2NoZWYuZ2l0bGFiLmNvbYIaZW1haWwuY3VzdG9tZXJzLmdp
dGxhYi5jb22CDmthcy5naXRsYWIuY29tghNyZWdpc3RyeS5naXRsYWIuY29tMA4G
A1UdDwEB/wQEAwIHgDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwewYD
VR0fBHQwcjA3oDWgM4YxaHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0Nsb3VkZmxh
cmVJbmNFQ0NDQS0zLmNybDA3oDWgM4YxaHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0Nsb3VkZmxhcmVJbmNFQ0NDQS0zLmNybDA+BgNVHSAENzA1MDMGBmeBDAECAjAp
MCcGCCsGAQUFBwIBFhtodHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwdgYIKwYB
BQUHAQEEajBoMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20w
QAYIKwYBBQUHMAKGNGh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9DbG91ZGZs
YXJlSW5jRUNDQ0EtMy5jcnQwDAYDVR0TAQH/BAIwADCCAX0GCisGAQQB1nkCBAIE
ggFtBIIBaQFnAHUAKXm+8J45OSHwVnOfY6V35b5XfZxgCvj5TV0mXCVdx4QAAAGB
xronxwAABAMARjBEAiA5IrJ05KRTZhpfPx4F0q+4fPYuXsURNIwl2+IDIXGfPgIg
GNMXcN5Ls2LPn5I8FJQ5PRzG+N/ipXIKym3BaHLNnP8AdgBByMqx3yJGShDGoToJ
QodeTjGLGwPr60vHaPCQYpYG9gAAAYHGuigJAAAEAwBHMEUCIE2XyQm+tVEVrmPk
qOTjT4LZR1cjzflhSWd6Gm9ydHG6AiEArEldk9K6kKLCLr1TsniI5FbyDoG86s83
+O1XPL+5swkAdgDfpV6raIJPH2yt7rhfTj5a6s2iEqRqXo47EsAgRFwqcwAAAYHG
uifLAAAEAwBHMEUCIAtAsPWw7ExMrIDeYiFYLuckjrqyTl4aRdVf3EdmgXFiAiEA
gMfVbiysb2r+G6pqUIm3IKZf1qzUCMgcQE0QDxd02mgwCgYIKoZIzj0EAwIDSQAw
RgIhAJq7U7Fasv+fIk/j1dlplJxovxE3YQTThZ/WnGmylkt/AiEAmLbVsjNPtKZW
sSwAYSAKFTsYqEWJLHbP9zi2dCvHtH4=
-----END CERTIFICATE-----
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com

issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2707 bytes and written 391 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 9C07AE13B35CC3B44511B862D6C7083EE93D31133EBD9F682FB30B0F1DA91E9D
    Session-ID-ctx: 
    Resumption PSK: E0B24D21A2A377E06447A3B20C7447A56BC133D6B75EC1BC8CA4CF3E6E722FB12662F483059482B5C6057DA1820F1E2D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - 71 47 2c 7d 94 7f 92 d4-33 25 f1 62 e7 d2 57 9f   qG,}....3%.b..W.
    0010 - bd 77 57 d9 4b c8 03 b9-6c 89 cb 5d 81 26 14 2b   .wW.K...l..].&.+
    0020 - 04 07 c5 50 97 41 0d 25-47 12 fc bf 24 df ad 51   ...P.A.%G...$..Q
    0030 - cc 97 86 6a 8e 13 ed 56-cc 4f d2 cd 98 04 69 43   ...j...V.O....iC
    0040 - ab 04 86 d9 c4 a1 83 f2-32 42 9a 90 37 05 ec 53   ........2B..7..S
    0050 - ea 21 6c 93 93 e2 2a 0c-d1 b2 0e f5 4b d7 7a b2   .!l...*.....K.z.
    0060 - 81 ce f4 7b 6c 9a 97 d6-f6 f2 e9 33 97 13 d6 ba   ...{l......3....
    0070 - 8e 48 8b 2f d3 6d 04 39-4c 0e b3 b6 93 b6 25 43   .H./.m.9L.....%C
    0080 - bd 23 cc 09 e7 ce 59 9f-70 2f 35 1c d2 84 d2 11   .#....Y.p/5.....
    0090 - 67 f0 3a 09 79 11 c1 c5-7a 6f b6 1c a1 65 4d 1e   g.:.y...zo...eM.
    00a0 - ee cf f7 e0 25 c1 b0 17-aa 17 8a 10 0f 81 fd 4a   ....%..........J
    00b0 - ac a3 8b c6 2d ae e1 bd-92 8d 96 57 38 fc c3 80   ....-......W8...

    Start Time: 1658303898
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 87A43A4B37249CDE9AF0A32C90A9CB1C47CAF0E7284A6E8D30E15C2774754A3E
    Session-ID-ctx: 
    Resumption PSK: 294DC23FDCAEF66DF26354A6357C9EE757D17FEDAD889F48D867C0D79CECBB3EAB54F1584A6D6BC6ACC031AB65A0DE5D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - 71 47 2c 7d 94 7f 92 d4-33 25 f1 62 e7 d2 57 9f   qG,}....3%.b..W.
    0010 - 77 29 13 5a 4b 87 cd 76-13 f6 77 5b 44 6a 81 23   w).ZK..v..w[Dj.#
    0020 - 73 27 b7 1b 54 09 ff 12-f6 d0 b9 8c bc 30 3c d4   s'..T........0<.
    0030 - ba 35 b6 6a 68 24 88 f7-ae 5e 1a e1 f7 2a f5 5a   .5.jh$...^...*.Z
    0040 - d8 64 77 91 2f 4b 68 60-28 67 c5 2d 00 66 dc 2e   .dw./Kh`(g.-.f..
    0050 - 41 c8 2a 32 cf 32 49 7b-86 a0 54 fe fd 12 9b 3b   A.*2.2I{..T....;
    0060 - 0c 34 17 76 ad df 17 5c-77 1b 5f d9 a7 32 3a 18   .4.v...\w._..2:.
    0070 - 0e 3a d2 8f d9 4d bb 20-ca 9b b7 4f 91 68 7e 65   .:...M. ...O.h~e
    0080 - 06 9d 05 df f9 da b6 85-6f 3c cd 85 76 2a ba 54   ........o<..v*.T
    0090 - cd 17 fe d4 df d1 99 70-c1 e9 b1 03 5a 2b 22 f1   .......p....Z+".
    00a0 - a8 27 f3 fd 9e b6 ce 76-d6 52 db 6b ad e3 ae 90   .'.....v.R.k....
    00b0 - d3 b3 bf 38 0e d7 c8 a5-38 ab 5a 8b 69 ad 36 5c   ...8....8.Z.i.6\

    Start Time: 1658303898
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
docker pull  registry.gitlab.com/pipeline-components/yamllint
Using default tag: latest
latest: Pulling from pipeline-components/yamllint
339de151aab4: Retrying in 1 second 
f10fb83f8d87: Retrying in 1 second 
f6e0acdc69e4: Retrying in 1 second 
70bbe8425d57: Waiting 
16dfeab63973: Waiting 
b5861c94c899: Waiting 
353d9ab46cbe: Waiting 
4f4fb700ef54: Waiting 
db027001410d: Waiting 
943155464b8c: Waiting 
error pulling image configuration: download failed after attempts=6: x509: certificate signed by unknown authority

so your OS ca bundle is OK, no issue there. Also please note that you need to restart Docker daemon after making any changes to OS CA trust bundle. Docker loads it only during startup.

Do you have anything in /etc/docker/certs.d/?

I’m thinking along the same lines as @balonik, so we might overlook the same problem (but it does lower the chance of that).

It won’t fix your issue, but a way to check whether your OS recognises the cert is to visit registry.gitlab.com in a browser. If you get an error message, something doesn’t recognise it (but as it is valid to everyone else, I would say that means that something (which is probably your OS) is broken), if you get a blank page (you’re not supposed to access that page so there is nothing there), something has accepted the cert

I don’t recommend using browsers, because some browsers have their own CA trust bundles and are not using OS ones (like Firefox and it’s clones). Unless you know that your browser is really using the OS one.

The correct-correct-correct way of checking it is to also specify against what CAs openssl should verify server cert:

openssl s_client -servername www.example.com -connect www.example.com:443 -CApath /etc/ssl/certs

but usually skipping the -CApath is ok, because Linux distro bundled openssl is using it by default.

For distros where openssl is not available I usually use curl :smiley:

I also don’t recommend using Java or Python based apps, both are using their own CA bundles.

@balonik I downloaded and installed gitlab.com chain certificate

ls /etc/docker/certs.d/*/*
/etc/docker/certs.d/registry.gitlab.com/gitlab-ca.crt

I use curl also and it seems working

curl -v https://registry.gitlab.com
*   Trying 104.18.26.123:443...
* TCP_NODELAY set
* Connected to registry.gitlab.com (104.18.26.123) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=gitlab.com
*  start date: Jul  4 00:00:00 2022 GMT
*  expire date: Oct  2 23:59:59 2022 GMT
*  subjectAltName: host "registry.gitlab.com" matched cert's "registry.gitlab.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b8319ae2f0)
> GET / HTTP/2
> Host: registry.gitlab.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Wed, 20 Jul 2022 09:04:54 GMT
< content-length: 0
< cache-control: no-cache
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=hZsr2kP5Ds0EqPvFjYfQ4mfvs6QeRAgU%2F8aDwYY8gADk5kYQX9iH7zc7nlT8dNk65XjUn3BF2DVIGgLdznSsr%2BlT1xRTFnsNUCpwj%2FifCyxFZ7j7b55VUT7v6jbw%2Bcx%2FNAPTyOQ%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
< strict-transport-security: max-age=31536000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 72da88349f550fd6-MRS
< 
* Connection #0 to host registry.gitlab.com left intact
openssl s_client -servername registry.gitlab.com -connect registry.gitlab.com:443 -CApath /etc/ssl/certs

CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
 1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFgzCCBSigAwIBAgIQBiPZw4be7paOmVGMBVHSLDAKBggqhkjOPQQDAjBKMQsw
CQYDVQQGEwJVUzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEgMB4GA1UEAxMX
Q2xvdWRmbGFyZSBJbmMgRUNDIENBLTMwHhcNMjIwNzA0MDAwMDAwWhcNMjIxMDAy
MjM1OTU5WjBqMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjET
MBEGA1UEAxMKZ2l0bGFiLmNvbTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABACu
ir0FpY/Td6QwLGHdBMssRJ9jEvqFqYj2y6K9ZQkiKbgzITxLPzUX5it8/ctNGiqn
+KOhQQefGKkRWYTzs5qjggPOMIIDyjAfBgNVHSMEGDAWgBSlzjfq67B1DpRniLRF
+tkkEIeWHzAdBgNVHQ4EFgQUp7B0Ubbe4FBSgpW+ucHTDarKhd4wgZQGA1UdEQSB
jDCBiYITcGFja2FnZXMuZ2l0bGFiLmNvbYIUY3VzdG9tZXJzLmdpdGxhYi5jb22C
CmdpdGxhYi5jb22CD2NoZWYuZ2l0bGFiLmNvbYIaZW1haWwuY3VzdG9tZXJzLmdp
dGxhYi5jb22CDmthcy5naXRsYWIuY29tghNyZWdpc3RyeS5naXRsYWIuY29tMA4G
A1UdDwEB/wQEAwIHgDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwewYD
VR0fBHQwcjA3oDWgM4YxaHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0Nsb3VkZmxh
cmVJbmNFQ0NDQS0zLmNybDA3oDWgM4YxaHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0Nsb3VkZmxhcmVJbmNFQ0NDQS0zLmNybDA+BgNVHSAENzA1MDMGBmeBDAECAjAp
MCcGCCsGAQUFBwIBFhtodHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwdgYIKwYB
BQUHAQEEajBoMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20w
QAYIKwYBBQUHMAKGNGh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9DbG91ZGZs
YXJlSW5jRUNDQ0EtMy5jcnQwDAYDVR0TAQH/BAIwADCCAX0GCisGAQQB1nkCBAIE
ggFtBIIBaQFnAHUAKXm+8J45OSHwVnOfY6V35b5XfZxgCvj5TV0mXCVdx4QAAAGB
xronxwAABAMARjBEAiA5IrJ05KRTZhpfPx4F0q+4fPYuXsURNIwl2+IDIXGfPgIg
GNMXcN5Ls2LPn5I8FJQ5PRzG+N/ipXIKym3BaHLNnP8AdgBByMqx3yJGShDGoToJ
QodeTjGLGwPr60vHaPCQYpYG9gAAAYHGuigJAAAEAwBHMEUCIE2XyQm+tVEVrmPk
qOTjT4LZR1cjzflhSWd6Gm9ydHG6AiEArEldk9K6kKLCLr1TsniI5FbyDoG86s83
+O1XPL+5swkAdgDfpV6raIJPH2yt7rhfTj5a6s2iEqRqXo47EsAgRFwqcwAAAYHG
uifLAAAEAwBHMEUCIAtAsPWw7ExMrIDeYiFYLuckjrqyTl4aRdVf3EdmgXFiAiEA
gMfVbiysb2r+G6pqUIm3IKZf1qzUCMgcQE0QDxd02mgwCgYIKoZIzj0EAwIDSQAw
RgIhAJq7U7Fasv+fIk/j1dlplJxovxE3YQTThZ/WnGmylkt/AiEAmLbVsjNPtKZW
sSwAYSAKFTsYqEWJLHbP9zi2dCvHtH4=
-----END CERTIFICATE-----
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com

issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2707 bytes and written 391 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 0D1330D2127464DA0ABC5784FF68ED9B0C22FDF59A260F5E5EF4CAFB455B68A8
    Session-ID-ctx: 
    Resumption PSK: 81D999F3AE40AB60ECC2C43D7A189604ED7694221C2511FC34D07A94A8B4A235AE65AC1226C185B4C7AEFD3EA3F66824
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - 74 65 3b b4 8f 26 af 96-93 91 17 b6 78 70 14 40   te;..&......xp.@
    0010 - 84 c5 59 fe ad 46 19 b6-00 68 7b 05 29 54 03 52   ..Y..F...h{.)T.R
    0020 - be 21 89 08 85 61 7b 3a-67 20 43 89 b0 c0 2b 60   .!...a{:g C...+`
    0030 - 90 10 05 4b 91 aa 09 62-4e 45 2e ed 09 7a 29 ae   ...K...bNE...z).
    0040 - 69 8c 93 5d 5b 32 fd e5-9b e9 1e 55 5d e8 8f 96   i..][2.....U]...
    0050 - 14 07 7e 8b c7 74 a0 bb-28 f7 9f c1 00 43 0e a1   ..~..t..(....C..
    0060 - f5 70 17 ba c3 8e c5 40-8f 33 17 97 20 d3 4b 77   .p.....@.3.. .Kw
    0070 - ab 54 97 69 f4 96 be 5d-3a de 6f f3 a9 2c 85 a4   .T.i...]:.o..,..
    0080 - d1 c7 9a f6 25 3e 35 29-49 b6 eb a9 6c 96 31 00   ....%>5)I...l.1.
    0090 - f7 99 22 eb 79 3f d6 5c-36 a5 28 20 fb 85 7c 2c   ..".y?.\6.( ..|,
    00a0 - 1d 08 f7 f3 b8 ab 84 6e-5e 10 e6 0f bf db 65 60   .......n^.....e`
    00b0 - 01 85 de 1f c3 6b 0d d9-03 cf f8 28 d5 24 ff 3c   .....k.....(.$.<

    Start Time: 1658308024
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: ED6E4AE2C0EFF15E2BF263791BDACBC36798561047237CCF96AAC540318F7C37
    Session-ID-ctx: 
    Resumption PSK: 468AC38420A667AE674367C60C5DACAD626F5B963C2ED57E6379F5F945F02E01AD7206530F555331AE59BDFEE6C4C887
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - 74 65 3b b4 8f 26 af 96-93 91 17 b6 78 70 14 40   te;..&......xp.@
    0010 - 48 1e 53 2f e8 29 2e 4d-74 ba d0 a4 af 0c 5f 2b   H.S/.).Mt....._+
    0020 - b1 8d 53 cd de 1a 60 27-b0 ad fa 14 72 1a 0b da   ..S...`'....r...
    0030 - a7 2e ed 51 44 0f 8c 16-89 fa 3e 1f 70 b1 70 0d   ...QD.....>.p.p.
    0040 - 21 d9 42 57 64 2f 54 5c-0a 11 43 ea c0 b7 62 ef   !.BWd/T\..C...b.
    0050 - 93 70 ec 49 89 54 a3 4c-d8 f9 7c 15 f5 2d 3b e2   .p.I.T.L..|..-;.
    0060 - e4 30 3a ce 3e e1 3a c5-cf 57 5f d4 07 ad d6 95   .0:.>.:..W_.....
    0070 - a3 54 7b d2 88 e4 e8 f5-ef 3a 00 a9 50 73 5d b3   .T{......:..Ps].
    0080 - e8 5d ab e7 44 99 28 29-7c 36 c8 44 b5 22 aa 47   .]..D.()|6.D.".G
    0090 - 27 1b 9b 81 a5 ca a6 cf-2a 77 2b 8e e4 5b dc 65   '.......*w+..[.e
    00a0 - 0a 24 9a 5c 07 c5 04 70-9c 08 6b 59 75 99 70 53   .$.\...p..kYu.pS
    00b0 - fb b8 37 10 ef 11 c2 52-94 74 4f 63 9e ce 87 6a   ..7....R.tOc...j

    Start Time: 1658308024
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

Just for my curiosity where did you downloaded the gitlab.com chain from? GitLab is using Cloudflare.

The solution here is to remove the /etc/docker/certs.d/registry.gitlab.com directory completely. It most likely contains incorrect certs and it is not needed since Docker will use OS chain by default.

Also remove any other changes to Docker daemon config. And don’t forget to restart Docker daemon :smiley:

I did it and yes restarted docker at any modification :slight_smile:
I’ll try it again hoping it would work this time

thank you :slight_smile:

Update : samething it didn’t work

docker pull  registry.gitlab.com/pipeline-components/yamllint
Using default tag: latest
latest: Pulling from pipeline-components/yamllint
339de151aab4: Retrying in 1 second 
f10fb83f8d87: Retrying in 1 second 
f6e0acdc69e4: Retrying in 1 second 
70bbe8425d57: Waiting 
16dfeab63973: Waiting 
b5861c94c899: Waiting 
353d9ab46cbe: Waiting 
4f4fb700ef54: Waiting 
db027001410d: Waiting 
943155464b8c: Waiting 
error pulling image configuration: download failed after attempts=6: x509: certificate signed by unknown authority

Is there an HTTP_PROXY or HTTPS_PROXY configured for the Docker environment? It looks like something is intercepting the traffic, but only within Docker itself (curl, openssl ruled out the system environment tests).

/etc/docker/* configurations would be interesting. Also, Systemd seems involved, the service unit environment configuration is needed.

cat /etc/docker/daemon.json

systemctl show docker.service

Which version of Docker and how was it installed?

docker -v
docker info 
apt-cache show docker 

Knowing these details will help reproduce the environment, and potentially make better suggestions.

Hi @dnsmichi no there is no proxy configured. I created a config file daemon.json in which I added my private registry

ls /etc/docker             
certs.d  daemon.json  key.json

sudo cat /etc/docker/daemon.json 
{
    "registry-mirrors": ["https://registry.mydomain.com"],
    "live-restore": true
}

systemd informations

sudo systemctl show docker.service
Type=notify
Restart=always
NotifyAccess=main
RestartUSec=2s
TimeoutStartUSec=infinity
TimeoutStopUSec=infinity
TimeoutAbortUSec=infinity
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=395819
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ReloadResult=success
CleanResult=success
UID=[not set]
GID=[not set]
NRestarts=0
OOMPolicy=continue
ExecMainStartTimestamp=Wed 2022-07-20 11:51:51 CET
ExecMainStartTimestampMonotonic=162940150181
ExecMainExitTimestampMonotonic=0
ExecMainPID=395819
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[Wed 2022-07-20 11:51:51 CET] ; stop_time=[>
ExecStartEx={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[Wed 2022-07-20 11:51:51 CET] ; stop_time=[n/a] ; p>
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReloadEx={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=42864640
CPUUsageNSec=[not set]
EffectiveCPUs=
EffectiveMemoryNodes=
TasksCurrent=55
IPIngressBytes=[no data]
IPIngressPackets=[no data]
IPEgressBytes=[no data]
IPEgressPackets=[no data]
IOReadBytes=18446744073709551615
IOReadOperations=18446744073709551615
IOWriteBytes=18446744073709551615
IOWriteOperations=18446744073709551615
Delegate=yes
DelegateControllers=cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
CPUQuotaPeriodUSec=infinity
AllowedCPUs=
AllowedMemoryNodes=
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=infinity
IPAccounting=no
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=infinity
LimitNOFILESoft=infinity
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=infinity
LimitNPROCSoft=infinity
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=23291
LimitSIGPENDINGSoft=23291
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=-500
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUAffinity=
CPUAffinityFromNUMA=no
NUMAPolicy=n/a
NUMAMask=
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
LogRateLimitIntervalUSec=0
LogRateLimitBurst=0
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadc>
AmbientCapabilities=
DynamicUser=no
RemoveIPC=no
MountFlags=
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectKernelLogs=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
PrivateMounts=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0755
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
TimeoutCleanUSec=infinity
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictSUIDSGID=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
ProtectHostname=no
KillMode=process
KillSignal=15
RestartKillSignal=15
FinalKillSignal=9
SendSIGKILL=yes
SendSIGHUP=no
WatchdogSignal=6
Id=docker.service
Names=docker.service
Requires=docker.socket sysinit.target system.slice containerd.service
Wants=network-online.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=containerd.service docker.socket firewalld.service systemd-journald.socket basic.target sysinit.target system.slice network-online.target
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/lib/systemd/system/docker.service
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Wed 2022-07-20 11:51:52 CET
StateChangeTimestampMonotonic=162940694568
InactiveExitTimestamp=Wed 2022-07-20 11:51:51 CET
InactiveExitTimestampMonotonic=162940150624
ActiveEnterTimestamp=Wed 2022-07-20 11:51:52 CET
ActiveEnterTimestampMonotonic=162940694568
ActiveExitTimestamp=Wed 2022-07-20 11:51:51 CET
ActiveExitTimestampMonotonic=162940135922
InactiveEnterTimestamp=Wed 2022-07-20 11:51:51 CET
InactiveEnterTimestampMonotonic=162940146022
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Wed 2022-07-20 11:51:51 CET
ConditionTimestampMonotonic=162940147495
AssertTimestamp=Wed 2022-07-20 11:51:51 CET
AssertTimestampMonotonic=162940147496
Transient=no
Perpetual=no
StartLimitIntervalUSec=1min
StartLimitBurst=3
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=1b540411e7324714a3fc2345d66fe703
CollectMode=inactive
docker info   
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-122-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 5.772GiB
 Name: Docker-Server1
 ID: WVEM:V2OO:IL3C:PRFQ:U7ED:A5S5:IRQA:LU3P:XQUC:TWTW:FFRA:33QH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: tba77
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://registry.endatamweel.tn/
 Live Restore Enabled: true

WARNING: No swap limit support
sudo apt-cache madison docker-ce
 docker-ce | 5:20.10.17~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:20.10.16~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:20.10.15~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:20.10.14~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:57 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:03 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Hello I changed my installation to a new server pulling images from registry.gitlab.com worked fine until I restored my gitlab backup to get my production environment back.
So the problems seems not related to my operation system but something changed in docker configuration I can’t explain what happened and why.

Thank you for your help

I’m running into the same issue, thought I had fixed it by bringing in CA certs and updating my ubuntu VM. But this week, the same error is occurring. Any other thoughts on what might be happening? I’ve enabled debug mode in my docker daemon.json, but was hoping for something more like javax.net.debug=ssl, so I could see more of the ssl handshake.