Lets Encrypt Error

CookieST · December 7, 2019, 9:54am

Hello I’m trying to setup Gitlab for the first time with the Docker image. I set my external url to
the one with https which should auto-deply a free Let’s Encrypt certificate, right?

Anyways I’m experiencing the following issue:

2019-12-07T09:40:11.279058811Z Recipe: letsencrypt::enable
2019-12-07T09:40:11.279409217Z   * ruby_block[http external-url] action run (skipped due to only_if)
2019-12-07T09:40:11.283909627Z   * directory[/etc/gitlab/ssl] action create (up to date)
2019-12-07T09:40:11.537955178Z   * acme_selfsigned[gitlab.DOMAIN] action create
2019-12-07T09:40:11.539566241Z     * file[gitlab.DOMAIN SSL selfsigned key] action create_if_missing (up to date)
2019-12-07T09:40:11.544357668Z     * file[gitlab.DOMAIN SSL selfsigned crt] action create_if_missing (up to date)
2019-12-07T09:40:11.544787505Z     * file[gitlab.DOMAIN SSL selfsigned chain] action create_if_missing (skipped due to not_if)
2019-12-07T09:40:11.545033309Z      (up to date)
2019-12-07T09:40:11.545238267Z Recipe: letsencrypt::http_authorization
2019-12-07T09:40:11.576538847Z   * letsencrypt_certificate[gitlab.DOMAIN] action create
2019-12-07T09:40:11.683970673Z     * acme_certificate[staging] action create
2019-12-07T09:40:11.685229115Z       * file[gitlab.DOMAIN SSL key] action create_if_missing (up to date)
2019-12-07T09:40:14.867445754Z
2019-12-07T09:40:14.867702769Z       ================================================================================
2019-12-07T09:40:14.867970100Z       Error executing action `create` on resource 'acme_certificate[staging]'
2019-12-07T09:40:14.868131905Z       ================================================================================
2019-12-07T09:40:14.868273388Z
2019-12-07T09:40:14.868457388Z       Acme::Client::Error::Malformed
2019-12-07T09:40:14.868594512Z       ------------------------------
2019-12-07T09:40:14.868776002Z       Method not allowed
2019-12-07T09:40:14.868928705Z
2019-12-07T09:40:14.869075815Z       Cookbook Trace:
2019-12-07T09:40:14.869215458Z       ---------------
2019-12-07T09:40:14.869368882Z       /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:77:in `block in class_from_file'
2019-12-07T09:40:14.869496664Z
2019-12-07T09:40:14.869652087Z       Resource Declaration:
2019-12-07T09:40:14.869790135Z       ---------------------
2019-12-07T09:40:14.870103203Z       suppressed sensitive resource output
2019-12-07T09:40:14.870229692Z
2019-12-07T09:40:14.870394451Z       Compiled Resource:
2019-12-07T09:40:14.870586852Z       ------------------
2019-12-07T09:40:14.870863494Z       suppressed sensitive resource output
2019-12-07T09:40:14.870969205Z
2019-12-07T09:40:14.871425871Z       System Info:
2019-12-07T09:40:14.871596631Z       ------------
2019-12-07T09:40:14.871812196Z       chef_version=14.13.11
2019-12-07T09:40:14.871955892Z       platform=ubuntu
2019-12-07T09:40:14.872059333Z       platform_version=16.04
2019-12-07T09:40:14.872155581Z       ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
2019-12-07T09:40:14.872264893Z       program_name=/opt/gitlab/embedded/bin/chef-client
2019-12-07T09:40:14.872412497Z       executable=/opt/gitlab/embedded/bin/chef-client
2019-12-07T09:40:14.872520797Z
2019-12-07T09:40:14.874171546Z
2019-12-07T09:40:14.874344604Z     ================================================================================
2019-12-07T09:40:14.874490974Z     Error executing action `create` on resource 'letsencrypt_certificate[gitlab.DOMAIN]'
2019-12-07T09:40:14.874728793Z     ================================================================================
2019-12-07T09:40:14.874840913Z
2019-12-07T09:40:14.874982901Z     Acme::Client::Error::Malformed
2019-12-07T09:40:14.875122756Z     ------------------------------
2019-12-07T09:40:14.875307999Z     acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: Acme::Client::Error::Malformed: Method not allowed
2019-12-07T09:40:14.875411284Z
2019-12-07T09:40:14.875564514Z     Cookbook Trace:
2019-12-07T09:40:14.875825390Z     ---------------
2019-12-07T09:40:14.876064236Z     /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:77:in `block in class_from_file'
2019-12-07T09:40:14.876173689Z
2019-12-07T09:40:14.876364659Z     Resource Declaration:
2019-12-07T09:40:14.876561139Z     ---------------------
2019-12-07T09:40:14.876670335Z     # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
2019-12-07T09:40:14.876769736Z
2019-12-07T09:40:14.876852990Z       5: letsencrypt_certificate site do
2019-12-07T09:40:14.876980708Z       6:   crt node['gitlab']['nginx']['ssl_certificate']
2019-12-07T09:40:14.877080457Z       7:   key node['gitlab']['nginx']['ssl_certificate_key']
2019-12-07T09:40:14.877175729Z       8:   notifies :run, "execute[reload nginx]", :immediate
2019-12-07T09:40:14.877272515Z       9:   notifies :run, 'ruby_block[display_le_message]'
2019-12-07T09:40:14.877367658Z      10:   only_if { omnibus_helper.service_up?('nginx') }
2019-12-07T09:40:14.877513353Z      11: end
2019-12-07T09:40:14.877647750Z
2019-12-07T09:40:14.877807635Z     Compiled Resource:
2019-12-07T09:40:14.877950481Z     ------------------
2019-12-07T09:40:14.878100998Z     # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:5:in `from_file'
2019-12-07T09:40:14.878207101Z
2019-12-07T09:40:14.878305970Z     letsencrypt_certificate("gitlab.DOMAIN") do
2019-12-07T09:40:14.878400607Z       action [:create]
2019-12-07T09:40:14.878497505Z       default_guard_interpreter :default
2019-12-07T09:40:14.878632957Z       declared_type :letsencrypt_certificate
2019-12-07T09:40:14.878733748Z       cookbook_name "letsencrypt"
2019-12-07T09:40:14.878841388Z       recipe_name "http_authorization"
2019-12-07T09:40:14.878941008Z       crt "/etc/gitlab/ssl/gitlab.DOMAIN.crt"
2019-12-07T09:40:14.879034991Z       key "/etc/gitlab/ssl/gitlab.DOMAIN.key"
2019-12-07T09:40:14.879128450Z       alt_names []
2019-12-07T09:40:14.879221162Z       cn "gitlab.DOMAIN"
2019-12-07T09:40:14.879315819Z       only_if { #code block }
2019-12-07T09:40:14.879410587Z     end
2019-12-07T09:40:14.879511427Z
2019-12-07T09:40:14.880026892Z     System Info:
2019-12-07T09:40:14.880180845Z     ------------
2019-12-07T09:40:14.880372257Z     chef_version=14.13.11
2019-12-07T09:40:14.880476958Z     platform=ubuntu
2019-12-07T09:40:14.880630668Z     platform_version=16.04
2019-12-07T09:40:14.880739581Z     ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
2019-12-07T09:40:14.884007762Z     program_name=/opt/gitlab/embedded/bin/chef-client
2019-12-07T09:40:14.884178224Z     executable=/opt/gitlab/embedded/bin/chef-client
2019-12-07T09:40:14.884295186Z

LFlo · December 7, 2019, 8:29pm

Seems to be a new bug, I have the same issue on a ubuntu 18.04 server (all up-to-date):

Starting Chef Client, version 14.13.11
resolving cookbooks for run list: ["gitlab::letsencrypt_renew"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - package (0.1.0)
  - postgresql (0.1.0)
  - redis (0.1.0)
  - monitoring (0.1.0)
  - registry (0.1.0)
  - mattermost (0.1.0)
  - consul (0.1.0)
  - letsencrypt (0.1.0)
  - praefect (0.1.0)
  - gitaly (0.1.0)
  - runit (4.3.0)
  - nginx (0.1.0)
  - acme (4.0.0)
  - crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 14 resources
Recipe: letsencrypt::enable
  * ruby_block[http external-url] action run (skipped due to only_if)
Recipe: <Dynamically Defined Resource>
  * service[nginx] action nothing (skipped due to action :nothing)
Recipe: nginx::enable
  * runit_service[nginx] action enable
    * ruby_block[restart_service] action nothing (skipped due to action :nothing)
    * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
    * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/sv/nginx] action create (up to date)
    * template[/opt/gitlab/sv/nginx/run] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/log] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/log/main] action create (up to date)
    * template[/opt/gitlab/sv/nginx/log/run] action create (up to date)
    * template[/var/log/gitlab/nginx/config] action create (up to date)
    * ruby_block[verify_chown_persisted_on_nginx] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/sv/nginx/env] action create (up to date)
    * ruby_block[Delete unmanaged env files for nginx service] action run (skipped due to only_if)
    * template[/opt/gitlab/sv/nginx/check] action create (skipped due to only_if)
    * template[/opt/gitlab/sv/nginx/finish] action create (skipped due to only_if)
    * directory[/opt/gitlab/sv/nginx/control] action create (up to date)
    * link[/opt/gitlab/init/nginx] action create (up to date)
    * file[/opt/gitlab/sv/nginx/down] action delete (up to date)
    * directory[/opt/gitlab/service] action create (up to date)
    * link[/opt/gitlab/service/nginx] action create (up to date)
    * ruby_block[wait for nginx service socket] action run (skipped due to not_if)
     (up to date)
  * execute[reload nginx] action nothing (skipped due to action :nothing)
Recipe: letsencrypt::enable
  * directory[/etc/gitlab/ssl] action create (up to date)
  * acme_selfsigned[gitlab.example.com] action create
    * file[gitlab.example.com SSL selfsigned key] action create_if_missing (up to date)
    * file[gitlab.example.com SSL selfsigned crt] action create_if_missing (up to date)
    * file[gitlab.example.com SSL selfsigned chain] action create_if_missing (skipped due to not_if)
     (up to date)
Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[gitlab.example.com] action create
    * acme_certificate[staging] action create
      * file[gitlab.example.com SSL key] action create_if_missing (up to date)
      
      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================
      
      Acme::Client::Error::Malformed
      ------------------------------
      Method not allowed
      
      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:77:in `block in class_from_file'
      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
      ------------------
      suppressed sensitive resource output
      
      System Info:
      ------------
      chef_version=14.13.11
      platform=ubuntu
      platform_version=18.04
      ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client
      
    
    ================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[gitlab.example.com]'
    ================================================================================
    
    Acme::Client::Error::Malformed
    ------------------------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: Acme::Client::Error::Malformed: Method not allowed
    
    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:77:in `block in class_from_file'
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
    
      5: letsencrypt_certificate site do
      6:   crt node['gitlab']['nginx']['ssl_certificate']
      7:   key node['gitlab']['nginx']['ssl_certificate_key']
      8:   notifies :run, "execute[reload nginx]", :immediate
      9:   notifies :run, 'ruby_block[display_le_message]'
     10:   only_if { omnibus_helper.service_up?('nginx') }
     11: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:5:in `from_file'
    
    letsencrypt_certificate("gitlab.example.com") do
      action [:create]
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      crt "/etc/gitlab/ssl/gitlab.example.com.crt"
      key "/etc/gitlab/ssl/gitlab.example.com.key"
      alt_names []
      cn "gitlab.example.com"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=14.13.11
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 07 seconds

PhilNetring · December 9, 2019, 10:18am

We’re getting the same issue on our new CE omnibus install (moved server). CentOS 7.

Thankfully I still have access to our old server, so I’ve used the old certificates for now. Otherwise our site would be showing ERR_CERT_AUTHORITY_INVALID.

dnsmichi · December 9, 2019, 12:37pm

Hi,

Unable to renew letsencrypt certificate anymore (#4900) · Issues · GitLab.org / omnibus-gitlab · GitLab mentions a workaround next to the fix.

Edit: Bugfix releases coming.

Unable to renew letsencrypt certificate anymore (#4900) · Issues · GitLab.org / omnibus-gitlab · GitLab

This is being included, and backported, into the next releases of 12.2.x through 12.5.x

Cheers,
Michael

lucas.tieman · December 9, 2019, 3:19pm

having the same issue here

CoreyVincent · December 9, 2019, 4:30pm

I had the same issue. Which gave me some weird issues with my gitlab. I upgraded gitlab and then had to shut off letsencrypt altogether. I’m now just using certbot to create the SSL certification.
Instructions here for certbot: https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx

Linds · December 9, 2019, 9:55pm

Thanks for sharing your experience @CoreyVincent! You are officially a GitLab bug hunter. (Trust me: it’s a high honor). I appreciate you sharing what you have learned with others! Let me know if you need anything from me.

CoreyVincent · December 11, 2019, 4:04pm

No, thank you @Linds and your team for creating such a great platform. And for such a great community experience, you all are clearly doing something right.

CookieST · December 12, 2019, 2:31pm

@Linds when will this bug be fixed and how can we get our hands on the update?

dnsmichi · December 16, 2019, 8:49am

Best is to subscribe to the issue mentioned above. On the right bottom corner, you’ll see the notifications slider.

dnsmichi · December 17, 2019, 5:08pm

So we have 12.5.5 early

To everyone having modified the source code (workaround number 2), please revert these changes before running the upgrade.

Cheers,
Michael

pabellopez · January 24, 2020, 8:38am

I was having the same problem, but after searching and searching and finding nothing, I decided to make the certificates on my own.

To do this, I used the certbot error installed on my server, then I changed the configuration of ssl_certificate in the gitlab.rb and then applied a reconfigure and a restart gitlab.

Note: in order to run the certbot certonly I had to stop the git-lab services and then everything worked correctly.

I hope it works …

opt1 · January 27, 2020, 5:15pm

Found a work around in gitlab-ce (ee should work the same). The problem I faced was I could not upgrade the gitlab-ce packages using APT because gitlab-ctl reconfigure would fail during the upgrade and dpkg would error out. By disabling letsencrypt and SSL, this would allow you to update the gitlab to the latest version, which does not have this same malformed method error, and then reenabling SSL after and renewing the certs.

edited my /etc/gitlab/gitlab.rb file and disabled https external_url and letsencrypt settings:

external_url 'http://domain.com'
#external_url 'https://domain.com'
# letsencrypt['auto_renew'] = true
# letsencrypt['auto_renew_hour'] = 0
# letsencrypt['auto_renew_minute'] = nil # Should be a number or cron expression, if specified.
# letsencrypt['auto_renew_day_of_month'] = "*/4"

Then reconfigure gitlab

bash~# gitlab-ctl reconfigure

And upgrade gitlab to the latest version

bash~# apt update && apt upgrade

edit the /etc/gitlab/gitlab.rb file again and uncomment https/letsencrypt

external_url 'http://domain.com'
external_url 'https://domain.com'
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 0
letsencrypt['auto_renew_minute'] = nil # Should be a number or cron expression, if specified.
letsencrypt['auto_renew_day_of_month'] = "*/4"

Renew the certs and reconfigure (you may have to reconfigure one more time for the certs, I did for some reason)

bash~# gitlab-ce reconfigure
bash~# gitlab-ce renew-le-certs
bash~# gitlab-ce reconfigure

And now we are running the latest version of gitlab-ce and the certs have been renewed.

Hope this helps somebody!

Terrafire123 · January 30, 2020, 11:12am

@ opt1

Thank you!!!

It worked perfectly.

guillaume.duveau · February 19, 2020, 9:55pm

Thanks @opt1

Shoxc · July 26, 2021, 10:02pm

This issue still persists on the current docker image gitlab/gitlab-ee:latest (which is docker pull gitlab/gitlab-ee:14.1.0-ee.0)