[SOLVED] Changes made in gitlab.rb are not set on reconfigure/restart of the Docker container

How to Use GitLab
1

Environment

  • Host System: Ubuntu 16.04 LTS
  • Apache 2.24 on host w/ reverse proxy to the container
$ sudo docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ sudo docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 30
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67GiB
Name: <masked>
ID: CWMI:WONN:2RZP:5HFD:GN52:BO3S:GNNW:P6WJ:5MQJ:PQLG:ZTWR:4JHM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Issue

I have torn down and rebuilt this server so many times in the last week getting various other items to function in my particular setup. It seems like when I first built the server, it saved a lot of my configuration but now does not allow me to change it.

Here is the SMTP/Email portion of my gitlab.rb (with my actual fqdn replaced by “mydomain.com”:

### Email Settings
# gitlab_rails['gitlab_email_enabled'] = true
# gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.com'
# gitlab_rails['gitlab_email_display_name'] = 'GitLab'
# gitlab_rails['gitlab_email_reply_to'] = 'noreply@mydomain.com'
# gitlab_rails['gitlab_email_subject_suffix'] = ''

### GitLab email server settings
###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html
###! **Use smtp instead of sendmail/postfix.**

# gitlab_rails['smtp_enable'] = true
# gitlab_rails['smtp_address'] = "mysmtp"
# gitlab_rails['smtp_port'] = 587
# gitlab_rails['smtp_user_name'] = "<username>"
# gitlab_rails['smtp_password'] = "<pass>"
# gitlab_rails['smtp_domain'] = "mysmtp"
# gitlab_rails['smtp_authentication'] = "plain"
# gitlab_rails['smtp_enable_starttls_auto'] = true
# gitlab_rails['smtp_tls'] = true

###! **Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert'**
###! Docs: http://api.rubyonrails.org/classes/ActionMailer/Base.html
# gitlab_rails['smtp_openssl_verify_mode'] = 'none'

# gitlab_rails['smtp_ca_path'] = "/etc/ssl/certs"
# gitlab_rails['smtp_ca_file'] = "/etc/ssl/certs/ca-certificates.crt"

Based on the information from GitLab’s SMTP documentation, I should be able to test sending mail via the ruby console like this:

gitlab-rails console
irb(main):007:0> Notify.test_email('adam@adamwgay.com', 'Message Subject', 'Message Body').deliver_now

However, the output from the console is unexpected. None of my settings from gitlab.rb are present, and it seems to be using some default set of options (including attempting to use sendmail, which it spits up an error about right at the beginning).

irb(main):007:0> Notify.test_email('adam@adamwgay.com', 'Message Subject', 'Message Body').deliver_now

Notify#test_email: processed outbound mail in 1.2ms
sh: 1: /usr/sbin/sendmail: not found

Sent mail to adam@adamwgay.com (5.8ms)
Date: Tue, 25 Jul 2017 19:16:15 +0000
From: GitLab <gitlab@gitlab.mydomain.com>
Reply-To: GitLab <noreply@gitlab.mydomain.com>
To: adam@adamwgay.com
Message-ID: <597798ff6423a_9593fdebe2c710863b8@gitlab.mydomain.com>
Subject: Message Subject
Mime-Version: 1.0
Content-Type: text/html;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-generated
X-Auto-Response-Suppress: All

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><p>Message Body</p></body></html>

=> #<Mail::Message:70225704413280, Multipart: false, Headers: <Date: Tue, 25 Jul 2017 19:16:15 +0000>, <From: GitLab <gitlab@gitlab.mydomain.com>>, <Reply-To: GitLab <noreply@gitlab.mydomain.com>>, <To: adam@adamwgay.com>, <Message-ID: <597798ff6423a_9593fdebe2c710863b8@gitlab.mydomain.com.mail>>, <Subject: Message Subject>, <Mime-Version: 1.0>, <Content-Type: text/html; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>, <Auto-Submitted: auto-generated>, <X-Auto-Response-Suppress: All>>

There are a couple of issues that I see with this output, and something tells me that the SMTP settings from gitlab.rb are not being parsed/delivered to the GitLab system properly:

  • All the From and Reply-To fields in this output DO NOT match what I have set in gitlab.rb
  • This line, sh: 1: /usr/sbin/sendmail: not found, seems to indicate the system is still trying to use sendmail. Again, not what I have set in gitlab.rb

Looking further, It seems that nothing in my gitlab.rb file is being updated upon running gitlab-ctl reconfigure then restarting the container.

  • If I change settings directly (even if there’s a volume on my host), say the nginx conf files, then a restart always reverts the settings, despite what I have set in the gitlab.rb file.
  • If I add a file to try and force the SMTP settings on the container, then again a restart removes this file.

Is there another way to apply the settings from gitlab.rb to the container that I have missed?

My order of ops:

  1. Make my changes in gitlab.rb
  2. Run sudo docker exec gitlab gitlab-ctl reconfigure or just gitlab-ctl reconfigure inside the container
  3. Run sudo docker restart gitlab or gitlab-ctl restart inside the container

Even though both ways give me a successful reconfigure message, the settings are not changed. At some point, changes I made to gitlab.rb got stored into the container, as every time I rebuild it certain hostnames and settings are unchanged, and my instance returns fairly quickly with all the necessary parameters for certs and HTTPS. I just can’t seem to make any further changes to my gitlab container via the config files.

Here’s how I build the container, maybe this is part of the issue? Am I missing something simple?

sudo docker run --detach \
   --hostname gitlab.mydomain.com \
   --publish 10443:443 --publish 10080:80 --publish 10022:22 --publish 10587:587 --publish 10025:25 \
   --name gitlab \
   --restart always \
   --volume /srv/gitlab/config:/etc/gitlab \
   --volume /srv/gitlab/logs:/var/log/gitlab \
   --volume /srv/gitlab/data:/var/opt/gitlab \
   gitlab/gitlab-ce:latest

UPDATE 1

If I run gitlab-ctl diff-config in the container, it seems to tell me that there is indeed another gitlab.rb that it is pulling from, and that it is not pulling from mine in the /etc/gitlab/ directory within the container. It even shows me the values from my config compared to the incorrect values I see in the “package available config.”

UPDATE 2

Went down the path thinking this may be a permissions issue. I’m not sure exactly what the suggested ownership should be for any persistent volumes that live on the host system. Mine end up owned by root:root every time I restart the container. Even if I make the folders owned by root:docker, then reconfigure/reset, it reverts back to being root:root. No idea if permissions is the issue, but just posting the update for information purposes in case it helps me track down the issue.

UPDATE 3

Ok, found a really roundabout way to bandaid my issue. Again, I think this is just a bandaid and that my gitlab.rb should be handling this re-config for me, I’ll have to track that down when I have the time.

The gist of the bandaid: I edited each .yml and .rb file individually on the container file system that I needed to edit (just 2 for now), and have made my server rebuild script as follows. I had to do the gitlab-ctl restart command, as restarting the whole container overwrites these two files again.

sudo docker run --detach \
        --hostname gitlab.mydomain.com \
        --publish 10443:443 --publish 10080:80 --publish 10022:22 --publish 10587:587 \
        --name gitlab \
        --restart always \
        --volume /srv/gitlab/config:/etc/gitlab \
        --volume /srv/gitlab/logs:/var/log/gitlab \
        --volume /srv/gitlab/data:/var/opt/gitlab \
        gitlab/gitlab-ce:latest
sudo docker exec gitlab cp /etc/gitlab/gitlab.yml /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
sudo docker exec gitlab cp /etc/gitlab/smtp_settings.rb /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb
sudo docker exec gitlab gitlab-ctl restart

UPDATE 4 (tldr;)

I had my config files and all parameters commented out… removed the single # from what I needed and now all is well.

1 Like
2

This has also been posted to StackOverflow: