Local gitlab is not longer working

My private instance of gitlab stop working. I had it running without any problems by using the following docker-compose.yaml:

web:
  image: 'gitlab/gitlab-ce:latest'
  restart: always
  hostname: 'git.MYDOMAIN.COM'
  environment:
    GITLAB_OMNIBUS_CONFIG: |
      external_url 'https://git.MYDOMAIN.COM'
      gitlab_rails['smtp_enable'] = true
      gitlab_rails['smtp_address'] = "smtp.mailgun.org"
      gitlab_rails['smtp_port'] = 587
      gitlab_rails['smtp_authentication'] = "plain"
      gitlab_rails['smtp_enable_starttls_auto'] = true
      gitlab_rails['smtp_user_name'] = "postmaster@email.MYDOMAIN.COM"
      gitlab_rails['smtp_password'] = "MY_KEY"
      gitlab_rails['smtp_domain'] = "email.MYDOMAIN.COM"
      gitlab_rails['gitlab_shell_ssh_port'] = 2224
  ports:
    - '80:80'
    - '443:443'
    - '2224:22'
  volumes:
    - '/nfs/config/gitlab:/etc/gitlab'
    - '/nfs/config/gitlab/logs:/var/log/gitlab'
    - '/nfs/config/gitlab/data:/var/opt/gitlab'


I tried to stop and start the instance manually and I’m getting the following error:

web_1  | Upgrading the existing database to 10 failed and was reverted.
web_1  | Please check the output, and open an issue at:
web_1  | https://gitlab.com/gitlab-org/omnibus-gitlab/issues
web_1  | If you would like to restart the instance without attempting to
web_1  | upgrade, add the following to your docker command:
web_1  | -e GITLAB_SKIP_PG_UPGRADE=true

I tried adding GITLAB_SKIP_PG_UPGRADE: 1 to my environment but it did not help.

Hi,

What was your GitLab version until now? Maybe you have unintentionally downloaded the latest version, and it attempts to upgrade your instance?

Hi @alexk,

Yes, I’m sure this happended when i pulled a new image. My docker file always had gitlab/gitlab-ce:latest I’m not sure which version was running before this stop working. I do know that my instance was working last week.

Is there a way to recover my repos? I would hate to loose all that data…

I suggest you to run sudo docker image list. You will get a list of locally stored images, like that:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
gitlab/gitlab-ee    latest              f5a3a4b0edd3        4 weeks ago         1.97GB
gitlab/gitlab-ee    12.1.0-rc1.ee.0     360fce1c2414        7 weeks ago         2GB
gitlab/gitlab-ee    <none>              1b8c909293a3        2 months ago        2GB
hello-world         latest              fce289e99eb9        7 months ago        1.84kB

One of those images is probably one you have used before. If you can identify it, you can try to run it. If it has no tag, you can use Image ID instead (see https://stackoverflow.com/questions/18497688/run-a-docker-image-as-a-container).

Anyway, make a backup first!

Good luck!

Thanks for the suggestion @alexk. I tried using this:

gitlab/gitlab-ce              11.11.7-ce.0        6d03211f059d        3 weeks ago         1.85GB

by changing the image of my yaml to: image: 'gitlab/gitlab-ce:11.11.7-ce.0'

Alas, I’m getting the following error now:

web_1  |     System Info:
web_1  |     ------------
web_1  |     chef_version=13.6.4
web_1  |     platform=ubuntu
web_1  |     platform_version=16.04
web_1  |     ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
web_1  |     program_name=/opt/gitlab/embedded/bin/chef-client
web_1  |     executable=/opt/gitlab/embedded/bin/chef-client
web_1  |     
web_1  | Recipe: gitlab::gitlab-rails
web_1  |   * execute[clear the gitlab-rails cache] action run
web_1  |     [execute] `/var/opt/gitlab` is not writable.
web_1  |               Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.
web_1  |     - execute /opt/gitlab/bin/gitlab-rake cache:clear
web_1  | 
web_1  | Running handlers:
web_1  | There was an error running gitlab-ctl reconfigure:
web_1  | 
web_1  | runit_service[redis] (redis::enable line 65) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[restart_log_service] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 69) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
web_1  | ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/redis/log ----
web_1  | STDOUT: timeout: down: /opt/gitlab/service/redis/log: 0s, normally up, want up
web_1  | STDERR: 
web_1  | ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/redis/log ----
web_1  | Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/redis/log returned 1
web_1  | 
web_1  | Running handlers complete
web_1  | Chef Client failed. 59 resources updated in 01 minutes 02 seconds

Any ideas?

Hy @ignacio82, I am also quite lost here. But I see that there is a problem with /var/opt/gitlab protection. Worth checking it…

I tried running:

ignacio@server:/nfs/config/docker/gitlab$ docker exec -ti gitlab bash
root@git:/# chmod -R 2770 /var/opt/gitlab/

But it did not help. I also tried

docker exec -it gitlab update-permissions
docker restart gitlab

No luck.

:frowning:

Have you a backup from the time it worked? I think that the unsuccessful upgrade damaged your data.

Maybe this was not the correct version as well. You can check your actual version at:
$ cat /var/opt/gitlab/gitlab-rails/VERSION

Depending on where you mounted /var/opt/gitlab on your host machine. I expect that with a rollback, gitlab also rollsback the VERSION file.

BTW: Making a backup before upgrading is always the best idea. You could look into LVM Snaphot backups, which is an easy way to make backups and restore them if needed.

Thanks @m.hoogenboom
The version that i have in my backup is 11.11.8. I modified my docker-compose accordingly and now I’m getting a new error (i guess that is progress?)

web_1  | I, [2019-08-26T10:49:50.672479 #658]  INFO -- : master process ready
web_1  | I, [2019-08-26T10:49:50.784239 #827]  INFO -- : worker=1 ready
web_1  | I, [2019-08-26T10:49:50.788853 #830]  INFO -- : worker=2 ready
web_1  | I, [2019-08-26T10:49:50.798169 #824]  INFO -- : worker=0 ready
web_1  | I, [2019-08-26T10:49:50.811525 #833]  INFO -- : worker=3 ready
web_1  | I, [2019-08-26T10:49:50.845574 #836]  INFO -- : worker=4 ready
web_1  | 
web_1  | ==> /var/log/gitlab/unicorn/current <==
web_1  | 2019-08-26_10:49:51.71131 adopted new unicorn master 658
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-rails/production.log <==
web_1  | Started GET "/-/metrics" for 127.0.0.1 at 2019-08-26 10:49:59 +0000
web_1  | Processing by MetricsController#index as HTML
web_1  | Completed 200 OK in 66ms (Views: 62.7ms | ActiveRecord: 27.9ms)
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-rails/production_json.log <==
web_1  | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":69.6,"view":62.7,"db":27.92,"time":"2019-08-26T10:49:59.698Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"ca3c8a01-a1e4-425f-92c3-0c6ee929a906"}
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <==
web_1  | [2019-08-26 10:50:03] 127.0.0.1 - - [26/Aug/2019:10:50:03 UTC] "GET /metrics HTTP/1.1" 200 664 "-" "Prometheus/2.8.1"
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-rails/production.log <==
web_1  | Started GET "/" for 192.168.86.198 at 2019-08-26 10:50:11 +0000
web_1  | Processing by RootController#index as HTML
web_1  | Completed 500 Internal Server Error in 296ms (ActiveRecord: 0.0ms)
web_1  |   
web_1  | NoMethodError (undefined method `sentry_enabled?' for #<ApplicationSetting:0x00007faec03fc660>
web_1  | Did you mean?  sentry_enabled):
web_1  |   
web_1  | lib/gitlab/current_settings.rb:15:in `method_missing'
web_1  | lib/gitlab/sentry.rb:7:in `enabled?'
web_1  | lib/gitlab/sentry.rb:11:in `context'
web_1  | app/controllers/application_controller.rb:511:in `sentry_context'
web_1  | lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
web_1  | lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
web_1  | lib/gitlab/metrics/transaction.rb:57:in `run'
web_1  | lib/gitlab/metrics/rack_middleware.rb:17:in `call'
web_1  | lib/gitlab/middleware/multipart.rb:103:in `call'
web_1  | lib/gitlab/request_profiler/middleware.rb:16:in `call'
web_1  | lib/gitlab/middleware/go.rb:20:in `call'
web_1  | lib/gitlab/etag_caching/middleware.rb:13:in `call'
web_1  | lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
web_1  | lib/gitlab/middleware/correlation_id.rb:15:in `call'
web_1  | lib/gitlab/middleware/read_only/controller.rb:40:in `call'
web_1  | lib/gitlab/middleware/read_only.rb:18:in `call'
web_1  | lib/gitlab/middleware/basic_health_check.rb:25:in `call'
web_1  | lib/gitlab/request_context.rb:26:in `call'
web_1  | lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
web_1  | lib/gitlab/middleware/release_env.rb:12:in `call'
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-rails/production_json.log <==
web_1  | {"method":"GET","path":"/","format":"html","controller":"RootController","action":"index","status":500,"error":"NoMethodError: undefined method `sentry_enabled?' for #<ApplicationSetting:0x00007faec03fc660>\nDid you mean?  sentry_enabled","duration":297.66,"view":0.0,"db":0.0,"time":"2019-08-26T10:50:11.904Z","params":[],"remote_ip":"192.168.86.198","user_id":null,"username":null,"ua":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","queue_duration":6.5,"correlation_id":"H0zuJwqhvO5"}
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-workhorse/current <==
web_1  | 2019-08-26_10:50:12.20753 git.ignacio.website 127.0.0.1:0 - - [2019/08/26:10:50:12 +0000] "GET / HTTP/1.1" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" 0.320
web_1  | 
web_1  | ==> /var/log/gitlab/nginx/gitlab_access.log <==
web_1  | 192.168.86.198 - - [26/Aug/2019:10:50:12 +0000] "GET / HTTP/1.1" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
web_1  | 
web_1  | ==> /var/log/gitlab/gitlab-rails/production.log <==
web_1  | Started GET "/-/metrics" for 127.0.0.1 at 2019-08-26 10:50:14 +0000
web_1  | Processing by MetricsController#index as HTML
web_1  | Completed 200 OK in 11ms (Views: 6.2ms)
web_1  | 

Could this be the problem?

web_1  | 2019-08-26_10:49:08.37477 `/var/opt/gitlab` is not writable.
web_1  | 2019-08-26_10:49:08.37485 Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.

If so, how can I fix it? I tried

docker exec -it gitlab update-permissions
docker restart gitlab

with no luck.

Can you check what the permissions on the host file system are of /nfs/config/gitlab/data where you mounted /var/opt/gitlab ? And if you have selinux enabled: can the container write in this folder as root?

ignacio@server:~$ ls -la /nfs/config/gitlab/data
total 16
drwxr-xr-x 1 root root 484 Aug 26 10:49 .
drwxrwxr-x 1 root root 344 Feb 28 11:00 ..
drwxr-x--- 1  992  992  40 Aug 26 10:49 alertmanager
drwx------ 1  998  998   0 Feb 28 10:58 backups
-rwxr-xr-x 1  998  998  38 Feb 28 11:00 bootstrapped
drwxr-sr-x 1  998  998   0 Feb 28 10:58 .bundle
drwx------ 1  998  998  82 Aug 26 10:49 gitaly
-rw-r--r-- 1  998  998 296 Feb 28 10:58 .gitconfig
drwx------ 1  998  998  24 Feb 28 10:58 git-data
drwxr-xr-x 1  998  998  12 Feb 28 10:58 gitlab-ci
drwxr-xr-x 1  998  998  60 Aug 26 10:49 gitlab-monitor
drwxr-xr-x 1  998  998 148 Aug 26 10:49 gitlab-rails
drwx------ 1  998  998  20 Aug 26 10:49 gitlab-shell
drwxr-x--- 1  998  999  48 Aug 26 10:49 gitlab-workhorse
drwxr-sr-x 1  998  998 144 Aug 12 20:43 grafana
drwx------ 1  998  998  82 Aug 26 20:59 logrotate
drwxr-x--- 1 root  999 176 Aug 26 10:49 nginx
drwx------ 1  996  998  24 Aug 26 10:49 postgres-exporter
drwxr-xr-x 1  996  996  86 Aug 26 10:49 postgresql
drwxr-x--- 1  992  992  46 Aug 26 10:49 prometheus
-rwxr-xr-x 1  998  998 226 Aug 26 10:49 public_attributes.json
drwxr-x--- 1  997  998  60 Aug 26 21:51 redis
drwx------ 1  998  998  70 Mar 12 16:02 .ssh
-rwxr-xr-x 1  998  998  40 Aug 25 21:17 trusted-certs-directory-hash

ignacio@server:~$ ls -la /nfs/config/gitlab
total 124
drwxrwxr-x 1 root    root      344 Feb 28 11:00 .
drwxrwxrwx 1 ignacio ignacio   200 Aug 25 20:18 ..
drwxr-xr-x 1 root    root      484 Aug 26 10:49 data
-rwxr-xr-x 1    1026 users   86016 Feb 28 10:58 gitlab.rb
-rw------- 1    1026 users   15367 Aug 26 10:49 gitlab-secrets.json
drwxr-xr-x 1     998 users     356 Jun 22 14:35 logs
-rwxr-xr-x 1    1026 users     227 Feb 28 10:58 ssh_host_ecdsa_key
-rwxr-xr-x 1    1026 users     186 Feb 28 10:58 ssh_host_ecdsa_key.pub
-rwxr-xr-x 1    1026 users     419 Feb 28 10:58 ssh_host_ed25519_key
-rwxr-xr-x 1    1026 users     106 Feb 28 10:58 ssh_host_ed25519_key.pub
-rwxr-xr-x 1    1026 users    1679 Feb 28 10:58 ssh_host_rsa_key
-rwxr-xr-x 1    1026 users     406 Feb 28 10:58 ssh_host_rsa_key.pub
drwxr-sr-x 1    1026 users     154 Feb 28 11:00 ssl
drwxr-xr-x 1    1026 users       0 Feb 28 10:58 trusted-certs

Thanks @m.hoogenboom. I really hope there is a way of fixing this so i don’t lose all my data…

Is selinux enabled on your server?
Check this with the following command:

$ sudo getenforce

Other thing that could be wrong are the nfs permissions: you might have to set “no_root_squash” in the /etc/exports file on your nfs server. And you may have to check the mount options on the gitlab server in the file /etc/fstab. Is your nfs share (still) mounted as rw?

Can you try if root can write files on the nfs share:

$ echo "test" > /nfs/config/gitlab/data/test1
$ chown 998:998 /nfs/config/gitlab/data/test1

The following commands will start a gitlab container with your current version that only tries to create a file and change the permissions:

$ cd /nfs/config/gitlab/data
$ docker run --rm -v $PWD:/var/opt/gitlab -w /var/opt/gitlab gitlab/gitlab-ee:11.11.8-ce.0 echo "test" > test2
$ docker run --rm -v $PWD:/var/opt/gitlab -w /var/opt/gitlab gitlab/gitlab-ee:12.11.8-ce.0 chown 998:998 test2

Is the container allowed to create the file and change the permissions?

If your docker container is still unable to write in the mounted volume, is it possible to copy the data on a local disk of this gitlab server instead of using the nfs share? Note you have to preserve the permissions when you copy the data.

I also noticed that you actually mounted the /nfs/config/gitlab several times in your container by the following lines in your docker-compose.yml. The first volume also contains data, and logs:

volumes:
  - '/nfs/config/gitlab:/etc/gitlab'
  - '/nfs/config/gitlab/logs:/var/log/gitlab'
  - '/nfs/config/gitlab/data:/var/opt/gitlab'

It would be better to move all the configuration files in its own folder and mount this one instead of the /nfs/config/gitlab (but I don’t think this is related to the error you are seeing, I suggest you make this change after it works again):

$ sudo mkdir /nfs/config/gitlab/config
$ sudo cd /nfs/config/gitlab
$ mv gitlab.rb gitlab-secrets.json ssh* ssl trusted-certs config

Then change the docker-compose.yml to:

volumes:
  - '/nfs/config/gitlab/config:/etc/gitlab'
  - '/nfs/config/gitlab/logs:/var/log/gitlab'
  - '/nfs/config/gitlab/data:/var/opt/gitlab'

I’m sure it can be solved. Please let me know the results of the tests.

It is not:

ignacio@server:~$ sudo getenforce
[sudo] password for ignacio: 
sudo: getenforce: command not found

After installing it and restarting this is what I get:

ignacio@server:~$ sudo getenforce
[sudo] password for ignacio: 
Disabled

This is the relevant line in my /etc/fstab

192.168.86.198:/volume1/config /nfs/config nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.86.198,mountvers=3,mountport=892,mountpr
oto=udp,local_lock=none,addr=192.168.86.198

And this is a screenshot of my synology configuration:

I guess this is the problem:

ignacio@server:~$ echo "test" > /nfs/config/gitlab/data/test1
-bash: /nfs/config/gitlab/data/test1: Permission denied

ignacio@server:~$ sudo !!
sudo echo "test" > /nfs/config/gitlab/data/test1
-bash: /nfs/config/gitlab/data/test1: Permission denied

How could I fix it? Thanks a lot!

I did this and now I’m getting a new error :frowning:

web_1  | 2019-08-29_10:42:09.37368 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing LoginService" logger=server
web_1  | 2019-08-29_10:42:09.37368 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing QuotaService" logger=server
web_1  | 2019-08-29_10:42:09.37369 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing RemoteCache" logger=server
web_1  | 2019-08-29_10:42:09.37370 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing ServerLockService" logger=server
web_1  | 2019-08-29_10:42:09.37370 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing TracingService" logger=server
web_1  | 2019-08-29_10:42:09.37371 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing UsageStatsService" logger=server
web_1  | 2019-08-29_10:42:09.37372 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server
web_1  | 2019-08-29_10:42:09.37374 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing CleanUpService" logger=server
web_1  | 2019-08-29_10:42:09.37374 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing NotificationService" logger=server
web_1  | 2019-08-29_10:42:09.37680 t=2019-08-29T10:42:09+0000 lvl=info msg="Initializing ProvisioningService" logger=server
web_1  | 2019-08-29_10:42:09.37937 t=2019-08-29T10:42:09+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Datasource provisioning error: attempt to write a readonly database"

My guess is that i need to change the permissions of some folder but i’m not sure what to do exactly…

@m.hoogenboom do you have any suggestion??

Try:

docker exec -it gitlab update-permissions
docker restart gitlab

No luck. I get the same error…

gitlab | 2019-08-29_11:28:19.66482 t=2019-08-29T11:28:19+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Datasource provisioning error: attempt to write a readonly database"
gitlab | 
gitlab | ==> /var/log/gitlab/gitaly/current <==
gitlab | {"level":"info","msg":"Wrapper started","time":"2019-08-29T11:28:20Z","wrapper":1772}
gitlab | {"level":"info","msg":"finding gitaly","pid_file":"/var/opt/gitlab/gitaly/gitaly.pid","time":"2019-08-29T11:28:20Z","wrapper":1772}
gitlab | {"error":"open /var/opt/gitlab/gitaly/gitaly.pid: permission denied","level":"fatal","msg":"find gitaly","time":"2019-08-29T11:28:20Z","wrapper":1772}
gitlab | 
gitlab | ==> /var/log/gitlab/grafana/current <==
gitlab | 2019-08-29_11:28:20.70517 t=2019-08-29T11:28:20+0000 lvl=info msg="Starting Grafana" logger=server version=6.1.1 commit=eff01d2 branch=HEAD compiled=2019-04-05T09:42:24+0000
gitlab | 2019-08-29_11:28:20.70521 t=2019-08-29T11:28:20+0000 lvl=info msg="Config loaded from" logger=settings file=/var/opt/gitlab/grafana/conf/defaults.ini
gitlab | 2019-08-29_11:28:20.70522 t=2019-08-29T11:28:20+0000 lvl=info msg="Config loaded from" logger=settings file=/var/opt/gitlab/grafana/grafana.ini
gitlab | 2019-08-29_11:28:20.70524 t=2019-08-29T11:28:20+0000 lvl=info msg="Path Home" logger=settings path=/var/opt/gitlab/grafana
gitlab | 2019-08-29_11:28:20.70525 t=2019-08-29T11:28:20+0000 lvl=info msg="Path Data" logger=settings path=/var/opt/gitlab/grafana/data
gitlab | 2019-08-29_11:28:20.70527 t=2019-08-29T11:28:20+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/gitlab/grafana
gitlab | 2019-08-29_11:28:20.70528 t=2019-08-29T11:28:20+0000 lvl=info msg="Path Plugins" logger=settings path=/var/opt/gitlab/grafana/data/plugins
gitlab | 2019-08-29_11:28:20.70530 t=2019-08-29T11:28:20+0000 lvl=info msg="Path Provisioning" logger=settings path=/var/opt/gitlab/grafana/provisioning
gitlab | 2019-08-29_11:28:20.70532 t=2019-08-29T11:28:20+0000 lvl=info msg="App mode production" logger=settings
gitlab | 2019-08-29_11:28:20.70598 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing HTTPServer" logger=server
gitlab | 2019-08-29_11:28:20.70662 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing SqlStore" logger=server
gitlab | 2019-08-29_11:28:20.70668 t=2019-08-29T11:28:20+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
gitlab | 2019-08-29_11:28:20.70713 t=2019-08-29T11:28:20+0000 lvl=info msg="Starting DB migration" logger=migrator
gitlab | 2019-08-29_11:28:20.71564 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing InternalMetricsService" logger=server
gitlab | 2019-08-29_11:28:20.71570 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing SearchService" logger=server
gitlab | 2019-08-29_11:28:20.71571 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing PluginManager" logger=server
gitlab | 2019-08-29_11:28:20.71573 t=2019-08-29T11:28:20+0000 lvl=info msg="Starting plugin search" logger=plugins
gitlab | 2019-08-29_11:28:20.72833 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing RenderingService" logger=server
gitlab | 2019-08-29_11:28:20.72840 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server
gitlab | 2019-08-29_11:28:20.72841 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing HooksService" logger=server
gitlab | 2019-08-29_11:28:20.72843 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing LoginService" logger=server
gitlab | 2019-08-29_11:28:20.72844 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing QuotaService" logger=server
gitlab | 2019-08-29_11:28:20.72846 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing RemoteCache" logger=server
gitlab | 2019-08-29_11:28:20.72848 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing ServerLockService" logger=server
gitlab | 2019-08-29_11:28:20.72850 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing TracingService" logger=server
gitlab | 2019-08-29_11:28:20.72854 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing UsageStatsService" logger=server
gitlab | 2019-08-29_11:28:20.72857 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server
gitlab | 2019-08-29_11:28:20.72860 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing CleanUpService" logger=server
gitlab | 2019-08-29_11:28:20.72862 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing NotificationService" logger=server
gitlab | 2019-08-29_11:28:20.72962 t=2019-08-29T11:28:20+0000 lvl=info msg="Initializing ProvisioningService" logger=server
gitlab | 2019-08-29_11:28:20.73055 t=2019-08-29T11:28:20+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Datasource provisioning error: attempt to write a readonly database"

Wat are the permissions now on your folders and especially “<your new host path>/gitlab/data/grafana”

This is from the server:

ignacio@server:~/config/gitlab/data$ sudo ls -la 
total 16
drwxr-xr-x 1 root root 484 Aug 29 11:46 .
drwxrwxr-x 1 root root 344 Aug 29 11:02 ..
drwxr-x--- 1  992  992  40 Aug 29 11:46 alertmanager
drwx------ 1  998 root   0 Aug 29 11:01 backups
-rwxr-xr-x 1 root root  38 Aug 29 11:02 bootstrapped
drwxr-xr-x 1  998  998   0 Aug 29 10:56 .bundle
drwx------ 1  998 root  82 Aug 29 11:46 gitaly
-rw-r--r-- 1  998  998 296 Aug 29 10:56 .gitconfig
drwx------ 1  998  998  24 Aug 29 10:56 git-data
drwxr-xr-x 1  998 root  12 Aug 29 11:01 gitlab-ci
drwxr-xr-x 1  998 root  60 Aug 29 11:46 gitlab-monitor
drwxr-xr-x 1  998  998 148 Aug 29 11:46 gitlab-rails
drwx------ 1  998  998  20 Aug 29 11:46 gitlab-shell
drwxr-x--- 1  998  999  48 Aug 29 11:46 gitlab-workhorse
drwx------ 1  992 root 144 Aug 29 11:46 grafana
drwx------ 1 root root  82 Aug 29 11:46 logrotate
drwxr-x--- 1 root  999 158 Aug 29 11:46 nginx
drwx------ 1  996 root  24 Aug 29 11:46 postgres-exporter
drwxr-xr-x 1  996  996 126 Aug 29 11:46 postgresql
drwxr-x--- 1  992  992  46 Aug 29 11:46 prometheus
-rwxr-xr-x 1 root root 226 Aug 29 11:46 public_attributes.json
drwxr-x--- 1  997  998  36 Aug 29 11:46 redis
drwx------ 1  998  998  70 Aug 29 10:56 .ssh
-rwxr-xr-x 1 root root  40 Aug 29 11:26 trusted-certs-directory-hash

And this is from inside the container

root@git:/etc/gitlab/data# ls -la
total 16
drwxr-xr-x 1 root              root              484 Aug 29 11:49 .
drwxrwxr-x 1 root              root              344 Aug 29 11:02 ..
drwxr-xr-x 1 git               git                 0 Aug 29 10:56 .bundle
-rw-r--r-- 1 git               git               296 Aug 29 10:56 .gitconfig
drwx------ 1 git               git                70 Aug 29 10:56 .ssh
drwxr-x--- 1 gitlab-prometheus gitlab-prometheus  40 Aug 29 11:49 alertmanager
drwx------ 1 git               root                0 Aug 29 11:01 backups
-rwxr-xr-x 1 root              root               38 Aug 29 11:02 bootstrapped
drwx------ 1 git               git                24 Aug 29 10:56 git-data
drwx------ 1 git               root               82 Aug 29 11:49 gitaly
drwxr-xr-x 1 git               root               12 Aug 29 11:01 gitlab-ci
drwxr-xr-x 1 git               root               60 Aug 29 11:49 gitlab-monitor
drwxr-xr-x 1 git               git               148 Aug 29 11:49 gitlab-rails
drwx------ 1 git               git                20 Aug 29 11:49 gitlab-shell
drwxr-x--- 1 git               gitlab-www         48 Aug 29 11:49 gitlab-workhorse
drwx------ 1 gitlab-prometheus root              144 Aug 29 11:49 grafana
drwx------ 1 root              root               82 Aug 29 11:49 logrotate
drwxr-x--- 1 root              gitlab-www        176 Aug 29 11:49 nginx
drwx------ 1 gitlab-psql       root               24 Aug 29 11:49 postgres-exporter
drwxr-xr-x 1 gitlab-psql       gitlab-psql       188 Aug 29 11:49 postgresql
drwxr-x--- 1 gitlab-prometheus gitlab-prometheus  46 Aug 29 11:49 prometheus
-rwxr-xr-x 1 root              root              226 Aug 29 11:49 public_attributes.json
drwxr-x--- 1 gitlab-redis      git                60 Aug 29 11:49 redis
-rwxr-xr-x 1 root              root               40 Aug 29 11:26 trusted-certs-directory-hash

In case it helps:

root@git:/etc/gitlab/data/grafana/data# ls -la
total 564
drwxr-xr-x 1 root              root     40 Aug 29 11:02 .
drwx------ 1 gitlab-prometheus root    144 Aug 29 11:49 ..
-rwxr-xr-x 1 root              root 577536 Aug 29 11:02 grafana.db
drwxr-xr-x 1 root              root      0 Aug 29 11:02 plugins
drwxr-xr-x 1 root              root      0 Aug 29 11:02 png

Thanks for helping!

gitlab uses
/var/opt/gitlab/grafana/data
not
/etc/gitlab/data/grafana/data, and I think the update-permissions might go wrong, because you mounted the complete data folder below /etc/gitlab in your compose file.

Permissions in our instance are different:

total 576
-rw------- 1 gitlab-prometheus gitlab-prometheus 577536 Aug 29 11:53 grafana.db
drwx------ 2 gitlab-prometheus gitlab-prometheus   4096 Aug 16 03:46 plugins
drwx------ 2 gitlab-prometheus gitlab-prometheus   4096 Aug 16 03:46 png

Please try my suggestion to change your docker-compose.yml and put the configuration files in it’s own folder and mount this instead of the complete gitlab folder with all data. After that try first

docker exec -it gitlab update-permissions
docker restart gitlab

And if that doesn’t work manually fix the permissions on the grafana folder.