Hi there I was wondering if anyone could help.
I have upgraded our gitlab instance from gitlab 9.1.7 (Source) to 9.1.7 (docker) then incrementally updated the server to 12.10.0 successfully.
System Specs : Ubuntu 18LTS 500GB HD 4 core 64GB Memory
However when I try and upgrade to 12.10.1 using the docker script below ;
docker run --detach --hostname localhost --publish 443:443 --publish 80:80 --publish 1022:22 --publish 587:587 --name gitlab --volume /srv/gitlab/config:/etc/gitlab --volume /srv/gitlab/logs:/var/log/gitlab --volume /srv/gitlab/data:/var/opt/gitlab gitlab/gitlab-ce:12.10.1-ce.0
the container starts and then stops after about 2 minutes through looking through the logs it looks like its a permissions and DB Issue so I tried running
sudo docker exec gitlab update-permissions
sudo docker restart gitlab
but the container refuses to start and the only way I can get it to start is to use the -e variable on the docker run to skip the DB update
docker run --detach --hostname localhost --publish 443:443 --publish 80:80 --publish 1022:22 --publish 587:587 --name gitlab -e GITLAB_SKIP_PG_UPGRADE=true --volume /srv/gitlab/config:/etc/gitlab --volume /srv/gitlab/logs:/var/log/gitlab --volume /srv/gitlab/data:/var/opt/gitlab gitlab/gitlab-ce:12.10.1-ce.0
Once this container is running I try the update permissions again and the container refuses to restart and I have to delete the image and re run the docker run command.
Any help appreciated I can add any config files or logs as required
Thanks in advance
Hi has anyone got any advice on this subject ???
The only way i can get the container to start is to use the -e flag and skip the PSQL update and the update-permissions command also causes the container to fail to restart
Also my images in issues have disappeared they exist in uploads but can’t be seen in the user interface
Thanks in advance
This is the database output
gitlabhq_production=> \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------------------+-------------------+----------+---------+---------+---------------------------------
gitlabhq_production | gitlab | UTF8 | C.UTF-8 | C.UTF-8 |
mattermost_production | gitlab_mattermost | UTF8 | C.UTF-8 | C.UTF-8 |
postgres | gitlab-psql | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | gitlab-psql | UTF8 | C.UTF-8 | C.UTF-8 | =c/"gitlab-psql" +
| | | | | "gitlab-psql"=CTc/"gitlab-psql"
template1 | gitlab-psql | UTF8 | C.UTF-8 | C.UTF-8 | "gitlab-psql"=CTc/"gitlab-psql"+
| | | | | =c/"gitlab-psql"
(5 rows)
gitlabhq_production=>
Hi,
can you please share the logs or the terminal output when it comes to
Cheers,
Hi Michael
Many thanks for getting back to me - what log do you require Production.log ??? or just all logs from that date
Stefano
Anyone of these
Hi Matthew
Sorry what log did you require the above is a list of all logs in gitlab-rails for the day that the migration failed can zip the lot !!! or if you know which one is the one you need can send that
Kind regards
Exception log
{"severity":"ERROR","time":"2020-04-24T19:07:16.243Z","correlation_id":"acab85e1-4c95-44b7-a2e3-d0ebc8417b61","tags.correlation_id":"acab85e1-4c95-44b7-a2e3-d0ebc8417b61","tags.locale":"en","extra.server":{"os":{"name":"Linux","version":"#97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020","build":"4.15.0-96-generic","kernel_version":"Linux localhost 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux"},"runtime":{"name":"ruby","version":"ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]"}},"exception.class":"GRPC::Unavailable","exception.message":"14:failed to connect to all addresses","exception.backtrace":["lib/gitlab/gitaly_client.rb:192:in `execute'","lib/gitlab/gitaly_client.rb:170:in `block in call'","lib/gitlab/gitaly_client.rb:198:in `measure_timings'","lib/gitlab/gitaly_client.rb:169:in `call'","lib/gitlab/gitaly_client/repository_service.rb:19:in `exists?'","lib/gitlab/git/repository.rb:98:in `exists?'","app/models/repository.rb:517:in `exists?'","lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'","lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'","lib/gitlab/repository_cache_adapter.rb:179:in `block (2 levels) in cache_method_output_asymmetrically'","lib/gitlab/safe_request_store.rb:12:in `fetch'","lib/gitlab/repository_cache.rb:25:in `fetch'","lib/gitlab/repository_cache_adapter.rb:178:in `block in cache_method_output_asymmetrically'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","lib/gitlab/repository_cache_adapter.rb:192:in `block in memoize_method_output'","lib/gitlab/repository_cache_adapter.rb:201:in `no_repository_fallback'","lib/gitlab/repository_cache_adapter.rb:191:in `memoize_method_output'","lib/gitlab/repository_cache_adapter.rb:177:in `cache_method_output_asymmetrically'","lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'","lib/gitlab/git_access.rb:259:in `check_repository_existence!'","lib/gitlab/git_access.rb:77:in `check'","app/controllers/repositories/git_http_controller.rb:106:in `access_check'","app/controllers/application_controller.rb:479:in `set_current_admin'","app/controllers/application_controller.rb:464:in `set_locale'","lib/gitlab/error_tracking.rb:48:in `with_context'","app/controllers/application_controller.rb:559:in `sentry_context'","lib/gitlab/application_context.rb:52:in `block in use'","lib/gitlab/application_context.rb:52:in `use'","lib/gitlab/application_context.rb:20:in `with_context'","app/controllers/application_controller.rb:455:in `set_current_context'"]}
Hi,
I meant the host which is running Docker specifically, and where you run the restart commands for the container. The database migrations should be output on stdout/stderr if I am not mistaken.Also, please refrain from sending me a PM after replying here, this won’t speed up my replies. I have many things on my task list and will respond with a helping answer as soon as possible.
less isn’t installed in Docker base images, neither is zless to open a gzipped log file. You’ll need the package manager inside the session, e.g. apt-get install gzip.
Reading the error message 14:failed to connect to all addresses with Gitaly in mind would mean that the migrations are being run but the Gitaly service is stopped for some reason.
While googling the error, I have found another Docker image where this is being discussed. Since Gitaly is a new core component since 2019, this might not be covered yet from your journey from 9.x.
sameersbn:master ← solidnerd:fix-backup-restore-and-create
opened 04:53PM - 07 Jul 19 UTC
Some more things to try within Docker:
https://gitlab.com/gitlab-org/gitaly/-/issues/1649 - pid file emptyhttps://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4059 - healthcheck problemhttps://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5070 - tmp dir not createdhttps://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4290 - since you explicitly use localhost as hostname, this is also is a good guess.
Cheers,
Apologies for the PM could not attach the logs will read up in the links you sent many thanks
@dnsmichi I have run a few checks see below the output of ./gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.6.5p114
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.7
Git Version: 2.26.2
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.10.1
Revision: e658772bd63
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.12
URL: https://git.techiaith.bangor.ac.uk
HTTP Clone URL: https://git.techiaith.bangor.ac.uk/some-group/some-project.git
SSH Clone URL: git@git.techiaith.bangor.ac.uk:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 12.2.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
And my gitlab.rb (Have put XXXX where there is sensitive login information
## Change the external_url to the address your users will type in their browser
external_url 'XXX'
#Email
gitlab_rails['gitlab_email_from'] = 'XXX'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['smtp_enable'] = true
#gitlab_rails['smtp_address'] = "XXXX"
gitlab_rails['smtp_address'] = "XXXX"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = "XXXXXX"
gitlab_rails['smtp_password'] = "XXXXX"
gitlab_rails['smtp_domain'] = "XXX"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_signin_enabled'] = true
# LDAP
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_host'] = 'XXX'
gitlab_rails['ldap_port'] = 636 #was 389
gitlab_rails['ldap_uid'] = 'XXXX'
gitlab_rails['ldap_method'] = 'ssl' #was plain
#gitlab 9.1.7 does not seem to do certificate verification at all
#gitlab_rails['verify_certificates'] = true #NEW LINE
#gitlab_rails['ca_file'] = '/etc/gitlab/trusted-certs/XXXX.xxx'
#gitlab_rails['ldap_bind_dn'] = 'XXXXX'
#gitlab_rails['ldap_password'] = 'XXXXXX'
gitlab_rails['ldap_user_filter'] = 'XXX"'
gitlab_rails['ldap_allow_username_or_email_login'] = true
gitlab_rails['ldap_base'] = 'XXX'
gitlab_rails['gitlab_default_theme'] = 5
gitlab_rails['gitlab_default_projects_limit'] = 9999
gitlab_rails['gitlab_default_projects_features_visibility_level'] = 'internal'
# Backups
gitlab_rails['backup_keep_time'] = 172800 # keep backups for 2 days (2*24*60*60)
# NGINX
nginx['redirect_http_to_https'] = true
nginx['client_max_body_size'] = '9000m'
nginx['ssl_certificate'] = "/etc/gitlab/ssl/XXX.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/XXX.key"
# postgresql
postgresql['shared_buffers'] = "400MB"
# Mattermost
mattermost_external_url 'XXX.com'
mattermost_nginx['redirect_http_to_http'] = true
mattermost_nginx['ssl_certificate'] = '/etc/gitlab/ssl/XXX.crt'
mattermost_nginx['ssl_certificate_key'] = '/etc/gitlab/ssl/XXXX.key'
mattermost['service_use_ssl'] = true
mattermost['gitlab_enable'] = true
mattermost['gitlab_id'] = "XXXXX"
mattermost['gitlab_secret'] = "XXXXX"
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "https://XXXX/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://XXX/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://XXXX/api/v4/user"
And the results of ./gitlab-rake gitlab:check SANITIZE=true
hecking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Anonymous. No `bind_dn` or `password` configured
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 10 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
12/1 ... yes
12/2 ... yes
6/3 ... yes
16/4 ... yes
16/5 ... yes
12/6 ... yes
12/7 ... yes
6/8 ... yes
18/9 ... yes
16/11 ... yes
10/12 ... yes
11/16 ... yes
10/17 ... yes
10/18 ... yes
10/19 ... yes
12/20 ... yes
10/21 ... yes
10/22 ... yes
10/23 ... yes
5/24 ... yes
13/25 ... yes
14/26 ... yes
12/28 ... yes
12/29 ... yes
12/30 ... yes
10/31 ... yes
18/32 ... yes
4/34 ... yes
18/35 ... yes
13/36 ... yes
19/37 ... yes
4/38 ... yes
24/39 ... yes
18/40 ... yes
11/41 ... yes
12/42 ... yes
19/43 ... yes
71/44 ... yes
20/47 ... yes
10/49 ... yes
10/50 ... yes
11/51 ... yes
13/52 ... yes
11/53 ... yes
12/54 ... yes
18/56 ... yes
22/57 ... yes
22/58 ... yes
23/59 ... yes
23/60 ... yes
11/61 ... yes
10/62 ... yes
22/63 ... yes
22/64 ... yes
12/65 ... yes
24/66 ... yes
24/67 ... yes
30/68 ... yes
30/69 ... yes
12/70 ... yes
18/71 ... yes
18/72 ... yes
26/73 ... yes
11/74 ... yes
11/75 ... yes
23/76 ... yes
22/77 ... yes
22/78 ... yes
27/80 ... yes
27/81 ... yes
27/82 ... yes
24/83 ... yes
13/84 ... yes
27/85 ... yes
27/86 ... yes
11/87 ... yes
18/88 ... yes
11/89 ... yes
11/92 ... yes
5/94 ... yes
10/95 ... yes
10/96 ... yes
11/97 ... yes
11/98 ... yes
5/99 ... yes
12/101 ... yes
12/102 ... yes
12/103 ... yes
26/104 ... yes
18/105 ... yes
29/106 ... yes
10/107 ... yes
6/108 ... yes
6/109 ... yes
16/111 ... yes
16/112 ... yes
6/113 ... yes
10/114 ... yes
18/115 ... yes
13/116 ... yes
14/117 ... yes
10/118 ... yes
13/119 ... yes
24/120 ... yes
22/121 ... yes
10/122 ... yes
7/123 ... yes
10/124 ... yes
10/125 ... yes
13/126 ... yes
29/127 ... yes
10/128 ... yes
10/129 ... yes
10/130 ... yes
6/131 ... yes
32/132 ... yes
12/133 ... yes
29/134 ... yes
18/135 ... yes
18/136 ... yes
4/137 ... yes
71/138 ... yes
10/139 ... yes
7/140 ... yes
10/141 ... yes
18/142 ... yes
12/143 ... yes
4/144 ... yes
4/145 ... yes
18/146 ... yes
18/147 ... yes
22/148 ... yes
18/149 ... yes
18/150 ... yes
45/151 ... yes
12/152 ... yes
4/153 ... yes
4/154 ... yes
29/155 ... yes
12/156 ... yes
22/157 ... yes
4/158 ... yes
10/159 ... yes
32/160 ... yes
29/161 ... yes
45/162 ... yes
18/163 ... yes
50/164 ... yes
4/165 ... yes
39/166 ... yes
7/167 ... yes
45/169 ... yes
23/170 ... yes
44/171 ... yes
18/172 ... yes
23/173 ... yes
23/174 ... yes
39/175 ... yes
44/176 ... yes
45/177 ... yes
45/178 ... yes
45/179 ... yes
23/180 ... yes
22/181 ... yes
12/182 ... yes
47/183 ... yes
47/184 ... yes
10/185 ... yes
13/186 ... yes
18/187 ... yes
4/188 ... yes
4/189 ... yes
24/190 ... yes
27/191 ... yes
12/192 ... yes
24/193 ... yes
18/194 ... yes
14/195 ... yes
18/196 ... yes
18/197 ... yes
4/198 ... yes
12/199 ... yes
32/200 ... yes
14/201 ... yes
22/202 ... yes
14/203 ... yes
23/204 ... yes
12/205 ... yes
12/206 ... yes
4/207 ... yes
32/209 ... yes
12/210 ... yes
12/211 ... yes
12/213 ... yes
12/214 ... yes
18/215 ... yes
29/216 ... yes
18/217 ... yes
71/218 ... yes
45/219 ... yes
23/220 ... yes
45/221 ... yes
4/222 ... yes
4/223 ... yes
16/224 ... yes
50/225 ... yes
29/226 ... yes
39/227 ... yes
50/228 ... yes
9/229 ... yes
10/230 ... yes
4/231 ... yes
4/232 ... yes
9/233 ... yes
10/234 ... yes
32/235 ... yes
23/237 ... yes
23/238 ... yes
4/239 ... yes
18/240 ... yes
4/241 ... yes
4/242 ... yes
12/244 ... yes
23/245 ... yes
54/247 ... yes
14/248 ... yes
7/249 ... yes
45/250 ... yes
54/251 ... yes
14/252 ... yes
18/253 ... yes
18/254 ... yes
10/255 ... yes
23/256 ... yes
32/257 ... yes
14/258 ... yes
23/259 ... yes
4/260 ... yes
18/261 ... yes
18/262 ... yes
23/263 ... yes
23/264 ... yes
4/265 ... yes
4/266 ... yes
50/267 ... yes
68/268 ... yes
68/269 ... yes
68/270 ... yes
68/271 ... yes
71/272 ... yes
71/273 ... yes
4/274 ... yes
18/275 ... yes
23/276 ... yes
4/277 ... yes
22/278 ... yes
7/279 ... yes
69/280 ... yes
69/281 ... yes
22/282 ... yes
4/283 ... yes
7/284 ... yes
4/285 ... yes
45/288 ... yes
4/289 ... yes
18/290 ... yes
22/291 ... yes
14/292 ... yes
14/293 ... yes
18/294 ... yes
22/295 ... yes
6/296 ... yes
32/297 ... yes
32/298 ... yes
32/299 ... yes
32/300 ... yes
4/301 ... yes
18/302 ... yes
32/303 ... yes
10/304 ... yes
4/305 ... yes
4/306 ... yes
74/307 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 37
Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
The only error I can see in that is the one below which is permissions on the upload Directory
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Any help appreciated
forgot to ask I am assuming that its not a wise idea to try and fix the permissions at runtime ???
Can anyone help me I fixed the permissions as suggested below
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Images still dont show up in attachments and If i try
sudo docker exec gitlab update-permissionssudo docker restart gitlab
It always fails to restart with the previous issues however the results of
./gitlab-rake gitlab:check SANITIZE=true comes back with no errors
If i try and restart the container goes up then crashes the only way I can get it to work is by running the following command
docker run --detach --hostname localhost --publish 443:443 --publish 80:80 --publish 1022:22 --publish 587:587 --name gitlab -e GITLAB_SKIP_PG_UPGRADE=true --volume /srv/gitlab/config:/etc/gitlab --volume /srv/gitlab/logs:/var/log/gitlab --volume /srv/gitlab/data:/var/opt/gitlab gitlab/gitlab-ce:12.10.1-ce.0
any help appreciated
