Gitlab is not starting up - technical break

I have installed gitlab on Ubuntu 20.04 using the following commands:
sudo apt update -y
sudp apt upgrade -y
sudo apt install -y curl openssh-server ca-certificates tzdata postfix
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
sudo EXTERNAL_URL=“https://tmp.address.pl” apt install gitlab-ee
sudo apt install docker
/opt/gitlab/embedded/bin/runsvdir-start &
sudo gitlab-ctl reconfigure
sudo gitlab-ctl start

Then after doing sudo gitlab status everything was running:
run: alertmanager: (pid 1648) 553s; run: log: (pid 1647) 553s
run: gitaly: (pid 1673) 553s; run: log: (pid 1670) 553s
run: gitlab-exporter: (pid 1651) 553s; run: log: (pid 1650) 553s
run: gitlab-workhorse: (pid 1702) 552s; run: log: (pid 1698) 552s
run: grafana: (pid 1671) 553s; run: log: (pid 1667) 553s
run: logrotate: (pid 1656) 553s; run: log: (pid 1655) 553s
run: nginx: (pid 1672) 553s; run: log: (pid 1669) 553s
run: node-exporter: (pid 1661) 553s; run: log: (pid 1660) 553s
run: postgres-exporter: (pid 1649) 553s; run: log: (pid 1646) 553s
run: postgresql: (pid 1676) 553s; run: log: (pid 1674) 553s
run: prometheus: (pid 1668) 553s; run: log: (pid 1665) 553s
run: puma: (pid 1686) 553s; run: log: (pid 1685) 553s
run: redis: (pid 1683) 553s; run: log: (pid 1682) 553s
run: redis-exporter: (pid 1666) 553s; run: log: (pid 1664) 553s
run: sidekiq: (pid 1681) 553s; run: log: (pid 1675) 553s
However when I open localhost in browser it always shows the message “Technical Break…” (or actually “Przerwa serwisowa…” as for some reason it defaulted to Polish)

How can I make it so that it runs?

The first startup after install can take a while to fully initialize the application, due to several database interactions, etc. running as pre-requisites.

What do you observe when you tail the GitLab logs and perform a retry of accessing the web page in your browser as it runs?

sudo gitlab-ctl tail

Nothing in particular I’m afraid. This is what I got the logs:

==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <==
{"type":"gitaly-ruby","grpc.start_time":"2020-12-05T15:31:07Z","grpc.time_ms":0.142,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":9784,"correlation_id":"4435520d49044a79701bd33c7297dea4","time":"2020-12-05T15:31:07.413Z"}
{"type":"gitaly-ruby","grpc.start_time":"2020-12-05T15:31:07Z","grpc.time_ms":0.156,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":9786,"correlation_id":"e5dae16d8f77042effe06bd30cfa087d","time":"2020-12-05T15:31:07.413Z"}

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2020-12-05 16:31:10 +0100
Processing by MetricsController#index as HTML
Completed 200 OK in 4ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 643)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2020-12-05T15:31:10.490Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"5d5057aa-0dad-4d25-a9e3-ebe3970f8ff8","cpu_s":0.01,"db_duration_s":0.0,"view_duration_s":0.00055,"duration_s":0.00426,"db_count":0,"db_write_count":0,"db_cached_count":0}

==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2020-12-05T15:31:21.078Z","pid":9930,"message":"PumaWorkerKiller: Consuming 2575.37890625 mb with master and 2 workers."}

==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <==
{"type":"gitaly-ruby","grpc.start_time":"2020-12-05T15:31:22Z","grpc.time_ms":0.181,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":9786,"correlation_id":"2a72b549f43cfedcba3645e751476479","time":"2020-12-05T15:31:22.420Z"}
{"type":"gitaly-ruby","grpc.start_time":"2020-12-05T15:31:22Z","grpc.time_ms":0.169,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":9784,"correlation_id":"97ee8bbbeac1c0a0622571f686a325c5","time":"2020-12-05T15:31:22.420Z"}

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2020-12-05 16:31:25 +0100
Processing by MetricsController#index as HTML
Completed 200 OK in 5ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 643)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2020-12-05T15:31:25.500Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"6e2b1df7-e2f3-4628-a02c-b7fcf66c0f20","cpu_s":0.02,"db_duration_s":0.0,"view_duration_s":0.0005,"duration_s":0.0047,"db_count":0,"db_write_count":0,"db_cached_count":0}

I do get (some time after sending the request) those INFOs (don’t know if they provide anything thou):

==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2020-12-05T15:33:15.937Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"4ac33b2afe6f1d7a0651d21f","created_at":"2020-12-05T15:33:15.870Z","meta.caller_id":"Cronjob","correlation_id":"df417e1ccecf00826a1980017e7de65c","enqueued_at":"2020-12-05T15:33:15.870Z","pid":9952,"message":"UpdateAllMirrorsWorker JID-4ac33b2afe6f1d7a0651d21f: start","job_status":"start","scheduling_latency_s":0.066699}
{"severity":"INFO","time":"2020-12-05T15:33:15.940Z","queue":"cronjob:schedule_merge_request_cleanup_refs","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"be533a35f6573555b798f418","created_at":"2020-12-05T15:33:15.908Z","meta.caller_id":"Cronjob","correlation_id":"ea961bde4dce12184535da27d632e9f5","enqueued_at":"2020-12-05T15:33:15.909Z","pid":9952,"message":"ScheduleMergeRequestCleanupRefsWorker JID-be533a35f6573555b798f418: start","job_status":"start","scheduling_latency_s":0.030482}
{"severity":"INFO","time":"2020-12-05T15:33:16.038Z","queue":"cronjob:elastic_index_bulk_cron","args":[],"class":"ElasticIndexBulkCronWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"3b5b39907882fdbed44debc1","created_at":"2020-12-05T15:33:15.832Z","meta.caller_id":"Cronjob","correlation_id":"fd57a0b6bda7ed6900a548bdc8f7c9a0","enqueued_at":"2020-12-05T15:33:15.833Z","pid":9952,"message":"ElasticIndexBulkCronWorker JID-3b5b39907882fdbed44debc1: start","job_status":"start","scheduling_latency_s":0.204403}
{"severity":"INFO","time":"2020-12-05T15:33:16.132Z","queue":"cronjob:geo_sidekiq_cron_config","args":[],"class":"Geo::SidekiqCronConfigWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"50656501ffcfc8fa15643be6","created_at":"2020-12-05T15:33:15.846Z","meta.caller_id":"Cronjob","correlation_id":"9eb38516fec965d01769c3aa09a611bb","enqueued_at":"2020-12-05T15:33:15.847Z","pid":9952,"message":"Geo::SidekiqCronConfigWorker JID-50656501ffcfc8fa15643be6: start","job_status":"start","scheduling_latency_s":0.285468}
{"severity":"INFO","time":"2020-12-05T15:33:16.185Z","queue":"cronjob:schedule_merge_request_cleanup_refs","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"be533a35f6573555b798f418","created_at":"2020-12-05T15:33:15.908Z","meta.caller_id":"Cronjob","correlation_id":"ea961bde4dce12184535da27d632e9f5","enqueued_at":"2020-12-05T15:33:15.909Z","pid":9952,"message":"ScheduleMergeRequestCleanupRefsWorker JID-be533a35f6573555b798f418: done: 0.244722 sec","job_status":"done","scheduling_latency_s":0.030482,"redis_calls":2,"redis_duration_s":0.000399,"redis_read_bytes":2,"redis_write_bytes":212,"redis_queues_calls":2,"redis_queues_duration_s":0.000399,"redis_queues_read_bytes":2,"redis_queues_write_bytes":212,"extra.schedule_merge_request_cleanup_refs_worker.merge_requests_count":0,"db_count":1,"db_write_count":0,"db_cached_count":0,"duration_s":0.244722,"cpu_s":0.005153,"completed_at":"2020-12-05T15:33:16.185Z","db_duration_s":0.03694}
{"severity":"INFO","time":"2020-12-05T15:33:16.188Z","queue":"cronjob:elastic_index_bulk_cron","args":[],"class":"ElasticIndexBulkCronWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"3b5b39907882fdbed44debc1","created_at":"2020-12-05T15:33:15.832Z","meta.caller_id":"Cronjob","correlation_id":"fd57a0b6bda7ed6900a548bdc8f7c9a0","enqueued_at":"2020-12-05T15:33:15.833Z","pid":9952,"message":"ElasticIndexBulkCronWorker JID-3b5b39907882fdbed44debc1: done: 0.150654 sec","job_status":"done","scheduling_latency_s":0.204403,"redis_calls":5,"redis_duration_s":0.036377,"redis_read_bytes":4,"redis_write_bytes":572,"redis_queues_calls":2,"redis_queues_duration_s":0.000766,"redis_queues_read_bytes":2,"redis_queues_write_bytes":200,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.035611,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":372,"extra.elastic_index_bulk_cron_worker.records_count":0,"db_count":1,"db_write_count":0,"db_cached_count":0,"duration_s":0.150654,"cpu_s":0.00526,"completed_at":"2020-12-05T15:33:16.188Z","db_duration_s":0.000925}
{"severity":"INFO","time":"2020-12-05T15:33:16.141Z","queue":"cronjob:elastic_index_initial_bulk_cron","args":[],"class":"ElasticIndexInitialBulkCronWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"e8032ecb89b7563a5981144a","created_at":"2020-12-05T15:33:16.121Z","meta.caller_id":"Cronjob","correlation_id":"18e27c405386813346c8a4584283ed64","enqueued_at":"2020-12-05T15:33:16.122Z","pid":9952,"message":"ElasticIndexInitialBulkCronWorker JID-e8032ecb89b7563a5981144a: start","job_status":"start","scheduling_latency_s":0.018584}
{"severity":"INFO","time":"2020-12-05T15:33:16.193Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"4ac33b2afe6f1d7a0651d21f","created_at":"2020-12-05T15:33:15.870Z","meta.caller_id":"Cronjob","correlation_id":"df417e1ccecf00826a1980017e7de65c","enqueued_at":"2020-12-05T15:33:15.870Z","pid":9952,"message":"UpdateAllMirrorsWorker JID-4ac33b2afe6f1d7a0651d21f: done: 0.256205 sec","job_status":"done","scheduling_latency_s":0.066699,"redis_calls":5,"redis_duration_s":0.000982,"redis_read_bytes":5,"redis_write_bytes":492,"redis_queues_calls":2,"redis_queues_duration_s":0.00039,"redis_queues_read_bytes":2,"redis_queues_write_bytes":195,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000592,"redis_shared_state_read_bytes":3,"redis_shared_state_write_bytes":297,"db_count":1,"db_write_count":0,"db_cached_count":0,"duration_s":0.256205,"cpu_s":0.009846,"completed_at":"2020-12-05T15:33:16.193Z","db_duration_s":0.014415}
{"severity":"INFO","time":"2020-12-05T15:33:16.209Z","queue":"cronjob:elastic_index_initial_bulk_cron","args":[],"class":"ElasticIndexInitialBulkCronWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"e8032ecb89b7563a5981144a","created_at":"2020-12-05T15:33:16.121Z","meta.caller_id":"Cronjob","correlation_id":"18e27c405386813346c8a4584283ed64","enqueued_at":"2020-12-05T15:33:16.122Z","pid":9952,"message":"ElasticIndexInitialBulkCronWorker JID-e8032ecb89b7563a5981144a: done: 0.068314 sec","job_status":"done","scheduling_latency_s":0.018584,"redis_calls":5,"redis_duration_s":0.0007,"redis_read_bytes":4,"redis_write_bytes":589,"redis_queues_calls":2,"redis_queues_duration_s":0.00023,"redis_queues_read_bytes":2,"redis_queues_write_bytes":208,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.00047,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":381,"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"db_count":1,"db_write_count":0,"db_cached_count":0,"duration_s":0.068314,"cpu_s":0.004338,"completed_at":"2020-12-05T15:33:16.209Z","db_duration_s":0.001538}
{"severity":"INFO","time":"2020-12-05T15:33:16.203Z","queue":"cronjob:geo_sidekiq_cron_config","args":[],"class":"Geo::SidekiqCronConfigWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"50656501ffcfc8fa15643be6","created_at":"2020-12-05T15:33:15.846Z","meta.caller_id":"Cronjob","correlation_id":"9eb38516fec965d01769c3aa09a611bb","enqueued_at":"2020-12-05T15:33:15.847Z","pid":9952,"message":"Geo::SidekiqCronConfigWorker JID-50656501ffcfc8fa15643be6: done: 0.070528 sec","job_status":"done","scheduling_latency_s":0.285468,"redis_calls":33,"redis_duration_s":0.028058,"redis_read_bytes":34381,"redis_write_bytes":7032,"redis_queues_calls":33,"redis_queues_duration_s":0.028058,"redis_queues_read_bytes":34381,"redis_queues_write_bytes":7032,"db_count":0,"db_write_count":0,"db_cached_count":0,"duration_s":0.070528,"cpu_s":0.033307,"completed_at":"2020-12-05T15:33:16.203Z","db_duration_s":0.0}

Is there anything in particular I should look out for?

Just some thoughts that may of help in pursuing a resolution here:

The logs indicate that your browser’s requests aren’t even reaching GitLab webserver application. Furthermore, I couldn’t find an error message string in Ruby on Rails sources, nor in GitLab sources, that match the literal (language) string “Przerwa serwisowa” in it (if it was literally that), so perhaps it is coming from another layer or service your browser may be hitting?

Does the DNS resolution of tmp.address.pl from your work machine (where the browser runs) match the IP used by the webserver?

Note that the log does appear to show that the webserver is up and responding to internal scheduled requests (cronjob, metrics calls, etc.) that GitLab performs for its own maintenance. Its just not appearing to receive any external-to-host calls. Perhaps its also worth checking what address your GitLab’s webserver has ended up listening to:

~> netstat -anp | grep -F 443

Run the following on both, your workstation (client machine) and webserver:

~> getent hosts tmp.address.pl

Do the returned addresses match?

Another thing to check may be your HTTP(S) proxy configuration if there is one in use.

I have determined that the reason for that string literal is because of the network configuration on our side. It since has been changed and the message did go away in place of an error 503 (Service unavaliable)

netstat doesn’t show anything on port 443 (it does however show something for port 80). I think this is the underlying issue there.

As for DNS it is discoverable - the actual address that is used is from an already registered domain, but with the additional redirection to gitlab (so any additional redirections need and are done on our side)

All right, so after I edited /etc/gitlab/gitlab.rb to enable letsencrypt and let it run a bit it started working

Thanks for your help!