Enabling puma in GITLab 12.85 causing pull/push error

How to Use GitLab
1

Enabling puma in GITLab 12.85 causing pull/push error

*I was trying to disable unicorn and enable puma (used https://docs.gitlab.com/12.8/omnibus/settings/puma.html ) *

  • clone or pull command gives following error
    Cloning into ‘xxxxxxxxx’…
    remote:
    remote: ========================================================================
    remote:
    remote: Internal API error (403)
    remote:
    remote: ========================================================================
    Please make sure you have the correct access rights
    and the repository exists.
    remote:
    fatal: Could not read from remote repository.

ssh to the server using “git” user is showing following error

remote:
remote: ========================================================================
remote:
remote: Failed to get username: Internal API error (403)
remote:
remote: ========================================================================
remote:

Once I change puma to unicorn clone/pull/push works fine

I am using a self managed gitlab 12.8.5-ee
Nothing found in search relating puma to the specific issue

Thanks for taking the time to be thorough in your request, it really helps! :blush:

2

Hi @jpd and welcome to the GitLab Community forum!

Can you clarify: are you able to git clone or git pull using HTTP(S) protocol instead of SSH?

I suggest a sudo gitlab-ctl reconfigure followed by a sudo gitlab-ctl restart again? This can often clear up lingering issues when making changes to config that affect the backend.

If you reconfigure and restart and the error persists, please:

  • check /var/log/gitlab/gitlab-shell/gitlab-shell.log for additional details (for clone/pulls over SSH).
  • share the output of: grep -E "unicorn|puma|8080" /etc/gitlab/gitlab.rb | grep -Ev "^\s*$|^\s*#"

Thanks!

3

pull/clone works with http url path (tried with root user/pass of gitlab). gitlab-ctl reconfigure f sudo gitlab-ctl restart didnt solved issue. There is no file in gitlab-shell log folder

Following is what I can see it gitlab-ctl tail

==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.232Z”,“queue”:“cronjob:geo_sidekiq_cron_config”,“class”:“Geo::SidekiqCronConfigWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“fff7405011811fb63cad55c4”,“created_at”:“2020-04-24T16:43:43.147Z”,“correlation_id”:“512011a0b2e5adbb20e23a1e6f99e5e0”,“enqueued_at”:“2020-04-24T16:43:43.231Z”,“pid”:20611,“message”:“Geo::SidekiqCronConfigWorker JID-fff7405011811fb63cad55c4: start”,“job_status”:“start”,“scheduling_latency_s”:0.000821}
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.257Z”,“queue”:“cronjob:update_all_mirrors”,“class”:“UpdateAllMirrorsWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“d75c6d40ebe6d507977a61d3”,“created_at”:“2020-04-24T16:43:43.255Z”,“correlation_id”:“8ddbe9bdb07f7eb850426b86fe5b21b5”,“enqueued_at”:“2020-04-24T16:43:43.256Z”,“pid”:20611,“message”:“UpdateAllMirrorsWorker JID-d75c6d40ebe6d507977a61d3: start”,“job_status”:“start”,“scheduling_latency_s”:0.001313}
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.553Z”,“queue”:“cronjob:update_all_mirrors”,“class”:“UpdateAllMirrorsWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“d75c6d40ebe6d507977a61d3”,“created_at”:“2020-04-24T16:43:43.255Z”,“correlation_id”:“8ddbe9bdb07f7eb850426b86fe5b21b5”,“enqueued_at”:“2020-04-24T16:43:43.256Z”,“pid”:20611,“message”:“UpdateAllMirrorsWorker JID-d75c6d40ebe6d507977a61d3: done: 0.295342 sec”,“job_status”:“done”,“scheduling_latency_s”:0.001313,“duration”:0.295342,“cpu_s”:0.026519,“completed_at”:“2020-04-24T16:43:43.553Z”,“db_duration”:14.2018748447299,“db_duration_s”:0.0142018748447299}
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.555Z”,“queue”:“cronjob:geo_sidekiq_cron_config”,“class”:“Geo::SidekiqCronConfigWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“fff7405011811fb63cad55c4”,“created_at”:“2020-04-24T16:43:43.147Z”,“correlation_id”:“512011a0b2e5adbb20e23a1e6f99e5e0”,“enqueued_at”:“2020-04-24T16:43:43.231Z”,“pid”:20611,“message”:“Geo::SidekiqCronConfigWorker JID-fff7405011811fb63cad55c4: done: 0.323298 sec”,“job_status”:“done”,“scheduling_latency_s”:0.000821,“duration”:0.323298,“cpu_s”:0.032668,“completed_at”:“2020-04-24T16:43:43.555Z”,“db_duration”:9.357288014143705,“db_duration_s”:0.009357288014143705}

==> /var/log/gitlab/sidekiq/current <==
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.232Z”,“queue”:“cronjob:geo_sidekiq_cron_config”,“class”:“Geo::SidekiqCronConfigWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“fff7405011811fb63cad55c4”,“created_at”:“2020-04-24T16:43:43.147Z”,“correlation_id”:“512011a0b2e5adbb20e23a1e6f99e5e0”,“enqueued_at”:“2020-04-24T16:43:43.231Z”,“pid”:20611,“message”:“Geo::SidekiqCronConfigWorker JID-fff7405011811fb63cad55c4: start”,“job_status”:“start”,“scheduling_latency_s”:0.000821}
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.257Z”,“queue”:“cronjob:update_all_mirrors”,“class”:“UpdateAllMirrorsWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“d75c6d40ebe6d507977a61d3”,“created_at”:“2020-04-24T16:43:43.255Z”,“correlation_id”:“8ddbe9bdb07f7eb850426b86fe5b21b5”,“enqueued_at”:“2020-04-24T16:43:43.256Z”,“pid”:20611,“message”:“UpdateAllMirrorsWorker JID-d75c6d40ebe6d507977a61d3: start”,“job_status”:“start”,“scheduling_latency_s”:0.001313}
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.553Z”,“queue”:“cronjob:update_all_mirrors”,“class”:“UpdateAllMirrorsWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“d75c6d40ebe6d507977a61d3”,“created_at”:“2020-04-24T16:43:43.255Z”,“correlation_id”:“8ddbe9bdb07f7eb850426b86fe5b21b5”,“enqueued_at”:“2020-04-24T16:43:43.256Z”,“pid”:20611,“message”:“UpdateAllMirrorsWorker JID-d75c6d40ebe6d507977a61d3: done: 0.295342 sec”,“job_status”:“done”,“scheduling_latency_s”:0.001313,“duration”:0.295342,“cpu_s”:0.026519,“completed_at”:“2020-04-24T16:43:43.553Z”,“db_duration”:14.2018748447299,“db_duration_s”:0.0142018748447299}
{“severity”:“INFO”,“time”:“2020-04-24T16:43:43.555Z”,“queue”:“cronjob:geo_sidekiq_cron_config”,“class”:“Geo::SidekiqCronConfigWorker”,“retry”:false,“queue_namespace”:“cronjob”,“jid”:“fff7405011811fb63cad55c4”,“created_at”:“2020-04-24T16:43:43.147Z”,“correlation_id”:“512011a0b2e5adbb20e23a1e6f99e5e0”,“enqueued_at”:“2020-04-24T16:43:43.231Z”,“pid”:20611,“message”:“Geo::SidekiqCronConfigWorker JID-fff7405011811fb63cad55c4: done: 0.323298 sec”,“job_status”:“done”,“scheduling_latency_s”:0.000821,“duration”:0.323298,“cpu_s”:0.032668,“completed_at”:“2020-04-24T16:43:43.555Z”,“db_duration”:9.357288014143705,“db_duration_s”:0.009357288014143705}

==> /var/log/gitlab/puma/puma_stdout.log <==
{“timestamp”:“2020-04-24T16:43:49.714Z”,“pid”:20601,“message”:“PumaWorkerKiller: Consuming 1963.75390625 mb with master and 2 workers.”}

=====================
remote:
remote: ========================================================================
remote:
remote: Failed to get username: Internal API error (403)
remote:
remote: ========================================================================
remote:

grep -E “unicorn|puma|8080” /etc/gitlab/gitlab.rb | grep -Ev “^\s*$|^\s*#”

unicorn[‘enable’] = false
puma[‘enable’] = true
puma[‘port’] = 8088