Can't log in after upgrading to 17.0.1

:hugs: Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

I upgraded a system from 16.8.1 to 16.11.3 to 17.0.1 and now I can’t log in.

I’ve attempted to clear cookies, reset my user’s password from the command line, reguardless of what I try, I get invalid login or password.

The only thing I see in the logs:

{“severity”:“INFO”,“time”:“2024-05-29T03:19:51.234Z”,“correlation_id”:“01HZ15RXQBPDYG1AQY9X2MYN12”,“meta.caller_id”:“SessionsController#new”,“meta.remote_ip”:“192.168.10.131”,“meta.feature_category”:“system_access”,“meta.client_id”:“ip/192.168.10.131”,“message”:“Failed Login: username=myuser ip=192.168.10.131”}

==> /var/log/gitlab/gitlab-workhorse/current <==
redis: 2024/05/28 20:20:11 pubsub.go:168: redis: discarding bad PubSub connection: EOF
{“error”:“keywatcher: pubsub receive: EOF”,“level”:“error”,“msg”:“”,“time”:“2024-05-28T20:20:11-07:00”}

I’ve reset the user password. Cleared cache. Ran reconfigure many times. Restarted gitlab.

Configuration

Provide screenshots from the GitLab UI showing relevant configuration, if applicable.
On self-managed instances, add the relevant configuration settings or changes.

  • [X ] Self-managed
1 Like

Also seeing this:

==> /var/log/gitlab/gitlab-exporter/current <==
2024-05-29_05:17:25.35545 ::1 - - [28/May/2024:22:17:25 PDT] “GET /ruby HTTP/1.1” 200 1089
2024-05-29_05:17:25.35561 - → /ruby
2024-05-29_05:17:28.67511 E, [2024-05-28T22:17:28.675030 #11863] ERROR – : Error connecting to the database: PQsocket() can’t get socket descriptor
2024-05-29_05:17:28.67557 E, [2024-05-28T22:17:28.675538 #11863] ERROR – : Error connecting to the database: PQsocket() can’t get socket descriptor
2024-05-29_05:17:28.67587 E, [2024-05-28T22:17:28.675835 #11863] ERROR – : Error connecting to the database: PQsocket() can’t get socket descriptor
2024-05-29_05:17:28.70942 ::1 - - [28/May/2024:22:17:28 PDT] “GET /database HTTP/1.1” 200 0
2024-05-29_05:17:28.71444 - → /database

[2024-05-28 22:17:39] INFO WEBrick::HTTPServer#start: pid=13721 port=8092
{“severity”:“DEBUG”,“time”:“2024-05-29T05:17:39.880Z”,“message”:“Client Middleware: Sentry::Sidekiq::SentryContextClientMiddleware, Gitlab::SidekiqMiddleware::WorkerContext::Client, Labkit::Middleware::Sidekiq::Client, Gitlab::Database::LoadBalancing::SidekiqClientMiddleware, Gitlab::SidekiqMiddleware::PauseControl::Client, Gitlab::SidekiqMiddleware::ConcurrencyLimit::Client, Gitlab::SidekiqMiddleware::DuplicateJobs::Client, Gitlab::SidekiqStatus::ClientMiddleware, Gitlab::SidekiqMiddleware::AdminMode::Client, Gitlab::SidekiqMiddleware::SizeLimiter::Client, Gitlab::SidekiqMiddleware::ClientMetrics”}
{“severity”:“DEBUG”,“time”:“2024-05-29T05:17:39.882Z”,“message”:“Server Middleware: Sidekiq::Metrics::Middleware, Sentry::Sidekiq::SentryContextServerMiddleware, Marginalia::SidekiqInstrumentation::Middleware, Gitlab::SidekiqMiddleware::SizeLimiter::Server, Gitlab::SidekiqMiddleware::ShardAwarenessValidator, Gitlab::SidekiqMiddleware::Monitor, Labkit::Middleware::Sidekiq::Server, Gitlab::SidekiqMiddleware::RequestStoreMiddleware, Gitlab::SidekiqMiddleware::ServerMetrics, Gitlab::SidekiqMiddleware::ExtraDoneLogMetadata, Gitlab::SidekiqMiddleware::BatchLoader, Gitlab::SidekiqMiddleware::InstrumentationLogger, Gitlab::SidekiqMiddleware::AdminMode::Server, Gitlab::SidekiqMiddleware::QueryAnalyzer, Gitlab::SidekiqVersioning::Middleware, Gitlab::SidekiqStatus::ServerMiddleware, Gitlab::SidekiqMiddleware::WorkerContext::Server, Gitlab::SidekiqMiddleware::PauseControl::Server, ClickHouse::MigrationSupport::SidekiqMiddleware, Gitlab::SidekiqMiddleware::ConcurrencyLimit::Server, Gitlab::SidekiqMiddleware::DuplicateJobs::Server, Gitlab::Database::LoadBalancing::SidekiqServerMiddleware, Gitlab::SidekiqMiddleware::SkipJobs”}
bundler: failed to load command: sidekiq (/opt/gitlab/embedded/lib/ruby/gems/3.1.0/bin/sidekiq)
[2024-05-28 22:17:39] INFO going to shutdown …
[2024-05-28 22:17:39] INFO WEBrick::HTTPServer#start done.
{“severity”:“INFO”,“time”:“2024-05-29T05:17:39.962Z”,“message”:“stopped”,“memwd_reason”:“background task stopped”,“memwd_handler_class”:“Gitlab::Memory::Watchdog::Handlers::SidekiqHandler”,“memwd_sleep_time_s”:3,“pid”:13721,“worker_id”:“sidekiq_0”,“memwd_rss_bytes”:730128384,“retry”:0}
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:103:in visit': stack level too deep (SystemStackError) from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:162:in visit’
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:107:in binary' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:109:in visit_CAT’
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:103:in visit' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:162:in visit’
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:117:in unary' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:119:in visit_GROUP’
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8.1/lib/action_dispatch/journey/visitors.rb:103:in visit' ... 10428 levels... from /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in with_friendly_errors’
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.9/exe/bundle:20:in <top (required)>' from /opt/gitlab/embedded/bin/bundle:25:in load’
from /opt/gitlab/embedded/bin/bundle:25:in `’

Any ideas on this? I’m basically locked out of my github instance at the moment and all I did was upgrade.

I get same issue

redis: 2024/08/27 14:22:34 pubsub.go:168: redis: discarding bad PubSub connection: EOF
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2024-08-27T14:22:34Z"}
{"backend_id":"rails","content_type":"text/html; charset=utf-8","correlation_id":"01J6A3FAASDZA11K6WS6KBGD0Y","duration_ms":229,"host":"gitlab.devanox.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","route_id":"default","status":200,"system":"http","time":"2024-08-27T14:22:41Z","ttfb_ms":229,"uri":"/help","user_agent":"curl/8.9.1-DEV","written_bytes":27374}
{"client_mode":"local_tempfile","copied_bytes":886,"correlation_id":"01J6A3FYP91ZRZTAVD9CYB99MS","filename":"icon.png","is_local":true,"is_multipart":false,"is_remote":false,"level":"info","local_temp_path":"/opt/gitlab/embedded/service/gitlab-rails/public/uploads/tmp","msg":"saved file","remote_id":"","time":"2024-08-27T14:23:02Z"}
{"backend_id":"rails","content_type":"text/html; charset=utf-8","correlation_id":"01J6A3FYP91ZRZTAVD9CYB99MS","duration_ms":1121,"host":"gitlab.devanox.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"http://gitlab.devanox.com/admin/application_settings/appearance","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","route_id":"default","status":500,"system":"http","time":"2024-08-27T14:23:03Z","ttfb_ms":1121,"uri":"/admin/application_settings/appearance","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36","written_bytes":1624}
newLocalFile: remove file "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/tmp/gitlab-workhorse-upload3399506552": remove /opt/gitlab/embedded/service/gitlab-rails/public/uploads/tmp/gitlab-workhorse-upload3399506552: no such file or directory{"correlation_id":"01J6A3FZW8EBW8EV0KTWPY4ENB","encoding":"","file":"/opt/gitlab/embedded/service/gitlab-rails/public/-/error-illustrations/error-500-lg.svg","level":"info","method":"GET","msg":"Send static file","time":"2024-08-27T14:23:03Z","uri":"/-/error-illustrations/error-500-lg.svg"}
{"backend_id":"rails","content_type":"image/svg+xml","correlation_id":"01J6A3FZW8EBW8EV0KTWPY4ENB","duration_ms":6,"host":"gitlab.devanox.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://gitlab.devanox.com/admin/application_settings/appearance","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/-/","route_id":"dash","status":200,"system":"http","time":"2024-08-27T14:23:03Z","ttfb_ms":0,"uri":"/-/error-illustrations/error-500-lg.svg","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36","written_bytes":6506}
{"backend_id":"rails","content_type":"text/html","correlation_id":"01J6A3G09R59Z32MGBAMEWJV7Q","duration_ms":105,"host":"gitlab.devanox.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://gitlab.devanox.com/admin/application_settings/appearance","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","route_id":"default","status":301,"system":"http","time":"2024-08-27T14:23:04Z","ttfb_ms":105,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36","written_bytes":175}

I now facing the same issue. Have you found a solution?

br/andré