Hello,
since 2017 we are using Gitlab CE on a selfe-hosted debian server. Gitlab is at 16.9.2 CE.
A few weeks ago I noticed, root login at web site is not possible anymore.
The error is: Invalid login or password.
I have changed the root passwort at CLI with command: sudo gitlab-rake ‘gitlab:password:reset[root]’
But without any effect. Login as root on the website is still not possible.
Thanks for Help
You have already resetting the password by using the sudo gitlab-rake. please check you log file for error and warning message related information.
Yes, I have changed the password via sudo gitlab-rake ‘gitlab:password:reset[root]’
I got also the mail for Administrator that the Password was changed by administrator.
But, still no login possible.
Which log-files do you mean I should check?
I’d recommend limiting firewall access to your instance, and review whether you are affected by this security problem.
I’d start with these and make my way through more log files.
If you happen to have the log files from the date when you noticed that root login does not work anymore – could also be helpful. Do not post them here, before reviewing them for sensitive data.
Dear dnsmichi, thank you for helping.
The server is behind a firewall and additionally is restricted to a range of IP addresses.
A root account takeover is not impossible, but hard to imagine.
I had a look at application_json.log files.
At 2024-02-12 was the last successfull root login.
{"severity":"INFO","time":"2024-02-12T07:28:47.152Z","correlation_id":"01HPE3FSDVJKF8BNPBQPF4A403","meta.caller_id":"SessionsController#create","meta.remote_ip":"xxx.xxx.xxx.xxx","meta.feature_category":"system_access","meta.client_id":"ip/xxx.xxx.xxx.xxx","message":"Successful Login: username=root ip=xxx.xxx.xxx.xxx method=standard admin=true"}
{"severity":"INFO","time":"2024-02-12T07:29:22.389Z","correlation_id":"01HPE3GXDTHQWHHVRBMMVE5ZQF","meta.caller_id":"SessionsController#destroy","meta.remote_ip":"xxx.xxx.xxx.xxx","meta.feature_category":"system_access","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","message":"User Logout: username=root ip=xxx.xxx.xxx.xxx"}
Due to the fact I’m not often do a root login, at 2024-03-05 I noticed, no root login is possible.
In The application_json.log is no entry or error when I try to login as root.
In the production.log files I cannot find any noticeable entrys.
Numerous entrys are
Raven 3.1.2 configured not to capture errors: DSN not set
Rendered layout layouts/mailer.html.haml (Duration: 140.1ms | Allocations: 43678)
Rendered layout layouts/mailer.text.erb (Duration: 3.1ms | Allocations: 778)
Delivered mail 65cbf087d562_6356fea076387@xyz.mail (362.3ms)
Migrating to xyz (20240108072545)
...
So I still have no plan where to go on with trouble-shooting
I’ve found an issue where it said that the rake task for the root account might not work, and should use Reset a user's password | GitLab instead.
It could also be the case that the root user is disabled. This is common security practice to avoid exploits,. I have shared an alternative approach with 2FA in Block built-in Administrator user - #3 by dnsmichi
Maybe the root login is also tied to LDAP/AD/SSO (through email or other fields) and does not allow resetting the password.
Dear dnsmichi,
I tested your first proposal and tried it with the gitlab-rake task
gitlab-rake "gitlab:password:reset[root]"
I got the mail with:
Hello, Administrator!
An administrator changed the password for your GitLab account on …
… but the login as root was still impossible with: Invalid login or password.
Your third point, root login ist tied to somethin else may be possible.
I tried some times ago LDAP but never used it.
The settings I did in gitab.rb are:
### LDAP Settings
###! Docs: https://docs.gitlab.com/omnibus/settings/ldap.html
###! **Be careful not to break the indentation in the ldap_servers block. It is
###! in yaml format and the spaces must be retained. Using tabs will not work.**
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = true
###! **remember to close this block with 'EOS' below**
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main: # 'main' is the GitLab 'provider ID' of this LDAP server
label: 'xyz'
host: 'xyz.de'
port: 636
uid: 'uid'
# bind_dn: '_the_full_dn_of_the_user_you_will_bind_with'
# password: '_the_password_of_the_bind_user'
encryption: 'simple_tls' # "start_tls" or "simple_tls" or "plain"
# verify_certificates: true
# smartcard_auth: false
# active_directory: true
allow_username_or_email_login: false
lowercase_usernames: false
block_auto_created_users: true
base: 'ou=xyz,ou=xyz,o=xyz'
user_filter: '(&xyz)'
# ## EE only
# group_base: ''
# admin_group: ''
# sync_ssh_keys: false
#
# secondary: # 'secondary' is the GitLab 'provider ID' of second LDAP server
# label: 'LDAP'
# host: '_your_ldap_server'
# port: 389
# uid: 'sAMAccountName'
# bind_dn: '_the_full_dn_of_the_user_you_will_bind_with'
# password: '_the_password_of_the_bind_user'
# encryption: 'plain' # "start_tls" or "simple_tls" or "plain"
# verify_certificates: true
# smartcard_auth: false
# active_directory: true
# allow_username_or_email_login: false
# lowercase_usernames: false
# block_auto_created_users: false
# base: ''
# user_filter: ''
# ## EE only
# group_base: ''
# admin_group: ''
# sync_ssh_keys: false
EOS
But these settings and testing I did in early 2023. And I never had problems with root login so far.
Assuming these ldap settings are the cause why root login isn’t possible and password change either.
What ist the best way to gain back root login?
The user “root” does not exist in LDAP, but the assigned mail for root ist in LDAP available, but assigned to another user name.
Maybe a pointer - could it be that the root username was renamed?
The rake tasks allow to search for users by email, see the example in How do I change my profile to Admin? - #2 by dnsmichi
gitlab-rails console -e production
Find the user, and print its attributes.
user = User.find_by(email: 'root@domain.com')
pp user.attributes
Hi.
The root username is not renamed, it is still root. Also the mail-adress for root is still the same.
I checked the attributes for the user root. User root is “admin”=>true
root@xyz:/# sudo gitlab-rails console
--------------------------------------------------------------------------------
Ruby: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
GitLab: 16.10.1 (6231cc1c609) FOSS
GitLab Shell: 14.34.0
PostgreSQL: 13.14
------------------------------------------------------------[ booted in 34.08s ]
Loading production environment (Rails 7.0.8.1)
irb(main):001:0> u = User.find_by_username('root')
=> #<User id:1 @root>
irb(main):002:0> pp u.attributes
{"id"=>1,
"email"=>"xyz",
"encrypted_password"=>"xyz",
"reset_password_token"=>nil,
"reset_password_sent_at"=>nil,
"remember_created_at"=>nil,
"sign_in_count"=>537,
"current_sign_in_at"=>Mon, 12 Feb 2024 08:28:45.801207000 CET +01:00,
"last_sign_in_at"=>Mon, 29 Jan 2024 10:50:20.874358000 CET +01:00,
"current_sign_in_ip"=>"xyz",
"last_sign_in_ip"=>"xyz",
"created_at"=>Thu, 20 Apr 2017 15:36:02.117969000 CEST +02:00,
"updated_at"=>Thu, 28 Mar 2024 12:00:49.234232000 CET +01:00,
"name"=>"Administrator",
"admin"=>true,
"projects_limit"=>100000,
"failed_attempts"=>0,
"locked_at"=>nil,
"username"=>"root",
"can_create_group"=>true,
"can_create_team"=>false,
"state"=>"active",
"color_scheme_id"=>1,
"password_expires_at"=>nil,
"created_by_id"=>nil,
"last_credential_check_at"=>Mon, 12 Feb 2024 08:28:46.545967000 CET +01:00,
"avatar"=>nil,
"confirmation_token"=>"xyz",
"confirmed_at"=>Thu, 20 Apr 2017 15:36:02.559844000 CEST +02:00,
"confirmation_sent_at"=>Fri, 21 Apr 2017 11:37:02.011400000 CEST +02:00,
"unconfirmed_email"=>"xyz",
"hide_no_ssh_key"=>false,
"notification_email"=>"xyz",
"hide_no_password"=>false,
"password_automatically_set"=>false,
"encrypted_otp_secret"=>nil,
"encrypted_otp_secret_iv"=>nil,
"encrypted_otp_secret_salt"=>nil,
"otp_required_for_login"=>false,
"otp_backup_codes"=>nil,
"public_email"=>"",
"dashboard"=>"project_activity",
"project_view"=>"activity",
"consumed_timestep"=>nil,
"layout"=>"fixed",
"hide_project_limit"=>false,
"unlock_token"=>nil,
"otp_grace_period_started_at"=>nil,
"external"=>false,
"incoming_email_token"=>"xyz",
"require_two_factor_authentication_from_group"=>false,
"two_factor_grace_period"=>48,
"last_activity_on"=>Mon, 12 Feb 2024,
"notified_of_own_activity"=>false,
"preferred_language"=>"de",
"theme_id"=>nil,
"accepted_term_id"=>nil,
"feed_token"=>"xyz",
"private_profile"=>false,
"include_private_contributions"=>nil,
"commit_email"=>nil,
"auditor"=>false,
"admin_email_unsubscribed_at"=>nil,
"group_view"=>nil,
"managing_group_id"=>nil,
"note"=>nil,
"roadmap_layout"=>nil,
"static_object_token"=>nil,
"first_name"=>nil,
"last_name"=>nil,
"role"=>nil,
"user_type"=>"human",
"static_object_token_encrypted"=>nil,
"otp_secret_expires_at"=>nil,
"onboarding_in_progress"=>false,
"color_mode_id"=>1,
"otp_secret"=>nil}
=>
{"id"=>1,
"email"=>"xyz",
"encrypted_password"=>"xyz",
"reset_password_token"=>nil,
"reset_password_sent_at"=>nil,
"remember_created_at"=>nil,
"sign_in_count"=>537,
"current_sign_in_at"=>Mon, 12 Feb 2024 08:28:45.801207000 CET +01:00,
"last_sign_in_at"=>Mon, 29 Jan 2024 10:50:20.874358000 CET +01:00,
"current_sign_in_ip"=>"xyz",
"last_sign_in_ip"=>"xyz",
"created_at"=>Thu, 20 Apr 2017 15:36:02.117969000 CEST +02:00,
"updated_at"=>Thu, 28 Mar 2024 12:00:49.234232000 CET +01:00,
"name"=>"Administrator",
"admin"=>true,
"projects_limit"=>100000,
"failed_attempts"=>0,
"locked_at"=>nil,
"username"=>"root",
"can_create_group"=>true,
"can_create_team"=>false,
"state"=>"active",
"color_scheme_id"=>1,
"password_expires_at"=>nil,
"created_by_id"=>nil,
"last_credential_check_at"=>Mon, 12 Feb 2024 08:28:46.545967000 CET +01:00,
"avatar"=>nil,
"confirmation_token"=>"xyz",
"confirmed_at"=>Thu, 20 Apr 2017 15:36:02.559844000 CEST +02:00,
"confirmation_sent_at"=>Fri, 21 Apr 2017 11:37:02.011400000 CEST +02:00,
"unconfirmed_email"=>"xyz",
"hide_no_ssh_key"=>false,
"notification_email"=>"xyz",
"hide_no_password"=>false,
"password_automatically_set"=>false,
"encrypted_otp_secret"=>nil,
"encrypted_otp_secret_iv"=>nil,
"encrypted_otp_secret_salt"=>nil,
"otp_required_for_login"=>false,
"otp_backup_codes"=>nil,
"public_email"=>"",
"dashboard"=>"project_activity",
"project_view"=>"activity",
"consumed_timestep"=>nil,
"layout"=>"fixed",
"hide_project_limit"=>false,
"unlock_token"=>nil,
"otp_grace_period_started_at"=>nil,
"external"=>false,
"incoming_email_token"=>"xyz",
"require_two_factor_authentication_from_group"=>false,
"two_factor_grace_period"=>48,
"last_activity_on"=>Mon, 12 Feb 2024,
"notified_of_own_activity"=>false,
"preferred_language"=>"de",
"theme_id"=>nil,
"accepted_term_id"=>nil,
"feed_token"=>"xyz",
"private_profile"=>false,
"include_private_contributions"=>nil,
"commit_email"=>nil,
"auditor"=>false,
"admin_email_unsubscribed_at"=>nil,
"group_view"=>nil,
"managing_group_id"=>nil,
"note"=>nil,
"roadmap_layout"=>nil,
"static_object_token"=>nil,
"first_name"=>nil,
"last_name"=>nil,
"role"=>nil,
"user_type"=>"human",
"static_object_token_encrypted"=>nil,
"otp_secret_expires_at"=>nil,
"onboarding_in_progress"=>false,
"color_mode_id"=>1,
"otp_secret"=>nil}
irb(main):003:0>
If your server is publicly accessible from the internet, you will see people trying to login and block your account. I had this on mine.
Simple solution: disable root user, and create yourself a new login that you can use for administrator purposes.
Generally recommended now is never use root account, admin, or similar for users since they are too simple to guess. If root points to someone’s email address, change it, and disable the account. Give them an account for normal everyday usage, and a separate one for admin purposes - for example username-adm (replace username with whatever their username would be).
Whilst gitlab does offer the ability to ask an admin user to re-authenticate when gaining access to admin resources, I still find it far better to have a separate account for admin purposes to separate those roles. Also, if one account is compromised, there is still another one accessible.
I solved the problem.
First I created with the console in gitlab-rake a new user and granted these user admin rights.
Now I could log in with the new created admin user.
I found out that the root user had an Ldap provider assigned at his identities.
So I removed this Ldap Provider, and a root login was again possible.
Thanks for help and input