User suddenly gets "Your account has been blocked." when using SSH

How to Use GitLab Self-managed
1

Today one of the users on (one of) the instance(s) I administer started complaining about getting “Your account has been blocked.” messages when using SSH to pull/push a project in GitLab.

He has no problems accessing the webUI, and using HTTPS urls for projects also works for him.

When I check in the admin area, his account looks perfectly normal, and is not listed as blocked. I’ve tried blocking and unblocking him, with no change in beaviour.

Like most other accounts, his was created using the API, but was neither the first nor the last to be so.

There are many settings for each user, so they don’t all fit on one screen, so instead of providing a series of screenshots, here are the (lightly edited to protect his identity and the company we work for) result of doing a GET in the API of his account:

{
  "id": 250,
  "name": "SSSSSSS GGGGGG",
  "username": "ssssssss",
  "state": "active",
  "avatar_url": "https://secure.gravatar.com/avatar/5b............................bb?s=80&d=identicon",
  "web_url": "https://gitlab.xxx.xxx/ssssssss",
  "created_at": "2018-10-04T07:22:51.938Z",
  "bio": "",
  "bio_html": "",
  "location": null,
  "public_email": "",
  "skype": "",
  "linkedin": "",
  "twitter": "",
  "website_url": "",
  "organization": null,
  "job_title": "",
  "bot": false,
  "work_information": null,
  "followers": 0,
  "following": 0,
  "last_sign_in_at": "2021-08-05T15:52:41.691Z",
  "confirmed_at": "2018-10-04T08:04:57.122Z",
  "last_activity_on": "2021-08-06",
  "email": "ssssssss@xxx.xxx",
  "theme_id": 5,
  "color_scheme_id": 3,
  "projects_limit": 1000,
  "current_sign_in_at": "2021-08-06T10:11:28.179Z",
  "identities": [],
  "can_create_group": true,
  "can_create_project": true,
  "two_factor_enabled": false,
  "external": false,
  "private_profile": false,
  "commit_email": "ssssssss@xxx.xxx",
  "shared_runners_minutes_limit": null,
  "extra_shared_runners_minutes_limit": null,
  "is_admin": false,
  "note": null,
  "using_license_seat": true,
  "highest_role": 40,
  "current_sign_in_ip": "1.2.3.4",
  "last_sign_in_ip": "1.2.3.4",
  "sign_in_count": 117,
  "plan": "free",
  "trial": false
}

What can cause this?

(It sounds similar to

but that seems to be about the users needing to be confirmed, but as can be seen from the API output above, this user is confirmed, so it’s not the same issue?)

2

Hi,

I’ve edited your post and replaced the IP addresses in the sign_in fields.

Which GitLab version are you using, 14.1?

A few ideas:

  • Did they change their primary email address recently, and did not confirm it
  • Did someone report their account for abuse? (blind guess)
  • rails console - search for the user and see if they are blocked User.find(id).blocked? (to rule out an API problem not showing everything to the token used)

Cheers,
Michael

3

Thank you for replacing those IP addresses, I had though about them, but must have gotten away from it somehow.

This server unfortunatey still runs 13.12.9, because it’s an Ubuntu Xenial. (I’m working on a new server to replace it, and this coincided with a failed attempt to deploy the new server, but I’ve basically only stopped and started puma on this, so I don’t see how that can have triggered anything. - When I’ve gotten the new server running, I plan on upgrading it to 14.1 within days)

The email-address associated with this account looks like the one used when creating the user. The database also says that no confirmation mails have been sent (recently), I see that by doing sudo gitlab-psql and executing a simple query:

gitlabhq_production=# SELECT id, reset_password_sent_at, locked_at, password_expires_at, confirmed_at, confirmation_sent_at, unconfirmed_email FROM users WHERE id=250;
 id  | reset_password_sent_at | locked_at | password_expires_at |        confirmed_at        |    confirmation_sent_at    | unconfirmed_email 
-----+------------------------+-----------+---------------------+----------------------------+----------------------------+-------------------
 250 |                        |           |                     | 2018-10-04 08:04:57.122267 | 2018-10-04 07:22:51.939467 | 
(1 row)

It would be quite strange for one colleague to report another for abuse, so I don’t think so, but how do I check that?

I would probably have noticed it when I looked around in the database, if the user was actually locked, but the rails console confirm that:

> sudo gitlab-rails console                                                                                                       ~
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
 GitLab:       13.12.9-ee (cdec15d2a57) EE
 GitLab Shell: 13.18.1
 PostgreSQL:   12.6
--------------------------------------------------------------------------------
Loading production environment (Rails 6.0.3.7)
irb(main):001:0> User.find(250).blocked?
=> false
irb(main):002:0>

(that also provides confirmation that I’m using 13.12.9)

4

Without me having done anything (relevant to this), yesterday afternoon the user reported that SSH access had started working for him again?

5

Hi,

That’s odd. I had your question opened in a tab, I wanted to dig into the changelog and issues because I remembered a similar situation around SSH messages a while ago. Not the same as yours though - wanted to start reading the source code for the message actually to analyse potential root causes.

A very wild guess - does the server suffer from performance problems, e.g. hitting open file descriptor or I/O limits which could - somehow - influence SSH server problems? Although, that sounds impossible but might be a way to analyse the health of your instance.

Or it is a bug in 13.12.x - my thoughts go into the direction of the disabled self-registration, requiring approval. 2nd guess: SSH key expiration. My recommendation would be upgrading to 14.1 and see if the issue comes back. If yes, please open a bug report for our engineers, linking this topic. Thanks!

Kind regards,
Michael

6

I would like to know what was going too, but replacing this server with another one (and upgrading that) has higher priority (and will probably make it impossible to debug this, but so be it).

The server is a little low on disk space, and occasionally maxes it’s 1Gbit network interface (for very short periods of time, I noticed it on some monitoring graphs but nobody has ever been bothered by it. But that are the limits I’m aware of hitting. If that was the problem, I also don’t think it would affect all attempts of one particular user.

Self-registration is disabled, but approvals are not required (anymore, I sort of remember having an issue that required me to set every user as approved in the database).

“Your account has been blocked.” is a very bad error message if the SSH key had expired, but as the user only has one SSH key in GitLab, that was created friday night, that (he’s been an employee for ~3 years, so I guess he must have had an SSH key before, but as the message didn’t send me in that direction, I didn’t look, and this is not worth restoring an old backup for).

1 Like