"Your account has been blocked." via git pull/clone using SSH on 12.9.1 floss self-managed

Hi all,

Just recently we’ve started seeing:

remote: ========================================================================
remote: 
remote: Your account has been blocked.
remote: 
remote: ========================================================================

The user has always had access and is used in our ansible playbooks. We can see SSH is working, so it’s not issue #212178

debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
xxx
xxx
debug1: Authentication succeeded (publickey).

I can see the user DOES have access on the project at Developer level.

Where do I look to debug this? I note on the automated user accounts that have been created for this they have an unconfirmed email address. Would that be the issue? These aren’t really users that login via the gui.

Thanks,
Gavin.

I have personally had an account with an unconfirmed email address thta I could log into (I don’t think I ever used it for git operations though, it was mostly for testing), so I don’t think that can be the reason.

My guess is that the user has actually been blocked (that’s a specific state in GitLab).

In any case the instance adminstrator should search for the user under “Users” in the admin area (there’s a “Blocked” tab, that you probably need to select to search between blocked users). If the user has been blocked, there’s an “Unblock user” button. If there’s some other problem, you can probably also see that there, you might want to compare the problematic user with one who works. There should also be a “Confirm user” button, if the email address is unconfirmed, that just marks the email address as confirmed (the first time I tried, I expected it to sent a mail with instructions to complete the verification, and had therefore wasted time making sure that mail ended up somewhere I could be found).

2 Likes

It does sound like this user has been blocked.

You can view blocked users remove existing blocks in the GitLab admin web UI as Grove suggests above.

If you’re using LDAP for authentication, also check that this user has not been deleted in LDAP, as this can also cause a user to get blocked.

https://docs.gitlab.com/ee/administration/auth/ldap.html#user-deletion

1 Like

Thanks @grove. These are not blocked users as far as I can see. I have also forced the Confirm User like you said and I still get:

git pull
remote: 
remote: ========================================================================
remote: 
remote: Your account has been blocked.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

there are two legitimately blocked users. I’m going to re-enable one of them that I think may have created the key that is used to create all these other users via an ansible role we use in all our playbooks.

Thanks.

The whole LDAP thing made no difference. I followed this to remove LDAP identities and then unblock the user:

https:// gitlab .com/gitlab-org/gitlab-foss/-/issues/13179#note_4281467

Same issue remains.

Here’s the account and the project. All looks good. I have NO IDEA why it’s showing “Your account has been blocked.” Would this show in the logs why?

and

Definitely not blocked.

See account (it also has no LDAP identities):

and summary (only 1 blocked legitimate user, this is happening to quite a few created through the API which have worked for years):

Thanks.

I note here that the deny message is not related to the account being blocked via the GUI:

Thanks for the quick response!

I suspect the IP has been blocked, not the user.

Can you check /var/log/gitlab/gitlab-rails/auth.log for any block/throtted messages?

2 Likes

Thanks. I have:

/var/log/gitlab/gitlab-rails/audit_json.log

I see it’s supposed to be there and I’m on 12.9.1, but it’s not as I think that’s only on EE version:

https://docs.gitlab.com/ee/administration/logs.html#authlog

Any ideas now where to look other then me do a code trace to see what’s triggering this? So very strange.

Hi @ghenry,

I’d check /var/log/gitlab/gitlab-rails/production.log and /var/log/gitlab/gitlab-rails/api_json.log for any errors or messages around this user getting blocked.

zgrep is helpful for searching rotated/compressed logs example:

zgrep "username" /var/log/gitlab/gitlab-rails/production*.gz | grep "block"

To manually unblock the user, you can use the Rails console if the API isn’t an option: https://docs.gitlab.com/ee/security/unlock_user.html

1 Like

No difference:

xx@xxx:~$ sudo gitlab-rails console -e production
--------------------------------------------------------------------------------
 GitLab:       12.9.1 (63745c932cc) FOSS
 GitLab Shell: 12.0.0
 PostgreSQL:   9.6.17
--------------------------------------------------------------------------------
Loading production environment (Rails 6.0.2)
irb(main):001:0> user = User.where(id: 33).first
=> #<User id:33 @xxxxxx>
 
irb(main):002:0> user.unlock_access!
=> true

then tried a git pull again:

remote: 
remote: ========================================================================
remote: 
remote: Your account has been blocked.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

I appologize, you’d need to unblock the user and I sent you docs on how to unlock the user. Can you try the following?

sudo gitlab-rails console -e production
user = User.where(id: 33).first
user.state = "active"
user.save

If that doesn’t work, try this in the ruby console to see if its something specific to LDAP:

user.ldap_user?
user.ldap_blocked?
user.ldap_identity

Let us know how it goes either way!

2 Likes

No joy:

Loading production environment (Rails 6.0.2)
irb(main):001:0> user = User.where(id: 33).first
=> #<User id:33 @xxxxxx>
irb(main):002:0> user.state = "active"
=> "active"
irb(main):003:0> user.save
=> true
irb(main):004:0> user.ldap_user?
=> false
irb(main):005:0> user.ldap_blocked?
=> false
irb(main):006:0> user.ldap_identity
=> nil
irb(main):007:0> 

I still get:

git pull
remote: 
remote: ========================================================================
remote: 
remote: Your account has been blocked.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Hi,

please share the remote origin for the repository where you run git pull. The most convenient way would be

grep url .git/config

Also, side question - does it work with clone/push over https?

Cheers,
Michael

1 Like

Hi,

sure. As it’s always been for 3 years:

[remote "origin"]
	url = ssh://git@git.xxxx.net/xxxxx/xxx-xxx.git
	fetch = +refs/heads/*:refs/remotes/origin/*

These users that are used in our ansible projects are created like this via a customer lib (and have always worked, but only use ssh):

#!/usr/bin/python
# Only works with the root user currently
import string
import random
from socket import getfqdn
from gitlab import Gitlab
from gitlab import DEVELOPER_ACCESS
from gitlab.exceptions import GitlabCreateError

def get_user_id():
  for user in gl.users.list(search=getfqdn()):
    if getfqdn().split('.')[0] == user.username:
      return user.id
  if not found:
    module.fail_json(msg='User cannot be found' + user_id)

def deploy_key():
  user_id = get_user_id()
  try:
    with open(module.params['deploy_key'], 'rb') as keyfile:
      keycontents = keyfile.read()
      for key in gl.users.get(user_id).keys.list():
        if key.key.rstrip() == keycontents.rstrip():
          module.exit_json(changed=False, msg='Key exists')
      gl.users.get(user_id).keys.create({
          'title': getfqdn() + ' root key',
          'key': keycontents
          },
          user_id=user_id)
      module.exit_json(changed=True, msg='Key added')
  except EnvironmentError:
    module.fail_json(msg='Could not open key file')
  except GitlabCreateError as e:
    if 'has already been taken' in e.response_body:
      module.exit_json(changed=False, msg='Key Exists')
    else:
      raise e

def create_user():
  if len(gl.users.list(search=getfqdn() + ' root')) == 0:
    user = gl.users.create({
        'email': 'root@' + getfqdn(),
        'password': ''.join(random.choice(string.ascii_uppercase) for i in range(60)),
        'username': getfqdn().split('.')[0],
        'name':  getfqdn() + ' root'
        })
    user.save()
    module.exit_json(changed=True, msg='User successfully created')
  else:
    module.exit_json(changed=False, msg='User already exists')

def add_to_project():
  user_id = get_user_id()
  # check project exists
  try:
    project = gl.projects.get(module.params['project'])
  except:
    module.fail_json(msg='Could not find project')
  # check if already a member of the project
  for user in project.members.list():
    if user.id == user_id:
      module.exit_json(changed=False, msg='already member of the project')
  # add to project
  if project.members.create({ 'user_id': user_id, 'access_level': DEVELOPER_ACCESS }, project_id=project.id):
    module.exit_json(changed=True, msg='Successfully added to the project')
  else:
    module.fail_json(msg='Unable to add to project')

def main():
  global module, gl
  module = AnsibleModule(
    argument_spec = dict(
      operation = dict(required=True, choices=['deploy_key', 'create_user', 'add_to_project']), # Operation
      project = dict(required=False), # Project name, only relevant if add_to_project is selected
      deploy_key = dict(required=False, default='/root/.ssh/id_rsa.pub'), # path to deploy key, example /root/.ssh/id_rsa.pub, only relevant if 'deploy_user' is selected
      token = dict(required=True), # API Key
      host = dict(required=True), # GitLab hostname
      verify_ssl = dict(required=False, default=False, choices=[True, False]),
    )
  )

  gl = Gitlab(module.params['host'], module.params['token'], api_version=4)
  
  if module.params['operation'] == 'deploy_key':
    deploy_key()
  elif module.params['operation'] == 'create_user':
    create_user()
  elif module.params['operation'] == 'add_to_project':
    add_to_project()

from ansible.module_utils.basic import *
main()

So nothing to do with LDAP and this user has worked for for the past few users. It is every user created this way that has recently broken and gets this “Your account has been blocked” message. No expired keys as if you see further up as ssh is authing correctly. I note a password is created, so I could test via HTTPS.

Thanks,
Gavin.

Any ideas anyone. At a real loss where to look next.

So I just deleted this user and re-ran our ansibile playbook that checks the user exists and if it doesn’t, re-creates it. I can see the user was created today in the Users section, but I still get the “Your account is blcoked” error.