Using available logs provided by GitLab, it is possible to determine if a GitLab instance has been compromised through the exploitation of CVE-2021-22205. Note, this issue was remediated and patched in the GitLab 13.10.3, 13.9.6, and 13.8.8 release from April 14, 2021: GitLab Critical Security Release: 13.10.3, 13.9.6, and 13.8.8 | GitLab.
All information provided here should be considered examples of information that could be found in log files. It is not intended to be an exhaustive list of possible log entries. As such, log entries and indicators of compromise (IOC) will vary slightly depending upon deployment configurations and malicious actor exploitation tactics. By default, GitLab instances maintain 30 days of log data. If the activity associated with a compromised instance occurred outside of this retention window, it is unlikely that similar IOC log entries will exist. Additionally, since logs rotate daily the .*.gz files contained in each of the mentioned log directories should also be searched.
Log events indicating a compromise will exist in the GitLab Workhorse current directory /var/log/gitlab/gitlab-workhorse/current which will contain an ExifTool error log entry from the time the vulnerability was exploited. For example:
`{
"correlation_id": "01FEPNG60XWAQ9K5EE3GB909Q6",
"filename": "exploit.jpg",
"level": "info",
"msg": "running exiftool to remove any metadata",
"time": "2021-09-03T20:27:16Z"
}
{
"command": [
"exiftool",
"-all=",
"--IPTC:all",
"--XMP-iptcExt:all",
"-tagsFromFile",
"@",
"-ResolutionUnit",
"-XResolution",
"-YResolution",
"-YCbCrSubSampling",
"-YCbCrPositioning",
"-BitsPerSample",
"-ImageHeight",
"-ImageWidth",
"-ImageSize",
"-Copyright",
"-CopyrightNotice",
"-Orientation",
"-"
],
"correlation_id": "01FEPNG60XWAQ9K5EE3GB909Q6",
"error": "exit status 1",
"level": "info",
"msg": "exiftool command failed",
"stderr": "Error: Writing of this type of file is not supported - -\n",
"time": "2021-09-03T20:27:17Z"
}
Another example from gitlab-workhorse
logs following exploit of this vulnerability is:
{"command":["exiftool","-all=","--IPTC:all","--XMP-iptcExt:all","-tagsFromFile","@","-ResolutionUnit","-XResolution","-YResolution","-YCbCrSubSampling","-YCbCrPositioning","-BitsPerSample","-ImageHeight","-ImageWidth","-ImageSize","-Copyright","-CopyrightNotice","-Orientation","-"],"correlation_id":"01FKBH8HB3A5YR8S7PYYB5A8SN","error":"signal: killed","level":"info","msg":"exiftool command failed","stderr":"sh: 1: Trying: not found\nsh: 2: Connected: not found\nsh: 3: Escape: not found\nConnection closed by foreign host.\n","time":"2021-10-31T11:07:18-07:00"}
{"correlation_id":"01FKBH8HB3A5YR8S7PYYB5A8SN","error":"error while removing EXIF","level":"error","method":"POST","msg":"","time":"2021-10-31T11:07:18-07:00","uri":"/e7c6305189bc5bd5"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FKBH8HB3A5YR8S7PYYB5A8SN","duration_ms":7636442,"host":"10.0.0.7","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":422,"system":"http","time":"2021-10-31T11:07:18-07:00","ttfb_ms":7636436,"uri":"/e7c6305189bc5bd5","user_agent":"curl/7.47.0","written_bytes":2936}
The exiftool command failed
and "error":"error while removing EXIF" in
gitlab-workhorse` logs may be indicators of (attempted) exploitation of CVE-2021-22201.
Correlating the timestamps from the workhorse logs with the NGINX Access Logs in /var/log/gitlab/nginx/gitlab_access_log can be used to identify the IP address of the attacker. However, it should be understood that depending on the network architecture and instance deployment configuration, the identified IP address may not be that of that malicious actor.
192.168.1.50 - - [03/Sep/2021:06:27:08 +0000] "GET /users/sign_in HTTP/1.1" 200 3193 "" "python-requests/2.25.1" 2.46
192.168.1.50 - - [03/Sep/2021:06:27:08 +0000] "POST /users/sign_in HTTP/1.1" 200 3371 "" "python-requests/2.25.1" 2.46
192.168.1.50 - - [03/Sep/2021:06:27:09 +0000] "GET /api/v4/projects?per_page=1000 HTTP/1.1" 200 2 "" "python-requests/2.25.1" -
192.168.1.50 - - [03/Sep/2021:06:27:09 +0000] "GET /help HTTP/1.1" 302 120 "" "python-requests/2.25.1" -
192.168.1.50 - - [03/Sep/2021:06:27:10 +0000] "GET /users/sign_in HTTP/1.1" 200 3379 "" "python-requests/2.25.1" 2.46
In many exploitation cases investigated, the malicious actor created a new account on the instance. Attempts to login and access the instance through the API can be found and correlated in the production_json.log located at /var/log/gitlab/gitlab-rails/production_json.log using the IP address identified previously.
{
"method": "POST",
"path": "/users/sign_in",
"format": "*/*",
"controller": "SessionsController",
"action": "new",
"status": 200,
"time": "2021-09-03T06:27:08.960Z",
"params": [
{
"key": "utf8",
"value": "%E2%9C%93"
},
{
"key": "authenticity_token",
"value": "[FILTERED]"
},
{
"key": "user",
"value"{
"login": "dexbcxh",
"password": "[FILTERED]",
"remember_me": "0"
}
}
],
"remote_ip": "192.168.1.50",
"user_id": null,
"username": null,
"ua":"python-requests/2.25.1
}
Available in GitLab v12.3 and later, the auth.log located at /var/log/gitlab/gitlab-rails/auth.log can also be used to also identify attempted and successful user logins. While not an all inclusive list, some of the most common email addresses and usernames used by malicious actors are as follows:
User emails:
Usernames:
- dexbcx
- dexbcx818
- dexbcxh
- dexbcxi
- dexbcxa99
Additional information and details regarding logging with GitLab can be found online in our GitLab Docs; subsection Log system.