Unauthorized emails sent by gitlab user on self-hosted instance

Problem

A few days ago, someone accessed the server running my company’s self-hosted instance of GitLab to send out a bunch of spam emails to random email addresses (i.e. email addresses we don’t recognized ourselves). The content of the emails was a standard phishing scam. The emails were sent via our installed Postfix agent as our local gitlab system user. Here’s a sample of an unauthorized email logged in /var/log/mail.log.

Dec  1 00:28:20 HOSTNAME postfix/pickup[5811]: 9BD4A236025B: uid=1002 from=<gitlab>
Dec  1 00:28:20 HOSTNAME postfix/cleanup[22528]: 9BD4A236025B: message-id=<20211201082820.9BD4A236025B@HOSTNAME>
Dec  1 00:28:20 HOSTNAME postfix/qmgr[2076]: 9BD4A236025B: from=<gitlab@HOSTNAME>, size=31957, nrcpt=1 (queue active)
Dec  1 00:28:22 HOSTNAME postfix/smtp[22533]: 9BD4A236025B: to=<SOME_RECIPIENT@REDACTED.com>, relay=smtp.gmail.com[IP_ADDRESS]:587, delay=2, delays=0.15/0.02/0.66/1.2, dsn=2.0.0, status=sent (250 2.0.0 OK  1638347302 e3sm3533108otk.71 - gsmtp)
Dec  1 00:28:22 HOSTNAME postfix/qmgr[2076]: 9BD4A236025B: removed

Basically, I’m now trying to figure out if there’s a vulnerability within our GitLab instance that allowed an attacker to send these emails, or if the fact that the emails were sent by the GitLab users is a red herring covering a different security flaw.

Notes

  • Our GitLab version is GitLab Enterprise Edition 13.5.3-ee. I noticed this means we are potentially vulnerable to the exiftool exploit, but I don’t think that was the problem here. Our GitLab instance is private, and we don’t allow random users to register accounts. I also don’t see any indication of exiftool being called in our logs.
  • I don’t see anything in our access logs (/var/log/auth.log) suggesting someone successfully logged in as the gitlab systems user around the time the attack began, or had an open session from earlier.
  • I don’t see anything particularly striking around that time in any of GitLab’s various logs either. One problem, though, is that I’m not really sure what I should be looking for in which logs.

If anyone could offer any advice, I’d really appreciate it.

I would say do this:

cat /var/log/mail.log | grep 9BD4A236025B

that will filter the log for the entire message and you can see if it was a local connection or whether a public IP connected directly to postfix. Perhaps you have an open-relay and they didn’t need to do anything with Gitlab. But you would need to double-check the logs for that.

Thanks for the reply. Unfortunately, the snippet I posted above is all there is for ID 9BD4A236025B; no IP address information present. I don’t think we have an open-relay. Our firewall blocks non-local traffic to most ports, including port 25. I also, get a Relay access denied error when trying to send an email from another server via telnet. And the mail.log logs don’t look the same when connecting to smtpd directly (e.g. Dec 7 11:23:59 HOSTNAME postfix/smtpd[30362]: connect from unknown[192.168.2.10]).

It looks to me like the gitlab user was able to insert the messages into the mail queue locally. I don’t see anything in the logs suggesting someone was able to ssh in as the gitlab user, and the only other publicly open ports on this machine are LDAPS and HTTP/HTTPS. I don’t see anything strange in any of the GitLab access logs, but I know GitLab runs several internal services, and I’m wondering if one of them might have been compromised somehow. Do you know of any particular GitLab logs that might be worth checking?

If it was a standard phishing scam, ie: you saw the email data, then that would mean someone has compromised your system. Gitlab itself would not send scam or phishing emails. You said your server is not accessible externally? Somehow this has been compromised or someone on your internal network has caused an issue with your server.

Otherwise, this simply isn’t possible.

Hi @iwalker, the unfortunate resolution to this issue is that we found that the gitlab user had been compromised, perhaps sometime back in early November. It seems quite likely that our server was hit with the exiftool exploit after all. Here are the mistakes we made, and the lessons we learned.

  1. We saw a post about the exploit, but thought we were safe because unauthenticated users could not post images. This was incorrect, as it turns out that you don’t need to be logged in to access exiftool. Lesson learned: just keep GitLab up-to-date, especially when an exploit is patched.
  2. We keep our GitLab instance generally accessible from the internet so people can access it from home. This is fine for issues and wiki stuff, but we had a bit of a scare when we realized our code repository may have been compromised. Lesson learned: a login portal isn’t secure enough on its own for sensitive data.
  3. We noticed the phishing emails last Friday, but didn’t make a copy of the logs until Tuesday, so we lost some info to log rotation. Lesson learned: make a copy of all logs as soon as something goes wrong.

For reference, it looks like our system was infected with a variation of this worm, plus the phishing emails (which we did see, actually).

Thanks again for your help.

4 Likes