Monitoring For Gitlab Software Package Updates and CVEs?

Summary: I have a monitoring platform, which I’d like to alert me when there’s a critical CVE on my running gitlab-ce self-hosted server.

I have a monitoring platform (currently CheckMK), which can run arbitrary scripts on my Gitlab (CE) self-hosted server. I can use this to run apt list --upgradeable, pipe it through some greps and whatnot to tell me if there’s an update to Gitlab available. I can then tell CheckMK to show an OK, or Warning depending on what I find from that information - this is all fine.

I’d additionally like to make CheckMK show a “critical” if there’s a known CVE in my current version. I can find this out by logging on to the UI as an administrator, but is there a way to do that programatically instead? Maybe an API endpoint, or a Rails Console command to run or something? I suppose it may be a matter of querying gitlab.com - which is also fine with me - just so long as I can do it in a script.

Googling around didn’t turn much up (mostly things about “Releases” which is a feature for repos inside Gitlab, not about Gitlab itself).

Any ideas?

You might want to look at other third party security software that can be used for this kind of thing. They usually have up to date databases with security data that just needs the software’s version number to tell you if there is a security vulnerability. Some vendors have free tiers that allow you to do scans regularly.

If you really want to hack it together,

If checkMK can scan for incoming emails maybe sign up a email address to the gitlab security emails. I don’t know if there is a gitlab webhook or RSS feed out there somewhere.

if checkMK can web-scrape data then set up a base user with read access and just pull the data from the help section. “Update ASAP” usually means critical vulnerability. “Update available” just means basic patching.

image

I can indeed write a script to web-scrape the help page - that is (probably) feasible. I was rather hoping for an API endpoint, but if there is none, then that’ll have to do. Email is another option, but that feels like it’s going to be tricky to get it right.

I take your point about using a proper security platform to do this - definitely a better option, but beyond my little org’s reach at the moment. I’d also suspect that would be slower to respond that Gitlab itself (not that response time is super critical to me, mind you).

Found an RSS feed for security patches. You should be able to scrape data out of there about new security updates for specific versions and just compare it to your gitlab version.

Gitlab Version API:

Security RSS:
https://about.gitlab.com/security-releases.xml

The version API endpoint was replaced with metadata API endpoint, unfortunately it doesn’t include the field for what we see in the Admin WebUI. It would be nice if it did, at least would be easier to do something then. Perhaps opening an issue with Gitlab to add this info to the metadata API endpoint (Metadata API | GitLab) would be an option? Since they are able to show in the GUI that there is an update available or even that a critical update is available, that this info could also be obtained from the metadata API endpoint as well for monitoring/notifications.

EDIT: in fact I raised an issue for it here: Add to metadata API endpoint info about updates being available (#439531) · Issues · GitLab.org / GitLab · GitLab

Thanks for raising the issue - it seems like it would be a useful addition (and presumably not hard to add), so hopefully they’ll accept it.

The RSS feed looks useful - there’s no definite way to determine if there’s an update for my specific version, but I can probably do well enough by scraping version numbers from the titles of the articles.

Another option appears to be using the gitlab-rails console to run VersionCheck.new.response - if I read it right, it return nil if all is good, or else a hash if not - that hash may contain a critical_vulnerability key if a security update is required. I can’t find doco or a real use/example of it to be sure though (the use of it I found is here: app/helpers/version_check_helper.rb · master · GitLab.org / GitLab · GitLab which makes use of the code here lib/version_check.rb · master · GitLab.org / GitLab · GitLab).

Running gitlab-rails console is pretty slow, as is trying to run some Ruby which does the same sort of thing. I reckon it’ll be okay for a daily check though (which CheckMK is happy to do). I’ll see if I can put together a check using /usr/bin/gitlab-rails runner to run a Ruby fragment which does the version check and outputs the magic data that CheckMK needs. Whenever an update becomes available I’ll find out how to use the data returned by VersionCheck (!)

A REST endpoint would be far easier :wink:

…actually, possibly easier (although go easy on their endpoint):

VERSION=`echo '{"version": "16.8.1"}' | base64`
curl -e "https://myserver.some.domain" "https://version.gitlab.com/check.json?gitlab_info=${VERSION}"

You get back JSON like this:

{"latest_stable_versions":[],"latest_version":"16.8.1","severity":"success","critical_vulnerability":false,"details":""}

The gitlab code caches this request/response pretty aggressively, so as I say, don’t be hitting up their endpoint more than once or twice a day at most.

1 Like

That only works if you have set up your own GitLab server to tell the company which version you run. I know I’m not the only one who believes that what version we run at work is not their business.

@grove - yes, true - good point. They do seem to have interlocks that prevent version checking for people such as yourself. If you hand-craft the check (as above), you don’t need to give them a “real” referrer, but they’ll get your IP address of course, and could quite probably infer everything if they really want to.

I’m not sure I can immediately think of a way to check for upgrades without giving away the current version other than the original apt list --upgradeable method, but that won’t tell you if it’s ‘critical’ or not. I guess you could then cross-check with the RSS feed to see if the version is mentioned - although now I look in more detail, I see that 16.8.0 is not mentioned in the RSS feed, yet it was a release, and addressed a security problem (so this method isn’t likely to be fool proof). You’re probably in the realms of “use a security product” (as suggested above) to make this really robust - assuming budget allows.

Here’s my check script - very short of exception handling(!)

#!/usr/bin/python3

import subprocess
import re
import requests
import base64
import json

def get_gitlab_apt_package_version():
    cmd = ['/usr/bin/dpkg-query', '-f', '${version}', '-W', 'gitlab-ce']
    process = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    for line in iter(process.stdout.readline, b''):
        process.kill()
        out = line.decode('utf-8').strip()
        out = re.sub('-[ce]e\..*$', '', out)
        return out

def compare_version(current_version, server_version):
    if current_version == server_version:
        # Quick simple check
        return 0

    current = current_version.split('.')
    server = server_version.split('.')

    if server[0] > current[0]:
        # major version available
        return 3
    if server[1] > current[0]:
        # minor version available
        return 2
    if server[2] > current[2]:
        # bug fix available
        return 1
    return 0

gitlab_version = get_gitlab_apt_package_version()

b64_version = base64.urlsafe_b64encode(json.dumps({ "version": gitlab_version}).encode('utf-8')).decode('utf-8')

url = 'https://version.gitlab.com/check.json?gitlab_info={}'.format(b64_version)

out = requests.get(url, headers={'Referer': 'https://my.server.name'})

server_data = out.json()

compare = compare_version(gitlab_version, server_data['latest_version'])

# From here on, it's CheckMK specific

checkmk_status = 3
checkmk_message = "Unknown"

if compare == 0:
    checkmk_message = "Current version ({}) is the latest - up to date".format(gitlab_version)
    checkmk_status = 0
elif compare == 1:
    checkmk_message = "Bug fix version is available: {} (Current = {})".format(server_data['latest_version'], gitlab_version)
    checkmk_status = 0
elif compare == 2:
    checkmk_message = "Minor version upgrade is available: {} (Current = {})".format(server_data['latest_version'], gitlab_version)
    checkmk_status = 1
elif compare == 3:
    checkmk_message = "Major version upgrade is available: {} (Current = {})".format(server_data['latest_version'], gitlab_version)
    checkmk_status = 2

if server_data['critical_vulnerability']:
    checkmk_message = "Critical vulnerability fix is available: {} (Current = {})".format(server_data['latest_version'], gitlab_version)
    checkmk_status = 2

print('{} "Gitlab Version" - {}'.format(checkmk_status, checkmk_message))