Migrating to KAS - how to get a list of all affected projects?

I am migrating a self-hosted Gitlab instance from cert-based k8s to KAS. This instance has hundreds of projects scattered across dozens of groups. In some cases it’s the groups that have the k8s config and in others it’s the project. There’s even a few mixed cases.

Is there any way I can get a list of all groups and projects and which k8s configs they use, other than going through the interface and each project 1 by 1?

One way to automate this could be querying the GitLab API for all groups and projects underneath and checking whether clusters are configured.

I have hacked a small script that

  • Fetches the group and all sub groups from an ID (can be modified for self-managed to get all groups)
  • Collects all group clusters
  • Gets all projects from group and sub groups
  • Collects all project clusters
  • Prints a summary that includes IDs and web URLs

Requires python-gitlab via pip, a GITLAB_TOKEN with at least maintainer permissions, and a GROUP_ID.

Output formatting is raw, the cluster objects are “as is”. MR in Python: List Kubernetes Clusters (cert based, that are deprecated) and help migration (!11) · Merge requests · Michael Friedrich / api-playground · GitLab

Script

This works very well! I had to add get_all=True to the groups list call for a self-managed instance and then was able to generate a nice overview of what is where.

Thanks a lot!

1 Like

Wonderful, thanks for sharing the missing option too. I have added it into the script :slight_smile:

I have the issue that we have quadrillions of projects where I’m not a member at all, and all my approaches using the API left those projects out (although I’m an admin, which doesnt count for the api in those calls).
Any chance to get this working through the api? What worked for me is dumping the database and then fiddling around with psql to get the integrations, which is a bit cumbersome :slight_smile:

BTW: if someone needs it (don’t blame me if the following command erases your db :rofl:)

select c.cluster_type, c.name, p.name, p.id as Gitlab_Id from cluster_projects cp left join clusters c on cp.cluster_id = c.id left join projects p ON cp.project_id = p.id where cluster_type = '3';

cluster_type=3 is the k8s integration.
Returns Cluster Name, Project Name and the Gitlab ID.

I’m curious why this does not work. As an administrator, you can, for example, pass the sudo parameter to impersonate other users. That makes me believe that a PAT from an admin user will work - I don’t have access to a self-managed GitLab instance currently.

Can you share the API calls/script you are using? Maybe something else is filtering projects away.

Already tried the sudo param. What it does is that you can act as another (non-super)user which also will have access to only a project subset. (think of the impersonate function).
Thing is, you can’t even get a userlist for a specific project ( to then pick an appropriate user for that project and issue calls with sudo ) as long as you aren’t a member :slight_smile:

These are the fragments of my attempts :slight_smile:

import requests
import json
import os

gl_token = "Bearer " + os.environ['GITLAB_TOKEN']


def get_projects():
    headers = {"Authorization": gl_token, "Sudo": "redacted_user"}
    page = 1
    next_page = True
    while next_page:
        r = requests.get(
            "https://gitlab.redacted.de/api/v4/projects/?per_page=50&page=" + str(page), headers=headers)
        if r.headers['x-next-page']:
            page += 1
        else:
            next_page = False
        data = json.loads(r.text)
        for i in data:
            print(str(i['id']) + ": " + str(i['name']))
            get_integration(i['id'])


def get_integration(project_id):
    headers = {"Authorization": gl_token, "Sudo": "redacted_user"}
    r = requests.get(
        "https://gitlab.redacted.de/api/v4/projects/" + str(project_id) + "/clusters", headers=headers)
    data = json.loads(r.text)

    for i in data:
        #print(i['name'] + "   " + i['platform_kubernetes']['api_url'])
        print(i)
    print(data)

get_projects()

Yep, sorry for not being that clear with my thoughts, it was late. I thought of testing whether the PAT really has admin rights by trying sudo - when it errors out, the PAT does not have admin rights and it may offer research entry points. One way I would tackle it - create a new PAT in your profile, and give it full API access.

The Python snippet looks good, maybe you can try my script to see if it makes any difference? python-gitlab abstracts the project fetching away, and also supports collecting sub groups. That could also be an entry point.

ah ok. Already tried that. The PAT has sudo capabilities and it works (as designed…).

But it seems that the get_all option somehow does the job. Ran your script- unfortunately it crashes while iterating over some projects. I already identified one- archived projects. This is quite easy

.....
manageable_project = gl.projects.get(project.id)
if project.archived:
            continue
.....

within the for projects loop. Unfortunately it seems that there’s another reason that a
manageable_project.clusters.list
fails with a 403. Will investigate…