Hi everyone, I’m seeing a change in behavior in API calls to a locally installed Gitlab instance since the latest security update was installed last week and wonder if anyone has any ideas. After the update, calls to retrieve CI/CD job artifacts started failing with 404 errors after having worked for at least a year. I’m using calls of this form:
curl --fail --location --header “PRIVATE-TOKEN: {mytoken}” ‘https://{my.gitlab.host}/api/v4/projects/{url-encoded-project-name}/jobs/artifacts/main/raw/{artifact-path}?job={job-name}’
(Replacing everything between {} with an appropriate value, of course.) These commands are part of CI/CD jobs that retrieve dependencies, and they’ve worked reliably for well over a year. But executing the same commands directly on the command line also fails now.
The problem seems to apply to many projects, some which were rebuilt recently and others which have been unchanged for months. The artifacts do exist and can be downloaded via the Gitlab web interface, just no longer using this curl command.
I’ve also noticed that the project API returns a list of what appears to be a random subset of projects in my gitlab instance instead of all the projects. Using a token in my account (an administrator) I see a comparatively small number of projects belonging to different users, and omitting many (not all) of my own projects. The curl command I’m using is this:
curl --fail --location --header “PRIVATE-TOKEN: {mytoken}” ‘https://{my.gitlab.host}/api/v4/projects’
I hadn’t checked the “projects” query recently, but I recall it returned everything a year or more ago when I used the API for a massive refactoring job.
I tried creating a new token and granting it all the permissions possible but the results were the same.
Is anyone else using these API calls and seeing similarly strange behavior? I’d welcome any advice, as these failures have broken many of my CI/CD jobs.
1 Like
I’m not using artifacts, but the command you gave for listing all projects works fine. I’m using gitlab-ce-13.9.3. I used a personal access token with API rights assigned.
Please check /var/log/gitlab/gitlab-rails/api_json.log maybe something here will explain why it’s not working for you.
1 Like
I don’t see anything that suggests a problem in the api_json.log or any other logs. It just as if it’s simply looking up the wrong information. In fact, the situation got worse this morning when we applied the latest security patch. Now only 2 projects are returned via the API call. I should have mentioned we’re now running v13.9.4 via the official Docker images. I tried rebuilding the redis cache which had no impact, and checking that all the DB migrations had been properly applied (they had). We’re trying a full backup and restore to development instance to see if that improves the sitution.
I am having similar issues as well. Just updated to 13.9.3 but haven’t used the api here before so not sure how long it hasn’t been working as expected.
Curl commands and python using gitlab are returning the same five projects no matter what i do. When i get a project by id using an id from one of the five returned from the list works as expected however using an id found via the web interface looking at a project results in a 404 error.
I have used my PAT with api and/or api_read access as well as impersonated the Administrator and created a PAT for that user… same five project ids are returned.
thoughts? requests for code? (similar to what OP asked but different enough that i don’t want to divert to my code… but suggest there is an issue)
We were just able to get things working again a couple of days ago. It’s very much a mystery why things stopped working, but the issue that originally trigged my problem was that our httpd and nginx servers were normalizing our URLs which was removing the URL encoding of the project name and artifact path. This configuration hadn’t been changed in months, so why two weeks ago it broke is a mystery. During the testing I briefly noticed that one of the test tokens I had created was associated with a different user. Again, I have no idea how this happened as I’m pretty certain I wasn’t masquerading as that user when I created the token (although I had done so the previous day.) We also upgraded to 13.10.0 at the start of the week. All together after fixing the proxy configuration and performing the update we found everything working as it had before. Very much a Twilight Zone experience, though.
thanks paulmer, hopeful that an update here to 13.10 will also fix things.
no such luck here still just five same project ids from the api call
Hi,
Can you please share the full code, and in case you are using the Python GitLab API bindings, its version from the provider (pip, etc.).
Maybe a pagination offset problem, 13.0 introduced pagination limits. I assume you did not modify the default offset of 50000?
Which proxy, own HTTP proxy in front of the Docker setup? If so, how’s that configured with limits?
Cheers,
Michael
pip freeze shows
gitlab-client==0.0.1.dev5
pyapi-gitlab==7.8.5
python-gitlab==2.5.0
import os
import gitlab
gitlab_server_ip = "https://abc.xyz"
gl = gitlab.Gitlab(gitlab_server_ip, job_token=os.environ['TOKEN'], ssl_verify= False)
projects = gl.projects.list(all=True)
print(f"{type(projects)}")
print(f"{projects}")
or even with curl
curl -H “Authorization: token $TOKEN” https://xyz.abc/api/v4/projects
I am not aware of making a change to the pagination limits… when i try to add it in with curl i get the following response
{“error”:“Keyset pagination is not yet available for this type of request”}
ending the above url with ?pagination=keyset&per_page=50
thank you dnsmichi
Yes, this was our own HTTP proxy in front of the GitLab Docker containers. Once those were configured correctly we didn’t find any further issues with GitLab. Why that became a problem suddenly for our CI/CD builds is a bit of a mystery, but it made sense once we found it and didn’t seem to be a GitLab issue.
As to retrieving the projects via the API, I also noted the pagination parameter was affecting the number of results I was seeing, but I don’t think that was the only thing going on as I thought I saw 19 projects in my requests initially, but my memory is fuzzy since I was really paying attention to the CI/CD problem. I changed too many things at once to know precisely which changes impacted which issues. The bottom line was that everything ended up working as documented once we identified all the relevant documentation for Apache, Nginx and GitLab.
1 Like