After upgrading to 13.12 or 14.0 our code quality issues have disappeared and the merge request widget always shows “No changes to code quality”.
We run our self-managed GitLab using the Docker images and use the free version. When on 13.11 the quality issues worked fine. Upgrading to 13.12 made the quality issues disappear. Downgrading to 13.11 again made the issues visible again. We hoped that 14.0 would have solved the issue but after upgrading from 13.11 to 14.0 (via 13.12) the quality issues disappeared again. Also, I believe we have had the problem of disappearing quality issues in an other version prior 13.11 but I am not sure which version that was. I get the feeling that the problem might be related to the process of upgrading itself, and not to a particular version.
I am not sure whether this is a GitLab bug. I searched for an issue in the tracker but no one reported it yet although there seem to exist some old related issues. If everyone would have this problem I would expect an issue to already be filed. Also I looked into some log files to see whether there was an exception relating to code quality reports but I could not find one. But I am not absolutely sure where to look. Is it possible we hit a bug? Is this a known problem?
@PeterJanRoes Thanks for the post!
This is probably related to a bug where the state of the report isn’t taken into account and that MR Widget displays a non helpful message until the parsing is done. This is often resolved after waiting awhile and refreshing the page which is not a great experience, not a great experience and something we will get fixed soon.
Hopefully this helps!
-James H, GitLab Product Manager, Verify:Testing
@jheimbuck_gl Thank you for your answer. I think I already came across that bug before but thought that it did not apply to our situation. For starters, I have waited for over a day now for the quality issues to show up in a merge request refreshing the page every once in a while, but they never do. Also, when I look at the communication between browser and server, I see that the server seems to indicate there are no degradations or improvements, and the parsing has already finished. The following is the 200 OK response from codequality_reports.json for an MR for which I am certain that quality issues existed before.
{"status":"success","new_errors":[],"resolved_errors":[],"existing_errors":[],"summary":{"total":0,"resolved":0,"errored":0}}
So I suspect the problem is server side. I have to say we use a custom tool that generates the CodeClimate issues. Something might have changed between GitLab 13.11 and 13.12 that breaks the parsing. However, I would expect an error in such a case. I have also looked at the GitLab source a while ago to see whether I could find clues for the exact syntax that is accepted by GitLab. I believe I validated our report file against the JSON schema I found, and it seemed to conform.
Can I provide you with some details that would allow you to investigate further? Or do you still believe it is related to the bug you linked to?
@PeterJanRoes - thanks for the extra information.
My next step would be to download the report file and ensure there is data in it that the comparison can use.
We did make a change in 13.12 that the comparison is now done in the backend for performance. We haven’t seen any other reports of problems since that release.
Hopefully this helps!
-James H, GitLab Product Manager, Verify:Testing
I just checked and there is a 6.8 Mb quality report file and I have downloaded it. To me it looks fine. Can I easily test whether the backend parser would accept it? Actually, I think it would be a great (new?) feature for GitLab to be able to test a quality report using the GitLab interface (Code Quality Lint?) like you are able to do for the CI/CD configuration using CI Lint. This would be great for users like us that develop there own custom tool.
I also would like to download and check whether the code quality artifact is ok for the merge request baseline, but I am not sure how to do this. Is that possible through the GitLab user interface?
@PeterJanRoes - There’s not a linter or way to validate the report directly in GitLab no. The features are expecting a Codeclimate report so the CLI tool may prove useful though in that regard.
The job artifacts will be available to download.
-James H, GitLab Product Manager, Verify:Testing
@jheimbuck_gl, I believe the code quality artifacts are present in both the source and target branch for the merge request: I can properly download both. Also, I have checked both against the schema that I found in the codebase at app/validators/json_schemas/codeclimate.json using https://www.jsonschemavalidator.net/. Moreover, I believe there should actually be some degradations present because using git diff on both files shows some differences.
I also launched a gitpod with the GitLab code and browsed the code a bit. I tried to find a specific test to run for testing the backend code quality parser with a part of my report. However, I have to understand the way these tests are organized in the GitLab codebase a bit better. Maybe you can indicate which input test file I can change and which specific test I can run to test my report?
Finally, I am still a bit uncertain which report is used by GitLab as the baseline. I have a merge request for branch-a to master. Of course, the latest artifact for branch-a will be used to determine the degradations. But which one will be used for master? Is that the artifact for the latest pipeline on master? Or can I find the artifact from the page for the merge request itself? If I know that, I can be absolutely sure both artifacts are available and can compare them myself to see whether I would expect any degradations.
Thank you for your help!
@PeterJanRoes - The report for the target branch is from the last commit, so if there are commits after the latest one with a report that maybe the issue. I think this bullet from the troubleshooting section applies:
- The widgets use the pipeline of the latest commit to the target branch. If commits are made to the default branch that do not run the code quality job, this may cause the merge request widget to have no base report for comparison.
-James H, GitLab Product Manager, Verify:Testing
@jheimbuck_gl, thank you for the information. With this I checked the code quality artifacts for both the source and target branch. There are no intermediate commits on the target branch (master) without a code quality report: all changes to master are applied through merge requests and all of these generate the artifacts. Both code quality artifacts look fine and using diff I can see there are actually some degradations/improvements.
When downloading the artifacts from both branches I noticed a difference in file name when downloading using the download button for artifacts in the merge request/pipeline and when downloading the artifacts from the artifacts archive from the job page. When downloading from the job page the file has its original name, CodeQuality.json in our case. When downloading from the merge request the file has a different name, which is gl-code-quality-report.json. I cannot see how this could be the source of the problem but it may be.
I am starting to believe everything is set up correctly at our end and this could be a bug in GitLab introduced by switching from client-side parsing to server-side parsing. Or it may be a problem introduced by the upgrade process itself. Is it possible that a background migration did not finish properly and might result in this problem? I would expect more serious errors in that case though.
@PeterJanRoes thanks for the extra info!
I agree, I don’t think the file name upload is the issue and as long as the files are uploaded using the keyword artifacts:reports:codequality the features should make use of them.
At this point either opening an issue with additional logs and the .gitlab-ci.yml file OR opening a support ticket would be the way to go.
Thanks!
-James H, GitLab Product Manager, Verify:Testing
@jheimbuck_gl, I decided to try and debug GitLab in a Gitpod using debug log statements and I was able to track down the problem. Apparently the backend parser (lib/gitlab/ci/parsers/codequality/code_climate.rb · master · GitLab.org / GitLab · GitLab) is more strict than the frontend parser in one respect: it fails on a BOM (byte order mark) at the beginning of the JSON report. As said, we generate a CodeClimate JSON file using our own custom tool and it adds a BOM. The JSON specification says that a BOM MUST NOT be added to a JSON text. So I will remove the BOM from our reports.
However, the specification also says that a parser MAY ignore a BOM at the beginning of a document. It is up to you to decide whether you want to allow the BOM to maintain compatibility between the frontend and backend parser. At least now anyone who encounters the same problem knows to check for the BOM and remove it if present.
2 Likes
@PeterJanRoes thanks for digging in on this! We’ll get a note added to the docs about this.
-James H, GitLab Product Manager, Verify:Testing
1 Like
@jheimbuck_gl, great that a note will be added to the docs! A small question though. At this line the error message from the parser is set on the report. Is it possible to view that error message through the GitLab UI? It would have saved me quite some time if I was able to directly see the error message alongside the quality report in the merge request. Also, it is a bit confusing that an erroneous quality report results in zero degradations instead of an error message.
@PeterJanRoes - We’re not currently surfacing parsing errors but sounds like an issue to create to surface that.
-James H, GitLab Product Manager, Verify:Testing