Gitlab provides option to mute/quarantine flaky tests as mentioned in GitLab Documentation. I Understand its not the best of practices but do want to explore it.
Some of the examples I was able to find are here - Quarantine tests that are very flaky but in rubi language. Please help to understand if that can be done in languages like C# (Dotnet) or Java. An example with that will be highly appreciated.
As per Documentation:
If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be placed in quarantine by assigning the :quarantine metadata .
I am not able to figure out how we can do that in Tests using NUnit (dotnet) or JUnit(Java)
There appears to be a confusion here.
The Contributor and Development section of documentation pages carries entries that pertain to the development of GitLab (the product), and must not be read as product documentation intended for users of GitLab.
The page you link to describes the guidelines and procedures that the developers of GitLab (the product) need to perform when they encounter a flaky test in GitLab’s own codebase. Likewise the issue you’ve linked to is a development issue (actual flaky tests in GitLab that is relevant to its development, not general-purpose).
Conditional test management must be implemented directly into the project, and referenced in the command or configuration that executes the tests. For example, you can apply tags on identified tests in Java: https://www.baeldung.com/junit-filtering-tests, or perform various forms of filtering in .NET after enriching the test metadata: https://docs.nunit.org/articles/nunit/running-tests/Test-Selection-Language.html
GitLab does offer a better way to view test reports from languages that offer publishing a JUnit style XML test report which you may find useful.
Thanks a LOT for clarifying it.
I also came across one issue #52908 (closed) where the proposal is
Identify and track the results of tests (perhaps just in the “test” job?) and allow users to mark individual tests as “allowed_failure” (quarantined).
Is it something that is in planned or will be available in future ? If Yes any ETA ?
Note: The moved (open) issue for that is here: https://gitlab.com/gitlab-org/gitlab/-/issues/24789
It does not appear to be currently planned for implementation. However, I’d encourage you to add your ideas, use-case, and/or needs for the feature as a comment on the issue, and upvote it (
) to add towards its priority.
If you are a subscription customer, you can also try raising it through your account representative.