Hi,

Trying to make a curl request to gitlab.com api for linting .gitlab-ci.yaml file but receiving bad request response: {"status":400,"error":"Bad Request"}

#!/usr/bin/env bash


PAYLOAD=$( cat << JSON 
{ "content":
$(<$PWD/../.gitlab-ci.yml)
JSON
)

echo "Payload is $PAYLOAD"

curl --include --show-error --request POST --header "Content-Type: application/json" --header "Accept: application/json" "https://gitlab.com/api/v4/ci/lint" --data-binary "$PAYLOAD"

Has anyone managed to successfully lint a .gitlab-ci.yml via a bash script?

Update

I think what is happening is that the GitLab CI endpoint expects the contents of the .gitlab-ci yaml file to be converted to json for the POST request, as opposed to directly sending the contents of the yaml file. See here.

So…I modifed the script to use ruby to convert yaml to json before sending and this works for very simple .gitlab-ci.yml.

#!/usr/bin/env bash

json=$(ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < .gitlab-ci.yml)

# escape quotes
json_content=$(echo $json | perl -pe 's/(?<!\\)"/\\"/g')


# Add object contect for GitLab linter
json_content='{"content": "'${json_content}'"}'

echo "${json_content}"

curl --include --show-error --request POST \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "https://gitlab.com/api/v4/ci/lint" \
    --data-binary "$json_content"

However when using the yaml file for my project it gives an error: {"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 221"]}% When I use the gitlab web page for linting the file is valid.

{"content": "{ \"stages\": [ \"build\", \"test\", \"pages\", \"release\" ], \"variables\": { \"DOCKER_DRIVER\": \"overlay2\" }, \"services\": [ \"docker:19.03.11-dind\" ], \"build:plugin\": { \"image\": \"docker:19.03.11\", \"stage\": \"build\", \"before_script\": [ \"echo \"$CI_JOB_TOKEN\" | docker login -u gitlab-ci-token --password-stdin \"$CI_REGISTRY\"\" ].....

Column 221 is \"image\": \"docker:19.03.11\" in the above json extract, specifically at the closing escaped quote. Think it is a problem with incorrectly escaped quotes??

Also tried with other complex files and get the same error. Very confused on how to use the [api]((https://docs.gitlab.com/ee/api/lint.html) to correctly validate gitlab-ci.yml from file…Can anyone from GitLab help with this??

Second Update

Using the above bash script this yaml file:

stages:
  - test
test:
  stage: test
  script:
    - echo "test"

gets converted to this json:

{"content": "{ \"stages\": [ \"test\" ], \"test\": { \"stage\": \"test\", \"script\": [ \"echo \"test\"\" ] } }"}

When this is sent to the api the following json error response is received:

{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 62"]}% 

Can anyone explain how to use GitLab’s api correctly?

Got it working finally using the following script:

#!/usr/bin/env bash

json=$(ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' custom_hooks/valid.yml)

# escape quotes
json_content=$(echo $json | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))')
echo $json_content

# Add object contect for GitLab linter
json_content="{\"content\": ${json_content}}"

# Output escaped content to file
echo $json_content > custom_hooks/input.json
echo "Escaped json content written to file input.json"

curl --include --show-error --request POST \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "https://gitlab.com/api/v4/ci/lint" \
    --data-binary "$json_content"

N.B will be tweaking script to read file from system args rather than the fixed file location custom_hooks/valid.yml. Including this script on the offchance that it will help others.

The problem was that initially I was sending YAML contents of the file directly to the api:

{ "content": { <contents of .gitlab-yml> } }

It looks as though GitLab accepts YAML converted to an escaped JSON string in their API. So used ruby to convert the yaml to JSON and then used python to escape the resulting JSON produced by ruby. Finally was able to use curl to send the escaped JSON string to the GitLab API for validating…

Not sure if Ruby has something equivalent to python’s json.dumps … but this solution allows me to validate gitlab-ci…Next stage hookup to git pre-commit hooks / server side pre-receive (if possible!) to invalid .gitlab-ci.yml files breaking CI pipeline.

Newbie to ruby…since posting original answer have had a go at creating a ruby script that can be used from pre-commit hooks etc. Now only require bash and ruby. The following script can be invoked from bash script…Would need to add token for POSTing lint requests to private project lint url…

#!/usr/bin/env ruby


require 'json'
require 'net/http'
require 'optparse'
require 'yaml'


=begin
POST to GitLab api for linting ci yaml
Params:
+url+ :: Api url
+yaml+ :: Yaml payload for linting
Returns:
Json validation result from API for HTTP response Success
Aborts with HTTP Message for all other status codes
=end
def call_api(url, yaml)
    uri = URI.parse(url)
    
    req = Net::HTTP::Post.new(uri)
    req.content_type='application/json'
    req['Accept']='application/json'
    req.body = JSON.dump({"content" => yaml.to_json})
    
    https = Net::HTTP.new(uri.host, uri.port)
    https.use_ssl = true
    https.verify_mode = OpenSSL::SSL::VERIFY_PEER
    
    response = https.request(req)

    case response
        when Net::HTTPSuccess
            puts "request successful"
            return JSON.parse response.body
        when Net::HTTPUnauthorized
            abort("#{response.message}: invalid token in api request?")
        when Net::HTTPServerError
            abort('error' => "#{response.message}: server error, try again later?")
        when Net::HTTPBadRequest
            puts "Bad request..." + request.body
            abort("#{response.message}: bad api request?")
        when Net::HTTPNotFound
            abort("#{response.message}: api request not found?")
        else
            puts "Failed validation\nJSON payload :: #{request.body}\nHTTP Response: #{response.message}"
            abort("#{response.message}: failed api request?")
    end
end


=begin
Display exit report and raise the appropriate system exit code
Params:
+status+ :: Validation status string.  Legal values are valid or invalid
+errors+ :: String array storing errors if yaml was reported as invalid
Returns:
Exits with 0 when successful
Exits with 1 on validation errors or fails to parse legal status value
=end
def exit_report(status, errors)
    case status
        when "valid"
            puts ".gitlab-ci.yml is valid"
            exit(0)
        when "invalid"
            abort(".gitlab-ci.yml is invalid with errors:\n\n" + errors.join("\n"))
        else 
            abort("A problem was encountered parsing status : " + status)  
    end
end


=begin
Load yaml file from path and return contents
Params:
+path+ :: Absolute or relative path to .gitlab-ci.yml file
=end
def load_yaml(path)
    begin
        YAML.load_file(path)
    rescue Errno::ENOENT
        abort("Failed to load .gitlab-ci.yml")
    end
end

=begin
Parse command line options
Returns:
Hash containing keys: {:yaml_file,:url}
=end
def read_args()
    options = {}
    OptionParser.new do |opt|
        opt.on('-f', '--yaml YAML-PATH', 'Path to .gitlab-ci.yml') { |o| options[:yaml_file] = o }
        opt.on('-l', '--url GitLab url', 'GitLab API url') { |o| options[:url] = o }
    end.parse!

    options
end

=begin
Load yaml to send to GitLab API for linting
Display report of linting retrieved from api
Returns:
Exits with 0 upon success and 1 when errors encountered
=end
def main()
    # try and parse the arguments
    options = read_args()
    unless !options.has_key?(:yaml_file) || !options.has_key?(:url)
         # try and load the yaml from path
        puts "Loading file #{options[:yaml_file]}"
        yaml = load_yaml(options[:yaml_file])

        # make lint request to api
        puts "Making POST request to #{options[:url]}"
        response_data=call_api(options[:url], yaml)

        # display exit report and raise appropriate exit code
        unless !response_data.has_key?("status") || !response_data.has_key?("errors")
            exit_report response_data["status"], response_data["errors"]
        else
            puts "Something went wrong parsing the json response " + response_data
        end
    else
        abort("Missing required arguments yaml_file and url, use -h for usage")
    end
end

# start
main

Since the time of asking this question, I have developed and released a Ruby gem, served on rubygems.org. Details of the gem are available from here.

Install with:

gem install gitlab-lint-client

Usage for the CLI is:

glab-lint -h

The source code is available from here. The repository also serves a pre-commit hook rule called validate-gitlab-ci. Add a .pre-commit-config.yml file to the root of a repository and copy and paste the following into it:

repos:
- repo: https://github.com/dcs3spp/validate-gitlab-ci
  rev: v0.0.1
  hooks:
  - id: validate-gitlab-ci
    args: [--yaml=.gitlab-ci.yml, --base-url=https://gitlab.com]
    pass_filenames: false
    types: [yaml]
    files: .gitlab-ci.yml
    stages: [commit]

Install pre-commit:

pip install pre-commit

Request the pre-commit tool to download and configure the git hook:

pre-commit install

Optionally, explictly try the hook with:

pre-commit run validate-gitlab-ci --all-files

Upon first use this will configure a ruby environment and install the git pre-commit hook. Initially, this may take a few minutes but once installed the environment is reused.

From within a test repository, try it out by editing the .gitlab-ci.yml to have some invalid content. Then, try commiting the invalid .gitlab-ci.yml file. The commit should be rejected, preventing it from being pushed to the repository and breaking the CI build!