Context:

After migrating a previous Gitlab version (Community Edition 9.5.10) to 
a new GITLAB running docker image gitlab/gitlab-ee:15.0.5-ee.0
(https://gitlab.com/gitlab-org/gitlab/-/tags/v15.0.5-ee)
I got problem using CI Lint and pipeline with syntax with .gitlab-ci.yml.

(A) problem on CI LINT validation :
{group} > {my project} >
On the left sidebar, select CI/CD > Pipelines.
In the upper-right corner, select CI lint.
Validate your GitLab CI configuration
<copy/paste hello world below>
Validate
=> “Error: Request failed with status code 500”

	When Validate with [x] "Simulate a pipeline created for the default branch"
	=> "Syntax is incorrect. CI configuration validated, 
		including all configuration added with the includes keyword. More information
		Undefined error (01GWKKZHPAM19JR91WYSKJZA7W)"

(B) Pipeline Editor > Edit
https://{external domain}/{group}/{project}/-/ci/lint
=> OK

But 
	Pipeline Editor > visualize
	Pipeline Editor > Lint
	Pipeline Editor > View merged YAML 
		=> "We're experiencing difficulties and this tab content is currently unavailable."

(C) When commiting, the Pipeline Jobs immediatly fails with
=> “Found errors in your .gitlab-ci.yml: Undefined error (01GWH7GEZ15X5QWZ7NK0WTNMVK)”

(D) Google Chrome debug console when visualise pipeline

Contents of .gitlab-ci.yml

build1:
stage: build
script:
- echo Hello World

interresting logs / inside container

==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 138ms (ActiveRecord: 48.8ms | Elasticsearch: 0.0ms | Allocations: 34312)

==> /var/log/gitlab/gitlab-rails/production.log <==

OpenSSL::Cipher::CipherError (bad decrypt):

app/models/concerns/ci/has_variable.rb:44:in uncached_runner_variable' app/models/concerns/ci/has_variable.rb:38:in block in to_runner_variable’
lib/gitlab/safe_request_store.rb:12:in fetch' app/models/concerns/ci/has_variable.rb:38:in to_runner_variable’
app/models/concerns/ci/maskable.rb:22:in `to_runner_variable’
…

other tests

https://{external domain}/api/v4/projects/{group}%2f{projet}/ci/lint
=> {“message”:“500 Internal Server Error”}

http://{external domain}/api/v4/projects/{group}%2f{projet}/repository/branches/master
=> OK ( json provided )

/usr/local/bin/gitlab-runner --debug run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml
…
Checking for jobs… nothing runner=pFZYeLBA
Feeding runners to channel builds=0
=> seem OK

https://{external domain}/api/graphql
=> {“errors”:[{“message”:“Unexpected end of document”,“locations”:}]}

GraphQl api test using SSH tunnel
http://localhost:3000/api/graphql
=> “errors”:[{“message”:“Unexpected end of document”,“locations”:}]}

Special notes : brackets elements is for scrubbing the customer & project
{group} : GITLAB project group name for our customer
{project} : GITLAB projet name
{gitlab hostname} : hosting amazon instance
{external domain} : external URL of our registered DNS name and HTTPS signed certificate

Thanks for your help.

Hello, I pursue my investigations.
The problems described can be narrow down and be reproduced

root@ccsp-gitlab:/# gitlab-rails c

Ruby: ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [x86_64-linux]
GitLab: 15.10.1-ee (36dd7c9b36a) EE
GitLab Shell: 14.18.0
PostgreSQL: 12.12
------------------------------------------------------------[ booted in 42.08s ]
Loading production environment (Rails 6.1.7.2)

irb(main):005:0> project = Project.find_by_full_path(‘CCSP/ccsp-back’)
=> #<Project id:2 CCSP/ccsp-back>>

irb(main):007:0> content = project.repository.gitlab_ci_yml_for(project.repository.root_ref_sha)
=> “build1:\n stage: build\n script:\n - echo Hello World\n\n”

irb(main):008:0> Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content)
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final’: bad decrypt (OpenSSL::Cipher::CipherError)

irb(main):009:0> project.repository.root_ref_sha
=> “65353a8a2…75e20f11be5468”

Another observation when going to group >> CI/CD Settings
=> “There was an error fetching the variables.”

Note : I’d removed the NGINX config & log from description as it seems not to be the problem.
(external URLs works with gitlab to retrieve branches and console is ok).

I tried a workaround which solved the encountered problems which was :

• rails console : parsing using Gitlab::Ci::Lint.new [OK]
• gitlab / CI Lint is working [OK]
• gitlab pipeline is starting after puting back my .gitlab-ci.yml [OK]
• gitlab > group > CI/CD variable [OK]

Workaround :
A) backup of the table ci_group_variables
$ pg_dump -a -t ci_group_variables gitlabhq_production > ci_group_variables.psql
B)
$ psql gitlabhq_production
delete from ci_group_variables;
C) restart gitlab

Debug informations added manually in :
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/ci/has_variable.rb: line 55
def uncached_runner_variable
STDERR.puts ‘key=’+key
STDERR.puts ‘value=’ +value.inspect
STDERR.puts ‘file=’+file?.inspect

  { key: key, value: value, public: false, file: file? }
end

/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/encryptor-3.0.0/lib/encryptor.rb : line 96
…
begin
result = cipher.update(value)
result << cipher.final
result << cipher.auth_tag if cipher.authenticated? && encryption?(cipher_method)
rescue => ex
puts ex.full_message()
end
result
end
…

I didn’t investigate further in what is failing between initial Gitlab version from where I come:
CE 9.5.10
to the current image I’m using : gitlab/gitlab-ee:15.10.1-ee.0

Seems to be the encoding change in those variables stored on my Postgres DB :
SONAR_LOGIN
SONAR_URL
GS4JS_HOME