Problem to solve
I have 2 distinct problems that I think are related to each other.
Sadly I do not get any error messages (that I know of) and I do not know where to find the right log files for these specific parts.
Already thanks for anyone that can point me in the right direction.
Problem 1: Pipelines/jobs are not created
When I create a .gitlab-ci.yml file in a repository and push it to gitlab no Pipeline or jobs are created. The Pipeline editor validate function gives no errors. For example this code
variables:
APP_NAME: "demo"
stages:
- test
- build
test_job:
stage: test
script:
- echo "Testing $APP_NAME"
build_job:
stage: build
script:
- echo "Building $APP_NAME"
gives
I can create pipelines manually and this generates the right job files but these job files are not started automatically as they have not been triggered.
This pipeline is also not visible under the Build > Pipelines.
The only way to start a job is to cancel the job specifically and then restart it.
Problem 2: Jobs are not picked up by the runner
The runner only picks up jobs once every hours. . Gitlab does show these as online.
The config of the dDoxCnU6 runner is
concurrent = 100
check_interval = 10
log_level = "debug"
log_format = "text"
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "local runner"
url = "http://gitlab.ix-gitlab.svc.cluster.local/"
id = 4
token = "glrt-dDoxCnU6MuCgR84KTMJ03jvGGtyt5eAQ.0w0q0zddd"
token_obtained_at = 2025-04-22T16:00:32Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "ubuntu"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
network_mtu = 0
The log of the runner does this over and over every 10 seconds (which is expected)
2025-04-22 17:32:26.518252+00:00time="2025-04-22T19:32:26+02:00" level=debug msg="Checking for jobs...no content" runner=dDoxCnU6M status="204 No Content"
2025-04-22 17:32:36.518312+00:00time="2025-04-22T19:32:36+02:00" level=debug msg="Feeding runners to channel" builds=0 max_builds=100
2025-04-22 17:32:36.518414+00:00time="2025-04-22T19:32:36+02:00" level=debug msg="Feeding runner to channel" builds=0 max_builds=100 runner=dDoxCnU6M
2025-04-22 17:32:36.518421+00:00time="2025-04-22T19:32:36+02:00" level=debug msg="Processing runner" builds=0 max_builds=100 runner=dDoxCnU6M
2025-04-22 17:32:36.518426+00:00time="2025-04-22T19:32:36+02:00" level=debug msg="Acquiring job slot" builds=0 max_builds=100 runner=dDoxCnU6M
2025-04-22 17:32:36.518431+00:00time="2025-04-22T19:32:36+02:00" level=debug msg="Acquiring request slot" builds=0 max_builds=100 runner=dDoxCnU6M
2025-04-22 17:32:36.518435+00:00time="2025-04-22T19:32:36+02:00" level=debug msg="Acquiring executor from provider" builds=0 max_builds=100 runner=dDoxCnU6M
I created multiple runners on different machines and with different settings. This is what I already tried:
- Installing on windows with executor = “docker-windows”
- Installing on linux (Ubuntu) with executor = “docker”
- Changing the concurrent setting to 1, 3, 5 and 100
- Changing the check_interval setting to 1, 5, 10, 30, 100
- Connecting directly to local IP, trough the local reverse proxy and trough the external proxy
- Installing different git-runner versions 17.11.0 and 17.10.1
- Installing different gitlab versions 17.11.0-ee and 17.10.4-ee
Configuration
My setup is fully installed on docker containers in a TrueNas 25.04-RC.1 system.
Everything is running trough a SWAG (NGINX) reverse proxy and connected trough an external Cloudflare proxy.
The gitlab is accessible trough git.domain.be and the registry is accessible trough registry.domain.be.
These are the subcomponent versions in gitlab
And this is the gitlab.rb (sorry for the large file, I added all changes as I do not know what might cause this)
external_url 'https://git.mydomainname.be'
gitlab_rails['gitlab_ssh_host'] = 'git.mydomainname.be'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.gmail.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "myemail@gmail.com"
gitlab_rails['smtp_password'] = "the password"
gitlab_rails['smtp_domain'] = "smtp.gmail.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_pool'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'myemail@gmail.com'
gitlab_rails['gitlab_email_display_name'] = 'Info'
gitlab_rails['gitlab_email_reply_to'] = 'myemail@gmail.com'
gitlab_rails['gitlab_email_subject_suffix'] = ''
gitlab_rails['gitlab_email_smime_enabled'] = false
gitlab_rails['gitlab_username_changing_enabled'] = true
gitlab_rails['gitlab_default_theme'] = 2
gitlab_rails['gitlab_default_projects_features_issues'] = true
gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
gitlab_rails['gitlab_default_projects_features_wiki'] = true
gitlab_rails['gitlab_default_projects_features_snippets'] = true
gitlab_rails['gitlab_default_projects_features_builds'] = true
gitlab_rails['gitlab_default_projects_features_container_registry'] = true
gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?)|([A-Z][A-Z0-9_]+-\d+))+)"
gitlab_rails['gitlab_repository_downloads_path'] = 'tmp/repositories'
gitlab_rails['gravatar_plain_url'] = 'http://www.gravatar.com/avatar/%{hash}?s=%{size}&d=identicon'
gitlab_rails['gravatar_ssl_url'] = 'https://secure.gravatar.com/avatar/%{hash}?s=%{size}&d=identicon'
gitlab_rails['stuck_ci_jobs_worker_cron'] = "0 0 * * *"
gitlab_rails['expire_build_artifacts_worker_cron'] = "*/7 * * * *"
gitlab_rails['environments_auto_stop_cron_worker_cron'] = "24 * * * *"
gitlab_rails['pipeline_schedule_worker_cron'] = "19 * * * *"
gitlab_rails['ci_archive_traces_cron_worker_cron'] = "17 * * * *"
gitlab_rails['repository_check_worker_cron'] = "20 * * * *"
gitlab_rails['admin_email_worker_cron'] = "0 0 * * 0"
gitlab_rails['personal_access_tokens_expiring_worker_cron'] = "0 1 * * *"
gitlab_rails['personal_access_tokens_expired_notification_worker_cron'] = "0 2 * * *"
gitlab_rails['repository_archive_cache_worker_cron'] = "0 * * * *"
gitlab_rails['pages_domain_verification_cron_worker'] = "*/15 * * * *"
gitlab_rails['pages_domain_ssl_renewal_cron_worker'] = "*/10 * * * *"
gitlab_rails['pages_domain_removal_cron_worker'] = "47 0 * * *"
gitlab_rails['remove_unaccepted_member_invites_cron_worker'] = "10 15 * * *"
gitlab_rails['schedule_migrate_external_diffs_worker_cron'] = "15 * * * *"
gitlab_rails['ci_platform_metrics_update_cron_worker'] = '47 9 * * *'
gitlab_rails['analytics_usage_trends_count_job_trigger_worker_cron'] = "50 23 */1 * *"
gitlab_rails['member_invitation_reminder_emails_worker_cron'] = "0 0 * * *"
gitlab_rails['user_status_cleanup_batch_worker_cron'] = "* * * * *"
gitlab_rails['namespaces_in_product_marketing_emails_worker_cron'] = "0 9 * * *"
gitlab_rails['ssh_keys_expired_notification_worker_cron'] = "0 2 * * *"
gitlab_rails['ssh_keys_expiring_soon_notification_worker_cron'] = "0 1 * * *"
gitlab_rails['loose_foreign_keys_cleanup_worker_cron'] = "*/5 * * * *"
gitlab_rails['ci_runner_versions_reconciliation_worker_cron'] = "@daily"
gitlab_rails['ci_runners_stale_machines_cleanup_worker_cron'] = "36 * * * *"
gitlab_rails['ci_catalog_resources_process_sync_events_worker_cron'] = "*/1 * * * *"
gitlab_rails['ci_click_house_finished_pipelines_sync_worker_cron'] = "*/4 * * * *"
gitlab_rails['ci_click_house_finished_pipelines_sync_worker_args'] = [1]
gitlab_rails['webhook_timeout'] = 10
gitlab_rails['graphql_timeout'] = 30
gitlab_rails['trusted_proxies'] = []
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_path'] = "/var/opt/gitlab/gitlab-rails/shared/artifacts"
gitlab_rails['external_diffs_enabled'] = false
gitlab_rails['terraform_state_enabled'] = true
gitlab_rails['terraform_state_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/terraform_state"
gitlab_rails['ci_secure_files_enabled'] = true
gitlab_rails['ci_secure_files_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/ci_secure_files"
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false
gitlab_rails['ldap_servers'] = YAML.load <<-EOS
main:
label: 'LDAP'
host: 'openldap.ix-openldap.svc.cluster.local'
port: 389
uid: 'uid'
bind_dn: 'cn=admin,dc=mydomainname,dc=be'
password: 'the password'
encryption: 'plain'
verify_certificates: true
smartcard_auth: false
active_directory: true
allow_username_or_email_login: true
lowercase_usernames: false
base: 'dc=mydomainname,dc=be'
group_base: 'ou=Groups,dc=mydomainname,dc=be'
admin_group: 'admin'
sync_ssh_keys: false
EOS
gitlab_rails['gitlab_shell_ssh_port'] = 22
registry_external_url 'https://registry.mydomainname.be'
gitlab_rails['registry_enabled'] = true
registry_nginx['listen_port'] = 5005
registry_nginx['listen_https'] = true
registry_nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
registry['enable'] = true
sidekiq['enable'] = false
nginx['enable'] = true
nginx['redirect_http_to_https'] = false
nginx['listen_port'] = 80
nginx['listen_https'] = false
gitlab_ci['gitlab_ci_all_broken_builds'] = true
gitlab_ci['gitlab_ci_add_pusher'] = true
gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'
gitlab_kas['enable'] = false
Versions
- Self-managed
-
GitLab.comSaaS - Dedicated