Gitlab-rails console doesn't work

Problem to solve

Any attempt to use gitlab-rails in container just does not work. It looks like it hangs intinitely, but trying to run it with strace shows some intel.

At first i found this error:
Gem Load Error is: Could not find a JavaScript runtime. See https://github.com/rails/execjs`` for a list of available runtimes.in `’ ``

I installed JS inside container only to run into this one:
Bundler::GemRequireError: There was an error while trying to load the gem 'toogle'.
Gem Load Error is: cannot load such file -- toogle

I ran sed -i '/toogle/s/^/#/' Gemfile to turn it off only to run into the next one:
Bundler::GemRequireError: There was an error while trying to load the gem 'sigdump/setup'.
Gem Load Error is: cannot load such file -- sigdump/setup

At this point i created absolutely new compose and tried clear installment - still does not work.

Any ideas? It definetely does not look like a normal behaviour.

Everything seems like it’s working tho, UI works fine, gitlab-ctl shows this
run: alertmanager: (pid 2571) 1735s; run: log: (pid 1896) 1890s
run: gitaly: (pid 2542) 1741s; run: log: (pid 1079) 2103s
run: gitlab-exporter: (pid 2520) 1742s; run: log: (pid 1772) 1916s
run: gitlab-kas: (pid 1487) 2088s; run: log: (pid 1513) 2085s
run: gitlab-workhorse: (pid 2496) 1743s; run: log: (pid 1691) 1932s
run: logrotate: (pid 973) 2118s; run: log: (pid 996) 2115s
run: nginx: (pid 2507) 1743s; run: log: (pid 1735) 1925s
run: postgres-exporter: (pid 2582) 1734s; run: log: (pid 1958) 1882s
run: postgresql: (pid 1112) 2095s; run: log: (pid 1137) 2092s
run: prometheus: (pid 2537) 1741s; run: log: (pid 1861) 1897s
run: puma: (pid 1590) 1951s; run: log: (pid 1604) 1950s
run: redis: (pid 1003) 2112s; run: log: (pid 1027) 2109s
run: redis-exporter: (pid 2527) 1741s; run: log: (pid 1810) 1907s
run: sidekiq: (pid 1619) 1945s; run: log: (pid 1635) 1942s
run: sshd: (pid 41) 2134s; run: log: (pid 40) 2134s

Steps to reproduce

Install clear docker image, enter container via bash and try to execute any gitlab-rails command.

Configuration

docker-compose.yml:

services:
gitlab:
image: 'gitlab/gitlab-ce:${GITLAB_IMAGE_TAG}'
restart: always
hostname: '${GITLAB_DOMAIN}'

ports:
  - '80:80'                                 
  - '${GITLAB_HTTPS_PORT}:443'             
  - '${GITLAB_SSH_PORT}:22'                 

volumes:
  - '$GITLAB_HOME/config:/etc/gitlab'                  
  - '$GITLAB_HOME/logs:/var/log/gitlab'                
  - '$GITLAB_HOME/data:/var/opt/gitlab'     

.env:
GITLAB_HOME=/home/user/gitlab-docker
GITLAB_DOMAIN=``gitlab.example.com
GITLAB_HTTPS_PORT=443
GITLAB_SSH_PORT=2222
GITLAB_IMAGE_TAG=18.11.9-ce.0

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • 18.11.9-ce.0

Initial questions:

  1. Which commands are used to spawn the container, and then open gitlab-rails inside the container?
  2. Which OS/architecture is the host running the Docker container?

I cannot reproduce the problem locally on my macOS environment – asked an AI agent to verify with Docker (in Rancher Desktop).


• I tested the exact gitlab/gitlab-ce:18.11.9-ce.0 image on arm64 and could not reproduce a defect.

  • The console initially appeared frozen with no output.
  • After ~24 seconds it printed version information.
  • After ~34 seconds the IRB prompt appeared.
  • puts :console_ok executed successfully.

The supported command is:

  docker exec -it <container> gitlab-rails console

This matches the official GitLab documentation (Rails console | GitLab Docs).

The reported gdk-toogle and sigdump errors concern development/test gems. They are intentionally absent from the production image and excluded through:

  BUNDLE_WITHOUT:
    - development
    - test

Therefore, commenting them out—or installing JavaScript inside the container—is not an appropriate fix. The errors suggest that GitLab’s packaged Bundler configuration was ignored or altered.

I’d ask the reporter for:

  uname -m
  docker exec <container> gitlab-rails console
  docker exec <container> cat /opt/gitlab/embedded/service/gitlab-rails/.bundle/config
  docker exec <container> env | sort

Also ask for the exact command passed to strace. Running bundle exec rails console manually is not equivalent to the documented Docker command.

  1. About commands:
    1. docker compose up -d is used to spawn the container
    2. docker exec -it <container> bash to enter container
  2. OS:
    PRETTY_NAME="Ubuntu 24.04.3 LTS"
    NAME="Ubuntu"
    VERSION_ID="24.04"
    VERSION="24.04.3 LTS (Noble Numbat)"
    VERSION_CODENAME=noble
  3. Architecture amd64

I tried usual, intended command multiple time, it just hangs and at some point whole console just stops responding at all, even to something like Ctrl+C.

So, about other commands:

  1. uname -m: x86_64

  2. docker exec gitlab-rails console - no response, hangs forever (just getting a disconnect “client_loop: send disconnect: Connection reset”)
    but if i interrupt it early with Ctrl+C i get this:
    Ruby: ruby 3.3.10 (2025-10-23 revision 343ea05002) [x86_64-linux]
    GitLab: 18.11.9 (7d3d034bc81) FOSS
    GitLab Shell: 14.50.0
    PostgreSQL: 17.10
    ------------------------------------------------------------[ booted in 70.83s ]
    Loading production environment (Rails 7.2.3.1)
    Switch to inspect mode.

  3. docker exec cat /opt/gitlab/embedded/service/gitlab-rails/.bundle/config


    BUNDLE_BUILD__GPGME: "--use-system-libraries"
    BUNDLE_BUILD__NOKOGIRI: "--use-system-libraries --with-xml2-include=/opt/gitlab/embedded/include/libxml2 --with-xslt-include=/opt/gitlab/embedded/include/libxslt"
    BUNDLE_BUILD__RUBY___MAGIC: "--with-magic-flags=--disable-zstdlib"
    BUNDLE_FROZEN: "true"
    BUNDLE_WITHOUT: "development:test"

  4. docker exec env | sort
    EDITOR=/bin/vi
    GITLAB_ALLOW_SHA1_RSA=false
    HOME=/root
    HOSTNAME=gitlab.example.ru
    LANG=C.UTF-8
    PATH=/opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    TERM=xterm

About strace command: docker exec -it <container> bash -c "timeout 30 strace -f -o /tmp/strace.log su - git -c 'cd /opt/gitlab/embedded/service/gitlab-rails && env PATH=/opt/gitlab/embedded/bin:\$PATH bundle exec rake environment'"

Ngl, it’s deepseek generated, because at some point i ran out of ideas and tried to use AI to find a solution.

Just in case this info is relevant, docker settings are almost default, the only thing changed is daemon.json:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "100m",
"max-file": "3",
"compress": "true"
}
}

Also i checked Postgre and Redis inside container, they seem to work completely fine.

Hi Vladimir, thanks for the details.

The output confirms that Rails completes its boot:

[ booted in 70.83s ]
Loading production environment (Rails 7.2.3.1)

The SSH connection reset and the entire console becoming unresponsive point toward host resource pressure rather than a missing gem.

How much RAM and how many vCPUs does the host have? GitLab currently lists 16 GB RAM as the single-node baseline and 8 GB for constrained installations:

Please collect these details on the Docker host:

free -h
nproc
swapon --show
df -h
docker stats --no-stream <container>
docker inspect --format \
  'MemoryLimit={{.HostConfig.Memory}} OOMKilled={{.State.OOMKilled}}' \
  <container>
sudo journalctl -k --since "-30 min" \
  | grep -Ei 'oom|out of memory|killed process'

Please also run this non-interactive test:

time docker exec <container> \
  gitlab-rails runner 'puts :runner_ok'

echo "exit=$?"

Ideally, keep this running in a second SSH session during the test:

docker stats <container>

If SSH disconnects again, check the kernel journal after reconnecting or through the hosting provider’s console.

The generated strace command bypasses the supported gitlab-rails wrapper and invokes a raw Rake environment through su. It also does not set RAILS_ENV=production.

I would disregard the resulting execjs, toogle, and sigdump errors for now. They come from a different execution path.

Please share the command outputs, especially the memory totals and kernel journal. That should tell us whether the kernel killed the Rails or SSH process.

Server does have 32 GB Ram and 10 vCPUs.

  • free -h:

                   total        used        free      shared  buff/cache   available
    
    Mem:            31Gi       8.7Gi        21Gi        97Mi       2.1Gi        22Gi
    Swap:          4.0Gi          0B       4.0Gi
    
  • nproc: 10

  • swapon --show:
    NAME TYPE SIZE USED PRIO
    /swap.img file 4G 0B -2

  • df -h:
    Filesystem Size Used Avail Use% Mounted on
    tmpfs 3.2G 768K 3.2G 1% /run
    efivarfs 128M 33K 128M 1% /sys/firmware/efi/efivars
    /dev/mapper/ubuntu--vg-ubuntu--lv 98G 27G 67G 29% /
    tmpfs 16G 0 16G 0% /dev/shm
    tmpfs 5.0M 0 5.0M 0% /run/lock
    /dev/sda2 2.0G 200M 1.6G 11% /boot
    /dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
    tmpfs 3.2G 12K 3.2G 1% /run/user/1000

  • docker stats:
    CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
    111cb8b96cee gitlab-1 46.28% 8.286GiB / 31.34GiB 26.44% 2.84kB / 7.06kB 1.28GB / 726MB 468

  • docker inspect (exactly your command): MemoryLimit=0 OOMKilled=false

  • sudo journalctl -k --since “-30 min” | grep -Ei ‘oom|out of memory|killed process’: empty response

  • time docker exec:
    runner_ok
    0.02user 0.01system 1:02.50elapsed 0%CPU (0avgtext+0avgdata 28928maxresident)k
    304inputs+0outputs (2major+2802minor)pagefaults 0swaps
    echo shows exit=0

Dynamic docker stats showed mostly stable resource usage, except for CPU %. It fluctuated all the way from 1-5% up to 115%.

Also journalctl has a lot of entries like this (while using docker stats):
Aug 14 16:42:41 gitlab-vm containerd[860]: time="2026-08-14T16:42:41.965352143Z" level=error msg="unable to parse "max 0" as a uint from Cgroup file "/sys/fs/cgroup/system.slice/docker-111cb8b96ceed8e585890a3d43e66d3c06bf02da75140cfb221049091413c76a.scope/hugetlb.2MB.events""

Also i ran a bit modified version of your docker exec command and results seem a bit strange to me.
I noticed 1:02.50elapsed wasn’t quite true, since it take like 15 mins to get output message.
I then ran 3 times date +"%T.%N" && sudo docker exec gitlab-docker1-gitlab-1 gitlab-rails runner 'puts :runner_ok' && date +"%T.%N" with these results:

1st time:
16:54:38.856466129
runner_ok
16:55:42.535557215

2nd time:
16:56:05.908845450
runner_ok
16:57:07.693570049

3rd time:

17:23:06.564769264
runner_ok
17:24:13.022987763

But third time the problem is that it showed 2 min later than it actually executed, which confuses me even more tbh.

And i ran date +"%T.%N" && sudo time docker exec gitlab-docker1-gitlab-1 gitlab-rails runner 'puts :runner_ok' && date +"%T.%N"
result was
16:57:27.763125432
client_loop: send disconnect: Connection reset

with
Aug 14 17:14:53 gitlab-vm systemd[1]: session-6.scope: Deactivated successfully.
Aug 14 17:14:53 gitlab-vm systemd[1]: session-6.scope: Consumed 3.818s CPU time.
Aug 14 17:14:53 gitlab-vm systemd-logind[829]: Session 6 logged out. Waiting for processes to exit.
Aug 14 17:14:53 gitlab-vm systemd-logind[829]: Removed session 6.
in journalctl.

Thanks for the details.

I’m also curious which terminal you are using on the host, and the shell environment, and if tmux is involved.

To rule out any SSH / terminal hang issues, can you run the tests in a detached container and check the logs afterward?

  sudo docker exec -d gitlab-docker1-gitlab-1 \
    bash -lc '
      date +"start=%T.%N" > /tmp/rails-runner-test.log
      gitlab-rails runner "puts :runner_ok" \
        >> /tmp/rails-runner-test.log 2>&1
      echo "exit=$?" >> /tmp/rails-runner-test.log
      date +"end=%T.%N" >> /tmp/rails-runner-test.log
    '

Then inspect it later:

  sudo docker exec gitlab-docker1-gitlab-1 \
    cat /tmp/rails-runner-test.log

Three additional tests to verify the application behavior:

First, measure Docker and embedded Ruby without Rails:

  time sudo docker exec gitlab-docker1-gitlab-1 true

  time sudo docker exec gitlab-docker1-gitlab-1 \
    /opt/gitlab/embedded/bin/ruby -e 'puts :ruby_ok'

Then time the database and Redis connections:

  time sudo docker exec gitlab-docker1-gitlab-1 \
    gitlab-psql -c 'SELECT 1;'

  time sudo docker exec gitlab-docker1-gitlab-1 \
    gitlab-redis-cli ping

If the detached runner log remains incomplete, capture the correctly wrapped command with strace:

  sudo docker exec gitlab-docker1-gitlab-1 \
    timeout 180 strace -ff -tt -T -s 256 \
    -o /tmp/gitlab-rails-runner.strace \
    gitlab-rails runner 'puts :runner_ok'

Then list the generated traces:

  sudo docker exec gitlab-docker1-gitlab-1 \
    bash -lc 'ls -lh /tmp/gitlab-rails-runner.strace*'

I’m using SSH connection via Windows cmd, nothing unusual. There is also no tmux on host.

  1. rails-runner-test seems fine:

start=09:09:04.210048924
runner_ok
exit=0
end=09:10:20.945382170

2.time sudo docker exec gitlab-docker1-gitlab-1 true

real 0m0.137s
user 0m0.002s
sys 0m0.010s

3.time sudo docker exec gitlab-docker1-gitlab-1 /opt/gitlab/embedded/bin/ruby -e ‘puts :ruby_ok’

ruby_ok

real    0m0.221s
user    0m0.002s
sys     0m0.010s

4. time sudo docker exec gitlab-docker1-gitlab-1 \
gitlab-psql -c ‘SELECT 1;’

?column?

    1

(1 row)

real    0m0.148s
user    0m0.003s
sys     0m0.009s

5. time sudo docker exec gitlab-docker1-gitlab-1 \
gitlab-redis-cli ping

PONG

real 0m0.137s
user 0m0.002s
sys 0m0.010s

And i tried to launch it on host directly and it works just fine.
I’m genuinely confused, why it works that way, ngl.
Completely clear ubuntu 24.04, base gitlab image, usual SSH connection. Nothing out of place as far as i can see. Well, gonna work with console using host then.

If you wish to investigate further, i’ll do what i can to help you, if not - thanks a lot for helping me and your time.

Well, since there is no reply, i guess it can be marked as solved and closed.

For whoever runs into the same issue: use host console, do not use ssh console for whatever reason.

Thanks, I was just too busy, and only wanted to ask which solution you are now using. Thanks for coming back and resolving :slight_smile: