Gitlab-ce is missing `/opt/gitlab/embedded/include/` for CI

tldr;

pip3 stopped working because of missing /opt/gitlab/embedded/include/ folder after version 10.6.5-ce
Manually readding it seems to (temporarily) solve the problem.

I took over from my predecessor a fully set up gitlab-ce sever. So even basic tips might help me

Problem

We use gitlab-ci for a internal project, which used to work quite fine.
But the new version of gitlab-ce (we are currently on 10.7.4-ce) throws an error when using pip3 in the .gitlab-ci.yml

$ sudo /opt/gitlab/embedded/bin/pip3 install requests
Requirement already satisfied: requests in /opt/gitlab/embedded/lib/python3.4/site-packages
Requirement already satisfied: certifi>=2017.4.17 in /opt/gitlab/embedded/lib/python3.4/site-packages (from requests)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/gitlab/embedded/lib/python3.4/site-packages (from requests)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in /opt/gitlab/embedded/lib/python3.4/site-packages (from requests)
Requirement already satisfied: idna<2.6,>=2.5 in /opt/gitlab/embedded/lib/python3.4/site-packages (from requests)
Exception:
Traceback (most recent call last):
  File "/opt/gitlab/embedded/lib/python3.4/distutils/sysconfig.py", line 442, in _init_posix
    with open(filename) as file:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/gitlab/embedded/include/python3.4m/pyconfig.h'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/gitlab/embedded/lib/python3.4/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/opt/gitlab/embedded/lib/python3.4/site-packages/pip/commands/install.py", line 350, in run
    isolated=options.isolated_mode,
  File "/opt/gitlab/embedded/lib/python3.4/site-packages/pip/commands/install.py", line 436, in get_lib_location_guesses
    scheme = distutils_scheme('', *args, **kwargs)
  File "/opt/gitlab/embedded/lib/python3.4/site-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/opt/gitlab/embedded/lib/python3.4/distutils/command/install.py", line 283, in finalize_options
    (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
  File "/opt/gitlab/embedded/lib/python3.4/distutils/sysconfig.py", line 494, in get_config_vars
    func()
  File "/opt/gitlab/embedded/lib/python3.4/distutils/sysconfig.py", line 449, in _init_posix
    raise DistutilsPlatformError(my_msg)
distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /opt/gitlab/embedded/include/python3.4m/pyconfig.h (No such file or directory)
ERROR: Job failed: exit status 1

It seems that some files in /opt/gitlab/embedded/include/ are missing.

What I have tested
So I have looked around and found out that the whole folder /opt/gitlab/embedded/include/ is removed after version 10.6.5. When I add the folder in version 10.7.4 it seem to work again.

So my question is:

Is this a feature? (I found nothing helpful in the changelog)
Is this a bug? (I found nothing helpful in the issue tracker)
Is it a configuration problem? (bundle exec rake RAILS_ENV=production or similar healthcheck shows no problems regarding configuration)

Any help is appreciated
Even a redirect to a more suitable category or forum.

Additional info:
Gitlab runs on a Ubuntu 16.04.4 LTS Server
with gitlab-runner version 10.8.0

From what I can tell, this is a bug introduced in Gitlab 10.7. It appears as though the embedded Python 3.4 environment is corrupt / broken. Based on my initial investigation, it appears as though the Python version embedded in the app has a dependency on the python-devel packages but those packages are not being deployed with the installer. They used to be deployed correctly in the 10.6.x releases.

Also, I can confirm that this problem affects both CE and EE editions of the product.

Unfortunately this completely breaks the Python runtime environment within the product, which is currently a production-stop situation for us. We have custom commit hooks written in Python which can no longer be deployed on the new Gitlab version - at least not using the embedded Python runtime anyway. So if this could be fixed sooner rather than later that would be great!

Hi @TheFriendlyCoder ,
Nice to get some reaction It seems that not a lot of people use git-hooks written in python…

As mentioned in my post, you can work around this problem when you manually add the missing files (in our case the folder /opt/gitlab/embedded/include/). When downgrading and upgrading again the folder gets deleted, but when including the folder in version 10.7.4 and upgrading (again: in our case to the currently newest version 10.8.3) the files seem to persist.

Did you post in any other forum?
Did you find anything anywhere else?

HTH,
M Stein

Yeah, we’re considering doing something similar here. The problem for us is that we script the deployments of our Gitlab servers so we’d need to wrap those missing folders in Ansible or something, which is a bit of a pain. Plus depending on how / why the folder was removed, this hack may or may not continue to work in future releases.

On a related note I did notice that the folder is also missing from the Docker images for Gitlab on DockerHub, so users of those containers will also be affected.

To answer your questions, I have not found any other reference to this defect any place else yet nor have I posted any other defects / questions elsewhere. We are registered Gitlab EE users so I may report the problem under the EE edition to see if that gets any better feedback / attention. I’ll post back here if / when I do that.

For now I have to find a way to hack around this problem to prepare our production systems for the next upgrade.