Using nuget package from package registry in CI pipeline

I am trying to use two nuget packages from a package registry in my CI pipeline. For this, I create a nuget.config file at the root of my folder with the following content:

<?xml version="1.0" encoding="utf-8"?>
    <add key="my_registry value="https://<gitlab_instance>.com/api/v4/projects/<some_project_id>/packages/nuget/index.json" />
         <add key="Username" value="deploy_token_username" />
         <add key="ClearTextPassword" value="deploy_token" />

But this doesn’t seem to be enough because every time my pipeline gets to restoring the two packages, it hangs because it can’t access the feed. I didn’t really find anything in the docs regarding this. Any ideas?