Problem accessing MVN private repository hosted in AWS S3

Hello everyone, we are experiencing an error running a pipeline on our hosted GitLab (GitLab Enterprise Edition 12.7.6-ee).

The problem is an error when mvn, during build phase, tries to access to a needed dependency hosted in our private mvn AWS S3 repository, with this code and message:

Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 29789767C8B5B964, AWS Error Code: AccessDenied, AWS Error Message: Access Denied

This is our .gitlab-ci.yml script:

image: docker:17.11.0-ce

stages:
  - check-maven-repo
  - docker-build

services:
  - docker:17.11.0-ce-dind

check maven repository connection:
  stage: check-maven-repo
  image: python:latest
  variables:
    AWS_ACCESS_KEY_ID: $AWS_USER
    AWS_SECRET_ACCESS_KEY: $AWS_PASSWORD
    AWS_DEFAULT_REGION: "eu-west-1"
  script:
    - pip install awscli
    - aws --debug s3 cp s3://<dependency pom file> .

build docker image:
  stage: docker-build
  only:
    - branches
  script:
    - docker version
    - docker build
      --build-arg AWS_USER=$AWS_USER 
      --build-arg AWS_PASSWORD=$AWS_PASSWORD 
      -t $DOCKER_REGISTRY/$CI_PROJECT_NAME:$CI_COMMIT_BRANCH .

and the Dockerfile:

FROM python:latest

ARG AWS_USER

ARG AWS_PASSWORD

RUN pip install awscli

RUN AWS_ACCESS_KEY_ID=$AWS_USER AWS_SECRET_ACCESS_KEY=$AWS_PASSWORD AWS_DEFAULT_REGION="eu-west-1" aws --debug s3 cp s3://<dependency pom file> .

FROM maven:3.5-jdk-8-alpine as builder

ARG AWS_USER

ARG AWS_PASSWORD

WORKDIR /app

# Prepare maven settings

ADD pom.xml /app/pom.xml

ADD settings.xml /app/settings.xml

RUN mvn -s settings.xml help:effective-settings -DshowPasswords=true

# Adding source, compile and package into a fat jar

ADD src /app/src

RUN mvn -q -e -s settings.xml package

Now as you can see, in both pipeline and docker script we added a debugging phase to try the access to the resource that fails and the download with provided credentials does success in both cases.
This should prove that S3 permissions, used credentials and access from Docker are completely fine.

Also, before launching mvn package command we check the settings.xml provided to maven which results in:

 <settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
   <localRepository>/root/.m2/repository</localRepository>
   <servers>
     <server>
       <username>***</username>
       <password>***</password>
       <id>mmaven-release-repo</id>
     </server>
     <server>
       <username>***</username>
       <password>***</password>
       <id>mmaven-snapshot-repo</id>
     </server>
   </servers>
   <pluginGroups>
     <pluginGroup>org.apache.maven.plugins</pluginGroup>
     <pluginGroup>org.codehaus.mojo</pluginGroup>
   </pluginGroups>
 </settings>

as expected.

Despite this, mvn package command fails with the access denied problem shown above.

At last please note that we are migrating to GitLab pipelines from Jenkins CI, and the same docker script (with same provided settings.xml) gives success. Versions of Docker and Maven are the same.

Does anyone have any ideas or clues about the cause of this error?
Thank you for any support!

After further investigation we found the problem.
It seems that the maven-AWS connector that we used:

<extension>
    <groupId>org.springframework.build</groupId>
    <artifactId>aws-maven</artifactId>
    <version>5.0.0.RELEASE</version>
</extension>

gives priority to the global AWS credentials instead of the ones specified in the .m2/settings.xml.
Also it seems that GitLab runners hosted on AWS have configured global credentials and by that the connector just ignore our credentials causing the access denied error.

The fix we found is to use another connector like:

<extension>
     <groupId>com.github.seahen</groupId>
     <artifactId>maven-s3-wagon</artifactId>
     <version>1.3.1</version>
</extension>

which gives priority to the .m2/settings.xml credentials instead of the global’s.