Problem to solve
As you will see below, I have integration tests running with LocalStack and TestContainers. Locally, the tests run end-to-end successfully. In GitLab Pipelines, they don’t get past the configurations, where a table and queue are created on application start.
The errors are Error creating queue: Service returned HTTP status code 404 (Service: Sqs, Status Code: 404, Request ID: null) which indicates to me that the request is malformed in some way when running in pipeline, but I can’t seem to get much insight into what the request looks like when it fails. The LocalStack logs aren’t capturing the attempted call on pipeline fails, but they do log on successful runs locally.
Grateful for any insight anyone can provide.
Configuration
Gitlab CICD config:
stages: # List of stages for jobs, and their order of execution
- build
- test
- deploy
image: maven:3.9.9-eclipse-temurin-21
integration-test-job:
stage: test
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
script:
- ./mvnw -s mvn_ci_settings.xml integration-test
rules:
# Run only ONCE on every commit or merge request
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
artifacts:
reports:
junit: target/surefire-reports/TEST-*.xml
This test configuration creates the localStackContainer Bean
@TestConfiguration(proxyBeanMethods = false)
public class LocalStackConfig {
@Value("${amazon.region}")
private String awsRegion;
private static final DockerImageName LOCALSTACK_IMAGE_NAME = DockerImageName.parse("localstack/localstack:3.8.1");
private LocalStackContainer localStackContainer;
@Bean
public LocalStackContainer localStackContainer() {
// Setting defaults for LocalStackContainer. Access and Secret keys are fake.
// https://docs.localstack.cloud/references/credentials/
this.localStackContainer = new LocalStackContainer(LOCALSTACK_IMAGE_NAME)
.withEnv("LS_LOG", "trace-internal")
.withEnv("AWS_DEFAULT_REGION", awsRegion)
.withEnv("AWS_ACCESS_KEY_ID", "fake")
.withEnv("AWS_SECRET_ACCESS_KEY", "fake")
.withServices(DYNAMODB,SQS);
return this.localStackContainer;
}
@PreDestroy
public void closeLocalStackContainer() {
if (this.localStackContainer != null) {
this.localStackContainer.close();
}
}
}
This test configuration creates SQS and DynamoDB clients, creates a table and a queue on application starts, and logs LocalStack health checks.
@TestConfiguration(proxyBeanMethods = false)
public class XadsIntegrationTestConfig {
@Autowired
private LocalStackContainer localStackContainer;
@Value("${amazon.sqs.name}")
private String queueName;
@Value("${amazon.region}")
private String awsRegion;
private static final Logger LOG = System.getLogger(ConsentRepositoryImpl.class.getName());
@Bean
public SqsClient sqsClient() {
LOG.log(System.Logger.Level.INFO, "SQS endpoint: " + localStackContainer.getEndpointOverride(SQS));
return SqsClient.builder()
.region(Region.of(awsRegion))
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(localStackContainer.getAccessKey(), localStackContainer.getSecretKey())
)
)
.endpointOverride(localStackContainer.getEndpointOverride(SQS))
.build();
}
@Bean
public DynamoDbClient dynamoDbClient() {
LOG.log(System.Logger.Level.INFO, "DynamoDB endpoint: " + localStackContainer.getEndpointOverride(DYNAMODB));
return DynamoDbClient.builder()
.region(Region.of(awsRegion))
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(localStackContainer.getAccessKey(), localStackContainer.getSecretKey())
)
)
.endpointOverride(localStackContainer.getEndpointOverride(DYNAMODB))
.build();
}
@EventListener
public void onApplicationReady(ApplicationReadyEvent applicationReadyEvent) {
checkLSHealth();
ApplicationContext applicationContext = applicationReadyEvent.getApplicationContext();
LOG.log(System.Logger.Level.INFO, "Creating tables and sqsclient for integration tests");
// Create DynamoDB table for testing
LOG.log(System.Logger.Level.INFO, "Getting consentTable bean...");
DynamoDbTable<Consent> consentTable = applicationContext.getBean("consentTable", DynamoDbTable.class);
try {
consentTable.createTable();
LOG.log(System.Logger.Level.INFO, "consentTable created...");
String logs = localStackContainer.getLogs();
LOG.log(System.Logger.Level.ERROR, logs);
} catch (Exception e) {
String logs = localStackContainer.getLogs();
LOG.log(System.Logger.Level.ERROR, "Error creating queue: " + e.getMessage());
LOG.log(System.Logger.Level.ERROR, logs);
}
// Create SQS queue for testing
LOG.log(System.Logger.Level.INFO, "Getting sqsclient bean...");
SqsClient sqsClient = applicationContext.getBean("sqsClient", SqsClient.class);
CreateQueueRequest createQueueRequest = CreateQueueRequest.builder()
.queueName(queueName)
.build();
try {
CreateQueueResponse createQueueResponse = sqsClient.createQueue(createQueueRequest);
LOG.log(System.Logger.Level.INFO, "Queue created: " + createQueueResponse.queueUrl());
String logs = localStackContainer.getLogs();
LOG.log(System.Logger.Level.ERROR, logs);
} catch (Exception e) {
String logs = localStackContainer.getLogs();
LOG.log(System.Logger.Level.ERROR, "Error creating queue: " + e.getMessage());
LOG.log(System.Logger.Level.ERROR, logs);
}
}
private void checkLSHealth() {
LOG.log(System.Logger.Level.INFO, "LocalStack isRunning check...");
LOG.log(System.Logger.Level.INFO, "LocalStack is running: " + localStackContainer.isRunning());
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(localStackContainer.getEndpointOverride(DYNAMODB) + "/_localstack/health"))
.GET()
.build();
try {
// Send the request synchronously
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
LOG.log(System.Logger.Level.INFO, "LocalStack health check response: " + response.statusCode());
LOG.log(System.Logger.Level.INFO, "LocalStack health check response body: " + response.body());
} catch (IOException | InterruptedException e) {
LOG.log(System.Logger.Level.ERROR, "Error during LocalStack health check: " + e.getMessage());
Thread.currentThread().interrupt(); // Restore interrupted status
}
String endpoint = localStackContainer.getEndpoint().toString();
LOG.log(System.Logger.Level.INFO, "LocalStack endpoint: " + endpoint);
String endpointOverride = localStackContainer.getEndpointOverride(DYNAMODB).toString();
LOG.log(System.Logger.Level.INFO, "LocalStack endpointOverride: " + endpointOverride);
}
}
Running locally, my tests (not shown here) run successfully. Running in GitLab Pipelines, I get 404 errors on calls to DynamoDB and SQS.
Here are logs of the run on local and in pipeline. I’m excluding the output of the LocalStack Logging below for brevity.
Local Logs:
2025-01-07T00:57:19.728-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack isRunning check...
2025-01-07T00:57:19.740-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack is running: true
2025-01-07T00:57:21.952-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack health check response: 200
2025-01-07T00:57:21.952-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack health check response body: {"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "available", "dynamodbstreams": "available", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "disabled", "kinesis": "available", "kms": "disabled", "lambda": "disabled", "logs": "disabled", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "disabled", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "disabled", "ses": "disabled", "sns": "disabled", "sqs": "available", "ssm": "disabled", "stepfunctions": "disabled", "sts": "disabled", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "3.8.1"}
2025-01-07T00:57:21.952-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack endpoint: http://127.0.0.1:33121
2025-01-07T00:57:21.952-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack endpointOverride: http://127.0.0.1:33121
2025-01-07T00:57:21.952-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Creating tables and sqsclient for integration tests
2025-01-07T00:57:21.952-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Getting consentTable bean...
2025-01-07T00:57:25.114-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : consentTable created...
.
.
.
2025-01-07T00:57:25.144-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Getting sqsclient bean...
2025-01-07T00:57:25.243-05:00 INFO 9560 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Queue created: http://127.0.0.1:33121/queue/us-east-1/000000000000/xads-data-sharing-queue
GitLab Pipeline logs:
2025-01-07T05:40:53.906Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : DynamoDB endpoint: http://172.17.0.1:32781
2025-01-07T05:40:54.782Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : SQS endpoint: http://172.17.0.1:32781
2025-01-07T05:40:55.488Z INFO 179 --- [ main] g.f.x.q.DataSharingQueueIntegrationTests : Started DataSharingQueueIntegrationTests in 22.396 seconds (process running for 23.721)
2025-01-07T05:40:55.493Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack isRunning check...
2025-01-07T05:40:55.510Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack is running: true
2025-01-07T05:40:58.925Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack health check response: 200
2025-01-07T05:40:58.926Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack health check response body: {"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "available", "dynamodbstreams": "available", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "disabled", "kinesis": "available", "kms": "disabled", "lambda": "disabled", "logs": "disabled", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "disabled", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "disabled", "ses": "disabled", "sns": "disabled", "sqs": "available", "ssm": "disabled", "stepfunctions": "disabled", "sts": "disabled", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "3.8.1"}
2025-01-07T05:40:58.927Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack endpoint: http://172.17.0.1:32781
2025-01-07T05:40:58.927Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : LocalStack endpointOverride: http://172.17.0.1:32781
2025-01-07T05:40:58.927Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Creating tables and sqsclient for integration tests
2025-01-07T05:40:58.927Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Getting consentTable bean...
2025-01-07T05:40:59.643Z ERROR 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Error creating queue: Service returned HTTP status code 404 (Service: DynamoDb, Status Code: 404, Request ID: null)
.
.
.
025-01-07T05:40:59.646Z INFO 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Getting sqsclient bean...
2025-01-07T05:41:13.994Z ERROR 179 --- [ main] g.f.x.p.r.impl.ConsentRepositoryImpl : Error creating queue: Service returned HTTP status code 404 (Service: Sqs, Status Code: 404, Request ID: null)
Versions
Please select whether options apply, and add the version information.
- Self-hosted Runners
Versions