Sending Emails in Python via pipeline

Unable to send emails in python with gitlab pipeline

with smtplib.SMTP(host, port) as server:
# Start TLS for security
server.starttls()
# Login to the SMTP server
server.login(username, password)
# Send the email
server.sendmail(username, to_email, message.as_string())
this gives error
server.login(username, password)

96 File “/usr/local/lib/python3.12/smtplib.py”, line 750, in login

97 raise last_exception

98 File “/usr/local/lib/python3.12/smtplib.py”, line 739, in login

99 (code, resp) = self.auth(

100 ^^^^^^^^^^

101 File “/usr/local/lib/python3.12/smtplib.py”, line 662, in auth

102 raise SMTPAuthenticationError(code, resp)

103smtplib.SMTPAuthenticationError: (535, b’5.7.3 Authentication unsuccessful [BN9PR03CA0631.namprd03.prod.outlook.com 2023-12-22T03:20:03.907Z 08DC01B4F6AAE5BB]')
have checked username and password are correct, emails are sent on my local run. what am i missing?

From the error it seems like you are not passing in the username and password correctly,

What kind of executor are you using? Docker, shell, etc

How are you passing in the creds? File already on the runner, gitlab variable, environment file or variable, etc

Hi Matt,
Thank you for your response and apologies for incomplete details.
I am running my selenium tests from gitlab-ci.yml file which starts tests in docker container.
I have stored the credentials in CI/CD variables but have printed them on console to check there values they are correct and also when i run the same script in my local am getting the emails meaning SMTP and credentials are working. The only variable now in gitlab docker container. Do we need to add some extra code for emails to work from docker container. I my local windows machine i just triggered in .py file and i received the email.

Please let me know if any more information is needed to help me with my issue. Thank you for your time.

ahh, I ran into the same issue not too long ago. The variables are being passed to the gitlab runner but probably not being passed into the docker container.

docker run -e PASSWORD=$PASSWORD -e USER=$USER

or if its using docker compose, you need something like this in the compose.yml.

service:
  thing:
    environment: 
      - USER: $USER
      - PASSWORD: $PASSWORD

Hey Again Matt, no i feel variables are getting passed into docker container as i was able to print them on gitlab console, also this code is working(using mailgun api to send emails)

mailgun_api_key = os.environ.get(“MAILGUN_API_KEY”)
mailgun_domain = os.environ.get(“MAILGUN_DOMAIN”)
recipient_email = os.environ.get(“TO_EMAIL”)
mailgun_url = f"https://api.mailgun.net/v3/{mailgun_domain}/messages"
Here as well able to access CI/CD variables to get values and authenticate to mailgun server.

I still feel somehow Giltab is interfering in python communication with mail server. not sure how but just a hunch.

Are you able to show us more of the pipeline file (gitlab-ci.yml)? It would help a lot to identify the issue.

Can you tell me if your runner is a shell executor or a docker executor?

If you are running a shell executor then its possible to print them on the console but that is printing it before they are being passed into the docker container. You have to explicitly pass environment variables to the docker container with the -e flag or the compose file, so you should see that somewhere in your pipeline call.

I am talking about sendTestSummary stage(commented for now)

image: markhobson/maven-chrome:jdk-21

stages:

  • build
  • test
    #- sendTestSummary

cache:
paths:
- .m2/repository/

before_script:

  • apt-get update -yq
  • apt-get install -y zip

build:
stage: build
script:
- mvn compile --batch-mode

test:
stage: test
script:
- echo “$TAG”
- mvn test -Dheadless=true -Dbrowser=chrome -Dcucumber.filter.tags=“$TAG”
artifacts:
paths:
- reports/**
- target
when: always
expire_in: 1 day
allow_failure: true
only:
- main
- schedules
except:
- merge_requests

#sendTestSummary:

stage: sendTestSummary

image: python:3

script:

- pip install jinja2 requests

- python $CI_PROJECT_DIR/sendTestResults.py

artifacts:

paths:

- test_results.html

expire_in: 1 day

only:

- main

except:

- merge_requests

This stage triggers py file, the mailgin api works def send_test_results_email(html_content, attachment_path):
mailgun_api_key = os.environ.get(“MAILGUN_API_KEY”)
mailgun_domain = os.environ.get(“MAILGUN_DOMAIN”)
recipient_email = os.environ.get(“TO_EMAIL”)
mailgun_url = f"https://api.mailgun.net/v3/{mailgun_domain}/messages"
message_data = {
“from”: “Gitlab Runner USER@YOURDOMAIN.COM”,
“to”: recipient_email,
“subject”: “Test Results”,
“html”: html_content,
}
But SMTP fails
def smtp_password():
password = os.getenv(“SMTP_EMAIL_PASSWORD”)
if password is None:
raise RuntimeError(“SMTP password is not available. Please set the environment variable SMTP_EMAIL_PASSWORD.”)
return password

with smtplib.SMTP(host, port) as server:
# Start TLS for security
server.starttls()
# Login to the SMTP server
server.login(username, password)
# Send the email
server.sendmail(username, to_email, message.as_string())
Thanks again for your time Matt.

So did some quick testing on my end with this stage

test-docker-image:
  image: python:3.12-alpine
  stage: test
  
  variables:
    - TEST_A: "WORKS"
    # TEST_B: "WORKS" defined in CI/CD settings
  
  script:
    - echo $TEST_A
    - echo $TEST_B
    - echo 'from os import environ' > test.py
    - echo 'print(f"This is where we get pipeline variable, {environ.get("TEST_A", "DOES NOT WORK")}")' >> test.py
    - echo 'print(f"This is where we get settings variable, {environ.get("TEST_B", "DOES NOT WORK")}")' >> test.py
    - echo 'quit()' >> text.py
    - python test.py

Results in:

WORKS
WORKS
This is where we get pipeline variable, WORKS
This is where we get settings variable, WORKS

So, I agree the variables being passed in shouldn’t be the problem.

I am a little confused by your code,

def smtp_password():
  password = os.getenv(“SMTP_EMAIL_PASSWORD”)
  if password is None:
    raise RuntimeError(“SMTP password is not available. Please set the environment variable SMTP_EMAIL_PASSWORD.”)
    return password

If that is the only place you call password = os.getenv(“SMTP_EMAIL_PASSWORD”) then password is a function variable and can not be used globally outside the function.

Also I don’t see a section importing your username variable the same way you do with your other environment variables. Are you doing it the same way? Is there is a similar test case for your username variable? If not I would add it.

username = os.getenv(“SMTP_EMAIL_USERNAME”)
password = os.getenv("SMTP_EMAIL_PASSWORD")
server.login(username, password)

or should be

server.login(smtp_username(), smtp_password())

Hey Matt, thank you for your involvement, please find complete file below

def generate_html_report(json_data):
scenarios =
for scenario in json_data[0].get(“elements”, ):
scenario_info = {
“name”: scenario[“name”],
“steps_passed”: sum(1 for step in scenario[“steps”] if step[“result”][“status”] == “passed”),
“steps_failed”: sum(1 for step in scenario[“steps”] if step[“result”][“status”] == “failed”),
“status”: “Failed” if any(step[“result”][“status”] == “failed” for step in scenario[“steps”]) else “Passed”,
“error_message”: next((step[“result”][“error_message”] for step in scenario[“steps”] if step[“result”][“status”] == “failed”), “”),
}
scenarios.append(scenario_info)
template = Template(html_template_content)
rendered_html = template.render(scenarios=scenarios)
with open(“test_results.html”, “w”) as output_file:
output_file.write(rendered_html)
return rendered_html

def send_test_results_email(html_content, attachment_path):
mailgun_api_key = os.environ.get(“MAILGUN_API_KEY”)
mailgun_domain = os.environ.get(“MAILGUN_DOMAIN”)
recipient_email = os.environ.get(“TO_EMAIL”)
mailgun_url = f"https://api.mailgun.net/v3/{mailgun_domain}/messages"
message_data = {
“from”: “Gitlab Runner USER@YOURDOMAIN.COM”,
“to”: recipient_email,
“subject”: “Test Results”,
“html”: html_content,
}
shutil.make_archive(attachment_path, “zip”, “reports”)
with open(f"{attachment_path}.zip", “rb”) as attachment:
files = {“attachment”: (os.path.basename(f"{attachment_path}.zip"), attachment)}
# Send the email using Mailgun API with attachment
response = requests.post(
mailgun_url,
auth=(“api”, mailgun_api_key),
data=message_data,
files=files,
)
print(response.json())

if name == “main”:
with open(“target/cucumber.json”, “r”) as json_file:
json_data = json.load(json_file)
html_content = generate_html_report(json_data)
attachment_path = “reports” # Update this path based on your GitLab CI/CD configuration
send_test_results_email(html_content, attachment_path)

yes i have function to read username and password

I’m confused, the code you provided is missing any call to a SMTP library. Are you able to provide that code?

should be a section like below

def smtp_password():
  password = os.getenv(“SMTP_EMAIL_PASSWORD”)
  if password is None:
    raise RuntimeError(“SMTP password is not available. Please set the environment variable SMTP_EMAIL_PASSWORD.”)
  return password


def smtp_username():
  username = os.getenv(“SMTP_EMAIL_USERNAME”)
  if username is None:
    raise RuntimeError(“SMTP username is not available. Please set the environment variable SMTP_EMAIL_USERNAME.”)
  return username

server.login(smtp_username(), smtp_password())

Hey Matt, feeling guilty for wasting your time. My apologies. SMTP Code is below


def smtp_password():
    password = os.getenv("SMTP_EMAIL_PASSWORD")
    if password is None:
        raise RuntimeError("SMTP password is not available. Please set the environment variable SMTP_EMAIL_PASSWORD.")
    return password
    
def generate_html_report(json_data):
    # Extract relevant information from the JSON data
    scenarios = []
    for scenario in json_data[0].get("elements", []):
        scenario_info = {
            "name": scenario["name"],
            "steps_passed": sum(1 for step in scenario["steps"] if step["result"]["status"] == "passed"),
            "steps_failed": sum(1 for step in scenario["steps"] if step["result"]["status"] == "failed"),
            "status": "Failed" if any(step["result"]["status"] == "failed" for step in scenario["steps"]) else "Passed",
            "error_message": next((step["result"]["error_message"] for step in scenario["steps"] if step["result"]["status"] == "failed"), ""),
        }
        scenarios.append(scenario_info)

    # Create a Template object from the embedded HTML template content
    template = Template(html_template_content)

    # Render the HTML template with the extracted data
    rendered_html = template.render(scenarios=scenarios)

    # Save the rendered HTML to a file in the project directory
    with open("test_results.html", "w") as output_file:
        output_file.write(rendered_html)

    return rendered_html

def send_test_results_email(html_content, attachment_path):
    # Sender's email address
    username = "dummy@outlook.com"
    # Sender's password
    password = smtp_password()

    # Recipient's email address
    to_email = "dummy@icloud.com"

    # SMTP server settings
    host = "smtp.office365.com"
    port = 587

    # Create a MIMEText object for the email content
    message = MIMEMultipart()
    message['From'] = username
    message['To'] = to_email
    message['Subject'] = "Test Results Email"

    # Attach the HTML content to the email
    message.attach(MIMEText(html_content, 'html'))

    # Create a ZIP file containing the reports
    shutil.make_archive(attachment_path, "zip", "reports")

    # Attach the zip file
    attachment = MIMEBase('application', 'zip')
    with open(f"{attachment_path}.zip", "rb") as attachment_file:
        attachment.set_payload(attachment_file.read())
    encoders.encode_base64(attachment)
    attachment.add_header('Content-Disposition', f'attachment; filename={os.path.basename(f"{attachment_path}.zip")}')
    message.attach(attachment)

    # Set up the SMTP connection
    with smtplib.SMTP(host, port) as server:
        # Start TLS for security
        server.starttls()
        # Login to the SMTP server
        server.login(username, password)
        # Send the email
        server.sendmail(username, to_email, message.as_string())

    print("Test results email sent successfully.")

if __name__ == "__main__":
    # Read the JSON file
    with open("target/cucumber.json", "r") as json_file:
        json_data = json.load(json_file)

    # Generate and save the HTML report
    html_content = generate_html_report(json_data)

    # Example usage
    attachment_path = "reports"  # Update this path based on your GitLab CI/CD configuration
    send_test_results_email(html_content, attachment_path)

Can you edit your reply and add your code into a code section so I can better read it? should be a </> icon at the top of the post text box.

done, was not aware about this

Thank you,

I can’t really see anything wrong with your code that would be causing the Authentication unsuccessful. As long as you are 100% sure that there is no spelling errors on the variable name or value it looks like it should work.

Have you tried running the code from where the gitlab runner is installed?

SSH into whatever system its on, there should be a build folder that you can follow down until you find your project code. Mine looks like ~/builds/gyDGPEiE/0/user/project-repo

From there you should be able to just run the code to test it again. Just make sure to

export SMTP_EMAIL_PASSWORD='some_password'

Then run the python script to send the email or here is a test script with the results attachment pulled out.

import os, <add more for SMTP>

password = os.getenv("SMTP_EMAIL_PASSWORD")
username = "dummy@outlook.com"
to_email = "dummy@icloud.com"

# SMTP server settings
host = "smtp.office365.com"
port = 587

# Create a MIMEText object for the email content
message = MIMEMultipart()
message['From'] = username
message['To'] = to_email
message['Subject'] = "Test Results Email"

# Set up the SMTP connection
with smtplib.SMTP(host, port) as server:
    # Start TLS for security
    server.starttls()
    # Login to the SMTP server
    server.login(username, password)
    # Send the email
    server.sendmail(username, to_email, message.as_string())

print("Test results email sent successfully.")

i have just started learning/using gitlab and use shared runners provided by them for now, so cant ssh(not that i know of)

As mailgun is working, will stick with it for now. Thank you.