Automate Python Notifications for Model Training and Data Transfers

The author frequently writes Python scripts for data processing, data transfer, and model training. As data volume and complexity increase, these scripts can take considerable time, and you can perform other work while waiting for them to finish.

What is needed

The first step is to determine which tasks require notifications. For the author, three common time‑consuming tasks are model training, data processing/transfer, and financial modeling. Each case needs a specific analysis.

Model Training

Updates should be sent every n epochs and must include key metrics such as training and validation loss and accuracy, visualizations, and additional useful information like the model directory, runtime, architecture, and sample predictions (text or image). The example below sends an email every 100 epochs with loss plots and a generated image.

import notify
START = datetime.now()  # this line would be placed before model training begins
MODELNAME = "SynthwaveGAN"  # giving us our model name
NOTIFY = 100  # send an update notification every 100 epochs
if e % notify_epoch == 0 and e != 0:
    txt = (f"{MODELNAME} update as of "
           f"{datetime.now().strftime(%H:%M:%S)}.")
    msg = notify.message(
        subject=Synthwave GAN,
        text=txt,
        img=[
            f"../visuals/{MODELNAME}/epoch_{e}_loss.png",
            f"../visuals/{MODELNAME}/epoch_{e}_iter_{i}.png"
        ]
    )
    notify.send(msg)

Every 100 epochs, an email containing all the above information is sent. An example email is shown below:

Data Processing and Transfer

This step is often the most time‑consuming. The example demonstrates uploading batch data to SQL Server and sending a simple completion notification.

import os
import notify
from data import Sql

dt = Sql(database123, server001)  # setup the connection to SQL Server
for i, file in enumerate(os.listdir('../data/new')):
    dt.push_raw(f"../data/new/{file}")  # push a file to SQL Server
# once the upload is complete, send a notification
msg = notify.message(
    subject=SQL Data Upload,
    text=f"Data upload complete, {i} files uploaded.",
)
notify.send(msg)

If occasional errors occur, a try‑except block can capture them and include the error list in the update or completion email.

Financial Model

Financial modeling runs quickly, but for large workloads you may want a summary analysis. The example shows a cash‑flow model that sends a portfolio summary with sample loans and visualizations.

end = datetime.datetime.now()  # get the ending datetime
hours, rem = divmod((end - start).seconds, 3600)
mins, secs = divmod(rem, 60)
runtime = f"{hours:02d}:{mins:02d}:{secs:02d}"
notify.msg(
    subject="Cashflow Model Completion",
    text=(f"{len(model.output)} loans processed. "
          f"Total runtime: {runtime}"),
    img=[
        "../vis/loan01_amortisation.png",
        "../vis/loan07_amortisation.png",
        "../vis/loan01_profit_and_loss.png",
        "../vis/loan07_profit_and_loss.png"
    ]
)
notify.send(msg)

Code

All the above functionality is extracted from a script named notify.py. The examples use Outlook for sending email and require two Python libraries: email and smtplib.

MIME

The email message itself is built with a MIMEMultipart object from the email module. Three MIME subclasses are attached to the multipart object:

MIMEText : contains the plain‑text payload of the email.

MIMEImage : embeds images in the email.

MIMEApplication : adds file attachments.

Combined, they form the structure illustrated below:

SMTP

After constructing the email message, the smtplib module sends it. You need the appropriate SMTP server address and TLS port (e.g., Outlook’s smtp-mail.outlook.com:587). The code below shows initializing the SMTP connection, starting TLS, logging in, sending the email, and handling network errors.

import smtplib
import socket

def send(server="smtp-mail.outlook.com", port=587, msg):
    try:
        smtp = smtplib.SMTP(server, port)
        smtp.ehlo()
        smtp.starttls()
        with open('../data/email.txt', 'r') as fp:
            email = fp.read()
        with open('../data/password.txt', 'r') as fp:
            pwd = fp.read()
        smtp.login(email, pwd)
        smtp.sendmail(email, email, msg.as_string())
        smtp.quit()
    except socket.gaierror:
        print("Network connection error, email not sent.")

The ehlo() and starttls() commands greet the server and initiate a TLS‑encrypted session. After reading the email credentials from files, the script logs in, sends the message, and quits. Errors such as network failures are caught to prevent crashes.

Putting it all together

With the message builder and send function ready, you can dispatch notifications automatically during long‑running tasks:

# build a message object
msg = message(text="See attached!", img=important.png, attachment=data.csv)
send(msg)  # send the email (defaults to Outlook)

This completes the end‑to‑end process of sending email notifications and automating workflows with Python. Remember to use the correct public email server addresses and TLS ports.