Blumetech's Tech Blog

SFTP File Automation for Blumetech Client

I just finished a small scripting project for one of Blumetech’s clients. It was a small project but shows how simple scripts can be used to automate many tasks around the office.

 

The client wanted to

 

1. Upload files to an FTP server via SFTP
2. Wanted logging on all the uploads
3. Wanted success and failure email alerts

 

The work was simple to do. In this particular case I did similar work for another client and it was easy to reuse the code that was previously written. Hence the learning curve needed to handle the SFTP connection was straightforward. The work was done and deployed in under 6 hours.

 

Of course I chose Python to do the work. One of the reasons I chose python over a bash script, Powershell, or even a simple Windows batch file is the readability of the code. The code excerpt below which represents the core module is readable even to people with little experience with Python. This is important as I am counting on other engineers with little Python experience to be the tier 1 support for these applications. Python’s simplicity allows even neophytes to get started with the language very quickly.

 

I expect more and more of these scripts and more office automation in the coming future. Many offices find themselves doing routine business tasks where parts of their workflow can be automated to increase both speed and accuracy.

 

'''
Uploads files to complysci FTP site.
Created on Feb 14, 2012

@author: mraposa
'''
import thread
import sys
import traceback
import smtplib
import os
from email.mime.text import MIMEText

import pysftp  
from paramiko import SSHException

from LogFileHelper2x import getDefaultLogger
from GeneralUtilities import displayPID
from FileHelper2x import getFileList
  
LOG_FILENAME = "FtpUpload_log.txt"
LOG_NAME = "Main"
WORK_DIR = r"E:\Data\Shared\FTP Files"
EMAIL_TO_SUCCESS = ["",]
EMAIL_TO_FAIL = ["", ""]
FTP_UID = ""
FTP_PASS = ""
FTP_SERVER = ""
SMTP_SERVER = "10.0.1.6"

log = getDefaultLogger(LOG_FILENAME, 'Main', consoleLogging=False)

def send_simple_email(to, subject, message, fromAddr='[email protected]'):
    """
    Function sends a simple plain text message to multiple recipients
    to = string of recipients separated by a comma, e.g. '[email protected],[email protected]'
    """
    # Create a text/plain message
    msg = MIMEText(message)
   
    msg['Subject'] = subject
    msg['From'] = fromAddr
    msg['To'] = to
   
    # Send the message via our own SMTP server.
    s = smtplib.SMTP(SMTP_SERVER)
    s.sendmail(fromAddr, to.split(','), msg.as_string())
    s.quit()

def sftpUpload(f):
    try:
        log.info("Connecting to %s" % FTP_SERVER)
        srv = pysftp.Connection(FTP_SERVER, username=FTP_UID, password=FTP_PASS)
        log.info("Uploading %s" % f)
        srv.put(f)
        log.info("Getting file list on FTP Server")
        log.info(srv.listdir())
        log.info("Disconnecting from ftp server")
        srv.close()
        return 0
    except SSHException:
        log.error("Unable to connect - login fail")
        message = """
        "Unable to connect - login fail"
        Host: %s
        UID: %s
        """ % (FTP_SERVER, FTP_UID)
        subject = "Ftp Upload Error: Unable to connect - login fail"
        for to in EMAIL_TO_FAIL:
            send_simple_email(to, subject, message)
        return 1
    except AttributeError:
        log.error("Host unreachable:")
        message = """
        "Host unreachable:"
        Host: %s
        UID: %s
        """ % (FTP_SERVER, FTP_UID)
        subject = "Ftp Upload Error: Host unreachable:"
        for to in EMAIL_TO_FAIL:
            send_simple_email(to, subject, message)
        return 1

def execute_task():
    for f in getFileList(WORK_DIR):
        log.info("Working on file: %s" % f)
        if sftpUpload(f) == 0:
            log.info("File upload successful")
            log.info("Deleting file: %s" %f)

            os.remove(f)
            subject = "Ftp Upload Successful"
            message = """
            Successfully uploaded file
            File: %s
            Server: %s
            """ % (os.path.basename(f), FTP_SERVER)
            for to in EMAIL_TO_SUCCESS:
                send_simple_email(to, subject, message)
           

if __name__ == '__main__':
    try:
        thread.start_new_thread(displayPID, ())
        execute_task()
    except Exception:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        print >> sys.stderr, "Exception in user code:"
        print >> sys.stderr, '-'*60
        traceback.print_exc(file=sys.stderr)
        print >> sys.stderr, '-'*60
        print >> sys.stderr, "*** format_exception:"
        errorText = ""
        for tmp in traceback.format_exception(exc_type, exc_value, exc_traceback):
            errorText = errorText + "%s" % tmp
        print >> sys.stderr, errorText
        log.exception("Exception in %s" % LOG_FILENAME)
        msg = """
        An exception occurred with an automated FTP client.
        %s
        """ % (errorText,)
        log.info("Attempting to send email")
        for to in EMAIL_TO_FAIL:
                send_simple_email(to, "Automated FTP Client Exception", msg)
        log.info("E-Mail sent")
        sys.exit(1)
    print("\nAll done!")
    log.info("All done!")
    sys.exit(0)

PowerShell Script to VM Snapshots Older than X Days

I frequently see clients forget to delete snapshots after they are created. Over time as more and more data is changed on the VM, the size of the snapshot keeps growing. This growing snapshot size creates several problems

1. VMs tend to run slower when using snapshots especially those VMs that are disk intensive. The snapshot image needs to be updated for each write IO. So if you have 4 snapshots of a VM, that leads to 4 writes in comparison to a single write to the primary VMDK file with no snapshots
2. When left unchecked, snapshots can grow to use up the entire datastore. When the datastore runs out of space, your VMs will likely all crash. At a minimum new VMs will not be able to start as space is required to create the VSWP file.
3. When creating linked snapshots, i.e. a snapshot of a snapshot, you can quickly run out of space even when removing the snapshot. (Note: This issue is resolved in ESX v4.1 and later) For example if you have 4 snapshots all linked together at 10GB each and try to delete all the snapshots, you will need signifcantly more than 40GB before all the snapshots are deleted. That is because each snapshot needs to be copied into the previous snapshot and all these copies need to finish before even a single snapshot is deleted. For more information http://www.yellow-bricks.com/2010/07/05/changes-to-snapshot-mechanism-delete-all/
4. There is a risk especially with large snapshots of data corruption. It is exceedingly rare but can occur anytime you have multiple linked snapshots and the link fails or when you run out of disk space.

To avoid all of these potential problems, I've written a simple script that I use to e-mail me when a snapshot is getting old and either 1. someone forgot to delete 2. is probably using up too much space or 3. is making the VM run slower than normal.

To run the full script you'll need Windows Powershell v1 or better, which comes with Windows 2008 and Windows 7, VMware CLI, and the Powershell Community Extensions. You only need PSCX if you plan on using the email functionality of the script.

Here is the Script

Get-VM | Get-Snapshot | Where { $_.Created -lt (Get-Date).AddDays(-15)} | FL | Send-SmtpMail -to [email protected] -Subject "Virtual Machines with Snapshots Older than 15 Days" -Body $_

The details...

• Get-VM -- Gets a list of VMs
• Get-Snapshot -- Gets a list of all the snapshots on the VMs if any snapshots exist
• Where { $_.Created -lt (Get-Date).AddDays(-15)} -- This is really the meat of the script. Here the where-object, a.k.a "where", selects only those snapshots that were created ($_.Created) more than 15 days ago ((Get-Date).AddDays(-15)). You are free to change the number of days.
• FL -- Format-List formats the result into an easy to read table
• Send-SmtpMail -- Sends me an e-mail message with the results. If the body of the message is blank, no snapshots are older than 15 days.

I then schedule a batchfile to run the job weekly. I use Automate, but any good job scheduling tool will work just as well, e.g. Windows Task Scheduler, cron, etc.

There's a lot that could be done to make this a little more sexy, for example, instead of using a date use a snapshot size, only send a mail when there are snapshots older than 15 days, and make the number of days a variable instead of hard coded.

Some important reminders on snapshots...Don't use snapshots as a means for long-term backup. VMware snapshots are designed to be deleted in a short time frame, i.e. a few days at most, and hence are not an appropriate solution for long term storage.

Also, snapshots are stored with the VMX file and not necessarily with the VMDK you are snapshotting. Too often I see someone look at a datastore that has plenty of free space, take a snapshot, and then run out of space on a different datastore. They never realized that the snapshot was being stored on a separate datastore with the VMX file.