Amazon AWS and BotoHelper Script

Blumetech recently migrated all of its servers to Amazon AWS. The migration went very smoothly and we couldn’t be happier with the services we are getting from Amazon. For an SMB like Blumetech, Amazon AWS completely does away with worrying about

1. Hardware to run Hypervisors including Storage, CPU, and Memory. There is a certain piece of mind knowing that you’ll never have to replace a failed power supply or hard drive ever again.
2. The Hypervisor itself. No more yearly ESX upgrades
3. DR – The 5 9’s uptime provided by Amazon is all the uptime we need. And if we ever needed DR it’s simple to leverage another Amazon EC2 zone.

After the migration, I wrote a few scripts to manage the EC2 instances. Like most SMBs we don’t have any long term archival needs. A two week backup rotation is all we really need.

With Amazon AWS, snapshots are cheap. Only the differential data is stored in the snap. That data is stored on Amazon S3 at $.11/GB/Month. We take backups daily and store them for two weeks. Monthly cost for backup is something like $20. No more tapes. No more tape drives. No swapping tapes. No more backup software upgrades. No more scheduling backups for new servers. The script does it all.

 

I wrote a simple wrapper using the Python BOTO library. The BOTO library is the API interface to all thing Amazon AWS written in Python. The wrapper handles simple tasks like taking snapshots and checking the status of the EC2 Instances.

Here is the code:

import logging
from operator import attrgetter

import boto

from LogFileHelper2x import get_default_logger

class BotoHelper():
    def __init__(self, aws_access_key_id, aws_secret_access_key, logger_name=None):
        if logger_name == None:
            self.log = get_default_logger(logger_name='BotoHelper', console_logging=True)
        else:
            self.log = logging.getLogger("%s.BotoHelper" % logger_name)
        self.log.info("Connecting to Amazon EC2")
        self.ec2 = boto.connect_ec2(aws_access_key_id, aws_secret_access_key)

    def get_instance_volumes(self, instance_id):
        volumes = self.ec2.get_all_volumes(filters={'attachment.instance-id': instance_id})
        return volumes

    def get_instance_id(self, instance_name):
        for instance in self.get_all_instances(): 
            tags = instance.__dict__['tags']
            name = tags['Name']
            if instance_name.lower() == name.lower():
                return instance.id
    
    def get_instance_name(self, instance_id):
        for instance in self.get_all_instances(): 
            tags = instance.__dict__['tags']
            if instance_id == instance.id:
                return tags['Name']
    
    def get_all_instances(self):
        instances = []
        all_inst = self.ec2.get_all_instances()
        for res in all_inst: 
            # each reservation have a instance: 
            for instance in res.instances: 
                instances.append(instance)
        return instances
    
    def backup_instance(self, instance_name, max_snapshots=30, description_prefix="Automated_Backup"):
        volumes = self.get_instance_volumes(self.get_instance_id(instance_name))
        snapshot_description = "%s_%s" % (description_prefix, instance_name)
        for volume in volumes:
            snaps = self.ec2.get_all_snapshots(filters={"volume-id": volume.id, "description": snapshot_description})
            self.log.info("%s: Creating new snapshot." % volume.id)
            self.log.info("%s %s snapshots currently exist." % (len(snaps), snapshot_description))
            self.ec2.create_snapshot(volume.id, snapshot_description)
            #    Find oldest snapshots that exceed max_snapshots limit
            #purgeable = sorted(snaps, key=lambda x: x.start_time)[:-max_snapshots]
            purgeable = sorted(snaps, key=attrgetter("start_time"))[:-max_snapshots]
            if purgeable:
                self.log.info("Deleting snapshots for %s > %s: %s" % (volume, max_snapshots, purgeable))
            for snap in purgeable:
                self.log.info("Deleting old snapshot: %s" % snap.id)
                self.ec2.delete_snapshot(snap.id)
                
    def backup_all_instances(self, max_snapshots=30, description_prefix="Automated_Backup"):
        for instance in self.get_all_instances():
            tags = instance.__dict__['tags']
            name = tags['Name']
            self.log.info("Creating backup for instance: %s" % name)
            self.log.info("Instance ID: %s" % instance.id)
            self.backup_instance(name, max_snapshots, description_prefix)
    
    def get_all_instance_status(self):
        return self.ec2.get_all_instance_status()

Comments

[Scott Chapman]

Not sure if you will see this. I was wondering if you have any data to back up the 5 9's uptime. I need something for comparison to other solutions and can't find it anywhere. Thanks