Blumetech's Tech Blog

Major Improvements to Amazon Health Events

"Back in the day" to get alerted for any Amazon related health issues required a combination for "screen scraping" the AWS Status Health Dashboard and polling the AWS API for health related issues. The "screen scraping" was constantly prone to issues and required continual maintenance. Everytime Amazon made a change to their webpage the status alerting would have issues. The AWS API was only mildly better, but still required you to poll serveral different APIs for status. Want to know if EC2 has an issue? Poll the EC2 API. Want to know if RDS has an issue? Poll the RDS API. AWS has over 100 services and each one requires a different check. This is certainly not a method that scales well.

We are happy to announce a major improvement in how we handle Amazon alerting. Now we are leveraging Amazon's new Personal Health Dashboard to get events. Those events include:

1. Major outages with AWS in general
2. Specific issues with AWS infrastructure that affects only you, e.g. an EC2 instance has failed, direct connect outage, etc.
3. Scheduled maintenance events, e.g. an EC2 instance reboot, VPN upgrade, etc.
4. General notifications made by AWS for your account, e.g. an upgrade to REDIS, a new version of storage gateway, etc.

Our customers also provided us feedback that the alert volume can be too high at times. We've taken that feedback and made some adjustments to how we do alerting. Here are the major changes:

1. Major issues that effect your account will be sent every 30 mins.
2. Scheduled maintenance alerts will be sent only once per day
3. General notifications on your account will be sent only once per day for a maximum of 4 times per notification.

All of these new features are available immediately to our Managed Services customers. No additional fees or charges apply. Enjoy!

UPDATE - Lifecycle Management for Amazon EBS Snapshots

On July 12^th Amazon released “Lifecycle Management for Amazon EBS Snapshots”. You can read about this new service at https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/

This new service can be a replacement for the backup functionality that is included with Blumetech’s Managed Services for AWS. The BlumeTech backup service does have features (listed below) that you may not want to lose by switching to the AWS Lifecycle Management service.  Blumetech’s Managed Services does also provide monitoring and AWS related operational automation.

Below are some important differences between Blumetech’s backup service and Amazon’s.

First, Blumetech's can take snapshots of EBS volumes as often as every hour (and even more frequently upon request). Amazon's service can only snapshot every 12 hours. Therefore, with Amazon's service in the event a restore is needed, you could lose up to 12 hours worth of data. Blumetech believes that this is an unacceptable data loss for mission critical applications.

Second, Amazon's service does not offer a "default" catch-all backup option. When setting up a lifecycle rule you must specify the specific volumes by tag that you want to back-up. Blumetech services works in two very different ways. First our backups focus on the instance/server itself. We believe that this is the logical unit that customers care about and not EBS volumes. With Amazon's service you need to tag EBS Volumes and not servers. This forces you to tag all EBS volumes, which will probably be significantly more than the number of servers. Second, Blumetech has a default backup rule that ensures that even if no tags are applied to server it will always get backed up. Our experience has been that people naturally forget to tag their instances. With Amazon's service this would lead to the server not being backed up. With Blumetech, even without any tags we ensure that every server is backed up.

Third, Blumetech adds critical tags to the EBS snapshot that enable faster more reliable restores. Our snapshots include the source instance_id, the source instance name, the source instance description, and finally the device name. The device name is used to attach the volume to the instance at the correct location.

Having managed our service for many years, we know that all of these tags are critical to restoring an EBS volume. To do a restore you need to easily find all of the snapshots for a particular server. And then restore the volumes ensuring that the volume is mounted in the same exact place (device_name).  For example, if you had a server with 5 attached volumes, when doing a restore of the server, those 5 volumes need to mounted to the server in the SAME EXACT ORDER. If the order is different, then the server won’t restore. With 5 volumes, there are 120 different combinations to try before you land on the right one.

We also copy all the tags from the source volume to the EBS snapshot. If you have tags specified on your EBS volumes for any type of management, e.g. cost accounting, automation, etc., these tags are propagated automatically for you to the snapshot. Amazon’s service does not propagate tags from the original volume to the snapshot.

Fourth, Blumetech automatically monitors our snapshot process. As a fully managed service we ensure that our customer’s snapshots are occurring and will take steps automatically to rectify issues. Amazon’s solution moves the monitoring and support burden back to the customer.

Regardless of Amazon’s shortcomings, their service is free to their customers.

With that said we are dropping our price by 25% for Blumetech’s Managed Services fees for all customers starting August 1^st.

If a customer chooses to migrate their EBS backup operations from Blumetech to Amazon that customer will get an additional 25% reduction from their current Blumetech Managed Services cost for a total reduction of 50% from today’s rate. The new discounted fee corresponds to the remaining Blumetech Managed Services, i.e. monitoring and AWS automation. Please note that the missing snapshot functionality mentioned above with Amazon’s service will still apply.

NEW!! - Support for EBS Volume Health Monitoring and Alerting

AWS EC2 EBS are exceedingly fault tolerant virtual disks. Straight from Amazon's docs...

Amazon EBS volumes are designed to be highly available and reliable. At no additional charge to you, Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. For more details, see the Amazon EC2 and EBS Service Level Agreement.

The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS Snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.

With Blumetech Managed Services for AWS, we take snaphots of our client's EBS volumes as often as hourly. With all these snapshots the likelihood of a volume failure is extremely rare.

However in the words of Werner Vogels, CTO of Amazon, "Everything fails all the time". EBS volumes no matter how reliable will eventually have failures.

To better respond and be alerted to these failures, we are happy to announce that we will be monitoring all of our client's EBS volumes for any impaired or warning status. We scan for status changes by default every 5 minutes. If an impaired EBS volumes is detected, an e-mail is sent to the client.

This additional service is immediately available to all of Blumetech's Managed Services customers. There is no additional charge for the additional monitoring. Enjoy!

UPDATED!! -- AWS Service Health Monitor

By now you have probably heard about an S3 outage...Service Disruption...that happened on Feb 28th.

During the outage the AWS Service Health Dashboard was not operational. Although the page was physically up, AWS was unable to update the service health to reflect the outage caused by S3. The AWS Service Health Dashboard depends on S3 to operate properly and hence during the S3 outage the dashboard was not working properly. AWS has recently resolved this operational dependency.

The Blumetech AWS Managed Service uses the AWS Service Health Dashboard to report on outages for customers. Our service automatically "scrapes" the webpage, looks for outages on AWS, and then e-mails customers about the failure. This has become a very useful service for customers and quickly notifies them of service disruption. However, during the S3 event, since this dashboard was not working, our service was not able to e-mail customers about the problem.

Last week we resolved this issue. Our managed service now uses API calls to AWS to confirm service status. This process is much less brittle compared to the old screen "scraping" process. We are now confident that future, unplanned outages will be accurately detected with customers being properly and quickly notified.

UPDATED - Support for Longer Resource IDs

Amazon recently announced they would be implementing longer resource IDs in 2016. Resources that are affected include instances, snapshots, volumes, and reservations. You can read more on Amazon's detailed plans at https://aws.amazon.com/blogs/aws/theyre-here-longer-ec2-resource-ids-now-available/

The first phase of this migration to longer Resource IDs applies to newly launched instances.

As an AWS Advanced partner, Blumetech had notice of the impending change and prepared for a seamless transition. Blumetech has done extensive testing of our Managed Services product using these longer instance IDs. We are happy to report that an update was deployed earlier this month. This update fully supports the new instance and reservation ID format.

We will continue to monitor Amazon's ongoing migration to longer Resource IDs and are prepared to add support for longer volume and snapshot IDs as soon as they are announced.

Updated - Reduced E-mails for AWS Scheduled Events

AWS can schedule events for your instances, such as a reboot, stop/start, or retirement. These events do not occur frequently. If one of your instances will be affected by a scheduled event, AWS updates the AWS Console prior to the scheduled event with details about the event including the start and end date. Depending on the event, you may be able to take action to control the timing of the event.

Customers need to periodically check the AWS Console for new events in order to avoid unplanned downtime when AWS performs maintenance. Many AWS users are completely unaware of these scheduled events. 

To resolve this problem, AWS Managed Services from Blumetech monitors our customer's AWS accounts for scheduled events. When AWS schedules one of these events, Blumetech's Managed Service sends an e-mail alert to the customer.

However, customers were complaining about a flood of e-mail alerts that occur any time there is a scheduled event. Blumetech's Managed Services checks our customer accounts every 5 minutes and would send an e-mail alert for a scheduled event. Customer's were receiving e-mail messages for events that would not occur for days and sometimes even weeks later.

We have made adjustments to our e-mail sending algorithm as it pertains to scheduled events. Scheduled event e-mails will now only be sent once per day. This change will reduce the e-mail volume for scheduled events by over 99%.

Use it Now
This functionality was released earlier December and has automatically been applied to your account.

Updated - Faster Backups

As part of AWS Managed Services from Blumetech, you can easily backup up your EC2 Instances in a completely automated way. Our Managed Service takes care of the management of your AWS backups for you. The service allows you to set-up a schedule to backup your instances hourly, weekly, and daily. You configure your backup schedule the way you want it.

Previously we would backup our customers EC2 Instances one at a time. For some of our larger customers with over 500 EC2 instances, it was taking too long to complete the backup jobs. A backup job for this many instances would take up to 45 minutes to complete. For hourly backups, the backup job was just completing before the next backup executed.

We have integrated some major improvements into the the backup process so that up to 20 instances are backed up concurrently. This change has led to drastic reductions in the time it takes to complete backups. Backup job times have reduced by over 80%. For the majority of our customers this means that backups will complete within 5 minutes of execution. We have some customers with over 500TB of data that are backed up in under 10 minutes.

Use it Now
This functionality was released earlier this month and has automatically been applied to your account.

Automatically Encrypt AWS Instances

Encryption is an important part of any data protection strategy. Over the past two years, Amazon AWS has introduced many features that are designed to simplify the task of storing your cloud-based information in encrypted form. Many customers appreciate the fact that AWS makes it very easy for them to encrypt their data. They enable it as needed, and rely on AWS for the heavy lifting. 

Recently AWS announced that they can encrypt data stored on your EBS volumes (the virtual disks attached to your cloud servers). The stored data is encrypted, as is the data transfer path between the EBS volume and the EC2 instance. The data is decrypted on the instance on an as-needed basis, then stored only in memory.

This feature will aid your security, compliance, and auditing efforts by allowing you to verify that all of the data that you store on EBS is encrypted, whether it is stored on a boot volume or on a data volume. Further, because this feature makes use of KMS, you can track and audit all uses of the encryption keys.

Many years ago, I ran a project that required encryption at rest for all of the mission critical data. At the time to meet this business need required $200+K in hardware, software, and professional services. The installation took over 3 months.

With the recent announcements from AWS, encryption at rest is now as easy as checking a checkbox. Because encryption is now so easy to do, executive officers will expect that all business data is encrypted at rest. There is no additional cost or IT burden and hence no excuse not to implement the additional security.

Encrypting new AWS instances is as simple as a checkbox, but how to we encrypt existing instances that already have data? In some cases these instances have multiple terabytes of unencrypted data.

Blumetech asked one of our skilled AWS engineers to time how long it would take to convert an existing instance into an encrypted one.  The test had one big caveat: nothing about the instance could change except for encryption. We wanted the IP, instance ID, security groups, tags, placement, tenancy, etc. to all be exactly the same. We were concerned that if any of these values changed application related problems would result.

On average it took our engineer about an 1 hour to encrypt a machine with two EBS volumes. Since the instance is stopped during most of the work (we can't safely encrypt data that is in use), lots of downtime was required to do the encryption. In addition, the manual process was prone to simple errors.

To resolve all of the problems Blumetech create an application that will automatically encrypt instances. Some of the key features:

• Error free conversion - The process is completely automated 
• High Throughout - Over 50 instances/hr can be processed
• Built-in Multithreading
• Instance Verification Checks - Runs a series of checks to ensure that an instance can be encrypted
• Safe - Allows you to optionally retain your unencrypted data in case a rollback is required
• Detailed Logging 

Use it Now
This functionality was released earlier this month. Please contact us if you are interested in using this new application.

Updated - Reserved Instance Expiration Alerts

As part of AWS Managed Services from Blumetech, we recently announced a new feature that will alert you to any pending Reserved Instance expirations. We monitor your account for any Reserved Instances that will expire in the next 30 days. We then send you six reminder e-mail alerts that are spread-out evenly over the remainder of the Reserved Instance contract.

This new feature is free to you and is included for all AWS Managed Services customers. You can read more at http://blogs.blumetech.com/blumetechs-tech-blog/2015/12/new-reserved-instance-expiration-alerts.html

The response from clients was overwhelming. This simple feature had been missing from their AWS account management. Many customers were thankful that they would no longer be "burned" by AWS when the reserved instance was expiring.

Customers liked the new feature so much that they asked us to include reserved instances for RDS and Elasticache as well. Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache.

As part of AWS Managed Services from Blumetech, we are now adding a new feature that will alert you to any pending Reserved Instance expirations including RDS and Elasticache reserved instances. As with EC2 reserved instances, we will monitor your account for any Reserved Instances that will expire in the next 30 days. We then send you six reminder e-mail alerts to remind you of the upcoming expiration.

This new feature is free to you and is included for all AWS Managed Services customers.

Use it Now
This functionality was released earlier this month. Please note that this new functionality requires some additional AWS read-only permissions to your AWS account. Specifically we need to be able to list any of your RDS and ElastiCache reserverations. Contact us and we'll be happy to walk you through the process. It's very easy.

New - Reserved Instance Expiration Alerts

Amazon EC2 Reserved Instances allow you to reserve Amazon EC2 computing capacity for 1 or 3 years, in exchange for a significantly discounted hourly rate (up to 75%), compared to On-Demand Instance pricing.

Reserved Instances can significantly lower your computing costs for your workloads and provide a capacity reservation so that you can have confidence in your ability to launch the number of instances you have reserved when you need them.

One of the subtle downsides of Reserved Instances is that you can easily lose track on when an instance contract is going to expire. When that occurs AWS will automatically switch you to the higher On-Demand Instance pricing. At the higher price you can find yourself getting an AWS bill that is surprisingly higher than previous months because of the Reserved Instance expiration.

As part of AWS Managed Services from Blumetech, we are now adding a new feature that will alert you to any pending Reserved Instance expirations. We will monitor your account for any Reserved Instances that will expire in the next 30 days. We then send you six reminder e-mail alerts that are spread-out evenly over the remainder of the Reserved Instance contract.

This new feature is free to you and is included for all AWS Managed Services customers.

Use it Now
This functionality was released earlier this month and has automatically been applied to your account.