Blumetech's Tech Blog

Major Improvements to Amazon Health Events

"Back in the day" to get alerted for any Amazon related health issues required a combination for "screen scraping" the AWS Status Health Dashboard and polling the AWS API for health related issues. The "screen scraping" was constantly prone to issues and required continual maintenance. Everytime Amazon made a change to their webpage the status alerting would have issues. The AWS API was only mildly better, but still required you to poll serveral different APIs for status. Want to know if EC2 has an issue? Poll the EC2 API. Want to know if RDS has an issue? Poll the RDS API. AWS has over 100 services and each one requires a different check. This is certainly not a method that scales well.

We are happy to announce a major improvement in how we handle Amazon alerting. Now we are leveraging Amazon's new Personal Health Dashboard to get events. Those events include:

1. Major outages with AWS in general
2. Specific issues with AWS infrastructure that affects only you, e.g. an EC2 instance has failed, direct connect outage, etc.
3. Scheduled maintenance events, e.g. an EC2 instance reboot, VPN upgrade, etc.
4. General notifications made by AWS for your account, e.g. an upgrade to REDIS, a new version of storage gateway, etc.

Our customers also provided us feedback that the alert volume can be too high at times. We've taken that feedback and made some adjustments to how we do alerting. Here are the major changes:

1. Major issues that effect your account will be sent every 30 mins.
2. Scheduled maintenance alerts will be sent only once per day
3. General notifications on your account will be sent only once per day for a maximum of 4 times per notification.

All of these new features are available immediately to our Managed Services customers. No additional fees or charges apply. Enjoy!

UPDATE - Lifecycle Management for Amazon EBS Snapshots

On July 12^th Amazon released “Lifecycle Management for Amazon EBS Snapshots”. You can read about this new service at https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/

This new service can be a replacement for the backup functionality that is included with Blumetech’s Managed Services for AWS. The BlumeTech backup service does have features (listed below) that you may not want to lose by switching to the AWS Lifecycle Management service.  Blumetech’s Managed Services does also provide monitoring and AWS related operational automation.

Below are some important differences between Blumetech’s backup service and Amazon’s.

First, Blumetech's can take snapshots of EBS volumes as often as every hour (and even more frequently upon request). Amazon's service can only snapshot every 12 hours. Therefore, with Amazon's service in the event a restore is needed, you could lose up to 12 hours worth of data. Blumetech believes that this is an unacceptable data loss for mission critical applications.

Second, Amazon's service does not offer a "default" catch-all backup option. When setting up a lifecycle rule you must specify the specific volumes by tag that you want to back-up. Blumetech services works in two very different ways. First our backups focus on the instance/server itself. We believe that this is the logical unit that customers care about and not EBS volumes. With Amazon's service you need to tag EBS Volumes and not servers. This forces you to tag all EBS volumes, which will probably be significantly more than the number of servers. Second, Blumetech has a default backup rule that ensures that even if no tags are applied to server it will always get backed up. Our experience has been that people naturally forget to tag their instances. With Amazon's service this would lead to the server not being backed up. With Blumetech, even without any tags we ensure that every server is backed up.

Third, Blumetech adds critical tags to the EBS snapshot that enable faster more reliable restores. Our snapshots include the source instance_id, the source instance name, the source instance description, and finally the device name. The device name is used to attach the volume to the instance at the correct location.

Having managed our service for many years, we know that all of these tags are critical to restoring an EBS volume. To do a restore you need to easily find all of the snapshots for a particular server. And then restore the volumes ensuring that the volume is mounted in the same exact place (device_name).  For example, if you had a server with 5 attached volumes, when doing a restore of the server, those 5 volumes need to mounted to the server in the SAME EXACT ORDER. If the order is different, then the server won’t restore. With 5 volumes, there are 120 different combinations to try before you land on the right one.

We also copy all the tags from the source volume to the EBS snapshot. If you have tags specified on your EBS volumes for any type of management, e.g. cost accounting, automation, etc., these tags are propagated automatically for you to the snapshot. Amazon’s service does not propagate tags from the original volume to the snapshot.

Fourth, Blumetech automatically monitors our snapshot process. As a fully managed service we ensure that our customer’s snapshots are occurring and will take steps automatically to rectify issues. Amazon’s solution moves the monitoring and support burden back to the customer.

Regardless of Amazon’s shortcomings, their service is free to their customers.

With that said we are dropping our price by 25% for Blumetech’s Managed Services fees for all customers starting August 1^st.

If a customer chooses to migrate their EBS backup operations from Blumetech to Amazon that customer will get an additional 25% reduction from their current Blumetech Managed Services cost for a total reduction of 50% from today’s rate. The new discounted fee corresponds to the remaining Blumetech Managed Services, i.e. monitoring and AWS automation. Please note that the missing snapshot functionality mentioned above with Amazon’s service will still apply.

NEW!! - Support for EBS Volume Health Monitoring and Alerting

AWS EC2 EBS are exceedingly fault tolerant virtual disks. Straight from Amazon's docs...

Amazon EBS volumes are designed to be highly available and reliable. At no additional charge to you, Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. For more details, see the Amazon EC2 and EBS Service Level Agreement.

The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS Snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.

With Blumetech Managed Services for AWS, we take snaphots of our client's EBS volumes as often as hourly. With all these snapshots the likelihood of a volume failure is extremely rare.

However in the words of Werner Vogels, CTO of Amazon, "Everything fails all the time". EBS volumes no matter how reliable will eventually have failures.

To better respond and be alerted to these failures, we are happy to announce that we will be monitoring all of our client's EBS volumes for any impaired or warning status. We scan for status changes by default every 5 minutes. If an impaired EBS volumes is detected, an e-mail is sent to the client.

This additional service is immediately available to all of Blumetech's Managed Services customers. There is no additional charge for the additional monitoring. Enjoy!

UPDATED!! -- AWS Service Health Monitor

By now you have probably heard about an S3 outage...Service Disruption...that happened on Feb 28th.

During the outage the AWS Service Health Dashboard was not operational. Although the page was physically up, AWS was unable to update the service health to reflect the outage caused by S3. The AWS Service Health Dashboard depends on S3 to operate properly and hence during the S3 outage the dashboard was not working properly. AWS has recently resolved this operational dependency.

The Blumetech AWS Managed Service uses the AWS Service Health Dashboard to report on outages for customers. Our service automatically "scrapes" the webpage, looks for outages on AWS, and then e-mails customers about the failure. This has become a very useful service for customers and quickly notifies them of service disruption. However, during the S3 event, since this dashboard was not working, our service was not able to e-mail customers about the problem.

Last week we resolved this issue. Our managed service now uses API calls to AWS to confirm service status. This process is much less brittle compared to the old screen "scraping" process. We are now confident that future, unplanned outages will be accurately detected with customers being properly and quickly notified.

Helpful links to help keep with smart computing

Secure computing time requires that all users take an intelligent approach at all times.  We all at least know someone who has productivity or even lost files due to viruses and malware attacks.  Someone recently announced that the most important thing that anyone can do is to SLOW DOWN.  We are all so busy that sometimes we forget to think before clicking on hyperlinks that were emailed to us and open email attachments.  Take the time to look at where the hyperlinks are going to take you (hover your mouse over the hyperlink) and avoid using your phone to open any questionable emails.

The following links offer tools to check files and websites for safety and help to keep our computers and networks safe.

These are invaluable tools to check suspicious files and hyperlinks:

www.virustotal.com to scan files before opening if questionable

www.scanurl.net to check URL to see if it's safe to click on the hyperlink

http://www.passwordmeter.com/ password checker to help select secure password

www.recordedfuture.com – recommended to perform real-time checks of all browsing

Windows 7 update struggle

We’ve been struggling with some computers which won’t update.  During updates, the system hangs searching for updates but never completes.  After several hours of struggling with various suggestions, following this post finally did the trick!  Thank you plugable for this post.

http://plugable.com/2016/06/08/windows-7-wont-update-what-to-do/

Stop and eliminate the Windows 10 ‘free upgrade’ prompt

This may be old news but we continue to have user’s computers incessantly prompting them to upgrade to Windows 10.  Some may not mind this but most find this annoying enough that they are tempted to click on the link to upgrade just to make the icon go away.  Most of the time an upgrade to Windows 10 works without issue but not everyone wants Windows 10 and certainly don’t want a persistent reminder on their desktop.

Good news is that you can easily get rid of the reminder without upgrading.  You can uninstall a Windows Update KB3035583 and then hide it so that it doesn’t reinstall itself on the next update cycle.

Below is a link to a detailed article with step by step instructions and pictures on how to uninstall and hide the offending update.

http://www.howtogeek.com/218856/how-do-you-disable-the-get-windows-10-icon-shown-in-the-notification-tray/

Hope this helps!

UPDATED - Support for Longer Resource IDs

Amazon recently announced they would be implementing longer resource IDs in 2016. Resources that are affected include instances, snapshots, volumes, and reservations. You can read more on Amazon's detailed plans at https://aws.amazon.com/blogs/aws/theyre-here-longer-ec2-resource-ids-now-available/

The first phase of this migration to longer Resource IDs applies to newly launched instances.

As an AWS Advanced partner, Blumetech had notice of the impending change and prepared for a seamless transition. Blumetech has done extensive testing of our Managed Services product using these longer instance IDs. We are happy to report that an update was deployed earlier this month. This update fully supports the new instance and reservation ID format.

We will continue to monitor Amazon's ongoing migration to longer Resource IDs and are prepared to add support for longer volume and snapshot IDs as soon as they are announced.

Updated - Reduced E-mails for AWS Scheduled Events

AWS can schedule events for your instances, such as a reboot, stop/start, or retirement. These events do not occur frequently. If one of your instances will be affected by a scheduled event, AWS updates the AWS Console prior to the scheduled event with details about the event including the start and end date. Depending on the event, you may be able to take action to control the timing of the event.

Customers need to periodically check the AWS Console for new events in order to avoid unplanned downtime when AWS performs maintenance. Many AWS users are completely unaware of these scheduled events. 

To resolve this problem, AWS Managed Services from Blumetech monitors our customer's AWS accounts for scheduled events. When AWS schedules one of these events, Blumetech's Managed Service sends an e-mail alert to the customer.

However, customers were complaining about a flood of e-mail alerts that occur any time there is a scheduled event. Blumetech's Managed Services checks our customer accounts every 5 minutes and would send an e-mail alert for a scheduled event. Customer's were receiving e-mail messages for events that would not occur for days and sometimes even weeks later.

We have made adjustments to our e-mail sending algorithm as it pertains to scheduled events. Scheduled event e-mails will now only be sent once per day. This change will reduce the e-mail volume for scheduled events by over 99%.

Use it Now
This functionality was released earlier December and has automatically been applied to your account.

Updated - Faster Backups

As part of AWS Managed Services from Blumetech, you can easily backup up your EC2 Instances in a completely automated way. Our Managed Service takes care of the management of your AWS backups for you. The service allows you to set-up a schedule to backup your instances hourly, weekly, and daily. You configure your backup schedule the way you want it.

Previously we would backup our customers EC2 Instances one at a time. For some of our larger customers with over 500 EC2 instances, it was taking too long to complete the backup jobs. A backup job for this many instances would take up to 45 minutes to complete. For hourly backups, the backup job was just completing before the next backup executed.

We have integrated some major improvements into the the backup process so that up to 20 instances are backed up concurrently. This change has led to drastic reductions in the time it takes to complete backups. Backup job times have reduced by over 80%. For the majority of our customers this means that backups will complete within 5 minutes of execution. We have some customers with over 500TB of data that are backed up in under 10 minutes.

Use it Now
This functionality was released earlier this month and has automatically been applied to your account.