{"id":237,"date":"2014-05-18T05:09:36","date_gmt":"2014-05-18T05:09:36","guid":{"rendered":"http:\/\/blog.maclawran.ca\/?p=237"},"modified":"2014-05-19T07:40:47","modified_gmt":"2014-05-19T07:40:47","slug":"devops-12-helpful-hints-for-living-with-aws","status":"publish","type":"post","link":"https:\/\/blog.maclawran.ca\/?p=237","title":{"rendered":"12 helpful tips (and a few scripts) for AWS"},"content":{"rendered":"<p><a href=\"https:\/\/console.aws.amazon.com\/console\/home\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-248 size-full\" src=\"http:\/\/blog.maclawran.ca\/wp-content\/uploads\/2014\/05\/Screen-Shot-2014-05-18-at-2.48.02-AM.png\" alt=\"Screen Shot 2014-05-18 at 2.48.02 AM\" width=\"327\" height=\"241\" \/><\/a>I really like working with Amazon&#8217;s AWS.\u00a0 I&#8217;ve been taking care of Unix\/Linux systems for a very long time, and AWS for a few years&#8230; Here are a few tips to make your life with AWS easier&#8230;<\/p>\n<p>Giant sites with hundreds of machines can most likely skip this post, or just use it to remember back to when you were small.\u00a0 Remember these are my opinions about how things should be done.\u00a0 Your mileage may vary.\u00a0 Or I could just be completely wrong&#8230; please let me know if I am.<\/p>\n<ul>\n<li><strong>Tip #1: A couple of little shell scripts<\/strong> &#8211; one called <a href=\"http:\/\/kimosabe.net\/aws\/SPINUP.txt\" target=\"_blank\">SPINUP (click to view)<\/a> and the other called <a href=\"http:\/\/kimosabe.net\/aws\/BACKUP.txt\" target=\"_blank\">BACKUP (click to view)<\/a>.\u00a0 SPINUP creates instances based on the parameters you set in the shell script; it&#8217;s a wrapper for <a href=\"http:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/CommandLineReference\/ApiReference-cmd-RunInstances.html\" target=\"_blank\">ec2-run-instances<\/a>.\u00a0 BACKUP takes a snapshot of any running image &#8211; it&#8217;s a trivial shell around <a href=\"http:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/CommandLineReference\/ApiReference-cmd-CreateImage.html\" target=\"_blank\">ec2-create-image <\/a>program.\u00a0 You can run BACKUP then use SPINUP to make a clone.\u00a0 Instead of building my environments from scratch using Chef, I tend to improve on a base AMI and use that with BACKUP and SPINUP.\u00a0<a href=\"http:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/CommandLineReference\/set-up-ec2-cli-linux.html\" target=\"_blank\"> You&#8217;ll need to install the AWS Command line tools <\/a>and you should rename SPINUP.txt and BACKUP.txt to SPINUP.sh and BACKUP.sh respectively.<\/li>\n<li><strong>Tip #2: Useful Instance Descriptions with Dates\u00a0<\/strong> For quite a while I was guilty of naming a new image something like &#8220;NEW APP SERVER&#8221;.\u00a0 Fine, until the next iteration&#8230; then it&#8217;s something like &#8220;REALLY NEW APP SERVER&#8221;&#8230; and although if you look hard, AWS will tell you how long an instance has been running (like 7435 hours), it&#8217;s not really helpful in dispelling the self-induced confusion.\u00a0 So include the date and description in a consistent format, i.e.:\u00a0 &#8220;APP1 SERVER &#8211; MAY 18 2014&#8221;.<\/li>\n<li><strong>Tip #3: Use<a href=\"http:\/\/aws.amazon.com\/iam\/details\/mfa\/\" target=\"_blank\"> 2 factor Authentication<\/a> to login to the AWS console.<\/strong>\u00a0 For all accounts, always.\u00a0 Amazon has the <a href=\"https:\/\/play.google.com\/store\/apps\/details?id=com.amazonaws.mobile.apps.Authenticator\" target=\"_blank\">AWS Virtual MFA<\/a> app for Android and <a href=\"https:\/\/itunes.apple.com\/us\/app\/google-authenticator\/id388497605?mt=8\" target=\"_blank\">Google Authenticator for the iphone<\/a> to generate auth codes, and you&#8217;ve always got your phone, so like, there&#8217;s no reason not to do it.\u00a0 Remember if someone gets unauthorized access to your console they can destroy everything.<a href=\"http:\/\/readwrite.com\/2014\/04\/15\/amazon-web-services-hack-bitcoin-miners-github#awesm=~oEB5CAchZIM0EB\" target=\"_blank\">\u00a0 Or mine bitcoin.<\/a><\/li>\n<li><strong>Tip #4:Use <a href=\"http:\/\/aws.amazon.com\/route53\/\" target=\"_blank\">Amazon Route53 for your DNS<\/a><\/strong> if you can, it&#8217;s awesome.\u00a0 <a href=\"http:\/\/aws.amazon.com\/route53\/pricing\/\" target=\"_blank\">Incredibly cheap<\/a> (about 1\/100 of something like DYN), and updates propagate across Amazon&#8217;s internal network very very quickly (like in seconds).<\/li>\n<li><strong>Tip #5: Assign <a href=\"http:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/elastic-ip-addresses-eip.html\">Elastic IP addresses<\/a><\/strong> to each of your &#8220;main services&#8221;, and use those fixed addresses in Route53 above.\u00a0 This suggestion may appear to be a little bit counterintuitive &#8211; since using the CNAME to Amazon&#8217;s internal names would allow you to do something like &#8220;ssh app1.bobo.com&#8221; and return the public address if you&#8217;re outside of AWS, or the internal AWS address if you&#8217;re inside.\u00a0 My logic for doing this is that I&#8217;ve been burned by DNS propagation times &#8211; I&#8217;d make the change, ssh to app1.bobo.com and would still be pointed to the old machine.\u00a0 So I&#8217;d have to ssh in using the (new) IP address.\u00a0 Gets old fast.\u00a0 Especially when you power down the production machines my mistake.\u00a0 Gets even worse when you&#8217;re swapping machines around (dev-&gt;prod-&gt;old)&#8230; at least this way ssh to app1.bobo.com always goes to the right place.<\/li>\n<li><strong>Tip #6: Use your DNS to impose a sensible structure<\/strong>, especially if you&#8217;re in an environment with dev, production, etc.\u00a0 Trying to do this is a little like changing the oil in a moving car, so be careful.\u00a0 Originally, and in many places, I see machine names like &#8220;devapp1.bobo.com&#8221;; not the end of the world until you go to promote devapp1.bobo.com-&gt;app1.bobo.com.\u00a0 It tends to get confusing, and it&#8217;s harder to do programatically (using a shell script or chef\/puppet\/ansible).\u00a0 Instead, create subdomains for dev, and friends.\u00a0 So when we move a machine into production it goes from app1.pre.bobo.com -&gt; app1.bobo.com and app1.bobo.com -&gt; app1.old.bobo.com&#8230;\u00a0 and we change the IP addresses accordingly&#8230; so this way if something goes wrong we just move stuff from app1.old.bobo.com back to app1.bobo.com instead of trying to find where that old machine went.<\/li>\n<li><strong>Tip #7: Use a load balancer<\/strong> as the only publicly accessible endpoint for your webservices.\u00a0 The AWS loadbalancers are easy to program, if you want to do it that way, but manually putting a new machine into production is easy &#8211; load the new machine onto the load balancer and take the other one off.\u00a0 Voila, zero downtime.\u00a0 Of course, I&#8217;d point the loadbalancer to be &#8220;app.bobo.com&#8221; using Amazon Route53.\u00a0 This configuration will also allow you to add more servers (app1-&gt;appN) and divide the load seamlessly if the need arises.<\/li>\n<li><strong>Tip #8: Have a meaningful \/etc\/motd<\/strong> and prompt.\u00a0 Just like the old days (actually the old days were more fun, since we had<a href=\"http:\/\/dfrench.hypermart.net\/cgi-bin\/bashFortune\/mkFortune.cgi?FORTFILE=.\/mw1.txt\" target=\"_blank\"> various rude fortune programs running<\/a>), but the idea being, with AWS it&#8217;s entirely possible to have a zillion windows open, and it&#8217;s often hard to tell where you are.\u00a0 <a href=\"http:\/\/asciiset.com\/figletserver.html\" target=\"_blank\">Click here for an ASCII banner generator:<\/a><\/li>\n<\/ul>\n<blockquote>\n<pre><a href=\"http:\/\/asciiset.com\/figletserver.html\">   ###    ########  ########     ##   \r\n  ## ##   ##     ## ##     ##  ####   \r\n ##   ##  ##     ## ##     ##    ##   \r\n##     ## ########  ########     ##   \r\n######### ##        ##           ##   \r\n##     ## ##        ##           ##   \r\n##     ## ##        ##         ######<\/a><\/pre>\n<\/blockquote>\n<ul>\n<li><strong>Tip #9: Backups.<\/strong>\u00a0 You have a backup script, use it.\u00a0 App servers can more or less be regenerated, so that&#8217;s not such a big deal, but in places running MySQL databases?\u00a0 I&#8217;ve heard people say &#8220;we don&#8217;t really need backups, we&#8217;re running in master-slave mode with N slaves, with auto-failover, so what&#8217;s the point?&#8221; &#8211; the point is data corruption and being able to go back to that point in time *before* the new guy wiped out that table\/database\/whatever.\u00a0 So, how do you take a backup of a MySQL database?\u00a0 Simple, bring down MySQL and run the BACKUP script.\u00a0 But bring MySQL down first, because taking a snapshot forces a reboot, and you should never crash a machine if you can avoid it.<\/li>\n<li><strong>Tip #10: Expiration Dates<\/strong>. See Tip #9, Backups?\u00a0 I&#8217;m sort of sloppy; I just clone an entire running instance as my backup, I don&#8217;t just snapshot the volume, and re-attach it, I just grab the whole machine.\u00a0 <a href=\"https:\/\/aws.amazon.com\/ebs\/pricing\/\" target=\"_blank\">Space is pretty cheap on AWS,<\/a> and being able to run SPINUP to get my instance back is priceless.\u00a0 The downside is you&#8217;re going to end up with a lot of AMIs, Volumes, and Snapshots floating around, and cleaning them up is miserable &#8211; especially if backups were being taken sort of &#8220;organically&#8221; &#8211; See Tip #2.\u00a0 I recently had 32 TB of crap to go through&#8230; and that was incentive for this not to happen again.\u00a0 The solution is to use AWS&#8217;s tagging ability to add an Expiration Date to anything you can, i.e.<strong> EXPIRES=20140618.<\/strong>.\u00a0 For example, with backups, I have daily backups that expire in a week, weekly backups are kept for a month, and monthly backups are kept for a year.\u00a0<a href=\"http:\/\/kimosabe.net\/aws\/BACKUP.txt\" target=\"_blank\"> An example of tagging is in the BACKUP script.<\/a>\u00a0 <a href=\"http:\/\/kimosabe.net\/aws\/CLEANUP.txt\" target=\"_blank\">Here&#8217;s a copy of my CLEANUP script<\/a> which goes through AMIs tagged in the BACKUP script and removes anything that has expired.<\/li>\n<li><strong>Tip #11: Right-size your Instances<\/strong>.\u00a0 Similar to the above &#8211; audit your AWS environment to see if you&#8217;re using your AWS resources efficiently.\u00a0 In general, and in my case, I&#8217;ll often spin up an instances larger than I really need &#8220;just in case&#8221;.\u00a0 The problem is AWS prices double between levels, i.e. s large instance costs twice as much as a medium instance which costs twice as much as a small instance.\u00a0 On top of that <a href=\"https:\/\/aws.amazon.com\/about-aws\/whats-new\/2014\/01\/21\/announcing-new-amazon-ec2-m3-instance-sizes-and-lower-prices-for-amazon-s3-and-amazon-ebs\/\" target=\"_blank\">AWS has been bringing out new instances type (hello M3.medium),<\/a> which offer better value for money than the older generation of boxes they&#8217;re replacing.\u00a0 So if it can be run on a smaller instance, do it.\u00a0 In fact, better a couple of smaller instances hanging off a load balancer then a single giant point of failure.<\/li>\n<li><strong>Tip #12: Use<a href=\"https:\/\/aws.amazon.com\/ec2\/purchasing-options\/reserved-instances\/\" target=\"_blank\"> Reserved Instances<\/a>.<\/strong>\u00a0 A lot of places use AWS a little like their old machine room&#8230; spin &#8217;em up and let them run forever.\u00a0 If you&#8217;re going to use AWS that way, then take advantage of the ~40% discount they&#8217;ll give you for telling them you&#8217;ll keep the machine for a year.<\/li>\n<\/ul>\n<p>Between tips 11 and 12 you can potentially drop your AWS bill by half, without any sort of performance hit.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p class=\"excerpt\">I really like working with Amazon&#8217;s AWS.\u00a0 I&#8217;ve been taking care of Unix\/Linux systems for a very long time, and AWS for a few years&#8230; Here are a few tips to make your life with AWS easier&#8230; Giant sites with hundreds of machines can most likely skip this post, or just use it to remember&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"more-link\" href=\"https:\/\/blog.maclawran.ca\/?p=237\">Read more &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":248,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-237","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/posts\/237","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=237"}],"version-history":[{"count":19,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/posts\/237\/revisions"}],"predecessor-version":[{"id":262,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/posts\/237\/revisions\/262"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=\/wp\/v2\/media\/248"}],"wp:attachment":[{"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=237"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=237"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.maclawran.ca\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=237"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}