Saturday, August 10, 2013

Boto and CloudFormation

I've been working on a  small side project using AWS. In this project, I create a slew a resources. In addition to the typical EC2 instances, I also use things like VPC, RDS and S3. In my project, I always keep the same 'infrastructure' of resources, but I may change out the EC2 instances.

So I came to the conclusion that I wanted a 2 phase setup. One that sets up the infrastructure, and another that will set up my EC2 instances. I quickly discovered AWS's CloudFormation stuff, and it looked like it fit the bill. It allows me to quickly bring up an infrastructure, and even parameterize it. I quickly was able to create my "Stack" for my infrastructure.

When I moved onto the creation of the EC2 instances, I tried to get the stack to query the infrastructure for the existing resources. This would allow me to automatically connect to the database and other resources. It turns out that I can't do this. So I needed to find a scripting solution.

I looked briefly at things like Chef and Puppet, but they seemed like overkill for what I needed, and it would be a whole new thing to learn that may or may not have value later on. I decided to try and work with boto, a python interface to various AWS services. It supported talking to CloudFormation, and I know Python fairly well, so it seemed an ideal fit.

Unfortunately, it seems that if you're doing anything outside of creating, deleting or updating stacks via boto, there's a lack of examples. For instance, I needed to query the stacks for things like the outputs and tags. I could not find a single example. So after a couple hours of hacking, here's a script I came up with to query the stacks.

#!/usr/bin/env python



import boto

import boto.cloudformation

import argparse


class MyBaseException(Exception):

    msg ="MyBaseException"

    def __init__(self, value):

        self.value = value


        return "%s: %s" % (self.msg, self.value)


class MissingParamException(MyBaseException):

    msg ="Missing param"


class InvalidCommandException(MyBaseException):

    msg ="Invalid command"


class InvalidStackException(MyBaseException):

    msg ="Invalid stack"



def _create_cf_connection(args):

    # Connect to a cloudformation

    # Returns a cloudformation connection.

    # Throws exception if connect fails

    if not args.access_key:

        raise MissingParamException("access_key")


    if not args.secret_key:

        raise MissingParamException("secret_key")


    if not args.region:

        raise MissingParamException("region")


    conn = boto.cloudformation.connect_to_region(args.region, 

                                                 aws_access_key_id = args.access_key,

                                                 aws_secret_access_key = args.secret_key)


    return conn


def get_stacks(args):

    conn = _create_cf_connection(args)

    return conn.list_stacks()


def get_stack(args, stack):

    conn = _create_cf_connection(args)


    stacks = conn.describe_stacks(stack)

    if not stacks:

        raise InvalidStackException(stack)


    return stacks[0]


def print_stack(stack):

    print "---"

    print "Name:            %s" % stack.stack_name

    print"ID:              %s"% stack.stack_id

    print "Status:          %s" % stack.stack_status

    print "Creation Time:   %s" % stack.creation_time

    print"Outputs:         %s"% stack.outputs

    print "Parameters:      %s" % stack.parameters

    print"Tags:            %s"% stack.tags

    print "Capabilities:    %s" % stack.capabilities



def list_stacks(args):

    stacks = get_stacks(args)

    for stackSumm in stacks:

        print_stack(get_stack(args, stackSumm.stack_id))



def list_regions(args):

    regions = boto.cloudformation.regions()


    for r in regions:




command_list = { 'list-regions' :   list_regions,

                 'list-stacks'  :   list_stacks,




def parseArgs():

    parser = argparse.ArgumentParser()

    parser.add_argument("--region" )

    parser.add_argument("--command" )

    parser.add_argument("--access-key" )

    parser.add_argument("--secret-key" )

    args = parser.parse_args()


    if not args.command:

        raise MissingParamException("command")


    if args.command not in command_list:

        raise InvalidCommandException(args.command)







    except Exception, e:

        print e

1 comment:

  1. Hi there, thank you so much for this and it really help me to get going with the project that I am working on at the moment.

    I have spotted two syntax errors,

    12: def__str__(self): => def __str__(self):
    103: if__name__=='__main__': => if __name__=='__main__':