Automation tips on Shutting down EC2 and ELB instances

This post is in continuation of my previous post, in which I discussed automation techniques to start EC2 instances along with ELB etc.

Scenario

We have been able to setup an EC2 machine , which only receives request from a ELB. This is done by making a security group.

Next, we created a Launching configuration and an AutoScaling group.

What do we want now?

Now I will try to delete the instance programatically. Here is the code -

def delete_this_instance(*args):
   
    conn_reg = CONN_REG(region_name)
    conn_as = CONN_AS(region_name)
    conn_elb = CONN_ELB(region_name)

    # delete auto scaling group
    asg = conn_as.get_all_groups(names=[auto_scaling_group])[0]
    asg.shutdown_instances()
    while asg.instances:
        time.sleep(2)
        asg = conn_as.get_all_groups(names=[auto_scaling_group])[0]

    # check all activities of asg
    loop_var = True
    while loop_var:
        all_activities = conn_as.get_all_activities(asg)
        print 'all_activities-', all_activities, 'inside'
        loop_var = False
        for activity in all_activities:
            if activity.progress != '100':
                print 'activity id-%s, progress-%s'%(activity.activity_id, activity.progress)
                loop_var = True
        time.sleep(5)
    asg.delete()

    # delete load balancer
    lb = conn_elb.get_all_load_balancers([loadbalancer_name])[0]
    lb.delete()

    # delete launch config
    lc = conn_as.get_all_launch_configurations(names=[launch_configuration])[0]
    lc.delete()

    # wait for network interfaces to die
    sg = conn_reg.get_all_security_groups(groupnames=[security_group])[0]
    ni = conn_reg.get_all_network_interfaces(filters={'group-id':sg.id})
    while ni:
        time.sleep(2)
        ni = conn_reg.get_all_network_interfaces(filters={'group-id':sg.id})

    # delete ec2 sg first
    sg1 = conn_reg.get_all_security_groups(groupnames=['%s1'%security_group])[0]
    sg1.delete()

    # delete security group for elb
    sg.delete()

Important thing to note here is the sequence of deleting the instance. We deleted in the following sequence -

Shutdown EC2 Instances
Delete AutoScaling Group
Then the launch configuration
Then all network interfaces
Atlast delete the Security Groups

Difficulties

Deleting the instances programitically is not as straightforward as it looks.

There is a dependency of one type of instance on other. So you cant delete ELB until ASG groups have been deleted.
Also, each type of instance takes its own time, so we have to recursively check and then proceed.

For deleting AutoScaling Groups i.e EC2 instances-

Recursively check for groups after 2 secs.
Once the groups are down, check for any other pending activites. Recursively wait for this as well. This is the slowest of all steps.

Deleting ELB and Launch Configs is easy
Before Deleting all security groups, we will have to wait for all network interfaces to die.
Done!!.