Jul 7, 2015

AWS Tricks: Updating Route53 DNS for AutoScalingGroup using Lambda

AWS Lambda is a powerful code-on-demand tool allowing you to host code blocks that can be executed on demand with high throughput. One of the not so obvious capabilities is that code executing via Lambda has both access to the AWS API and to permission grants via IAM.
These can be combined to create some powerful infrastructure automation that no longer requires a hosted server to be running. Remember that Lambda is stateless, so you have to implement your code such that yours tasks executes end to end.

One trick I have found useful is to allow Auto Scaling Groups to update a corresponding Route 53 DNS entry during a scaling event. Specifically, I like to use this where I am ensuring an always-running server by creating an ASG that has a minimum and maximum instance count of 1. Traditionally, this could have been accomplished by having the instance utilizing an Elastic IP or have the instance itself call into the Route53 API to update the record. Utilizing Lamba, we can off load that code from the instance.

We are using the following AWS capabilities:
1. Auto Scaling Group
2. Auto Scaling Group Event Notifications – generates events whenever the ASG encounters a scaling event.
3. Simple Notification Service (SNS) – channel for publishing and subscribing to ASG Event Notifications
4. Lambda – code execution driven by SNS messages

Step 1

Create a new SNS topic to publish autoscaling events. From the AWS Console, select Services, SNS, Topics, then Create Topic. Enter the topic name and description then select Create Topic.

2015-06-30 at 7.47 AM

Step 2

Configure your autoscaling group to publish event notifications. This can be done through the AWS console by selecting the Notifications tab on the AutoScaling page. Configure the autoscaling group to publish events for Launch and Terminate. This allows us to handle both scale up and scale down events. Configuring the events to be sent to the SNS topic created in step 1.

2015-06-30 at 7.50 AM

Step 3

Create a new Lambda function to process the ASG events. The code block below is a good starting point.


var AWS = require('aws-sdk');
var async = require('async');

exports.handler = function (event, context) {
  var asg_msg = JSON.parse(event.Records[0].Sns.Message);
  var asg_name = asg_msg.AutoScalingGroupName;
  var instance_id = asg_msg.EC2InstanceId;
  var asg_event = asg_msg.Event;

  console.log(asg_event);
  if (asg_event === "autoscaling:EC2_INSTANCE_LAUNCH" || asg_event === "autoscaling:EC2_INSTANCE_TERMINATE") {
    console.log("Handling Launch/Terminate Event for " + asg_name);
    var autoscaling = new AWS.AutoScaling({region: 'us-east-1'});
    var ec2 = new AWS.EC2({region: 'us-east-1'});
    var route53 = new AWS.Route53();

    async.waterfall([
      function describeTags(next) {
        console.log("Describing ASG Tags");
        autoscaling.describeTags({
          Filters: [
            {
              Name: "auto-scaling-group",
              Values: [
                asg_name
              ]
            },
            {
              Name: "key",
              Values: ['DomainMeta']
            }
          ],
          MaxRecords: 1
        }, next);
      },
      function processTags(response, next) {
        console.log("Processing ASG Tags");
        console.log(response.Tags);
        if (response.Tags.length == 0) {
          next("ASG: " + asg_name + " does not define Route53 DomainMeta tag.");
        }
        var tokens = response.Tags[0].Value.split(':');
        var route53Tags = {
          HostedZoneId: tokens[0],
          RecordName: tokens[1]
        };
        console.log(route53Tags);
        next(null, route53Tags);
      },
      function retrieveASGInstances(route53Tags, next) {
        console.log("Retrieving Instances in ASG");
        autoscaling.describeAutoScalingGroups({
          AutoScalingGroupNames: [asg_name],
          MaxRecords: 1
        }, function(err, data) {
          next(err, route53Tags, data);
        });
      },
      function retrieveInstanceIds(route53Tags, asgResponse, next) {
        console.log(asgResponse.AutoScalingGroups[0]);
        var instance_ids = asgResponse.AutoScalingGroups[0].Instances.map(function(instance) {
          return instance.InstanceId
        });
        ec2.describeInstances({
          DryRun: false,
          InstanceIds: instance_ids
        }, function(err, data) {
          next(err, route53Tags, data);
        });
      },
      function updateDNS(route53Tags, ec2Response, next) {
        console.log(ec2Response.Reservations);
        var resource_records = ec2Response.Reservations.map(function(reservation) {
          return {
            Value: reservation.Instances[0].NetworkInterfaces[0].Association.PublicIp
          };
        });
        console.log(resource_records);
        route53.changeResourceRecordSets({
          ChangeBatch: {
            Changes: [
              {
                Action: 'UPSERT',
                ResourceRecordSet: {
                  Name: route53Tags.RecordName,
                  Type: 'A',
                  TTL: 10,
                  ResourceRecords: resource_records
                }
              }
            ]
          },
          HostedZoneId: route53Tags.HostedZoneId
        }, next);
      }
    ], function (err) {
      if (err) {
        console.error('Failed to process DNS updates for ASG event: ', err);
      } else {
        console.log("Successfully processed DNS updates for ASG event.");
      }
      context.done(err);
    })
  } else {
    console.log("Unsupported ASG event: " + asg_name, asg_event);
    context.done("Unsupported ASG event: " + asg_name, asg_event);
  }
};

To create a Lambda function follow the instructions here. You will need to create a new IAM role for the Lambda function. This IAM role provides the permissions that Lambda has on your account when executing. For our capability we need the following specific policies: AmazonEC2ReadOnlyAccess and AmazonRoute53FullAccess.
Note, you could restrict the policies further, but the above will get you started.

Step 3

Configure your Lambda function to be trigged by publications to the SNS topic by adding an event source for the function.

2015-06-30 at 8.00 AM
2015-06-30 at 8.01 AM

Step 4

The above Lambda function utilizes metadata that we’ve placed on the autoscaling group to indicate which Route 53 DNS record to update when a scaling event occurs. There are 2 pieces of information we need to know. The first is the HostedZoneId in Route 53 and the second is the specific record (i.e. www.example.com). The Lambda function we’ve defined expects these 2 pieces of information to be stored in a Tag called DomainMeta.

The format is for the tag is HostedZoneId:RecordName. For example, ZFBW5S4JKK3LA:www.example.com.

Update the autoscaling group to have a DomainMeta tag with the values for your DNS.

Step 5

Test out the your configuration by either increasing or decreasing the size of the ASG. You can then check the Lambda dashboard to see that your function was executed (additionally, you can view logs for Lambda functions in CloudWatch) and then verify that your DNS is updated.

About the Author

Object Partners profile.

One thought on “AWS Tricks: Updating Route53 DNS for AutoScalingGroup using Lambda

  1. Anonymous Coward says:

    Trying to get to square one with this but can’t work out what the module-name.export value should be for the function as required under ‘Handler’ in the configuration.

    Unable to import module ‘asg_dns_updater’: Error at Function.Module._resolveFilename (module.js:338:15) at Function.Module._load (module.js:280:25) at Module.require (module.js:364:17) at require (module.js:380:17) at Object. (/var/task/asg_dns_updater.js:2:13) at Module._compile (module.js:456:26) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.require (module.js:364:17)

  2. Robert says:

    The referenced Node.js script did not work for me. I used it as a base to make the same thing in python:

    from __future__ import print_function
    import json
    import logging
    import boto3

    logger = logging.getLogger()
    logger.setLevel(logging.INFO)

    def lambda_handler(event, context):
    autoscaling = boto3.client(‘autoscaling’)
    ec2 = boto3.client(‘ec2’)
    route53 = boto3.client(‘route53’)

    logger.info(json.dumps(event))
    message = json.loads(event[‘Records’][0][‘Sns’][‘Message’])
    asg_name = message[‘AutoScalingGroupName’]
    asg_event = message[‘Event’]

    if asg_event == “autoscaling:EC2_INSTANCE_LAUNCH” or asg_event == “autoscaling:EC2_INSTANCE_TERMINATE”:
    logger.info(“Getting Tags”)
    as_response = autoscaling.describe_tags(
    Filters=[
    {
    ‘Name’: ‘auto-scaling-group’,
    ‘Values’: [asg_name],
    },
    {
    ‘Name’: ‘key’,
    ‘Values’: [‘DomainMeta’],
    }
    ],
    MaxRecords=1
    )
    logger.info(“Processing ASG Tags”)
    if len(as_response[‘Tags’]) is 0:
    logger.error(“ASG: {} does not define Route53 DomainMeta tag”.format(asg_name))
    else:
    tokens = as_response[‘Tags’][0][‘Value’].split(‘:’)
    route53tags = {
    ‘HostedZoneId’: tokens[0].split(‘/’)[2],
    ‘RecordName’: tokens[1]
    }
    logger.info(“Found tags:”)
    logger.info(json.dumps(route53tags))
    logger.info(“Retrieving Instances in ASG”)
    as_response = autoscaling.describe_auto_scaling_groups(
    AutoScalingGroupNames=[asg_name],
    MaxRecords=1
    )
    instanceIds = []
    for instance in as_response[‘AutoScalingGroups’][0][‘Instances’]:
    instanceIds.append(instance[‘InstanceId’])
    ec2_response = ec2.describe_instances(InstanceIds=instanceIds)
    resourceRecords = []
    for reservation in ec2_response[‘Reservations’]:
    resourceRecords.append({‘Value’: reservation[‘Instances’][0][‘NetworkInterfaces’][0][‘PrivateIpAddress’]})
    logger.info(“Found Resources:”)
    logger.info(json.dumps(resourceRecords))
    logger.info(“Getting zone name”)
    r53_response = route53.get_hosted_zone(Id=route53tags[‘HostedZoneId’])
    zoneName = r53_response[‘HostedZone’][‘Name’]
    logger.info(“Found zone name: {}”.format(zoneName))
    logger.info(“Updating Route53 Record”)
    r53_response = route53.change_resource_record_sets(
    HostedZoneId=route53tags[‘HostedZoneId’],
    ChangeBatch={
    ‘Changes’: [
    {
    ‘Action’: ‘UPSERT’,
    ‘ResourceRecordSet’: {
    ‘Name’: “{name}.{domain}”.format(name=route53tags[‘RecordName’], domain=zoneName),
    ‘Type’: ‘A’,
    ‘TTL’: 10,
    ‘ResourceRecords’: resourceRecords
    }
    }
    ]
    }
    )

    1. Howard says:

      I’m looking to do something very similar. Any chance of sticking your Python script on Github or somewhere? The indentation and quotes are all lost in the commenting system!

  3. Victor says:

    Code contained 2-3 bugs ( at least with last versions of AWS APIs ):
    * If there were no auto-scaling groups with DomainMeta info, the code was not being stopped and crashed.
    * PublicIp is now retrieved from reservation.Instances[0].PublicIpAddress and not NetworkInterfaces…

    var AWS = require(‘aws-sdk’);
    var async = require(‘async’);

    exports.handler = function (event, context) {
    var asg_msg = JSON.parse(event.Records[0].Sns.Message);
    var asg_name = asg_msg.AutoScalingGroupName;
    var instance_id = asg_msg.EC2InstanceId;
    var asg_event = asg_msg.Event;

    console.log(asg_event);
    if (asg_event === “autoscaling:EC2_INSTANCE_LAUNCH” || asg_event === “autoscaling:EC2_INSTANCE_TERMINATE”) {
    console.log(“Handling Launch/Terminate Event for ” + asg_name);
    var autoscaling = new AWS.AutoScaling({region: ‘us-east-1’});
    var ec2 = new AWS.EC2({region: ‘us-east-1’});
    var route53 = new AWS.Route53();

    async.waterfall([
    function describeTags(next) {
    console.log(“Describing ASG Tags”);
    autoscaling.describeTags({
    Filters: [
    {
    Name: “auto-scaling-group”,
    Values: [
    asg_name
    ]
    },
    {
    Name: “key”,
    Values: [‘DomainMeta’]
    }
    ],
    MaxRecords: 1
    }, next);
    },
    function processTags(response, next) {
    console.log(“Processing ASG Tags”);
    console.log(response.Tags);
    if (response.Tags.length === 0) {
    return next(“ASG: ” + asg_name + ” does not define Route53 DomainMeta tag.”);
    }
    var tokens = response.Tags[0].Value.split(‘:’);
    var route53Tags = {
    HostedZoneId: tokens[0],
    RecordName: tokens[1]
    };
    console.log(route53Tags);
    next(null, route53Tags);
    },
    function retrieveASGInstances(route53Tags, next) {
    console.log(“Retrieving Instances in ASG”);
    autoscaling.describeAutoScalingGroups({
    AutoScalingGroupNames: [asg_name],
    MaxRecords: 1
    }, function(err, data) {
    next(err, route53Tags, data);
    });
    },
    function retrieveInstanceIds(route53Tags, asgResponse, next) {
    console.log(asgResponse.AutoScalingGroups[0]);
    var instance_ids = asgResponse.AutoScalingGroups[0].Instances.map(function(instance) {
    return instance.InstanceId;
    });
    ec2.describeInstances({
    DryRun: false,
    InstanceIds: instance_ids
    }, function(err, data) {
    next(err, route53Tags, data);
    });
    },
    function updateDNS(route53Tags, ec2Response, next) {
    console.log(ec2Response.Reservations);
    var resource_records = ec2Response.Reservations.map(function(reservation) {
    return {
    Value: reservation.Instances[0].PublicIpAddress
    };
    });
    console.log(resource_records);
    route53.changeResourceRecordSets({
    ChangeBatch: {
    Changes: [
    {
    Action: ‘UPSERT’,
    ResourceRecordSet: {
    Name: route53Tags.RecordName,
    Type: ‘A’,
    TTL: 10,
    ResourceRecords: resource_records
    }
    }
    ]
    },
    HostedZoneId: route53Tags.HostedZoneId
    }, next);
    }
    ], function (err) {
    if (err) {
    console.error(‘Failed to process DNS updates for ASG event: ‘, err);
    } else {
    console.log(“Successfully processed DNS updates for ASG event.”);
    }
    context.done(err);
    });
    } else {
    console.log(“Unsupported ASG event: ” + asg_name, asg_event);
    context.done(“Unsupported ASG event: ” + asg_name, asg_event);
    }
    };

  4. Turbo Fredriksson says:

    I have put this up at https://github.com/FransUrbo/Lambda-AWS-AutoScalingGroups-Route53 (keeping the changes in this blog post) with my own changes and fixes.

    Please feel free to contribute with code, comments or bug reports.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Android Development for iOS Developers
Android development has greatly improved since the early days. Maybe you tried it out when Android development was done in Eclipse, emulators were slow and buggy, and Java was the required language. Things have changed […]
Add a custom object to your Liquibase diff
Adding a custom object to your liquibase diff is a pretty simple two step process. Create an implementation of DatabaseObject Create an implementation of SnapshotGenerator In my case I wanted to add tracking of Stored […]
Keeping Secrets Out of Terraform State
There are many instances where you will want to create resources via Terraform with secrets that you just don’t want anyone to see. These could be IAM credentials, certificates, RDS DB credentials, etc. One problem […]
Validating Terraform Plans using Open Policy Agent
When developing infrastructure as code using terraform, it can be difficult to test and validate changes without executing the code against a real environment. The feedback loop between writing a line of code and understanding […]