Hi all!

Good night and happy holiday! Here I would like to share with all of you about something an interesting case study that maybe you’ve done or come across soon.

Imagine the Developer Team in your company is still doing Proof-of-Concept related to a specific project for the user, and the user is strict about the cost. The Dev Team is using EC2 Instance and the maximum budget for EC2 Instance is only $5 per day.

You —as an infrastructure engineer are tasked to create an alert when a budget amount is reached. The alert will send a notification via email, and at the same time the user wants any running EC2 Instance will automatically stop for further investigation.

For this case, we will be using several services to make the goal is reached. The diagram is below:

First, to ensure the team’s cost usage is under control and tracked easily, we are able to use AWS Budget. This service is provided by AWS and will show our maximum budget for all or specific services. We are also able to define the threshold and connect it to Amazon SNS (Simple Notification Service), so the team will be notified. For example, we defined the maximum budget as $2 per day, and the alarm was triggered when it reached 80%. This is really something interesting. We don’t necessarily check it manually, and the team is notified automatically via email (for example).

How about a task to stop the EC2 Instance at the same time? Yup, AWS Lambda Function. We are able to use that service to ensure the EC2 instance is stopped when the alarm is triggered 🙂

Let’s break it down!

First, we have to configure the notification using Amazon SNS. There are two components that we have to set up. Topic and Subscription. Topic is similar to a channel that will send a broadcast message to subscribers we defined in a subscription, for example email and AWS Lambda. Both will receive a notification when the budget amount is reached.

To create a topic, you can follow the full guide here. The thing you must pay attention to is access policy. Add an additional policy so that Amazon SNS will get permission to publish a notification when the budget is reached. Below:

{
  "Sid": "E.g., AWSBudgetsSNSPublishingPermissions",
  "Effect": "Allow",
  "Principal": {
    "Service": "budgets.amazonaws.com"
  },
  "Action": "SNS:Publish",
  "Resource": "your topic ARN",
   "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "<account-id>"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:budgets::<account-id>:*"
        }
      }
}

Below is an example of topic I’ve created:

For the first subscription, follow the guide here and ensure the email is confirmed.

If the Amazon SNS is configured, we move on to the AWS Budget.

From the Management Console, find an AWS Budget menu and just click create button. In this case, I’m using Customize and Cost budget type like the one below.

I’m set for the period is to daily, and the amount is $2.

For the budget scope, because in this case, the Dev team only using EC2 Instances, we can configure to filter EC2 Instances only.

We can define the specific threshold for the alert. Below the example, if the usage is reached 80% of the budgeted amount, it will send the notification.

Who will receive the notification? Put the ARN Amazon SNS that we created before in the form. Ensure it is valid.

Until this step, a team that has the email will get a notification when the budget amount is reached 80%. After this, we have to configure AWS Lambda. We have to create a function that will execute to stop the EC2 Instance when the alert is triggered.

First, create a new policy and role in IAM so the AWS Lambda will have permission to stop EC2 Instance directly. Below is the policy in a JSON format:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "ec2:StopInstances",
                "ec2:DescribeInstanceStatus"
            ],
            "Resource": "*"
        }
    ]
}

The policy I’ve created successfully:

Below is the role I’ve created successfully:

For the function in AWS Lambda, I’m using Python 3.7 runtime and for the execution role, choose the existing role we created before. Here, my existing role name is StopEC2Instance.

Below is the example code to stop EC2 instances. Paste into lambda_function.py editor.

import boto3
def GetInstanceRunning():
client = boto3.client('ec2')
ec2_regions = [region['RegionName'] for region in client.describe_regions()['Regions']]
for region in ec2_regions:
ec2 = boto3.resource('ec2')
instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
return [instance.id for instance in instances]
def StopInstance(ids=GetInstanceRunning()):
ec2 = boto3.client('ec2')
if not ids:
print("No Instance in the state Running or pending")
else:
ec2.stop_instances(InstanceIds=ids)
ec2.get_waiter('instance_stopped').wait(InstanceIds=ids)
print('instance {} was shutdown'.format(ids))
def lambda_handler(event, context):
StopInstance()
view raw stopec2instance.py hosted with ❤ by GitHub

Go to the Test tab and click Test. Wait for a moment, and below is the result. It shows that there is one instance is shutting down.

Go to EC2 Instance, and the instance status is stopped 🙂

But wait! The function is run successfully when we click the Test button. How about ensuring that it will run when the budget is reached?

Just add a new Trigger 🙂

Set a new trigger into Amazon SNS that we created before.

It was added successfully, and whenever the alert rings, the function will automatically run to stop EC2 Instances.

Thank you!

References:

  1. https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-sns-policy.html
  2. https://docs.aws.amazon.com/sns/latest/dg/sns-email-notifications.html
  3. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/ec2-example-managing-instances.html