So for the past three weeks while studying for the AWS Devops Professional certification, I've been punch-drunk with setting up automated pipelines. I've already automated my container image creation process now the next piece is to automate the AMI instance creation!
Why do I need to automate this when I'm using ECS? Well the main reason is because AWS updates the ECS agent frequently and one of the best ways to update the agent is by terminating the instance and having it recreated by the auto-scaling group. This would be ideal, however the default ECS Amazon Machine Image (AMI) is missing some stuff that I require, for example node-exporter for Prometheus to consume metrics, ssm-agent for automation and EFS mount.
The solution is to bake my very own ECS AMI that contains all the dependencies I need and to create it routinely. I will bake the new AMI using Hashicorp's Packer tool within CodeBuild! Packer will create an AMI based on the default ECS AMI download and configure the dependencies I need. CodePipeline will provide the "glue" or the automation aspect.
I will be basing my pipeline on this AWS blog post. I'll make edits to the code which will be used in both Lambda functions.
The new feature in this pipeline will incorporate Slack in the approval stage or as the "cool kids" like to call it – ChatOps! Instead of sending an approval email the message will be relayed to Slack via Lambda and the response will be captured by AWS API Gateway and invoke a second Lambda function that will send the approval to the pipeline.
The flow will go as follows:
- CloudWatch will be configured with a cron to trigger once a week.
- CodePipeline will be triggered by CloudWatch to execute the build.
- The code from CodeCommit will be collected and sent to the next stage.
- Before the code can be built in the CodeBuild stage, it needs to go through an approval stage. In this stage a specific group or individual can approve or reject a build from continuing. This acts as a failsafe to ensure only desired builds are pushed out into production.
- The approval stage triggers the first lambda function that will deliver a message to a specified Slack channel.
- Slack will send a message using a POST method to a specified API gateway endpoint.
- The API gateway will accept the message from Slack and invoke a second lambda function.
- The second Lambda function will send the approval response to CodePipeline which will signal the CodeBuild stage to build the AMI.
- The last and final stage is CodeBuild which will utilize Packer and build a new AMI with all of my specifications.
Before creating the pipeline make sure you have an ECR repository configured with all of your code loaded into it. Below are the four files needed by Packer: a shell script for Packer to execute, CodeBuild, the json file telling Packer what to do, and node-exporter service.
Script for packer to execute:
Building the Pipeline
Go ahead and create a new pipeline. I'll name my pipeline
pafable-ecs-ami and select the default options for it.
In the source stage, select AWS CodeCommit as the source provider. Select the repository you created earlier and make sure to select master for the branch name.
In the build stage select CodeBuild as the provider and the region you wish to deploy to. Next click on the "Create project" button, I'll call my project
In the environment section click on "Managed image" and select
Ubuntu for the operating system. For the time being, Amazon Linux cannot run Packer. You can leave the rest as default.
On the buildspec section choose "Use a buildspec file".
Along with the settings mentioned above, I'll be utilizing environment variables. However these are going to be sensitive credentials such as AWS access and secret keys.
I'll be storing the access and secret keys in Parameter Store as secure strings. My access and secret keys will be called
packer_secret_key respectively as they will be used by Packer only.
I'll give these environment variables the name
AWS_ACCESS. Next I will enter the ARN of the credentials in the value box and for type I'll select "Parameter". Now when I call
AWS_ACCESS within the buildspec file it will be replaced by the credentials in Parameter Store. This ensures my secrets are kept secure and not written in my code!
You can skip the CodeDeploy stage and finish your pipeline, I won't be using it in my pipeline.
Your pipeline should look something like below:
Creating the First Lambda Function
The first Lambda function will be triggered by an SNS topic (still needs to be created) that will send a message to my Slack channel. As mentioned in the beginning, the python code for Lambda can be found here.
I made some modifications to the code and I'll walk you through it now. Let's start with the libraries needed for this; the libraries are
b64decode. The next section are the environment variables, I tweaked this section slightly so that it will pull the Slack webhook and channel from Parameter Store.
logger the logging is set to
INFO. After the logging section comes the intersting bits! The first function (
lambda_handler) is rquired by Lambda, it will take an input sent from SNS in JSON format. I'll then parse through the JSON and look for the
Subject. Hopefully your dictonary parsing skills are on point!
Next it will parse and grab the
codepipeline_name. Quickly following that is the Slack mesage. This will be sent to my specified channel on Slack, it will ask for an input and the user may click on either "Deploy" or "Reject". Finally the message is packaged up into a request
req and sent off into the "wild blue yonder" of Slack land.
Before I close and save this Lambda code, I need to configure the environment variables and then make sure the entries for the Slack webhook and channel are in Parameter Store.
At the bottom of Lambda, I'll configure the environment variables like so:
In Parameter Store, I'll create two securestring entries with the names
Navigate to the SNS service and create a new topic. In the new topic create a new subscription. Select
AWS Lambda as the Protocol and select your first Lambda function as the endpoint.
Quickly hop back into your Lambda function and make sure the trigger is set to your newly created SNS topic.
Creating the Second Lambda Function
This second Lambda function will be triggered by API Gateway (will be created in the next section). Like before I'll be importing
boto3 libraries with the addition of
Just like before I'll be utilizing Parameter Store to supply the value for a secret. In this case that secret is
SLACK_VERIFICATION_TOKEN. In the
parse_qs will convert application/x-www-form-urlencoded into a dictionary which will then be parsed in json.
After parsing the data, if the
SLACK_VERIFICATION_TOKEN matches the token within the parsed data (payload) it willl call the
send_slack_message method and return a status code of 200 and a message will be seen in Slack saying "Aye understood, captain!".
send_slack_message will use boto3 to interact with my codepipeline build and signal it that I approved it and it can continue with the CodeBuild stage!
Just like in the first function, I'll create the environment variable. This second function will only need one.
NOTE: If you're troubleshooting why your Lambda function keeps on failing, take a look at CloudWatch Logs. There should be a log group for your function. Be very careful of printing credentials in your functions because they will appear in the logs!
Configuring the API Gateway
Time to configure something completely new to me – API Gateway. This is required because there is no way to invoke the second Lambda function without API Gateway. In AWS you can't directly invoke a Lambda function without another service acting as the trigger.
Open the API Gateway service console and click on the "Create API" button. In the following page that appears select the first option for "REST API". Note the descriptions, the other REST API option is restricted to the VPC only – you do not want to select this!
REST as the protocol, create a
New API, you can give your API a name and description, and lastly select
Regional for the endpoint type.
Click on the "Actions" button and create a new resource. Give the resource a name and make sure to check off the box for
Enable API Gateway CORS.
Once your resource has been created, click on the "Actions" button again and create a method. In the drop down option that appears, select
Integration type should be
Lambda Function and for the "Lambda Function" select the name of the second lambda function. Mine is called
approved_by_slack. You can leave the rest of the option default.
After creating the method, you should have a diagram like below. It will show you what will happen an what services will be invoked when a POST request is received. As you can see in mine, when a request is received my second Lambda function is invoked and a response is returned to the client.
Now you can deploy the API. Click on the "Actions" button once more and select "Deploy API". Select
[New Stage] for deployment stage and the rest can be what ever you want.
When the API is deployed you will receive a URL that can be invoked.
Creating the Slackbot
Now that the heavy lifting in AWS is out of the way, let's configure Slack. First you will need to make sure you're logged into Slack via web ui. Next navigate over to Slack's API page and go to your apps.
Click on the green button to create a new Slack App. Give it a name and select your Slack Workspace.
In the basic information page select "Incoming Webhooks".
Navigate down to the "Incoming Webhooks" section and confirm that it's
on. Also on the same page, click on the
Add New Webhook to Workspace button. This will give you the
SLACK_WEBHOOK_URL that you will need to upload to Parameter Store.
Head on over to the "Interactive Components" section and remember the invoke URL from API Gateway? You can copy and paste it into the "Request URL" box. Also make sure this is set to
Once you have that in place, go back into the "Basic Information" and copy the "Signing Secret". Make sure NOT to copy the deprecated "Verification Token"! I made references in my code to use the verification token, but according to Slack using it is no longer secure and best practice.
The sigining secret functions the same way as the verification token so there is no code change needed.
Time to Test!
Now it's time for my favorite part executing the pipeline and watching everything
burn work on the first try. Go into CodePipeline and select your AMI pipeline, next click on the "Release change" button to kick off the build.
Wait a few seconds and you should see a message pop up in your Slack channel. It will ask you if you want to Deploy your new AMI.
If you click on the "Deploy" button a pop-up window will appear confirming your deploy... in case you accidentally clicked on the deploy button for any reason.
After approving the deploy the text box will change to the confirmation message. This prevents other users from confusing previous unapproved messages.
Success! What a huge relief, it worked flawlessly. The build alerted me of an approval and all I had to do was click to deploy.
Now it's time to fully automate this pipeline. I don't want to start this build by clicking on the "Release change" button or by pushing an update to the git repository in CodeCommit. I simply want it to run once a week and start the process.
Open up CloudWatch and navigate to
Rules and select the rule for the pipeline.
In the next page click on the "Actions" button on the top right and select edit. In the next page change the event source from "Event Pattern" to "Schedule". In the cron expression box add
0 0 ? * 7 *. This expression will trigger the pipeline every Saturday at 12AM GMT or 7PM EST.
That is it! Now the pipeline can initiate on it's own and check with me if I want it to proceed with the deploy or not. Those of you who are lazy like me and do not want to log into the AWS console to approve builds, this is a great alternative to just receiving emails from SNS. There are so many applications you can do with this.
You can even take it a step further from here and literally deploy to ECS or even create new stacks with CloudFormation. This way your entire AWS estate stays fresh.
However before I let you all go off and wreak havoc on your Slack channels, I need to stress the importance of this again, DO NOT print out your environment variables if it is a secret in your Lambda functions! If you did, please delete your logs in CloudWatch immediately and change the secret in Parameter Store.