Category Archives: cloudformation

Obtain the name of the elastic beanstalk autoscaling group

Posted on by , Modified on January 15, 2019
Reply

It is possible to export the name of the autoscaling group which elastic beanstalk provisions as part of each elastic beanstalk environment. This exported value can then be used from other cloud formation stacks in order to add custom scaling triggers, etc. We can export values by adding an outputs section into a .ebextensions file.

If you have large numbers of stacks it is very helpful if one can use a consistent naming scheme for the exported values. There is a way to pass a namespace into elastic beanstalk by using a custom option.

Here is the CloudFormation yaml to be included in an elastic beanstalk extension file:

Parameters:
  BeanstalkASGName:
    Type: String
    Description: "The name to export the autoscaling group name under"
    Default:
      Fn::GetOptionSetting:
        OptionName: MyBeanstalkStackInfoName
        DefaultValue: unknown
Outputs:
  OutputAutoScalingGroupName:
    Description: Beanstalk AutoScalingGroup Name
    Value:
      Ref: "AWSEBAutoScalingGroup"
    Export:
      Name:
        Fn::Join:
          - "-"
          - - { "Ref" : "BeanstalkASGName" }
            - "AutoScalingGroup"

Note that Fn::GetOptionSetting does not seem to be allowed directly in the Outputs section. So we have to use it instead to set the value of a parameter in a Parameters section, and then use the value indirectly via the parameter.

Here is a snippet to use in your master CloudFormation file:

Namespace: aws:elasticbeanstalk:customoption
OptionName: MyBeanstalkStackInfoName
Value: anyvalue

Cloudformation example of an S3 bucket with attached SQS notifications

Posted on by , Modified on November 9, 2018
Reply

Creating an s3 bucket with an SQS queue attached is a simple and powerful configuration. Cloudformation allows one to express such a configuration as code and commit it to a git repository. I was not able to find a complete example of how to express such a configuration using Cloudformation. What follows is written using the Troposhere library. Please do not take this post to be an endorsement of using Troposhere.

t = self.template
 
# The queue which will handle the S3 event messages
t.add_resource(Queue(
    "MyQueue",
    VisibilityTimeout=30,
    MessageRetentionPeriod=60,
    QueueName=Sub("my-${AWS::Region}-${AWS::AccountId}")
))
 
# The bucket that will generate the s3 events. The NotificationConfiguration
# also supports SNS and Lambda. Notifications can also be filtered according
# the S3 key of the object to which the event relates.
t.add_resource(Bucket(
    "MyBucket",
    BucketName=Sub("my-${AWS::Region}-${AWS::AccountId}"),
    # Note that the queue policy must be created first
    DependsOn="MyQueuePolicy",
    NotificationConfiguration=NotificationConfiguration(
        QueueConfigurations=[
            QueueConfigurations(
                Event="s3:ObjectCreated:*",
                Queue=GetAtt("MyQueue", "Arn"),
            )
        ]
    )
))
 
# The queue policy will give access to the S3 bucket to send on the queue
# The queue policy can also be used to give permission to the message receiver
t.add_resource(QueuePolicy(
    "MyQueuePolicy",
    Queues=[Ref("MyQueue")],
    PolicyDocument={
        "Version": "2012-10-17",
        "Statement": [
            # Allow the S3 bucket to publish to the queue
            # https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#grant
            # -destinations-permissions-to-s3
            {
                "Effect": "Allow",
                "Principal": Principal("Service", ["s3.amazonaws.com"]),
                "Action": [
                    "SQS:SendMessage"
                ],
                "Resource": GetAtt("MyQueue", "Arn"),
                "Condition": {
                    "ArnLike": {
                        # have to construct the ARN from the static bucket name to avoid
                        # the circular dependency
                        # https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/
                        "aws:SourceArn": Join("", [
                            "arn:aws:s3:::",
                            Sub("my-${AWS::Region}-${AWS::AccountId}")
                        ])
                    }
                }
            },
            # Allow some user to read from the queue. This is just and example,
            # please change this to match the permissions your use case requires.
            {
                "Effect": "Allow",
                "Principal": AWSPrincipal(GetAtt("MyUser", "Arn")),
                "Action": [
                    "sqs:ReceiveMessage"
                ],
                "Resource": GetAtt("MyQueue", "Arn"),
            }
        ]
    }
))
 
# Allow some user to manipulate the S3 bucket. This is just and example,
# please change this to match the permissions your use case requires.
t.add_resource(BucketPolicy(
    "MyBucketPolicy",
    Bucket=Ref("MyBucket"),
    PolicyDocument={
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": AWSPrincipal(GetAtt("MyUser", "Arn")),
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject"
                ],
                "Resource": Join("", [GetAtt("MyBucket", "Arn"), "/*"])
            }
        ]
    }
))

Connecting SNS to a lambda function using CloudFormation

Posted on by , Modified on March 13, 2017
Reply

We are using Amazon CloudFormation to configure our infrastructure as code. We are doing video processing using a Lambda function triggered by a message in an SNS queue. The documentation on how to do this in CloudFormation is fairly poor. In this article I will show some troposphere code that shows how to do this.

First create the lambda function. This one just logs its invocation event into the cloudwatch logs:

def create_lambda(self):
    t = self.template
 
    code = [
        "exports.handler = function(event, context) {" +
        "    console.log(\"event: \", JSON.stringify(event, null, 4));" +
        "    context.succeed(\"success\");" +
        "}"
    ]
 
    return t.add_resource(Function(
        "LambdaFunction",
        Code=Code(
            ZipFile=Join("", code)
        ),
        Handler="index.handler",
        Role=GetAtt("LambdaExecutionRole", "Arn"),
        Runtime="nodejs4.3",
    ))

Creating the SNS topic is straightforward:

def subscribe_lambda_to_topic(self, topic, function):
    topic.Subscription = [Subscription(
        Protocol="lambda",
        Endpoint=GetAtt(function, "Arn")
    )]

The most complicated, and least well documented, part of the configuration is to give the relevant authorisations. The lambda function itself will require authorisation to use any resources it needs. In this example it is given authority to log to cloudwatch. It is also necessary to set a lambda permission that allows SNS to invoke the lambda in response to a message on the topic. Note that this permission is not a normal IAM role and policy, but something specific to lambda.

def give_permission_to_lambda(self, topic, function):
    t = self.template
 
    # This role gives the lambda the permissions it needs during execution
    lambda_execution_role = t.add_resource(Role(
        "LambdaExecutionRole",
        Path="/",
        AssumeRolePolicyDocument={"Version": "2012-10-17", "Statement": [
            {
                "Action": ["sts:AssumeRole"],
                "Effect": "Allow",
                "Principal": {
                    "Service": [
                        "lambda.amazonaws.com",
                    ]
                }
            }
        ]},
    ))
 
    lambda_execution_policy = t.add_resource(PolicyType(
        "LambdaExecutionPolicy",
        PolicyName="LambdaExecutionPolicy",
        PolicyDocument={
            "Version": "2012-10-17", "Statement": [
                {"Resource": "arn:aws:logs:*:*:*",
                 "Action": ["logs:*"],
                 "Effect": "Allow",
                 "Sid": "logaccess"}]},
        Roles=[Ref(lambda_execution_role)]
    ))
 
    t.add_resource(Permission(
        "InvokeLambdaPermission",
        FunctionName=GetAtt(function, "Arn"),
        Action="lambda:InvokeFunction",
        SourceArn=Ref(topic),
        Principal="sns.amazonaws.com"
    ))