-->

devops ,  aws codepipeline, git, aws_developer_tools_services

Introduction

This post is part of my mini-series about AWS Developer Tools Services

AWS CodePipeline

In the previous posts I was able to create a Git repository in CodeCommit, create a build and upload the artifacts to an S3 bucket using CodeBuild and deploy the artifacts to an EC2 instance. In this post I will use CodePipeline service to tie these steps together.

AWS CodePipeline is more like an umbrella service that it combines the the source, build and deploy steps together with some added benefits:

  • Ability to retry deployment: In CodeDeploy, if the deployment fails you have to create a new deployment manually but with Pipeline we have the option to retry the deployment which clones the failed deployment and runs the new one automatically.
  • Trigger build upon code changes: You can start builds manually or on schedule by creating triggers but you can not trigger a build when you push your code to the repository. CodePipeline handles this automatically.

Create a pipeline

Step 1: Give it a name

First step is to simply give the pipeline a name. CodePipeline can automatically create an IAM role with the required permissions. Required permissions are not really clear at this point as we haven’t connected any of the services yet. That’s why it’s very useful to keep the “Allow AWS CodePipeline to create a service role so it can be used with this new pipeline” checkbox ticked.

Step 2: Add source

Next, we choose the source provider. In this example, I will use CodeCommit. It also support Amazon ECR, Amazon S3 and GitHub.

At this stage, by using CloudWatch events we can automatically start the build. This option os provided because CodeCommit is an AWS service. If you choose GitHub as your provider you’ll see GitHub webhooks which do the same job.

Step 3: Add build

By default two build services are supported: CodeBuild and Jenkins. It’s easier to connect to CodeBuild as you just have to select your build project from the dropdown list:

You can also choose to skip this stage. If you are copying static assets from an S3 bucket for instance you wouldn’t need to build anything and would choose to skip this stage.

Step 4: Add deployment

At deployment stage, we have quite a few options:

As stated before, I will use the CodeDeploy deployment I created before:

Finally we review and create the pipeline.

Troubleshooting the pipeline

Even though my CodeBuild and CodeDeploy projects succeeded, running the CodePipeline at this point fails. The reason for this error is CodePipeline creates its own S3 bucket and CodeDeploy agent needs access to that bucket as well to be able to download the deployment artifacts. So I had to modify my policy as such:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::replace-with-your-s3-bucket-name/*",
        "arn:aws:s3:::aws-codedeploy-us-east-1/*",
        "arn:aws:s3:::codepipeline-us-east-1-XXXXXXXXXXX/*"
      ]
    }
  ]
}

See it in action

After fixing the policy I retried the deployment (which creates a new deployment, doesn’t actually retry the failed one)

Conclusion

In this mini-series, I investigated creating a full pipeline using AWS developer services. In future posts I will delve deeper into these services and start using other complementary services as well.

Resources

aws ,  devops codedeploy, git, aws_developer_tools_services

Introduction

This post is part of my mini-series about AWS Developer Tools Services

AWS CodeDeploy

CodeDeploy is a flexible deployment service that can integrate with a number of different services. It can deploy to EC2/On-premises instances as well as AWS Lambda and Amazon ECS. In my example I’ll use EC2 deployment on a Windows server.

Agent

CodeDeploy requires an agent application installed on the target machines. This is only a requirement for EC2 though. For containers (ECS) and serverless (Lambda) deployments this is not necessary.

As of this writing it’s officially supported by the following versions of Windows Server: 2016, 2012 R2, and 2008 R2.

The installer for the agent program can be downloaded from here

The agent setup is a bit unintuitive in the sense that once it’s done it just disappears without any confirmation of successful installation.

Creating an Application

Now that we have the EC2 instance running with the CodeDeploy agent we are ready to configure CodeDeploy service.

First we need to create our application and specify the target platform (EC2 / ECS / Lambda).

Create a Deployment Group

In order to create a deployment group we need to supply a few details.

IAM Role

We need to provide a CodeDeploy service role. So first we need go to IAM, create a role and select CodeDeploy from the AWS Services list. Then select the use case (in my case it’s the first option):

This comes with a role with AWSCodeDeployRole policy attached.

Finally we give it a name and copy the ARN and use it in CodeDeploy configuration.

Deployment Type

CodeDeploy supports 2 types of deployment:

  • In-place
  • Blue/green

In this example I will have 1 EC2 instance and I will use In-place deployment type. But Blue/green is very important to have a new system with no downtime so I’ll cover that bit in a separate post.

Environment Configuration

The way CodeDeploy finds the target instances is via tags. You can provide multiple groups of tags. In that case, for an instance to be targeted by CodeDeploy

Other Settings

There are more settings such as deployment configuration (deploy all at once, one at a time, half at a time) and configuring Load Balancer. Since I’m only covering the basics in this example I’ll not explore those features. They don’t mean much with a single EC2 instance anyway.

Deploying the Application

We have at this point defined the target of the deployment and now we need to define what to deploy.

To tie it to the previous configuration we have done, we first select the deployment group. Then we choose the location of the deployment package: In my example I’ll use the artifacts built by CodeBuild and uploaded to Amazon S3.

After you create the deployment, it automatically starts it. Make sure there are running instances with the correct tags other wise you might see an error like this:

For each deployment you need to create a new deployment. The good news is that you can an existing deployment and create it very quickly that way.

After the deployment starts on the instances you can see the progress on the screen right away:

IAM Instance Profile

If you are having an error like this

InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Missing credentials - please check if this instance was started with an IAM instance profile

that is probably because you skipped creating an instance profile for your EC2 instances like I did. It is explained here.

Basically the CodeDeploy agent running in EC2 instance needs to communicate with CodeDeploy service and Amazon S3 to get the deployment package.

To resolve this issue:

  1. Create an IAM policy like this:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::replace-with-your-s3-bucket-name/*",
        "arn:aws:s3:::aws-codedeploy-us-east-1/*"
      ]
    }
  ]
}
  1. Create an IAM service role for EC2. In permissions, attach the policy created in Step 1.

In EC2, select the instance and attach the role:

Make sure to restart the service after attaching the role.

AppSpec.yml

At this point the agent can locate the deployment package but it doesn’t know what to do with. We need to define installation steps. Just like we had to provide a buildspec.yml file to CodeBuild service, we need to provide an appspec.yml file to CodeDeploy service to define these deployment steps.

appspec.yml file must be placed in application source code’s root folder.

In this example, the deployment is basically just copying the new application to the destination and my appspec.yml file looks like this:

version: 0.0
os: windows
files:
  - source: \
    destination: c:\website

In addition to this basic configuration, it’s possible to run scripts before and after installation. So if there are dependencies or clean up tasks they can be carried out that way.

Troubleshooting

I created this sample project on a MacBook Pro and when I tested the deployment had the following error:

The problem is the line endings. Windows expects \r\n line endings whereas on unix-baseds systems the line ending is \n. So the solution for that is to save the appsepc.yml file with UTF-8 encoding and the agent was able to read the file correctly and automatically unzips the contents of the deployment package to the target folder:

Conclusion

In this post I used basic features of AWS CodeDeploy service to deploy my artifacts from the previous post and carry out the actual deployment. In future posts I will focus on more advanced features and other AWS code services.

Resources

aws ,  devops codebuild, git, aws_developer_tools_services

Introduction

This post is part of my mini-series about AWS Developer Tools Services

AWS CodeBuild

As the name implies, AWS CodeBuild is Amazon’s build service that compiles your code, runs unit tests etc. As this is quite broad and changes from project to project and technology to technology it requires some customization.

Hook up the source code

First step is to select the source. It’s quite flexible in the sense that it integrates with top Git providers such as GitHub and Bitbucket. The easiest way is to use CodeCommit because of the direct integration between services. No extra authentication is required to connect to the repository.

It also support Amazon S3 as source but I cannot think of a practical use for it.

Another interesting feature is that it supports multiple input sources. If your build depends on source from multiple repositories I would consider merging them into a single repository but if that’s not possible this is a nice option to make it work.

Hook up the Docker image with build tools

The build process takes place in a Docker container. By default AWS provides their own build images for different operating systems:

A couple of notes on build images:

  • AWS provides some build images but we can provide our own images as well.

  • How the managed images configured can be found here: Docker Images Provided by CodeBuild. They have build tools installed on these images for all runtimes they support. As of this writing dotnet core 3.0 is not supported in any of the images though.

  • The image availability may differ between regions. For example N.Virginia regions has 3 supported OS images whereas London region doesn’t have Windows image.
  • Running this build process is not free and pricing details can be found here. If you’re running long build processes that you run frequently these costs can quickly add up.

Configuring the build

If you run the build after selecting the source and the build image, you get the following error:

By default CodeBuild requires a special configuration file called buildspec.yml to reside at the root of the repository:

In my example, the folder structure is like this:

[Root]
	[Src]
   		[Sample.UI.Web]
   		[Sample.Core]
   		[Sample.Core.UnitTests]
   		Sample.sln
   		buildspec.yml

I’m using a dotnet core 2.2 project on Windows-based Docker image and my buildspec.yml looks like this:

version: 0.2

phases:
  build:
    commands:
      - cd ./src
      - dotnet restore
      - dotnet test
      - dotnet publish Sample.UI.Web -c Release -o ../output

artifacts:
  files:
    - '**/*'
  base-directory: ./src/output

Running the unit tests

As you can see in my buildspec file I’m running the unit tests using dotnet test command. There’s no built-in facility in CodeBuild to run unit tests as it’s a general-purpose tool. You have to provide your own mechanism to achieve this.

To test running the unit tests I deliberately broke a test and checked in the code. After running the build again I got this result:

So it actually runs the tests and fails the build if there’s a failing test. Ideally you want to run the tests whenever there are code changes. Looks like currently this is possible via CodePipeline service. I’ll explore that later and will cover running the build and unit tests automatically.

Artifacts

Now that we can build the project we need to upload the output of the build to somewhere so that we can deploy it. Currently you can only choose Amazon S3 to upload the artifacts or no artifacts at all.

If you had created a bucket already you can simply select it from the dropdown list.

The options are pretty basic. You can choose to compress the output and upload as a single zip file or the artifacts you defined in the buildspec in their original format.

You can also choose to create a folder named as the build GUID so that you can keep the unique outputs for each release. You can do this by specifying Build ID as the namespace type.

When you’re running the build make sure to view the full logs to get the latest status quickly. I noticed that if you just watch the main page and check out the results you may experience some delays.

Conclusion

In my experience CodeBuild doesn’t provide too much to help with the build process because by nature it’s a generic tool. But it provides flexibility and full integration with other AWS services such as CodeCommit and CodeDeploy so I believe it’s worht using.

Resources