awsdevops codedeploy, git, aws_developer_tools_services

Introduction

This post is part of my mini-series about AWS Developer Tools Services

AWS CodeDeploy

CodeDeploy is a flexible deployment service that can integrate with a number of different services. It can deploy to EC2/On-premises instances as well as AWS Lambda and Amazon ECS. In my example I’ll use EC2 deployment on a Windows server.

Agent

CodeDeploy requires an agent application installed on the target machines. This is only a requirement for EC2 though. For containers (ECS) and serverless (Lambda) deployments this is not necessary.

As of this writing it’s officially supported by the following versions of Windows Server: 2016, 2012 R2, and 2008 R2.

The installer for the agent program can be downloaded from here

The agent setup is a bit unintuitive in the sense that once it’s done it just disappears without any confirmation of successful installation.

Creating an Application

Now that we have the EC2 instance running with the CodeDeploy agent we are ready to configure CodeDeploy service.

First we need to create our application and specify the target platform (EC2 / ECS / Lambda).

Create a Deployment Group

In order to create a deployment group we need to supply a few details.

IAM Role

We need to provide a CodeDeploy service role. So first we need go to IAM, create a role and select CodeDeploy from the AWS Services list. Then select the use case (in my case it’s the first option):

This comes with a role with AWSCodeDeployRole policy attached.

Finally we give it a name and copy the ARN and use it in CodeDeploy configuration.

Deployment Type

CodeDeploy supports 2 types of deployment:

  • In-place
  • Blue/green

In this example I will have 1 EC2 instance and I will use In-place deployment type. But Blue/green is very important to have a new system with no downtime so I’ll cover that bit in a separate post.

Environment Configuration

The way CodeDeploy finds the target instances is via tags. You can provide multiple groups of tags. In that case, for an instance to be targeted by CodeDeploy

Other Settings

There are more settings such as deployment configuration (deploy all at once, one at a time, half at a time) and configuring Load Balancer. Since I’m only covering the basics in this example I’ll not explore those features. They don’t mean much with a single EC2 instance anyway.

Deploying the Application

We have at this point defined the target of the deployment and now we need to define what to deploy.

To tie it to the previous configuration we have done, we first select the deployment group. Then we choose the location of the deployment package: In my example I’ll use the artifacts built by CodeBuild and uploaded to Amazon S3.

After you create the deployment, it automatically starts it. Make sure there are running instances with the correct tags other wise you might see an error like this:

For each deployment you need to create a new deployment. The good news is that you can an existing deployment and create it very quickly that way.

After the deployment starts on the instances you can see the progress on the screen right away:

IAM Instance Profile

If you are having an error like this

InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Missing credentials - please check if this instance was started with an IAM instance profile

that is probably because you skipped creating an instance profile for your EC2 instances like I did. It is explained here.

Basically the CodeDeploy agent running in EC2 instance needs to communicate with CodeDeploy service and Amazon S3 to get the deployment package.

To resolve this issue:

  1. Create an IAM policy like this:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::replace-with-your-s3-bucket-name/*",
        "arn:aws:s3:::aws-codedeploy-us-east-1/*"
      ]
    }
  ]
}
  1. Create an IAM service role for EC2. In permissions, attach the policy created in Step 1.

In EC2, select the instance and attach the role:

Make sure to restart the service after attaching the role.

AppSpec.yml

At this point the agent can locate the deployment package but it doesn’t know what to do with. We need to define installation steps. Just like we had to provide a buildspec.yml file to CodeBuild service, we need to provide an appspec.yml file to CodeDeploy service to define these deployment steps.

appspec.yml file must be placed in application source code’s root folder.

In this example, the deployment is basically just copying the new application to the destination and my appspec.yml file looks like this:

version: 0.0
os: windows
files:
  - source: \
    destination: c:\website

In addition to this basic configuration, it’s possible to run scripts before and after installation. So if there are dependencies or clean up tasks they can be carried out that way.

Troubleshooting

I created this sample project on a MacBook Pro and when I tested the deployment had the following error:

The problem is the line endings. Windows expects \r\n line endings whereas on unix-baseds systems the line ending is \n. So the solution for that is to save the appsepc.yml file with UTF-8 encoding and the agent was able to read the file correctly and automatically unzips the contents of the deployment package to the target folder:

Conclusion

In this post I used basic features of AWS CodeDeploy service to deploy my artifacts from the previous post and carry out the actual deployment. In future posts I will focus on more advanced features and other AWS code services.

Resources

awsdevops codebuild, git, aws_developer_tools_services

Introduction

This post is part of my mini-series about AWS Developer Tools Services

AWS CodeBuild

As the name implies, AWS CodeBuild is Amazon’s build service that compiles your code, runs unit tests etc. As this is quite broad and changes from project to project and technology to technology it requires some customization.

Hook up the source code

First step is to select the source. It’s quite flexible in the sense that it integrates with top Git providers such as GitHub and Bitbucket. The easiest way is to use CodeCommit because of the direct integration between services. No extra authentication is required to connect to the repository.

It also support Amazon S3 as source but I cannot think of a practical use for it.

Another interesting feature is that it supports multiple input sources. If your build depends on source from multiple repositories I would consider merging them into a single repository but if that’s not possible this is a nice option to make it work.

Hook up the Docker image with build tools

The build process takes place in a Docker container. By default AWS provides their own build images for different operating systems:

A couple of notes on build images:

  • AWS provides some build images but we can provide our own images as well.
  • How the managed images configured can be found here: Docker Images Provided by CodeBuild. They have build tools installed on these images for all runtimes they support. As of this writing dotnet core 3.0 is not supported in any of the images though.

  • The image availability may differ between regions. For example N.Virginia regions has 3 supported OS images whereas London region doesn’t have Windows image.
  • Running this build process is not free and pricing details can be found here. If you’re running long build processes that you run frequently these costs can quickly add up.

Configuring the build

If you run the build after selecting the source and the build image, you get the following error:

By default CodeBuild requires a special configuration file called buildspec.yml to reside at the root of the repository:

In my example, the folder structure is like this:

[Root]
	[Src]
   		[Sample.UI.Web]
   		[Sample.Core]
   		[Sample.Core.UnitTests]
   		Sample.sln
   		buildspec.yml

I’m using a dotnet core 2.2 project on Windows-based Docker image and my buildspec.yml looks like this:

version: 0.2

phases:
  build:
    commands:
      - cd ./src
      - dotnet restore
      - dotnet test
      - dotnet publish Sample.UI.Web -c Release -o ../output

artifacts:
  files:
    - '**/*'
  base-directory: ./src/output

Running the unit tests

As you can see in my buildspec file I’m running the unit tests using dotnet test command. There’s no built-in facility in CodeBuild to run unit tests as it’s a general-purpose tool. You have to provide your own mechanism to achieve this.

To test running the unit tests I deliberately broke a test and checked in the code. After running the build again I got this result:

So it actually runs the tests and fails the build if there’s a failing test. Ideally you want to run the tests whenever there are code changes. Looks like currently this is possible via CodePipeline service. I’ll explore that later and will cover running the build and unit tests automatically.

Artifacts

Now that we can build the project we need to upload the output of the build to somewhere so that we can deploy it. Currently you can only choose Amazon S3 to upload the artifacts or no artifacts at all.

If you had created a bucket already you can simply select it from the dropdown list.

The options are pretty basic. You can choose to compress the output and upload as a single zip file or the artifacts you defined in the buildspec in their original format.

You can also choose to create a folder named as the build GUID so that you can keep the unique outputs for each release. You can do this by specifying Build ID as the namespace type.

When you’re running the build make sure to view the full logs to get the latest status quickly. I noticed that if you just watch the main page and check out the results you may experience some delays.

Conclusion

In my experience CodeBuild doesn’t provide too much to help with the build process because by nature it’s a generic tool. But it provides flexibility and full integration with other AWS services such as CodeCommit and CodeDeploy so I believe it’s worht using.

Resources

devopsaws codecommit, git, aws_developer_tools_services

Introduction

This post is part of my mini-series about AWS Developer Tools Services

AWS CodeCommit

I’m looking for GitHub alternatives and thought CodeCommit would be worth a shot.

Creating a Repository

Like any other Git provider, you start with creating a repository.

As a side-note, I always thought what was lacking in GitHub was a way of grouping repositories. You can use topics which are essentially labels

I was hoping there would be a way to create folders or another way to group related repositories together but the closest thing they provide is using tags:

Connecting to repositories

After creating the repository it shows all the information needed for different protocols (HTTPS / SSH) and platforms (MacOS / Windows / Linux). One thing to note is that because of the security risks, it doesn’t allow you to connect with root credentials so you have to be logged in as an IAM user with CodeCommit full access.

I’m writing this blog post on a Windows laptop and so followed this guide for generating public/private keys.

It’s similar to any other Git guide with the exception that you have to add your access key to your SSH config file (~/.ssh/config):

Host git-codecommit.*.amazonaws.com
  User APKAEIBAERJR2EXAMPLE
  IdentityFile ~/.ssh/codecommit_rsa

Then to test your connection, run:

ssh git-codecommit.us-east-2.amazonaws.com

After adding the host to known hosts list you should see this message:

You have successfully authenticated over SSH. You can use Git to interact with AWS CodeCommit. Interactive shells are not supported.Connection to git-codecommit.us-east-2.amazonaws.com closed by remote host.

Clone the repo

On the same, click on the Clone URL dropdown and choose the protocol you’d like to use (ideally SSH):

and try it in your shell (Powershell in my case). You should see your repository locally:

Troubleshooting

In my experience in Windows, everything worked find on command line but when I tried to use my Git client (Git Extensions) I started having connection issues. The fix for that was to add my AWS SSH Key Id in the URL:

git clone ssh://Your-SSH-Key-ID@git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo my-demo-repo

Resources