dev git, github flow comments

In this post I woud like to talk about GitHub Flow branching strategy:

Basic approach

  • Main diagram of the workflow:

  • It’s a branch-based workflow mostly targeting projects where deployments are made regularly. In other words, it’s more suitable for continuos deployment environments.
  • There’s only one rule about branching: anything in the master branch is always deployable. Because of this, it’s extremely important that your new branch is created off of master when working on a feature or a fix.
  • Branch names should be descriptive
  • Once the changes are done, create a pull request.
  • Discuss the pull request and apply changes where necessary
  • Merge into master and deploy straight away

Resources

dev git, git flow, sourcetree comments

In this post I woud like to talk about a common Git branching strategy: Git Flow.

Basic approach

  • Main diagram of the workflow:

  • The central repo holds two main branches with an infinite lifetime:

    • master
    • develop
  • origin/master is the main branch where the source code of HEAD always reflects a production-ready state
  • origin/develop is the main branch where the source code of HEAD always reflects a state with the latest delivered development changes for the next release (also called as integration branch)
  • When the source code in the develop branch reaches a stable point and is ready to be released, all of the changes should be merged back into master somehow and then tagged with a release number.
  • There are also supporting branches with limited lifetime such as:
    • Feature branches
    • Release branches
    • Hotfix branches
Supporting branch May branch off Must merge back into Naming convention Lifetime Remarks
Feature develop develop anything except master, develop, release-*, or hotfix-* as long as the feature is in development Features should never interact directly with master.
Release develop develop and master release-* preparation of the release Starts after feature-freeze. Allows minor bugfixes. Adding large new features here is strictly prohibited.
Hotfix master develop and master hotfix-* preparation of the release Similar to release in that used to prepare a production release but unplanned.

Automation

Most tools have plugins to support Git Flow. In the example below I will use SourceTree which is available in Windows and Mac.

In this example I’m going to use a GitHub repository called gitflow-automation-test. I’ve already created it in GitHub and in my development environment.

Initialization and working with features

First thing to do with Git Flow in SourceTree is to initialize the repository.

It sounds confusing at first because the git repo itself is already initialized. What it actually does is explained here. It shows a dialog box to configure the plugin:

It also created the local develop branch.

At this point we have the three types of branches explained above to choose from to start. Let’s start with a new feature named “users”

I just added a blank file that represents the changes required for that feature and commmited my changes.

Now that the feature has been implemented, I want to finish the feature. Similar to start, I went to Repository -> Git Flow -> Hg Flow -> Finish Feature

As the options in the window suggests, this action merges the changes into develop and deletes the local branch. It doesn’t push to remote so we have to push it ourselves.

Releasing new version

Now let’s assume we’ve implemented all the features planned for this release and we want to deploy a new version of our application.

To achieve this we start a new release:

Once we are ready to ship this release, we end the release similarly:

We can choose to push to the origin directly too. If we switch to detail view in Sourcetree output we can see a nice summary of actions that took place:

Summary of actions:
- Release branch 'release/1.0' has been merged into 'master'
- The release was tagged '1.0'
- Release tag '1.0' has been back-merged into 'develop'
- Release branch 'release/1.0' has been locally deleted
- 'develop', 'master' and tags have been pushed to 'origin'
- You are now on branch 'develop' 

Working with hotfixes

As the development continues, let’s assume we implemented another feature named “orders”. Finished the feature and pushed to origin. Now we have a new feature in develop. Suddenly we noticed there’s a bug in the production and we need to deploy a hotfix. We cannot create a new release branch off of develop because the new “orders” feature would be deployed along with the bugfix. So we create out hotfix branch off of master which always points to production.

As we can see the code reverted back to master:

So I added another file to represent the changes for the hotfix and finished the hotfix:

And the summary of actions looks like this:

Summary of actions:
- Hotfix branch 'hotfix/1.0-hotfix-1' has been merged into 'master'
- The hotfix was tagged '1.0-hotfix-1'
- Hotfix tag '1.0-hotfix-1' has been back-merged into 'develop'
- Hotfix branch 'hotfix/1.0-hotfix-1' has been locally deleted
- 'develop', 'master' and tags have been pushed to 'origin'
- You are now on branch 'develop'

As we can see the process is very similar to release. The only difference is the originating branch which was master in this case.

Working with pull requests

It’s a good practice to create pull requests and ask colleagues to review the code before merging into develop. This can still be part of Git Flow.

In this example I’ve created a new feature called “customers”. The changes are represented by the file “Implemented Customers Feature in This File.md”. I’ve commited my code to my feature branch and pushed it to GitHub.

Now I can click a new Pull Request by clicking Compare & pull request button:

I’ve clicked on Create Pull Request and now it’s open on GitHub.

At this point there are several strategies you can take such as reviewed handling the merge and the developer deleting the local branch themselves (as there won’t be a Finish Feature step).

My suggestion is for the reviewer to Approve the pull request but not to merge it. Once the PR is approved, the developer receives a notification. Also the status of the PR can be seen on GitHub as well:

At this point the developer can go back to Git Flow and finish feature as shown previously.

Now that we know customers feature have been approved we can select that and finish that feature:

Summary of actions:
- The feature branch 'feature/customers' was merged into 'develop'
- Feature branch 'feature/customers' has been locally deleted; it has been remotely deleted from 'origin'
- You are now on branch 'develop'

At this point developer needs to push the merged develop branch to GitHub otherwise the recent change (approved customers change) doesn’t appear in remote even though the pull request was merged.

Even if you’re in the middle of another feature and have uncommited files you can still finish the approved feature so it doesn’t interrupt with your work.

Command Line

All the Git Flow commands explained above are also supported by command line but to be able to use them you have to install it:

brew install git-flow

Then you can start using it as

git flow <subcommand>

In Windows, command line tools come with the installer so no extra installation is necessary.

Resources

hobby raspberry pi, productivity comments

The Problem

Snoozing is the worst possible thing you can do to yourself! You start the day with a feeling of guilt and being late for everything from the start. To avoid this I’ve been working on improving my toolkit and techniques. I posted this as a list of my current methods and tools.

Whilst these methods worked for me a while I’ve started having an issue with Alexa playing my playlist from Spotify: It always starts with the same song. My idea was to have a different song every morning so this bug in Spotify impeded me from achieving my goal.

The Idea

As the saying goes: If all you have is a hammer, everything looks like a nail!

Even though I have other means of developing tools and software, I wanted to use another Raspberry Pi to replace Alexa morning routine (it still turns on the light though).

The Execution

So I decided to dust off my OG Raspberry Pi 1 Model B and implement my own custom solution for playing a random song every morning to help wake me up.

Here’s how I implemented it from start to finish:

Step 01: Download Raspbian image and prepare SD/MicroSD card

I downloaded Raspbian Buster Lite as I don’t need desktop and other extra software for this project.

I followed the official guide from raspberrypi.org to burn the image. It’s basically just a few commands:

Insert the card and run the following to identify the card:

diskutil list

Then

sudo dd bs=1m if=path_of_your_image.img of=/dev/rdiskn conv=sync

where n is the number of disk which I found from the first step.

If you get “resource is busy” error, then run the following

diskutil unmountDisk /dev/diskn

Step 02: Carry Out Necessary Configuration Updates

It’s a good practice to change the default passwords so I ran

passwd

and updated the password.

Also for remote installations I enabled SSH:

sudo systemctl enable ssh
sudo systemctl start ssh

And I like to give a specific name to all my devices so that I can keep an eye on the machines connected to my network more easily.

To achieve that I ran:

sudo hostname NEW_HOSTNAME

I used raspi-config to connect to WiFi network

sudo raspi-config

and enter the details and rebooted to connect to the network.

In this MVP implementation I don’t even need Internet connectivity (after the initial installations below) as all my MP3 files are on the device already. But in my tests I noticed that when the Pi is disconnected the system time wasn’t updating so scheduled job wasn’t being triggered. This is probably because I’m using a Pi 1. I’m guessing the newer version can work offline as well.

Step 03: Install MP3 Player Software

In this simple project I’m going to install mpg123 and run it via a Python script

Installation is as simple as this:

sudo apt-get install mpg123 -y

Step 04: Develop the script

The following script points to a folder which supposedly have all the local MP3 files, picks one randomly and starts playing.

import os
import random

path ='/path/to/mp3/folder' 
files = os.listdir(path)
index = random.randrange(0, len(files))

os.system(f'/usr/bin/mpg123 {files[index]}')

Step 05: Copy music and script to Raspberry Pi

Save the script above in a USB drive along with the MP3 files. To copy the files from USB drive to Pi microSD card, I inserted my USB and ran the following:

sudo fdisk -l
sudo mkdir /media/usb
sudo mount /dev/sda1 /media/usb

Then

mkdir /home/pi/music
cp /media/usb/*.mp3 /home/pi/music

and copy the script to home folder

cp /media/usb/alarm-pi.py /home/pi

Step 06: Schedule the script

crontab -e

Then add to run at a certain time. The example below runs at 2:30am everyday:

30 2 * * * /usr/bin/python3 /home/pi/alarm-pi.py

If you want to run the script multiple times you can another one (like another one after 10 minutes as a snooze-prevention mechanism!)

Step 07: Test the schedule

Update your schedule temporarily and set it to a few minutes later and start playing the waiting game! Once you hear a random song playing make sure to set it back to your actual alarm time.

Conclusion

I meant to run this in a Docker container and develop the application with dotnet core but turns out playing MP3 with dotnet core in Linux is not easy. Also playing sound from inside a container has its own issues. So for those reasons I chose the easiest possible path to achieve the goal. If I find time I’ll improve this and make it more flexible but for the time being it does the job so I’ll assume MVP is done now.

Resources

security pi-hole, raspberry pi comments

For some time I’d been wanting to set up my own Pi Hole in my network.

In this previous post I implemented a Docker registry on Raspberry Pi. I was planning to install it using Docker on that Pi but then I decided to set up a new Pi from scratch due to performance concerns. After I get the hang of it I might migrate it to Docker.

Preparations: Download Raspbian image and prepare Micro SD card

As of this writing, the latest Raspbian release is Buster but in the official supported OS list the latest supported Raspbian is Stretch. So I went to this page to download Raspbian Stretch Lite.

I downloaded Raspbian Buster Lite as I don’t need desktop and other extra software for this project.I followed the official guide from raspberrypi.org to burn the image. It’s basically just a few commands:

I inserted the card and run the following to identify the card:

diskutil list

Then

sudo dd bs=1m if=path_of_your_image.img of=/dev/rdiskn conv=sync

where n is the number of disk which I found from the first step.

If you get “resource is busy” error, then run the following

diskutil unmountDisk /dev/diskn

Carry Out Necessary Configuration Updates

It’s a good practice to change the default passwords so I ran

passwd

and updated the password.

Also for remote installations I enabled SSH:

sudo systemctl enable ssh
sudo systemctl start ssh

And I like to give a specific name to all my devices so that I can keep an eye on the machines connected to my network more easily.

To achieve that I ran:

sudo hostname NEW_HOSTNAME

No need to set a static IP address at this point as Pi Hole setup takes care of that:

Install Pi-Hole

There is an easy one-step installation:

curl -sSL https://install.pi-hole.net | bash

I changed my IP to a different static IP and rebooted the Pi.

After that I could go to the admin page via http://{Pi Hole Static IP}/admin/ and see the dashboard:

Test

Now it’s time to see if it’s working at all. I have 2 MacBooks connected to the same network. I changed one of the laptop’s DNS server to point to Pi Hole and left the other one intact.

In the laptop that’s still using the old DNS settings, I opened a Chrome window in incognito mode and visited a site and at top it shows an ad:

And on the other laptop I repeated the process and I got the same page without ads this time:

When I go to the admin page I can see what calls are blocked:

Also a nice thing about it you can choose to Whitelist certain providers. For instance in the image below it looks like Google Analytics is blocked and I can whitelist it easily:

Resources

machine learning aws, deeplens comments

I recently bought this device in anticipation of getting accustomed with AWS Machine Learning services. I thought it would be a nice and fun introduction to build a DIY project using ML. This weekend I had a chance to play around with it.

Unboxing

Registering

First thing you need to do is register your DeepLens which can be done here

Alternatively, you can log in to AWS Console, go to AWS DeepLens service and click Register Device. As of this writing this service is only available in Tokyo, N.Virginia and Frankfurt. As closest to me is Frankfurt I chose that one.

Creating a Project

Now that the devices is ready, I went to Projects section and created a new project based on a template. For this initial exploration mission I chose Bird Classification template.

When you choose your template it shows a description of the template and how it works

Once you create the project it creates the necessary resources. For example in my template it created 2 Lambda functions:

Now we are ready to deploy the project to the device so we can simply click on Deploy to Device button on top right-hand corner in the project page:

After that you choose your DeepLens from the avialble device list (in my case I had one device) and click Review then Deploy

Testing the project

First I followed the steps to install custom certificate so that I can start viewing the video feed in my browser:

Because DeepLens doesn’t know if there’s a bird or not in the picture it constantly makes a guess. Even an empty carton box looks like a Long Tailed Jaeger with 2.1% confidence apparently!

When I presented an image I found from the Internet the results significantly improved though (thankfully, otherwise this would all be for nothing!)

According to Google the bird in the picture is indeed a Kingfisher.

Next test is the bird I saw on the roof and identified as a Mallard. Let’s see what DeepLens thinks about it:

So with 17.5% confidens DeepLenms agrees that it is a Mallard. One thing to note is that the angle of the image significantly changes the prediction. Even a slight change in the angle makes the prediction shift from 3% to 45%. But it’s just a test project template anyway so I’m not looking for too much accuracy at this point.

Final test: I downloaded the dataset used to train the algorithm. It’s a 1.1GB download full of images. To compare with the last Mallard test I got one picture from the Mallard folder and showed it to DeepLens:

As shown above the confidence increased to 55%. So probably if I use the pictures I took to train the algorithm I would be able to get much better results.

Conclusion

This was just a first step to set up and explore the device. Hopefully in the comin days I’ll be doing more meaningful projects with it and post those too.

Resources

docker registry, self-hosted comments

Working with Docker is great but when you want to deploy your applications to another server you need a registry to push your images so that you can pull them from the other end. To have that capability in my dev environment I decided to setup my own self-hosted Docker registry.

For the sake of brevity, I will omit creating Raspberry Pi SD Card and installing Docker on it. There are lots of gret videos and articles out there already.

Self-hosted vs Hosted

When it comes to hosted Docker registries, there are lots of free and paid options.

Benefits of self-hosted registry:

  • See the storage size and number of repos required for your system early on without having to pay anything
  • Push/pull images on the go without Internet connection during development phase
  • No privacy concerns: If you upload your image with your application in it you may have some sensitive data inside the image which may pose a risk if the 3rd party registry has full access to them
  • Free!

Benefits of hosted registry:

  • Hassle-free: No backups or server management

There are great Docker registries such as Docker Hub and Amazon ECR. I wouldn’t recommend usign a self-hosted registry for production. But if the price or privacy is a concern it can certainly be an option.

Creating Self-Hosted Registry

It sounds like it requires installing a server application but the nice thing about Docker is, even it is a Docker registry it can run in a container itself. So first off we pull the registry repo for Docker Hub:

docker pull registry

Now let’s create a container that will act as our registry:

docker run -d -p 5000:5000 --restart always --name registry registry

In my case the hostname of the Raspbeery Pi is

Now to test how we can push and pull images let’s download Docker’s hello-world image from Docker Hub:

docker pull hello-world

Now to push this inot our own registry running in Raspbeery Pi all we have to do is tag it with the server URL such as:

docker tag hello-world HOBBITON.local:5000/hello-world

At this point if we take look at the images on our local machine we can see the hello-world image is duplicated.

Now let’s push it to Pi:

docker push HOBBITON.local:5000/hello-world

This doesn’t work because of the following reason:

This is because the reigstry is considered to be insecure and by default it’s rejected by the client. We can confirm it’s deemed to be insecure if we check the server by running the following command:

docker info

At the bottom of the bottom we can see the localhost registry is insecure:

To address this we can add this registry to the list of insecure registries. For example in a Mac client we add go to Perefences –> Daemon and add the Raspbeery Pi registry as shown below:

After this, if we try once again to push to Pi we can see it succeded:

Now if we check the repository list on the registry again we can see the hello-world image hosted on our Pi:

Let’s now see if we can pull this image from another client.

And after pulling the image we can see it in the image list:

Resources

dev csharp, elasticsearch, docker, nest comments

I’ve been playing around with Elasticsearch on several occasions. This post is to organize those thoughts and experiences and show an easy way to setup ElasticSearch and start playing around with it.

Setup

Easiest way to setup Elasticsearch locally is using Docker. As of this writing the latest version of ElasticSearch is 7.2.0 and I’ll be using that in this example:

If you don’t already have the image, simply pull from Docker hub:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0

For development environment suggested command to run a container is

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.2.0

which keeps it very simple and straightforward but in my workout I’d like to insert a whole bunch of data and run some queries on it and I don’t want to re-generate my data over and over again. So I decided to persist my data on host.

Persisting Elasticsearch Data

Instead of running containers one by one in the command line a better approach is to create a docker-compose.yml file file and use Docker compose to start services. I used the sample YAML file provided in official Elastic documentation

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    container_name: es01
    environment:
      - node.name=es01
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    container_name: es02
    environment:
      - node.name=es02
      - discovery.seed_hosts=es01
      - cluster.initial_master_nodes=es01,es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata02:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local

networks:
  esnet:

This example creates an Elasticsearch cluster with 2 nodes and uses named volumes to persist data so next time when we bring this cluster up again we should be able to continue where we left off data-wise.

Sample Application

In my previous blog post I developed a simple test data generator to generate fake bank statement data with a library called Bogus. In this project, I will use that generator to generate lots and lots of test data, insert them into Elasticsearch and have fun with it!

When you start a C# project and start looking for a library to interact with Elasticsearch, it’s a bit confusing to find out there are actually two of them: Elasticsearch.net and NEST. The gist of it is NEST is a high-level library that uses Elasticsearch.net under the hood. It also exposes low-level client so that it actually enhances Elasticsearch.net and allows using strongly typed DSL queries. In the sample application I used NEST.

Creating Elasticsearch client

Creating a client with some basic settings is straightforward:

using (var connectionSettings = new ConnectionSettings(new Uri("http://localhost:9200")))
{
    var settings = connectionSettings
        .DefaultIndex("bankstatementindex")
        .ThrowExceptions(true);
	IElasticClient elasticClient = new ElasticClient(settings);
}

Indexing data

To index a single document **IndexDocument** method can be called. However, using this method to loop through a large number of documents is not recommended.

elasticClient.IndexDocument<BankStatementLine>(testData.First());

For multiple documents, IndexMany method should be called. If the data size too large then using BulkAll method and BulkAllObservable helper is recommended.

To see the difference I created a test to index 5,000 documents with a looping over the array and using BulkAll after that. Looping over the collection took around 26 seconds whereas bulk index took only 1.2 seconds as shown in the screenshot.

Also it displays “Done” 5 times because I set the size to 1,000 and I requested 5,000 documents to be indexed so it automatically divided the load into 5 and made 5 calls:

var bulkAll = elasticClient.BulkAll(testData, x => x
                .BackOffRetries(2)
                .BackOffTime("30s")
                .RefreshOnCompleted(true)
                .MaxDegreeOfParallelism(4)
                .Size(1000));

bulkAll.Wait(TimeSpan.FromSeconds(60),
    onNext: (b) => { Console.Write("Done"); }
);

Same result can also be achieved by subscribing to BulkAll observer:

var waitHandle = new CountdownEvent(1);

bulkAll.Subscribe(new BulkAllObserver(
    onNext: (b) => { Console.Write("."); },
    onError: (e) => { throw e; },
    onCompleted: () => waitHandle.Signal()
));

waitHandle.Wait();

Showing progress

In the sample code below I showed displaying progress using onNext action delegate:

var testData = dataGen.Generate(statementConfig.StartDate, statementConfig.EndDate, statementConfig.OpeningBalance, statementConfig.DebitTransactionRatio, statementConfig.TransactionDateInterval, statementConfig.NumberOfStatementLines);
var cancellationToken = new CancellationToken();
var batchSize = 250;
var bulkAll = elasticClient.BulkAll(testData, x => x
    .BackOffRetries(2)
    .BackOffTime("30s")
    .RefreshOnCompleted(true)
    .MaxDegreeOfParallelism(4)
    .Size(batchSize), cancellationToken);
var totalIndexed = 0;
var stopWatch = new Stopwatch();
stopWatch.Start();
bulkAll.Wait(TimeSpan.FromSeconds(60),
    onNext: (b) =>
    {
        totalIndexed += batchSize;
        Console.WriteLine($"Total indexed documents: {totalIndexed}");
    }
);

and the output looked like this:

Even though the numbers seem a bit wonky I think it’s a good example to illustrate the multi-threaded nature of BulkAll. Because I set the maximum degree of paralleism to 4 and first 1,000 were inserted in a mixed order suggesting that they were running in parallel.

Cancellation with bulk operations

BulkAll observer can also be cancelled for longer processes if necessary. The code excerpt below shows the relevant pieces to cancellation

var cancellationTokenSource = new CancellationTokenSource();
var cancellationToken = cancellationTokenSource.Token;
var batchSize = 250;
var bulkAll = elasticClient.BulkAll(testData, x => x
    .BackOffRetries(2)
    .BackOffTime("30s")
    .RefreshOnCompleted(true)
    .MaxDegreeOfParallelism(4)
    .Size(batchSize), cancellationToken);
var totalIndexed = 0;
var stopWatch = new Stopwatch();
stopWatch.Start();
Task.Factory.StartNew(() =>
    {
        Console.WriteLine("Started monitor thread");
        var cancelled = false;
        while (!cancelled)
        {
            if (stopWatch.Elapsed >= TimeSpan.FromSeconds(60))
            {
                if (cancellationToken.CanBeCanceled)
                {
                    Console.WriteLine($"Cancelling. Elapsed time: {stopWatch.Elapsed.ToString("mm\\:ss\\.ff")}");
                    cancellationTokenSource.Cancel();
                    cancelled = true;
                }
            }

            Thread.Sleep(100);
        }
    }
);

try
{
    bulkAll.Wait(TimeSpan.FromSeconds(60),
        onNext: (b) =>
        {
            totalIndexed += batchSize;
            Console.WriteLine($"Total indexed documents: {totalIndexed}");
        }
    );
}
catch (OperationCanceledException e)
{
    Console.WriteLine($"Taking longer than allowed. Cancelled.");
}

Querying Data

Querying data can be done by calling Search method of ElasticsearchClient. Here’s a few examples below. There are more in the sample accompanying source code:

// Get the first 100 documents
var searchResponse = elasticClient.Search<BankStatementLine>(s => s
    .Query(q => q
        .MatchAll()
    )
    .Size(100)
);
// Get transactions with date between 01/01/2018 and 10/01/2018
var searchResponse = elasticClient.Search<BankStatementLine>(s => s
    .Query(q => q
        .DateRange(x => x
            .Field(f => f.TransactionDate)
            .GreaterThanOrEquals(new DateTime(2018, 01, 01))
            .LessThanOrEquals(new DateTime(2018, 01, 10))
        )
    )
    .Size(10000)
);

Deleting data

For my tests I had to delete all frequently and it can be achieved by running the query below:

elasticClient.DeleteByQuery<BankStatementLine>(del => del
    .Query(q => q.QueryString(qs => qs.Query("*")))
);

Source Code

Sample application can be found under blog/ElasticsearchWorkout folder in the repository.

Resources

dev csharp, fake, test, data comments

Generating high-quality test data can have an impact on the accuracy of the tests overall. In this post I’ll show using a helpful C# library called Bogus

Showcase project: Bank Statement Generator

In this example I’ll generate fake bank statements. Normally they come in CSV files and have the following model:

public class BankStatementLine
{
    public DateTime TransactionDate { get; set; }
    public string TransactionType { get; set; }
    public string SortCode { get; set; }
    public string AccountNumber { get; set; }
    public string TransactionDescription { get; set; }
    public decimal? DebitAmount { get; set; }
    public decimal? CreditAmount { get; set; }
    public decimal Balance { get; set; }
}

I’ll use Bogus to generate realistic fake statement lines and finally save it as a CSV and see if it looks real.

Rules and restrictions

I want the fields in the model above conform to certain set of rules to be realistic:

  • Transaction Date must be within a certain range I provide as bank statements are generated for a date range.
  • Dates should be incremental and not random
  • Sort Code must be in the following format: NN-NN-NN and must be the same for the entire statement.
  • Account number must be an 8-digit number and same for the entire statement.
  • Transaction Description must be free text
  • Debit Amount and Credit Amount must be decimal numbers but only one of them can be present at any given line
  • Transaction Type must be one of the pre-defined values and also some types can be for credit and some for debit only.
  • Balance should be sum of all debit and credit amounts plus the first balance in the statement. So this value is dependent on the values that come before it.
  • The number of lines in a statement should be random.

Rule implementations

Some rules stated above are very straightforward and easy to implement. These are some samples of what Bogus is capable of. For the full documentation check out the GitHub repository.

Date range support

Generating a date between a range is simple:

.RuleFor(x => x.TransactionDate, f => f.Date.Between(startDate, endDate))

Enum and array support

For Transaction Type I want to select a random value from a list of set values. This can be done in 2 ways: By using an enum or an IEnumerable.

var transactionTypes = new[] { "FPO", "DEB", "DB", "FPI" };

and in the rule description it can be used as

.RuleFor(x => x.TransactionType, f => f.PickRandom(transactionTypes) )

Another way is using enums such as:

public enum TransactionType
{
    FPO,
    DEB,
    DB,
    FPI
}

and the rule becomes:

 .RuleFor(x => x.TransactionType, f => f.PickRandom<TransactionType>().ToString())

In my final implementation I used selecting from a list of objects. You can check out the sample code to see that version.

Number range

For the account number I need an 8-digit number which can be achieved with something like this rule:

.RuleFor(x => x.AccountNumber, f => f.Random.Long(100000000, 99999999).ToString())

Bogus API also has builtin support for account number so the following is a more elegant and expressive way of achieving the same:

.RuleFor(x => x.AccountNumber, f => f.Finance.Account())

Formatting string

Formatting Sort Code can be achieved by Field.Random.Replace method

.RuleFor(x => x.SortCode, f => f.Random.Replace("##-##-##"))

Similar to account number, it also has built-in support for sort code:

.RuleFor(x => x.SortCode, f => f.Finance.SortCode())

Null values

In my case in some fields I’d like to have null values too. This can be achieved by OrNull extension method. For example, in the code below it generates %20 of DebitAmount values null.

.RuleFor(x => x.DebitAmount, f => f.Random.Decimal(0.00m, 9999.00m).OrNull(f, 0.2f))

Common fields

In my case some values in each statement line repeat throughout the entire statement such as account number and sort code. To achieve that I created a “base” statement line and every fake statement line used these shared fields instead of generating new ones.

var commonFields = new Faker<BankStatementLine>()
    .RuleFor(x => x.AccountNumber, f => f.Finance.Account())
    .RuleFor(x => x.SortCode, f => f.Finance.SortCode())
    .Generate();


var fakeTransactions = new Faker<BankStatementLine>()
    .StrictMode(true)
    .RuleFor(x => x.AccountNumber, commonFields.AccountNumber)
    .RuleFor(x => x.SortCode, f => commonFields.SortCode)
	...
	...

Random number of objects

It’s more realistic to have varying number of lines in statements. With Generate method you can specify the exact number of items you want to generate which is good for unit tests. For my purposes I just wanted to create random of rows in each statement as I only needed the data to be imported. This can be achieved by GenerateBetween:

var statementLines = fakeTransactions.GenerateBetween(10, 20);

Dependent values

The tricky part in this scenario was the dependent values. Normally when you use RuleFor extension method it generates the value for that field alone in isolation. In my case, one restriction was Debit Amount and Credit Amount could not both have values in the same line. Also Balance depends on these values and needs to be calculated in each line.

As far as I can tell there’s no built-in support to define these dependencies. Based on my tests I was able to achieve this in 2 ways

  1. Update the values accordingly in FinishWith extension method
  2. Use Rules extension method to define multiple rules at once and implement the restrictions inside it.

I think the latter is a better solution as FinishWith sounds more like clean up, logging or similar extra activity where Rules sound more like actual business logic implementation.

So with that in mind my rules for Debit Amount, Credit Amount and Balance fields looked like this:

.Rules((f, x) =>
{
    var debitAmount = (decimal?)f.Random.Decimal(1, 100).OrNull(f, 1.0f - statementconfig.DebitTransactionRatio);
    if (debitAmount.HasValue) // Is it a debit transaction?
    {
        x.CreditAmount = null;
        x.DebitAmount = debitAmount.Value;
        balance -= x.DebitAmount.Value;

        x.TransactionType = f.PickRandom(TransactionType.AllTransactionTypes
            .Where(tt => tt.Direction == TransactionDirection.Debit || tt.Direction == TransactionDirection.DebitOrCredit)
            .Select(tt => tt.Code));
    }
    else
    {
        var creditAmount = f.Random.Decimal(1, 100);
        x.DebitAmount = null;
        x.CreditAmount = creditAmount;

        balance += x.CreditAmount.Value;

        x.TransactionType = f.PickRandom(TransactionType.AllTransactionTypes
            .Where(tt => tt.Direction == TransactionDirection.Credit || tt.Direction == TransactionDirection.DebitOrCredit)
            .Select(tt => tt.Code));
    }

    x.Balance = balance;
});

A caveat with this approach is that I cannot use StrictMode anymore as it complains about those 3 fields having null values. It specifically mentions that in the exception. If you use Rules you’re on your own to ensure that all fields are populated properly.

Another drawback of setting multiple rules at once is that it can easily make the code harder to read. Fortunately for me, the author of the library Brian Chavez kindly reviewed the code and suggested some refactorings one of which proved it was still possible to use RuleFor method and strict mode. I’ve updated the final source code with these refactorings. So with individual rules the implementation looks like this:

.RuleFor(x => x.DebitAmount, f =>
{
    return (decimal?)f.Random.Decimal(1, 100).OrNull(f, 1.0f - statementconfig.DebitTransactionRatio);
})
.RuleFor(x => x.CreditAmount, (f, x) =>
{
    return x.IsCredit() ? (decimal?)f.Random.Decimal(1, 100) : null;
})
.RuleFor(x => x.TransactionType, (f, x) =>
{
    if (x.IsCredit())
    {
        return RandomTxCode(TransactionDirection.Credit); ;
    }
    else
    {
        return RandomTxCode(TransactionDirection.Debit);
    }

    string RandomTxCode(TransactionDirection direction)
    {
        return f.PickRandom(TransactionType.AllTransactionTypes
            .Where(tt => tt.Direction == direction || tt.Direction == TransactionDirection.DebitOrCredit)
            .Select(tt => tt.Code));
    }
})
.RuleFor(x => x.Balance, (f, x) =>
{
    if (x.IsCredit())
        balance += x.CreditAmount.Value;
    else
        balance -= x.DebitAmount.Value;

    return balance;
});

IsDebit and IsCredit methods referred to above are extension methods defined like this:

public static class Extensions
{
   public static bool IsCredit(this BankStatementLine bsl)
   {
      return bsl.DebitAmount is null;
   }
   public static bool IsDebit(this BankStatementLine bsl)
   {
      return !IsCredit(bsl);
   }
}

Random text

For the transaction description for now I’ll go with random Lorem Ipsum texts. Bogus has support for this too

.RuleFor(x => x.TransactionDescription, f => f.Lorem.Sentence(3))

I probably will need to use a fixed list of descriptions soon but for the time being it’s fine. Also as shown below it’s very easy to switch to that too.

Incremental values

Similar to balance being dependent on the previous values, transaction date is also dependent as it needs to go in an incremental fashion. I couldn’t find built-in support for this so implemented it using my own shared variable like this:

.RuleFor(x => x.TransactionDate, f =>
{
    lastDate = lastDate.AddDays(f.Random.Double(0, statementconfig.TransactionDateInterval));
    if (lastDate.Date > statementconfig.EndDate)
    {
        lastDate = statementconfig.EndDate;
    }
    return lastDate;
})

Putting It All Together

So let’s see the output with the help of another nice library called Console Tables

Source Code

Sample application can be found under blog/GeneratingTestDataWithBogus folder in the repository.

Resources

docker devops comments

In this post I’d like to cover a wider variery of Docker commands and features. My goal is to be able to use it as a quick reference while working with Docker without duplicating the official documentation. It consists of the notes I took at various times.

Docker commands

  • docker ps: Shows the list of running containers
  • docker ps -a: Shows all containers including the stopped ones
  • docker images: Shows the list of local images
  • docker inspect {container id}: Gets information about the container. It can be filtered by providing –format flag.

    Example:

      Command: docker inspect --format="" abcdabcdabcd
    
      Output: sha256:abcdabcdabcdabcdabcdabcd.....
    

    The data is in JSON format so it can be filtered by piping to jq.

    Example:

      docker inspect 7c1297c30159 | jq -r '.[0].Path'
    
  • docker rm {container id or name}: Deletes the container

    run command accepts –rm parameter which deletes the container right after it finishes.

    To delete all the containers the following command can be used:

      docker rm $(docker ps -aq) 
    

    where docker ps -aq only returns the container ids.

  • docker rmi {image id or name}: Seletes the image (provided that there are no containers created off that image)

    To delete all images the following command can be used:

      docker rmi $(docker images -q)
    

    This command is equivalent of

      docker image rm {image name or id}
    
  • docker build: Creates a new image from Dockerfile

    It is equivalent of

      docker image build ...
    

    It can be used to build images straight from GitHub:

    Example:

      docker build https://github.com/docker/rootfs.git#container:docker
    
  • docker history {image name}: Shows the history of an image

  • docker run -it {image name} bash: Runs a container in interactive mode with bash shell

    Same as:

      docker container run -it ...
    
  • docker start/stop: Starts/stops an existing container

    If a command is specified the first time the container was run, then it uses the same command when it’s started.

    Example:

      docker run -it {image name} bash
    

    This creates a container in interactive mode

    We can exit the container by running exit in the terminal (leaving the container running in the background) ???? (or Ctrl P+Q??). We can then stop the container:

      docker stop {container id}
    

    Stop command sends a signal to the process with pid = 1 in the container to stop gracefully. Docker gives the container 10 seconds to stop before it forcefully stops it itself.

    We can start it again with interactive flag to run bash again docker run -i {container id}

    This runs bash automatically because that was the command we used while creating the container.

  • docker run -p {host port}:{container port}: Forwards a host port to container

  • docker run -v {host path}:{container path}: Maps a path on host to a path on container.

    If the container path exists it will be overwritten by the contents of host path. In other words, the container will see the host files while accessing that path incestead of its local files.

  • docker diff: Lists the differences between the container and the base image. Shows the added, deleted and changed files.

  • docker commit: Creates a new image based on the container

  • docker logs {container id}: Shows the logs of the container.

    Docker logging mechanism listens to stdout and stderr. We can link other logs to stdout and stderr so that we can view them using docker logs {container id}

    Example:

      RUN ln -sf /dev/stdout /var/log/nginx/access.log && ln -sf /dev/stderr /var/log/nginx/error.log
    
  • docker network:

    docker network create {network name}: Creates a new network

    docker network ls: Lists all networks

    docker network inspect {network name}: Gets information of a specific network

    docker run –network {network name}: Creates a container inside the specified network

    Containers can address each other by name within the network.

  • docker container exec: Can be used to run commands in a running container. Also can be used to connect to the container by running a shell

    Example:

    docker container exec {container id} sh

It starts a new process inside the container

Docker Compose

A tool for defining and running multi-container Docker applications.

  • Instead of stand-alone containers, “services” are defined in “docker-compose.yml” file
  • docker-compose up command will create the entire application with all the components (containers, networks, volumes etc)
  • docker-compose config: It is used to verify a docker-compose yml file.
  • docker-compose up -d: Starts the services in detached mode. Uses the current directory name to prepend the objects (networks, volumes, containers etc) created.
  • docker-compose ps: Shows the full stack with names, commands, states and ports
  • docker-compose logs -f: Shows logs of the whole stack. tails and shows the updates.
  • docker-compose down: Stops and removes the containers and networks. Volumes are NOT deleted.

Environment variables can be used in docker-compose.yml files in ${VAR_NAME} format.

  • docker-compose -f {filename}: Used to specify non-default config files.
  • Multiple YML files can be used and can be extended using “extends:” key in the following format:

      extends: 
          file: original.yml
          service: service name
    

General Concepts

Container: Isolated area of an OS with resource usage limits applied. VMs virtualize hardware resources, containers virtualize OS resources (file system, network stack etc)

Namespaces: Isolation (like hypervisors)

Control groups: Grouping objects and setting limits. In Windows they are also called Job Objects.

Daemon: Includes REST API.

containerd: Handles container operations like start, stop etc.. on Windows, Compute Services, includes runc

runtime: OCI (Linux -> runc) runc creates the container than exits

containerd cannot create containers by itself. That capability is with the OCI layer.

Windows containers are 2 kinds:

  • Native containers
  • Hyper-V

  • Layering
    • Base Layer: At the bottom (OS Files & objects)
    • App Code: Built on top of base layer

    Each layer is located under /var/lib/docker/aufs/diff/{layer id}

    Layer id is different from the hash value of the layer

  • Images live in registries. Docker defaults to Docker Hub

  • Docker Trusted Registry comes with Docker Enterprise Edition

  • Official images can be addressed as docker.io/image name or just the image name

  • Naming convention: Registry / Repo / Image (Tag)

  • Registry: docker.io is default
  • Image: latest is default

  • Image layers are read-only. Containers have their read-write layers. Container adds a writable layer on top of the read-only layers from the image. If they need to change something on a read-only layer, they use copy-on-write operation and create a copy of the existing layer and apply the change on that new layer. Anytime container needs to change a file, creates a copy of the file in its own writable layer and changes it there.

  • Cmd vs entrypoint: runtime arguments overrides default arguments. Entrypoint runtime arguments are appended

  • Dockerfile commands
    • FROM is always the first instruction
    • Good practice to label with maintainer email address
    • RUN: Executes command and creates a layer
    • COPY: Copies files into image in a new layer
    • EXPOSE: Specifies which port to liste to. Doesn’t create a new layer. Adds metadata.
    • ENTRYPOINT: Specifies what to run when a container starts.
    • WORKDIR: Specifies the default directory. Doesn’t create layer.
  • Multi-stage builds
    • Helps to remove unnecessary packages / files from the final production image.
    • It can get specific files from intermediary builds

Notes

  • A container will run one process only
  • If the command is a background process, the container will stop right after it ran. It needs to be something like a shell and not like a web server.
  • As of v1.12 Docker runs in either single-image mode or swarm mode.
  • Recent Docker versions support logging drivers. They forward the logs to external systems such as dusk of, splunk etc (–log-driver and –log-opts)

Resources

dev asp.net, dotnet core comments

Developing projects is fun and a great way to learn new stuff. The things is everything moves so fast, you constantly need to maintain the projects as well. In that spirit, I decided to re-write my old AngularJS DomChk53.

First, I developed an API. This way I won’t have to maintain my AWS Lambda functions. Even though nobody uses them, still opening a public API poses risks and maintenance hassle. Now everybody can run their own system locally using their own AWS accounts.

API Implementation

Full source code of the API can be found in DomChk53 repo under aspnet-api folder. There’s not much benefit in covering the code line by line but I want to talk about some of the stuff that stood out in one way or the other:

CS5001: Program does not contain a static ‘Main’ method” error

I started this API with the intention of deploying it to Docker so right after I created a new project I tested Docker deployment but it kept giving the error above. After some searching I found that the Dockerfile was in the wrong folder! I moved it one level up as suggested in this answer and it worked like a charm!

Reading config values

I appreciated the ease of using configuration files. IConfiguration provider is provided out-of-the-box which handles appsettings.json config file.

So for example I was able to inject it easily:

public TldService(IConfiguration configuration, IFetchHtmlService fetchHtmlService)
{
    this.configuration = configuration;
    this.fetchHtmlService = fetchHtmlService;
}

and use it like this:

var awsDocumentationUrl = configuration["AWS.DocumentationURL"];

Using dependency injection

ASP.NET Core Web API has built-in dependency injection so when I needed to use it all I had to do was register my classes in ConfigureServices method:

// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);

    services.AddTransient<ITldService, TldService>();
    services.AddTransient<IFetchHtmlService, FetchHtmlService>();
}

Documentation

Adding Swagger documentation was a breeze. I just followed the steps in the documentation and now I have a pretty documentation. More on that in the usage section below.

AWS Implementation

Even though the code will run locally in a Docker container, you still need to setup an IAM user with the appropriate permissions. So make sure the policy below is attached to the user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "route53domains:CheckDomainAvailability",
            "Resource": "*"
        }
    ]
}

Usage

Nice thing about Swagger is it can used as a full-blown API client as well. So instead of developing a dummy client I just used Swagger UI for simple domain availability enquiries. This is until I develop a better frontend on my own. For now at least it works.

To use Swagger, simply run the Domchk53.API application and change the URL to https://localhost:{port number}/docs, port number being whatever your local web server is running on.

curl usage:

Windows:

curl -i -X POST -H "Content-Type: application/json" -d "{\"domainList\": [\"volkan\"], \"tldList\": [\"com\", \"net\"]}" https://localhost:5001/api/domain

Mac:

curl --insecure -X POST -H "Content-Type: application/json" -d '{"domainList": ["volkan"], "tldList": ["com", "net"]}' https://localhost:5001/api/domain

Example Test data:

This is just a quick reference to be used as template:

{
  "domainList": ["domain1", "domain2"],
  "tldList": ["com", "net"]
}
{ "domainList": ["volkan"], "tldList": ["com", "net"] }

References

dev javascript, tdd, unit testing comments

As applications get more complex, they come with more dependencies. Mocking helps us isolate the subject under test so that our unit tests can run any external dependencies. In this example, I’d like to show a simple mocking example using Jasmine and Karma in an Angular app.

Case study: Mocking a HTTP dependency

In this example the service I want to test is called TldService. It basically gets a raw HTML from a URL and extracts some data using regex.

It looks something like this

export class TldService {

  constructor(private fetchHtmlService: FetchHtmlService) { }

  getAllSupportedTlds(): string[] {
    const rawHtml = this.fetchHtmlService.getRawHtml();
    const supportedTlds = this.parseHtml(rawHtml);
    return supportedTlds;
  }

  private parseHtml(html: string): string[] {
    const pattern = /some-regex-pattern/g;
    const regEx = new RegExp(pattern);
    let tldList = [];

    let match = regEx.exec(html);
    console.log(match);

    while (match !== null) {
      tldList.push(match[1]);
      match = regEx.exec(html);
    }

    return tldList;
  }
}

This service depends on FetchHtmlService which does the getting HTML part. Nice thing about injecting this dependency is that we can replace it with a fake one while testing. This way we can test it without even having to implement the dependency.

import { TestBed } from '@angular/core/testing';
import { TldService } from './tld.service';
import { FetchHtmlService } from './fetch-html.service';

describe('TldService', () => {
  beforeEach(() => TestBed.configureTestingModule({

  }));

  it('should be created', () => {
    const service: TldService = TestBed.get(TldService);
    expect(service).toBeTruthy();
  });

  it('should parse html and extract TLD list', () => {
    const fetchHtmlService: FetchHtmlService = new FetchHtmlService();
    spyOn(fetchHtmlService, 'getRawHtml').and.returnValue('<a href="registrar-tld-list.html#academy">.academy</a>');
    const service: TldService = new TldService(fetchHtmlService);
    expect(service.getAllSupportedTlds().length).toBe(1);
  });
});

In the 2nd test above we are creating a new FetchHtmlService and overriding the getRawHtml function behaviour by using Jasmine’s spyOn method.

That’s it! Now we don’t need a network connection to make actual calls while testing our code. We can develop and test our service in isolation independent from the dependency!

References

dev devops, git comments

When you work on the same repository on multiple computers it’s easy to forget to push code and leave some changes local. To avoid that I decided to develop a tiny project that loops through all the folders and pulls remote changes then commits and pushes all the local changes.

The script itself it very simple. It first pulls the latest and pushes the local changes:

Function Sync-Git {
    git fetch --all
    git pull
    git add .
    git commit -m 'git-sync'
    git push
}

$rootFolder = '/rootFolder'

Write-Host "Changing directory to root folder: $rootFolder"
Set-Location $rootFolder
Get-Location | Write-Host

Get-ChildItem -Directory | ForEach-Object { 
    Set-Location -Path $_ 
    Write-Host $_
    Sync-Git
    Set-Location $rootFolder
}

Write-Host "Done"

Dockerfile is based Microsoft’s Powershell image and it only needs to install Git on it:

FROM mcr.microsoft.com/powershell

RUN apt-get update && apt-get install -y git
RUN git config --global user.email "you@example.com"

RUN echo "    IdentityFile ~/.ssh/id_rsa" >> /etc/ssh/ssh_config
RUN echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config

RUN mkdir /home/git-sync
WORKDIR /home/git-sync

COPY ./git-sync.ps1 .

ENTRYPOINT ["pwsh", "git-sync.ps1"]

Build it at the root folder:

docker build . -t {image name}

And run it as:

docker run --rm -d -v /host/path/to/root/git/folder:/rootFolder -v ~/.ssh:/root/.ssh:ro {image name}

Or pull my image from Docker Hub:

docker pull volkanx/git-sync

Issues

fatal: unable to auto-detect email address

GitHub didn’t allow me to push my changes without an email address so I had to add the command below to Dockerfile:

git config --global user.email "you@example.com"

It doesn’t look good maybe but I don’t care too much about email addresses in commits anyway so I’m fine with it.

“WARNING: UNPROTECTED PRIVATE KEY FILE!” error

This might also be an issue in which case the solution is simple (assuming the name of the private key file is id_rsa):

chmod 600 ~/.ssh/id_rsa 

Bad configuration option

Make sure to have the following options in the config file in your .ssh directory:

Host *
  StrictHostKeyChecking no
  AddKeysToAgent yes
  IgnoreUnknown UseKeychain
  UseKeychain yes
  IdentityFile ~/.ssh/id_rsa

Source Code

References

dev javascript, tdd, unit testing comments

There are so many names of tools and frameworks are flying around in JavaScript if you’re a beginner it’s easy to get lost.

First things first: Identify what’s what

Basically we need the following key components to write and run unit tests:

  • Assertions: We run a method in subject under test and compare the result with some known expected value
  • Mocking: We need to mock the external dependencies (network services, database, file system etc) so that we can reliably test subject under test and the tests can run everywhere and without requiring any credentials or permissions.
  • Test runner: Our test is some block of code. Something needs to make sense of the unit tests we wrote, execute them and show the results (fail/pass)

Choosing the weapons: Mocha and Chai

In this example I’m going to use Mocha as my test runner and Chai as my assertion library.

They can simply be installed via NPM:

npm install --save-dev chai

To be able use Mocha in the Integrated Terminal inside Visual Studio we have to install Mocha as a global package:

npm install -g mocha

To put these to test I created this simple Maths service that calculates the factorial of an integer.

module.exports = class MathsService {
    factorial(num) {
        if (num < 0) 
            return -1;
        else if (num == 0) 
            return 1;
        else {
            return (num * this.factorial(num - 1));
        }
    }
};

And my unit test to test this method looks like this:

const MathsService = require('../maths-service.js');
const chai = require('chai');
const expect = chai.expect; 

describe('Factorial', function() {
    it('factorial should return correct value', function() {
        const mathsService = new MathsService();
    
        expect(mathsService.factorial(-1)).to.equal(-1);
        expect(mathsService.factorial(0)).to.equal(1);
        expect(mathsService.factorial(1)).to.equal(1);
        expect(mathsService.factorial(3)).to.equal(6);
        expect(mathsService.factorial(5)).to.equal(120);
    });
});

Chai supports 3 styles:

  • Should
  • Expect
  • Assert

They all server the same purpose so it’s just a matter of taste at the end of the day. For this demo I used expect style.

By default Mocha searches test folder. So you can run mocha without parameters if you put your tests under test folder or you can specify the folder. For example, in the screenshot below I moved the test to the root folder and entered “mocha .” and that worked fine as well.

Conclusion

In this post I want to show the very basics of unit testing in JavaScript just enough to see a passing test. In future posts I’ll build on this and explore other frameworks and other aspects of TDD.

References

linux macbook, ubuntu comments

This is my old MacBook Mid 2009. It was sitting in the closet for so long I thought I could invent some purpose and use it before it finally dies. Looks like latest macOS version don’t even support this device so I thought maybe I could use it as a learning tool for Ubuntu.

Since SSDs are so cheap these days I just decided to keep the old macOS drive as backup and install Ubuntu on one of these babies:

I think around £18 is a small price to pay for a brand new SSD so went with it.

Installing Ubuntu

  1. Download the ISO here

  2. Burn the ISO to a USB Drive. Instead of installing extra software I followed this guide: Making a Kali Bootable USB Drive. It’s for Kali Linux but this bit works for burning any Linux distro.
  3. For the rest follow the steps here starting with Step 9

And after the installation this is what my MacBook looks like:

References

hobby productivity, apple watch, alexa, philips hue, iot comments

My #1 rule for productivity is “No Snoozing!”. If you snooze, it means you are late for everything you planned to do and that is a terrible way to start your day. This post is about a few tools, tips and techniques I use to prevent snoozing.

Tip #1: Sleep well

This is generally easier said done but there’s no way around it. You MUST get enough sleep. Otherwise, sooner or later your will power will get weaker and weaker. Eventually you’ll succumb to the sweet temptation of more sleep.

Tip #2: Place the alarm away from your bed

This way when your alarm (most likely your phone) goes off you have to make a deliberate attempt to get up and turn it off. If you have to get out of the bed you’re more likely to not to get back to it straight away.

Tip #3: Use multiple alarms with different sounds

It gets easier to wake up if you can surprise yourself! Human beings are so good at adapting to every condition we very easily start getting used and ignore the same alarm sound going off at the exact same time every day. I find it useful to change the alarm times and sounds every now and then.

Tip #4: Use Apple Watch

After Apple Watch Series 4 was released I got myself one.

I’m not sure if it’s worth the cost but when it comes to waking up a little vibration on your wrist can do miracles apparently!

When you have it pair with your iPhone, by default you can stop the alarms from your watch. This may be a nice convenience feature in some cases, but when it comes to waking up we are trying to make it as hard as possible for ourselves to turn the alarms off.

My trick is:

First I disable “Push alerts from iPhone” in the Watch app.

Then I create a separate alarm on watch for the same time.

This way I get 2 alarms at the same time. It’s easy to stop the watch as it’s within my arm’s reach. While the haptic feedback of the watch wakes me up the alarm on the phone also goes off. Now I have to physically get out of the bed to stop that one as well.

Tip #5: Use Alexa

Another gizmo to set an alarm is Alexa but you can do much more than just that with Routines.

Tip #5.1: Play a playlist

This tip requires Spotify Premium subscription.

First, create yourself a nice, loud and heavy playlist of “waking up” music. I prefer energetic Heavy Metal songs from Lamb of God and Slayer. The trick here is to play a random song every morning. Similar to Tip #3, the same song every morning becomes very boring very quickly. But having a random one keeps you surprised every morning. I use this command to play my playlist in shuffle mode:

Shuffle Playlist '{Playlist name}'

Tip #5.2: Turn the lights on

A good sleep tip is to keep your bedroom as dark as possible. That’s why I have all black curtains in my room and it’s quite dark. The downside is it’s so good for sleep it makes waking up even harder!

That’s why I bought myself a Philips Hue smart bulb and as part of my waking up routine Alexa turns it on along with playing the Spotify playlist.

This is what my routine looks like:

Conclusion

For me snoozing is a cardinal sin so I’m always on the lookout for improving my arsenal to fight against snoozing. Hope you find something useful in this post too. If you have tips on your own feel free to leave a comment.

Resources

docker devops, github, backup comments

A while back I created a PowerShell script to backup my GitHub account and blogged about it here. It’s working fine but it requires some manual installation and setup and I didn’t want to do that every time I needed to deploy it to a new machine. That’s why I decided to Dockerize my script so that everything required can come in an image pre-installed.

TL;DR:

  • There’s a Powershell script that allows you to back up your GitHub account including private repositories here: Source Code
  • There’s a Docker image that encapsulates the same script which can be found here: Docker Image
  • Below are my notes on Docker that I’ve been taking while working on Docker-related projects. Proceed if are interested in some random tidbits about Docker.

Lessons Learned on Docker

  • Shortcut to leave container without stopping it: Ctrl P + Ctrl Q

  • Build a new image from the contents of the current folder

      docker image build -t {image_name} .
    
  • Every RUN command creates a new layer in the image. Image layers are read-only.

  • Connect to an already running container:

      docker attach {container id or name}
    
  • Save a running container as an image:

      docker commit -p 78727078a04b container1
    
  • Remove image

      docker rmi {image name}
    

    This requires the image doesn’t have any containers created off of it. To delete multiple images based on name:

      docker rmi $(docker images |grep 'imagename')
    
  • List running containers

      docker container ls
    
  • To list all containers including the stopped ones:

      docker ps -a
    
  • Delete all stopped containers

      docker container prune
    
  • To delete all unused containers, images, volumes, networks:

      docker system prune
    
  • Copy files to and from a container

      docker cp foo.txt mycontainer:/foo.txt
      docker cp mycontainer:/foo.txt foo.txt
    
  • To overwrite the entrypoint and get an interactive shell

      docker run -it --entrypoint "/bin/sh" {image name}
    
  • Tip to quickly operate on images/containers: Just enter the first few letters of the image/container. For example if your docker ps -a returns something like this

      1184d20ee824        b2789ef1b26a                  "/bin/sh -c 'ssh-k..."   18 hours ago        Exited (1) 46 seconds ago                         happy_saha
      7823f76352e3        github-backup-04              "/bin/sh"                18 hours ago        Exited (255) 21 minutes ago                       objective_thompson
    

    you can start the first container by entering

      docker start 11
    

    This is of course provided that there aren’t any other containers whose ID start with 11. So no need to enter the full ID as long as the beginning is unique.

  • To get detailed info about an object

      docker inspect {object id}
    

    This returns all the details. If you interested in specific details you can tailor the output by using –format option. For example the following only returns the LogPath for the container:

      docker inspect --format ''  {container id}
    
  • To get the logs of the container:

      docker logs --details --timestamps  {container id}
    
  • Docker for Mac actually runs inside a Linux VM. All docker data is stored inside a file called Docker.qcow2. The paths that are returned are relative paths in this VM. For instance if you inspect the LogPath of a container it would look something like

      /var/lib/docker/containers/{container-id}/{container-id}-json.log
    

    But if you check your host machine, there is no /var/lib/docker folder.

    In Docker preferences it shows where the disk image is located:

    This command let me to go into VM

      screen  ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
    

    Then I was able to navigate to /var/lib/docker and peek inside the volumes where the data is persisted. This virtualization does not exist in Linux and you can view everything on the host machine straight away.

  • Enter a running container (Docker 1.3+):

      docker exec -it {container-id} bash
    
  • Copy an image from one host to another: Export vs Save

    Save, saves a non-running container image to a file:

      docker save -o <save image to path> <image name>
      docker load -i <path to image tar file>
    

    Export, saves a container’s running or paused instance to a file

      docker export {container-id} | gzip > {tar file path}
      docker import {tar file path}
    

Resources

linux ec2, ebs comments

I recently decided to purchase a reserved t3.nano instance to run some Docker containers and for general testing purposes. In addition to the default volume I decided to add a new one to separate my files from the OS. It required a few steps to get everything in place so I decided to post this mostly for future reference!

Attach a volume during creation

First I added a new volume to the instance while creating it.

Connecting to instance

Now we have to connect to the instance to format the new volume. To achieve that we must have access to the private we generated while we created the instance. So to SSH into the machine we run this command:

ssh -i {/Path/To/Key/file_name.pem} ec2-user@{public DNS name of the instance}

Format the volume

I found some AWS documentation to achieve this which was very useful: Making an Amazon EBS Volume Available for Use on Linux

No need to repeat every command in that documentation. It’s a simple step-by-step guide. Just follow it and you have a volume in use which is also mounted at start up.

Install and configure Docker

Installing Docker is as simple as running this:

sudo yum update -y
sudo yum install -y docker

To be able to use Docker without sudoing everything ad ec2-user to docker group:

sudo usermod -aG docker ec2-user

We need to make sure that Docker daemon starts on reboot too. To achieve this run this:

sudo systemctl enable docker

Copy files to the instance

To copy some files to the new instance I used SCP command:

sudo scp -i {/Path/To/Key/file_name.pem} -r {/Path/To/Local/Folder/} ec2-user@{public DNS name of the instance}:/Remote/Folder

The issue was ec2-user didn’t initially have access to write on the remote folder. In that case you can run the following command to have access:

setfacl -m u:ec2-user:rwx /Remote/Folder

Resources

aws cloudtrail, security, audit comments

Another important service under Management & Governance category is CloudTrail.

A nice thing about this service is that it’s already enabled by default with a limited capacity. You can view all AWS API calls made in your account within the last 90 days. This is completely free and enabled out of the box.

To get more out of CloudTrail we need to create trails

Inside Trails

A couple of options about trails:

  • It can apply to all regions
  • It can apply to all accounts in an organization
  • It can record all management events
  • It can record data events from S3 and Lambda functions

Just be mindful about the possible extra charges when you log every event for all organization accounts:

Testing the logs

I created a trail that logs all events for all organization accounts. I created an IAM user in another account in the organization. In the event history of the local account events look like this:

These events now can be tracked from master account as well. In the S3 bucket these events are organized by account id and date and they are stored in JSON format:

Conclusion

Having a central storage of all events in across all regions and accounts is a great tool to have. Having the raw data is a good start but making sense of that data is even more important. I’ll keep on exploring CloudTrail and getting more out of it to harden my accounts.

Resources

aws security, aws config, audit comments

In my previos blog post I talked about creating an IAM admin user and using that instead of root user all the time. Applying such best practices is a good idea which also begs the question: How can I enforce these rules?

AWS Config

The official description of the service is: “AWS Config is a service that enables you to assess, audit, and evaluate the configurations of your AWS resources”

What this means is you select some pre-defined rules or implement your custom rules and AWS config constantly runs checks against your resources and notify you if you have non-compliant resources.

Since currently I’m interested in hardening IAM users in the next example I’m going to use an IAM check

Use case: Enforcing MFA

As of this writing there are 80 managed config rules. To enforce MFA, I simply searched MFA in the “Add rule” anf got 5 matches of which I selected only 3:

After I accepted the default settings it was able to identify my IAM user without MFA:

And it comes with a nice little dashboard that shows all your non-compliant resources:

It also supports notifications via SNS. It creates a topic and all you have to do is subscribe to that via an email address and after confirming your address you can start receiving emails.

I was only expecting to get emails about non-compliant resources but it’s bit noisy as it sends emails with subjects “Configuration History Delivery Completed” or “Configuration Snapshot Delivery Started” which didn’t mean much to me.

Pricing

I think the price is exteremely high. The details can be found on their pricing page but in a nutshell a single rule costs $2/month. So for the above example I paid $6 which is a lot of money in terms of resources used.

Conclusion

I like the idea of having an auditing system with notifications but for this price I don’t think I will use it.

I will keep on exploring though as I’m keen on implementing my custom rules with AWS config and also implementing them without AWS config and see if this service adds any benefit over having scheduled Lambda functions.

Resources

aws iam, security, best practices comments

When you create a new AWS account you are the root user who has unlimited access to everything. Using such a powerful user as root on a day-to-day basis is not such a good idea because if it gets compromised you may not have a way to override and/or undo the changes done by the hacker.

Using IAM user instead of root account

Instead, suggested best practice is to create an admin-level IAM account and use it for normal operations. At first I was hesitant to adopt this practice. I didn’t see the point and thought attaching AdministratorAccess policy awould make the use as powerful as root. But there’s a whole list of things that even the most powerful IAM user cannot do. Here’s the list: AWS Tasks That Require AWS Account Root User Credentials

So as you can see root user has important privileges such as closing the account and changing billing information. Even if your account gets compromised and some mailicous person gains access using an IAM account, you can still log in as root and take necessary action.

In a nutshell, based on AWS documentation the following practices are recommended:

  • Use the root user only to create your first IAM user
  • Create an IAM user for yourself as well, give that user administrative permissions, and use that IAM user for all your work.

In addition to Eric Hammond suggests in his blog to delete the root account password as well and use Forgot Password option to create a new one when needed. I keep my passwords in a password manager so if that application is compromised, the hacker can reset my password as well so I don’t follow this practice but it might come in handy if you have to write your password down somewhere.

Templated IAM user creation

It’s a good practice to create an IAM user right after you create your AWS account. It’s even a better practice to automate this process. To achieve this I created a CloudFormation template. The YAML template below does the following:

  • Creates an IAM group named administrators
  • Creates a user named admin
  • Attaches AdministratorAccess policy to the group
  • Forces the user to change their password first time they log in (by attaching IAMUserChangePassword policy to the user)
Resources:
  AdministratorsGroup:
    Type: AWS::IAM::Group
    Properties:
      GroupName: "administrators"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess
      Path: /

  AdminUser:
    Type: AWS::IAM::User
    Properties: 
      Groups:
        - !Ref AdministratorsGroup
      LoginProfile:
        Password: "CHANGE_THIS"
        PasswordResetRequired: true
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/IAMUserChangePassword
      Path: /
      UserName: "admin"

Resources