# Density-based Clustering with DBSCAN Algorithm

My previous post was about grid-based clustering. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) takes another approach called density-based clustering. It grows regions with high density (above threshold provided) into clusters and discovers clusters of arbitrary shape.

### DBSCAN Implementation

First off the data is loaded and a distance matrix is calculated based on the data points.

The algorithm visits every data point and finds its neighbours. If the neighbours are dense enough than the cluster is expanded to include those points as well.

So data points that are close enough to each other are included in the same cluster.

Joining dense clusters is a similar approach taken in grid clustering. The difference is this way the clusters can have arbitrary shapes.

# Ingress

I’ve been playing this game for about 6 months. It’s an interesting game in a number of ways.

### Ingress Basics

I discussed the basics of the game in my previous post. If you want to find out more about the rules and game play you can check out that post here. For me the definite guide for the game is a site called decodeingress.me. You can find the latest news and updates about the game. It is updated quite frequently so it pays off to keep an eye on the latest changes in the game. Anyway, what I want to log in this post is mainly my experiences.

Feeling proud of achievements in a virtual world is a bit silly but I guess it goes for all sorts of games. Here’s a snapshot of the fields I created a while back:

In theory the goal is to create and try to maintain those fields so that you capture “mind units”. The team who has more mind units wins (actually the game never ends so I guess there are no winners or losers. The worst case of scenario is your team falls behind). Once you start playing it’s all about levelling up but after a certain level you seek more. Here’s my view of pros and cons of the game:

### Pros

• Exercise: This is the main reason I started the game. It motivates you walk more. I stopped taking the bus from town centre and started walking home because there are some portals that need to be captured along the way!</li>
• Get to know the city better: Before this game, I knew where my house was, where the nearest train station was and where the nearest Sainsbury’s store was. Now I know all the churches, parks and many other public buildings around my neighbourhood and around the city.
• New friends: You can play the game alone with no problems and can have fun at first. But after a while it becomes a bit boring because you do the same activities repeatedly. By joining the local community you get to know the people playing around you.

### Cons

• Repetition: The portals have a mitigation cap meaning that you cannot keep enforcing the portals. So it can be taken down by a single player.After some time it becomes quite repetitive. You capture the portals and create fields. A few hours later someone blows them up and the next morning you have to do the same thing all over again. This might get the fun away especially If you are playing alone.
• Can be time-consuming: Because of the above fact, portals require maintenance especially in crowded areas. If you become obsessed about keeping the portals yours you might end up spending a lot of time.

For me pros of the game far outweigh the cons. I can control the time I spend on the game and play with other people so it becomes a social event.

### Some Tips

Just off the top of my head here are a few tips for beginners:

• Deploy resonators as far as possible from the center so that they don’t all get hit hard at the same time.
• When firing XM bursters, long press fire button and release it when XM is accumulated in the center. It allows you to get bonus firepower up to %20
• Links make portals stronger so try to create as many links and fields you can.

# Grid Clustering with CLIQUE Algorithm

CLIQUE (CLustering In QUEst) algorithm is a grid-based clustering algorithm that partitions each dimension of the dataset as a grid.

### CLIQUE Implementation

The algorithm starts by assigning each data point to the closest grid in the matrix.

In order for a grid cell to be considered as “dense” it has to have data points more than or equal to the threshold. In this sample the threshold was provided as 2. So after the adjacent dense cells are joined together we get 3 clusters:

# Rough set implementation

A rough set is an approximation of a set that gives the lower and upper borders of the set.

### Rough set implementation

The sample implementation starts off with reading the data set and the configuration file. The object and attribute index lists are acquired from the configuration. The items that have indices specified in the object list are considered as the set of objects and the rest of the items are the complement of the data set.

If a given index in attribute index list equal in both object list and complementary object list are not equal the object is considered to belong to the negative border, if not it belongs to the positive border.

# Genetic Algorithm Implementation

Genetic algorithm sounds fascinating to me as in a way it mimics evolution. It’s quite easy to implement and understand.

### Genetic Algorithm

The algorithm has 3 stages:

• Selection: Every generation is subjected to a selection process which eliminates the unfit ones.
• Cross-over: This happens randomly among the selected generation and some genomes swap genes.
• Mutation: Some genes are mutated randomly based on a given threshold value for mutation.

### Implementation

The implementation starts with generating random bit sequences to represent genomes. The length of the genome and the number of genomes in a generation are specified in the configuration.

First generation is calculated randomly and their fitness is calculated based on the fitness function the system uses. After the groundwork has been done we have a genome pool each with their own fitness values and selection ranges. The width of the selection range is determined by the fitness of the genome. So that the larger the range the more likely they are selected for the next generation.

In selection process, a random value is generated and the genome whose selection range includes that value is selected. Same genome can be selected more than once.

After the next generation is selected and their new fitness values are calculated cross-over starts. In cross-over 2 random genomes are picked and they swap genes. The point in the genome string is specified in the configuration (3 in this example)

The final phase of the algorithm is mutation. In this step a number of genomes are picked and some bits is flipped randomly.

The most important thing about this algorithm is having a good fitness function that can accurately compute the fitness values for a given feature.

# AGNES Implementation

Another data mining algorithm: AGNES (Agglomerative Nesting)

### AGNES Algorithm

AGNES takes a different approach than k-means which was the subject of my previous post. Initially it considers all data points as a separate cluster.

Then finds the minimum distance between clusters and merges the closest clusters:

The resulting cluster is added to the all cluster list the merged clusters are removed as they are no longer valid.

# k-means Implementation

I’m keeping on reviving my old projects. This is the second data mining algorithm implementation. It is another clustering algorithm called k-means.

### k-means Algorithm

Algorithm groups and creates k clusters from n data points. First the cluster centres are picked randomly from the data points. Then the entire dataset is iterated and all points are assigned to their closest cluster. Closest cluster is determined by measuring the distance of the data point to the centroid of the clusters. This process is repeated until there is no change in the dataset and all points are assigned to the closest ones.

### Implementation

The project contains 6 libraries:

• VP.KMeans.Core: Core library including the algorithm implementation
• VP.KMeansClient.GUI: User interface for entering the parameters and plotting the clusters
• VP.KMeansClient.Console: Console user interface. No fancy plots, just an output file is generated
• VP.KMeans.DataGenerator.Core: Library to generate test data
• VP.KMeans.DataGenerator.Console: Console application to feed the core library to generate test data
• CPI.Plot3D: External library to plot the results

# An Apriori implementation

I was digging through some old projects and found out the Data Mining and Machine Learning projects I implemented. Instead of letting them gather dust(!) in the hard disk I decided to review and publish the source code. This will also give me a chance to revise my data mining knowledge. Let’s start with Apriori algorithm.

### Apriori Algorithm

It is an algorithm to determine to most frequent items in a given collection. The term “frequent” is defined by a given threshold or “minimum support count”.

### Implementation

The project is implemented in Java. It uses a Hashtable to keep count of itemsets in the database. The algorithm starts with finding the most frequent 1-item sets first. Then using the previous most frequent item list of item size k, it generates candidate item list of size k+1.

For example, for the sample data below and a threshold 4, 1,2,3,4,5 are all frequent 1-itemsets. From this list we generate a 2-item candidate list (all 10 combinations) and check if the subsets are also frequent. For 1-itemsets they are all frequent so they all pass pruning. Then we count the occurences of these candidates. Only 7 of them are equal to or greater than the threshold. From this list we generate our 3-item candidates. Such as 1,2 and 1,4 combined to generate 1,2,4. Then we count the occurrence of 3-itemsets and prune the results by checking all of its subsets.

The idea of pruning is if the there are some infrequent subsets inside an itemset then the larger set cannot be frequent so it is removed from the candidate list. (1,2,3 is a candidate but as 2,3 is not a frequent 2-itemset it is removed from the candidate list) This process helps improve the performance of the algorithm as it reduced the number of iterations.

1,3,4,5

1,2,4,5

1,2,4

1,3

3

1,5

1

3

1,3,5

4

1,2,4

2

1,2

3,4

3,5

1,3,4

2

3,5

1,2,3,4

Results:

### Source Code

I created a public repository for the source code. You can check it out here if it tickles your fancy!

### Resources

I bought this about 2 months ago. I wore it every single day since then and I just loved it! It is basically a motion sensor that detects and keeps track of your daily movements. You can set your own daily goals steps you walked, distance you took or calories you burnt.

I can keep track of distance by using my Garmin ForeRunner 10 (which I reviewed here) but this one is easier to use because it does everything in the background. Garmin takes a few minutes to start because it needs to find your GPS coordinates but that’s not the case for Flex.

Flex lets you keep track of distance, active minutes, calories and steps.

Also you can log your weight, other exercises and food intake so that you can calculate the net calories throughout a time period.

Another great feature about it is tracking your sleep quality. You can use this data in conjunction with your daily activities.

And here is my favourite feature: Alarm! It turns out if something on your wrist start to vibrate you wake up. Instantly! Of course I keep my phone’s alarm still running as a fallback method but this one works pretty good.

### Conclusion

I bought it for £68 and as of now it is listed as £83 on Amazon. Apparently the price fluctuates a bit but I think £70 – 80 price range is good for this product. I charge once every 3 days or so. Other than that I completely forget about it while it does its job in the background. It motivates you to reach your goals and be more active in general and the silent alarm is absolutely fantastic. I’d recommend this to anyone who would like to have more exercise.

# Lamb of God, O2 Brixton Academy, 18 January 2014

It’s been 5 months since I’ve gone to a metal show. The last one was Bloodstock Festival and Lamb of God was headlining that. The playlist on this show had no surprises (Just the way we like it). O2 Academy Brixton twitter account kindly tweeted the playlist:

### “Open up the dance floor!”

Apparently this was a sold out show. It was quite crowded and people went nuts the second LoG hit the stage. Normally there is a small area for moshpit in front of the stage right behind the first 2-3 rows or people. But from where I stood it looked like the half of the floor was a giant moshpit area. Check this short clip to get a sense of the atmosphere (sound quality is awful so make sure you don’t start it at full volume).

### Notes

• Number of girls seemed to be a lot more than a usual metal concert. Number of girls in the moshpit was definitely a lot more than a usual metal concert!
• Guitarist Mark Morton didn’t take the stage. At the beginning of the show Randy introduced Paul Waggoner as his temporary replacement and explained Mark’s absence as “He had to attend to a family business and family always comes first in this band”
• Beer prices were ridiculous (or maybe I should say tragic): £4.75 for a Carlsberg. I only had two. The feeling of being ripped off ruins the fun of it but mainly the long queues in the front of the bars were a huge turn off.

# More notes on productivity

I’ll keep rambling about my opinions, observations and experiences about productivity. Here goes:

### 01. Don’t try to cut back sleep to gain time

We all fee like we need more time to accomplish things but sleep is not the right item for savings. I tried it in the past. Tried to get away with sleeping 3-4 hours every night. At first it felt like the days were longer but towards the end of the week I was feeling so tired that I couldn’t perform well during the day. So I was up for more hours but my productivity had reduced significantly. I strongly suggest to have a good night’s sleep around 6 – 8 hours.

### 02. Two pomodoros of reading a day makes you (feel) smarter

In my last post I talked a bit about the Pomodoro technique. I love using it to divide small tasks into short work units. The idea of spending two pomodoros for reading is similar to saving money in a piggy bank. It’s easy to use 50 minutes a day for reading no matter how busy your day is. You wouldn’t notice and it wouldn’t become a huge burden once you make it a habit. But over time it would amount to a bunch finished books that would be gathering dust on your desk otherwise (or iPad or Kindle or phablet or.. anyway, you get the point)

### 03. Review your processes often

Setting goals for the day and week is a good practice but to make it work you have to review the outcomes periodically. In the excellent book called 30 days of Getting Results, J.D.Meier names is as “Friday Reflection”. Every week the status of the tasks that were targeted to be completed must be reviewed and more importantly the process must be updated the following week based on lessons learned from the previous week.

### 04. Keep learning

Everyday we are wasting a tremendous amount of time doing chores, commuting etc. At this day and age we have access to so many different means of accessing technology. We have small computers in our podcasts that are almost always connected to Internet. It’s a great chance we have to make use of the idle time by watching a PluralSight video for example. There are similar online courses of course but it just happens to be my favourite with apps for all mobile platforms. Another way it listening to a podcast. There are a lot of podcasts about technology and development. A few of my favourites are:

• .NET Rocks
• Hanselminutes
• Yet Another Podcast by Jesse Liberty

I’m planning to compile a list of my favourite shows so I hope this list will look more comprehensive.

The reason I added this item is using these learning mechanisms help you keep sharp and on track. Generally it’s not possible to really learn something by just listening to it but the idea here is to keep an eye of the latest to research about it.

Trying to do multiple things at the same time creates an illusion of going fast and accomplishing a lot but I don’t believe that’s the case. Of course I’m not talking about doing something meaningful along with a chore. Like as in #4, it’s perfectly acceptable to watch a technical video on your way to work. The type of multitasking I’m against is doing it while coding for example. My idea is divide such tasks into pomodoros, focus on one thing at a time and nothing else. This would improve the quality of that unit of work dramatically and will eventually lead to accomplishing more.

# Book Review: Framework Design Guidelines

### Framework Design Guidelines

Having a common framework is quite important to reduce code reuse. But designing that framework properly and consistently is a huge challenge. Even though we are living in a RESTful world now, I think having a framework or a set of common libraries for personal or commercial projects is still relevant. A well-designed well-tested framework would significantly improve any application built on top of it.

I had referred to this book partially before but this time decided to read it from cover to cover and make sure I “digest” all. It contains countless gems that every developer should know. Anyone developing even small libraries can benefit from this book a lot. You don’t need to design the .NET framework (like the authors). Also it comes with a DVD full of presentations of authors.

### Some notes

The book is full of gems and very useful tips. Here are just a few:

• Keep it simple: “You can always add, you cannot ever remove”
• There is no perfect design: You always have to make some sacrifices and consider the trade-offs
• Well-designed frameworks are consistent
• Scenario-driven design: Imagine scenarios to visualize the real-world use of the API When in doubt leave the feature out, add it later. Conduct usability studies to get developers’ opinions
• Keep type initialization as simple as possible. It lowers the barrier of entry.
• Throw exceptions to communicate the proper usage of API. It makes the API self-documenting and supports the learning-by-doing approach.
• Going overboard with abstractions may deteriorate the performance and usability of the framework.

Even though it’s been a few years since this books was released it is still a very helpful resource.

# Ingressercise

Yet another horrible term coined by inaptly fusion of two words! I was going to call dibs on it but a quick Google search showed other people used it before. Someone even registered the .com domain for it! Anyway, obviously the term is an abomination of words “Ingress” and “Exercise”.

### What is Ingress

Ingress is an augmented reality game developed by Google (Well, actually by a company called Niantic Labs that is owned by Google). To play the game you need to install the free app on an Android device. (I saw iOS support is coming in 2014 here)

For me the beauty of this game is having exercise without even noticing it. Even having a desire to do it. Last Sunday, for example, I walked around at least 30 minutes hacking a bunch of portals. To an outsider I might have looked like I was enjoying a nice walk in the park but actually I was trying to keep to world free from the unreliable Shapers! I didn’t even notice I walked for that long.

### The story

An alien race called Shapers use Portals to teleport to earth. There are two teams: The Resistance and The Enlightened. The resistance are opposed to Shapers’ whereas Enlightened believe they should be embraced.

### The goal

The ultimate goal is to control as many portals as you can. You can create links between the portals your team controls and the triangular area between the portals forming the link is called “control field” and any player located inside this area is considered to be under your team’s control. They are called “mind units”. That’s essentially how the current control rates on the intel page is calculated. (As a proud Resistance agent it just saddened me to see we are down to 43% of global control)

### The gameplay

Portals can be found anywhere. They can be controlled by either of the teams or they can be unclaimed. To claim a portal at least one resonator must be placed. To capture a portal controlled by the opponent team their resonators must be destroyed by firing XMP. You progress to higher levels by collecting AP (Action Point). After you start hacking the portals you will notice some items in your inventory.

### Basic strategy

Okay here’s a guide to the absolute beginner: Just walk or run around and hack every portal you see. Don’t attack powerful enemy portals, you will just end up using all your weapons and energy. If you find unclaimed portals (the gray ones) jump on them instantly. You will gain a lot of APs by deploying resonators and linking them to other portals. Before attacking enemy portals check out the deployment and mod status. Once you have enough weapons in your arsenal and move up a few levels try to find fairly weak portals and zap the resonators. You can earn some sweet APs by creating a control field and don’t let enemy portals surround you because links cannot intersect and if they have a link going through your portals you cannot establish a link between them. So in that sense best defense is offense. Just pick your fights carefully.

### Glossary

• Portal: Places where Shapers can teleport to earth. The main goal is finding and hacking them.

• Hacking: You hack the portals to get Action Points and add items to your inventory. You can hack any portal but you only get AP by hacking other people’s portals. You get more items by hacking your own portals.

• Resonator: Every portal has 8 resonators. The resonators deplete over time and eventually the portal becomes unclaimed unless they are recharged. This helps your team to keep control of the portal for a longer period of time.

• Portal Key: When you hack a portal you might get portal key (there is 15% chance you will). They are very important because you need them to create links between portals.

• Mod: There are various mod items that let you upgrade a portal. Every portal have 4 mod slots. The mods are portal shields (to defends portals against XMP attacks), force amplifier (increases force of the portal), link amplifier (increases the portal link range), multi-hack (increases the number of times a player can hack a portal), heat sink (decreases the duration between hacks) and turret (increases the portal attack rate).

• XMP Burster: Weapons to fire XMP on enemy portals. You can destroy resonators, links and control fields with them

• Power Cube: Used to store XM so you can recharge your XM reserves.

### Conclusion

It’s an enjoyable game for me but I can see myself getting bored of it pretty soon. But I have a great motivation to keep playing. It makes exercise just go by. I cannot wait for spring when I can start using it while running which would allow me to hack so many more portals that I can by walking.

# Moving S3 bucket to another account

Amazon Web Services, Cloud Computing comments

I have two AWS accounts and I made a mistake by using mixing the usage of services. More specifically, I hosted an application on one account but used S3 on the other. So I perpetually had to switch back and forth between accounts to access all services I used. First I thought fixing it would be a non-issue but it proved to be a rather daunting task.

### Bucket naming in S3

In S3, all buckets must have unique names. You cannot use a name if it’s already taken (much like domain names). Since I was using the bucket already, creating the same bucket in the other account and copying its contents was not an option. The second idea was to create the target bucket with a temporary name, copy the contents, delete the first one and rename the target bucket. Well, guess what? You cannot rename a bucket either! Another problem is when you delete a bucket you can create a new one with the same name right away. I’m guessing this is because of the redundancy S3 provides. It takes time to propagate the operation to all the nodes. My tests showed that I could re-create the bucket in the other account only after 45 – 50 minutes.

### To develop or not to develop

My initial instinct was to develop a tool to handle this operation but I decided to check out the what’s already available. I was occasionally using Cloudberry but wanted to check its competitors hoping one of the tools would support the functionality I need.

#### Cloudberry Explorer for Amazon S3

I find this tool quite handy. It has lots of functions and a nice intuitive. It comes in flavours: Free and Pro version. I used free version so far and unless you are a big enterprise it seems sufficient. It allows you to manage multiple AWS account. It allows copying objects among accounts but not moving a bucket (actually after my findings above I wasn’t very hopeful anyway)

As you can see in the menu bar, it supports lots of features.

#### S3 Browser

This one comes with a free version too as well as a paid version. The free version is limited to 2 accounts and you can only see one account at a time.

I tried to copy a file and paste to another but it got an Access Denied error. I could do the same thing with Cloudberry in seconds by simply dragging and dropping to the target folder.

#### Bucket Explorer

Third candidate only has a 30-day trial version as opposed to a free one. The second I installed it I knew it was a loser for me because it doesn’t support multiple accounts. Also as you can see below the UI is hideous so this is not a tool for me.

### ..and the winner is

Cloudberry won by a landslide! It looks much more superior than both of the other tools combined.

### Operation Bucket Migration

So I backed up everything locally and deleted the source bucket so that I could create the same one in the new account. After periodically checking for 45 minutes I finally created the bucket and uploaded the files. Set the permissions and the operation was completed without any casualties.. Well, at least I thought that was the case..

### Nobody is perfect!

After I uploaded the images I reloaded my blog. The first image re-appeared and I was ready for the celebrations which were abruptly interrupted by the missing images in the second post. The images were nowhere to be found locally in none of the two backups I took. I think Cloudberry has a bug when handling filenames with hyphens. I’m still not certain that is the case but that’s the only characteristic that differs from the other files. Anyway, the moral of the story is triple-check everything before you’re initiating a destructive process and don’t trust external tools blindly.

# Book Review: The Bloody Reign of SLAYER!

## SLAAAAYERRR!

This book had been on my queue for quite a long time. Luckily Slayer never gets old so this book is still relevant! It’s been 30 years since they released their debut album (Show No Mercy). I’ve listened to all of their albums countless times yet whenever I need something fast and heavy the name Slayer comes to mind first. Now that Jeff Hanneman is dead and Dave Lombardo has left the band (yet again), they are close to the end of their careers at least in terms of releasing new material. So I’m glad I got to finish the book before they disbanded!

### The Book

This book was written in 2008 and it covers all but the last album (World Painted Blood). Honestly I finished the book just because it is about one of my favourite bands, not because it was a page-turner. In the beginning, the sections about the early years of the band are rather interesting but later on it doesn’t offer much.

It’s like a collection of album reviews. I liked the album review parts, analyzing all songs helps understand them better. But I think there is just too much boring stuff. The stuff that might make sense if you read it in an article about a recent article. But pointless and irrelevant after the album is released 15 years ago.

It was also a bit disappointment to see how boring the lives of the band members are! They are just 4 regular guys that come together to make the records and then live their separate lives. Of course, as the author mentions, this is not an official book endorsed by the band. So maybe it’s coming across boring because of the lack of information released by the members.

Anyway, I’d not recommend this book unless you are a diehard Slayer fan.  Even in that case I’d recommend to buy it just because it would look cool on your bookshelf. It’s a nice-looking hardcover book but the contents are just too mundane. There are not much interesting notes I remember from the book but here’s some trivia that I found enjoyable:

• Tom Araya’s birthday is on June 6 (Sixth day of the sixth month!)
• In the early years they stole lights from nearby houses to use them in their shows
• The producers of the debut album are Brian SLAgel and Bill MetoYER (get it?)
• Lombardo recorded Show No Mercy without cymbals because of the studio’s inadequacy. Later he overdubbed the cymbals.
• Slayer name was owned by a San Antonio-based band. They even supported LA Slayer one night.
• Kerry King played for Dave Mustaine’s Megadeth in some shows. Mustaine tried to convince him to join his band permanently (but failed in the end obviously)
• Dave Lombardo keft the band in 1986 because he wasn’t allowed to bring his wife on tour. A few months later he joined back only to quit again in 1992.
• They toured with Megadeth, Anthrax, Suicial Tendencies in Tge Big Four of Thrash and had personal issues with Dave Mustaine. Turns out Mustaine had issues with everybody else too in the industry!
• The idea of Wall of Blood (Sprinkler for fake blood at the beginning of Raining Blood) came up to Hanneman while watching the movie Blade

# Show Review: Ed Byrne

Ed Byrne is one of my favourite comedians. I like his jokes and his especially his style. He doesn’t offend people with his jokes. Being offensive is not necessarily a bad comedy in my opinion. I do like Jim Jefferies, for example, and he is offensive as it gets. But most of them don’t have his ability to present their jokes in a funny and not-so-annoying way. Anyway, back to Ed Byrne..

### Row X!

Unfortunately, my seat was way back in the stalls so I can’t say I clearly saw the stage. That was a bummer but maybe it’s not as important in comedy shows as it’s more about listening. There is not much action on the stage anyway.

### The Show

The show had a very interesting setup. First, Ed Byrne came to stage at 20:10 to warm up the crowd for his support act! The Ben Norris came and warmed us up for another 20 minutes. Then there was an intermission followed by a 5 minute set by Ben Norris. Finally, the one-hour long main attraction started. I liked Ben Norris and will try to keep an eye on his tours. Especially his jokes about the Post Office were killer, probably because we could relate to them! There is not much to say about Ed Byrne. I saw his DVDs before and this show was very much like them in general. About half of the show was about marriage jokes which were quite funny even though I didn’t relate.

### The Venue

It’s right across the Hammersmith tube station so it’s hard to miss. It’s generally a nice place but he seat was not too comfortable. As the seat being an important factor I’m afraid I’ll have to lower its grade in my eyes.

# Surviving the Minecraft

I recently decided to give Minecraft a shot as there are a few people playing it already (and by a few I mean 12,995,53 as of now, according to its official site). The problem is I’m a casual gamer and I don’t want to spend too much time on it. Anyway, we’ll see what I can do with the amount of time I can spare.

### Enter Minecraft

As it is a game encouraging creativity (well, I hope it does at least) when you first start it looks like just a blank slate. There is just some wood and grass around you. The problem is I didn’t have any goals! After spending a few minutes wandering around pointlessly I closed it and forgot about it some time. Until tonight!

Sometimes I forget that little thing called the Internet. I found a great tutorial in Youtube called “How to survive your first night in Minecraft”. First I thought it was referring to the actual night you are playing the game. And I thought by surviving it meant getting the hang of it and keep playing instead of being overwhelmed by it and leaving it for good as I was almost doing. But turns out there are days and nights in the game! And at night time some creatures emerge and they attack and kill you if you are not prepared for it. The tutorial I watched was only 15 minute long but it gave great insight about the game. As you can see it even gathered a lot of resources and built some tools using them already.

### Didn’t make it through the night :-(

Well, apparently pausing the game doesn’t stop time. When I was watching the tutorial in paused state the night fell and I was surrounded with all sorts of creatures like skeletons and zombies. Out in the open, without any tools I didn’t stand a chance!

### Conclusion

Even though I couldn’t survive the night a learned a lot about the game. It looks like a lot of fun. So maybe I should invest some time on it during the weekends. I know if I play at all nights I will be addicted very quickly. So I guess if I just limit it to weekends from the get-go I can stay loyal to my not spending too much time on games rule. Also I know there is an interest in Raspberry Pi community about running Minecraft servers on RPis. It sounds like it would be a nice use case for my idle Pi.

# Auto-Scaling using AWS Management Console

Auto-scaling has always been a feature of Amazon Web Services (AWS). Until today, it could be done in 2 ways:

• Using command line tool (See resources section for the link)
• Using Elastic BeanStalk to deploy your application

Yesterday (10/12/2013) they announced they added Auto-Scaling support to AWS console. I was planning to create auto-scaling my blog anyway so I cannot think of a better time to apply this.

## Auto-scaling using AWS Management Console

### Step 01: Launch Configuration

First we tell AWS what we want to launch. This step is a lot like creating a new EC2 instance. First you select an AMI. So before I started I created an AMI of my current blog and selected that one for the launch configuration. Then we select the instance properties. In this wizard we have the option for using spot instances. They are not suitable for Internet-facing applications so I’ll skip that part.

### Step 02: Auto Scaling Group

At the end of Launch Configuration wizard we can select create auto-scaling group with that launch configuration and jump right into Step 2. First we specify the name and the initial instance count for the group. Also we need to choose at least 1 availability zone. I always select all of them, I’m not sure if there is any trade-off with narrowing down your selection.

An important point to pay attention here is to expand Advanced Details section because it contains the load balancer selection. For web applications auto-scaling makes sense when the instances are behind a load-balancer. Otherwise new instances could not be reached anyway. Once you create the auto-scaling group you cannot associate it with an ELB so make sure you select your load balancer at this step.

After comes another important step: Specifying scaling policies. Basically, telling AWS the action to take when it needs to scale up or down and when to do it. “When” is defined by CloudWatch alarms. For scaling up, I added an alarm for average CPU utilization over 80% for 5 minutes and for scaling up CPU utilization under 20% for 5 minutes. When high CPU alarm goes off it will take the action we select, which in my case is adding 1 more instance. And scaling down is just the opposite: remove 1 instance from the existing machine farm.

On next step we define the notifications we want to receive when an AS event is triggered. I would definitely would like to know everything that happens to my machines so I requested an email for all events.

That’s all it takes to create an AS group using the wizard.

### Testing the scaling

The easiest way to test auto-scaling group is to terminate the instance it just launched. As you can see below once I killed the instance it immediately launched another one to match the minimum number requirement of AS group. So auto-scaling group is working but how can I be sure that it will launch a new instance when I need it most. Time to make it sweat a little! But first we have to setup an environment to create load on the system:

### Installing Siege

The easiest and simplest load testing tool I know is a Linux-based one called Siege. To prepare my simple load testing environment I quickly downloaded siege:

wget http://www.joedog.org/pub/siege/siege-latest.tar.gz

tar -xzvf siege-latest.tar.gz

It requires a C compiler which doesn’t come out-of-the-box with an Amazon Linux AMI. So first we need to install that:

yum install gcc*

And configure it by

./configure

At the end of the configuration it instructs us to run the following commands:

So after running make Siege is ready to go. We can check the configuration by

/usr/local/bin/siege -C

It should display the current version and other details about the tool.

Now, we have a micro instance running Siege and a small instance launched by auto-scaling.

The auto-scaling is supposed to launch another instance and add it to load balancer if the CPU usage is too high on the existing one. Let’s see if it’s really working.

### Under Siege!

I first created a URL file from my sitemap so that the load can be more realistic. I fired up 20 threads and it started to bombard my site:

When I try to load my site it was incredibly slow. The CPU usage kept rising on the single instance until the CloudWatch alarm went off. It triggered auto-scale to launch a new instance.

Now, I had 2 instances to share the load but that could only happen if the new instance was added to the Elastic Load Balancer (ELB) automatically. After a few minutes it passed the health checks and went in service.

At this point I had 2 instances and when I tried to load posts from my blog I noticed it was quite fast again. The CPU usage graph below tells how it all went down:

My first instance (orange) was running silently and peacefully until it was attacked by Siege. After a few minutes of hard times the cavalry came to rescue (blue instance) and started getting its fair share of the load. Then ELB distributed load as evenly as possible making the system running smoothly again. OK, so the system can withstand a spike and scale itself but it costs money. What’s going to happen after the storm. So I stopped Siege and sure enough, as we’d expect, after a few minutes Low CPU alarm kicked off and set the instance count back to 1 by terminating one of the instances.

Also, I was notified in every step of this process. So that I could be able to keep track of my instances at all times.

### Architecture of the system

So at this point the architecture of the system looks like this:

I’m planning to cover some basics (EC2, RDS, S3) in more detail in a later post. Also I’ll try to add more AWS services and enhance this architecture as I go along.

### Final Words

• If you are planning to use auto-scaling in production environment make sure to backup all your stuff externally. Also create snapshots for all the volumes.
• Even though network traffic is cheap it still costs. So for extended tests I suggest you keep an eye on your billing statement
• In Amazon Linux AMI Apache and MySQL don’t start automatically so you may need to update your configuration like I did. I used the script I found here.

# DevOps

DevOps (Development + Operations) is one of most popular terms in the IT world recently. From what I’ve read and listened to so far, my understanding is it is all about continuous deployment (or delivery). Basically, you have to automate everything from development to deployment to practice DevOps.

### Current problem

Traditionally, successful deployment is a huge challenge. It is mostly a manual and cumbersome process. Because of its sensitive nature the system admins are not huge fans of deployments. Also, another challenge is the miscommunication (or no communication in some cases) between system admin and development teams. They are generally run by different high-level executives and their priorities conflict most of the time.

### Solution

On the philosophical side, DevOps is bringing these teams together and work in harmony. Having social events with both teams’ attendance is a key to build confidence among team members. As Richard Campbell (from RunAsRadio and .NET Rocks podcasts) says “Pizza and beer is a global lubricant”.

#### Dev…

On the development side, the key requirement is continuous integration. You have to able to run unit tests and acceptance tests automatically on build servers. This means development has to be done in short sprints in an agile way with frequent check-ins. One step further of this stage is continuous deployment.

####…Ops This is where the IT team comes into play. When the whole system is automated, deploying to production frequently and without much headache becomes possible. Cloud computing is one of the core technologies that makes DevOps possible. Ability to manage virtual machines programmatically (i.e. AWS, OpenStack) leads to a whole bunch of possibilities.

This is a fairly complex topic encompassing many disciplines and technologies. Also it’s quite dynamic and open to innovation. Definitely worth keeping an eye on.

# Notes on productivity

I’ve been trying to improve my time management techniques perpetually. This post is a small compilation of notes, tips and tools I came up with. This is an on-going process and I will keep updating these. I strongly recommend reading Getting Results the Agile Way and visiting the sites in the resources section. I learned most of the stuff below from that book. And there is plenty of more good advice in there. So here is my list:

#### 01. Be realistic!

I think this is the most important thing I have learned so far. Yes, we all want to do more tasks and write more code, develop the killer apps we were thinking of but the reality is we have a limited time we can spare. This is not about aiming low but just don’t start everything at the same time because you are too excited about them at the time. It’s very likely that you will end up with a bunch of unfinished projects. Over time they will keep haunting you. First you will never feel the sense of accomplishment even though you spent a ton of time and effort. Also they will remain somewhere in your to-do list only making everything worse.

#### 02. Rule of 3 rocks!

I think if there is one thing to take away from the book Getting Results the Agile Way, it is the Rule of 3. It is a very simple and effective idea. Don’t put too many things on your plate. Every month pick 3 major tasks to accomplish. Then divide them into smaller weekly tasks and even smaller daily tasks. If you only aim to complete 3 tasks every day you can steadily move toward accomplishing the greater goals at the end of the month. For me, it’s not always to plan things so clearly. Generally monthly and weekly tasks are left unfulfilled. But it is easy and very achievable to plan the next day every night and pick only 3 things. At least that one works pretty well.

#### 03. Determine your categories (hot spots)

I have to admit I tweaked rule of 3 a little bit! I wanted to have a more balanced plan so I determined main categories in my life like health, work, career, chores etc. I’m picking 1 task per category each day. This way I believe I can make progress in every aspect of life.

#### 04. Keep it simple

Simplify your process as much as possible. When I first started keeping lists of outcomes I created templates so detailed that it took half an hour to fill out completely. Instead of overly complicating things and spending too much time on managing just try to get rid of as much as possible. In the end the ultimate goal is to use time for things that matter. Even a blank note with 3 items written on it can be a very good way to plan the day. Just do the 3 things on the list, don’t think about anything else.

#### 05. Keep your backlog clean and short

Learn how to let things go. This is another crucial thing. Let’s be honest we will never be realistic and we will put a ton of items to our to-do list turning it into a bottomless pit. There were times I found items in my list that didn’t even mean anything anymore. Or so outdated that it had no value anymore. The result is you keep those in your list all that time, they keep taking your time to manage also they weigh on your shoulders as they pile up. And you end up not doing them anyway. Essentially you stall and stall then delete them once they make no sense anymore. The golden rule to keep in mind is that if it’s important it will surface again! So don’t worry about forgetting a very important task. You will be reminded it in one way or another.

#### 06. Pen & paper with colour coding helps

Keeping notes on Evernote is great but sometimes the ease of use makes it also easier to neglect or forget tasks. After all they become invisible once you minimize the window. I recently started using coloured post-its.They are coded in their order of importance. So for example I cannot do green tasks before the purple ones are completed. Also it is annoying to have a bunch of notes sticking around so trying to get rid of them is an extra motivation!

#### 07. Use Pomodoro on small tasks

I love this technique since the day I heard about it. It’s quite simple: Just divide tasks small unbreakable units of time. The default is 25 minute work and 5 minute rest. I use Focus Booster on my desktop computer

On iPad I use an app which doesn’t have a specific name and I cannot find it in the app store anymore. It looks like this:

It works fine and looks stylish so I’m not planning to find another one. There are a ton of similar apps in the app store though.

I cannot use this method for large tasks. Also I cannot work non-stop with 25-5 periods. I get distracted by something else eventually. So I just use this for short 2-4 pomodoro-long tasks. For example spending 2 pomodoros a day for reading is a great way to spare time to read.