Development comments edit

If you have multiple accounts for git providers (i.e multiple accounts on Github and/or Bitbucket) then you’d need to update your SSH configuration to be able to access all your repositories seamlessly. Of course you can use HTTPS but then you’d have to enter username and password every time.

If you don’t specify which key to use for each account SSH agent will try the default key if there is one (id_rsa) and will most likely fail if you didn’t grant access to that key in your git provider settings.

SSH Access Denied

To resolve the issue you need to create a config file under .ssh folder that looks like this:

SSH Config

If you are just using one account per provider you don’t need to create multiple keys, you can just use id_rsa for both accounts. But if you have multiple accounts for a provider you’d need a key for each account. In the example configuration above I used a new key for BitBucket anyway. After creating the config file and adding the keys to your accounts you can start cloning repositories from various sources.

Final step to accomplish this is to use the hostname you set in the config file when cloning the repository. For example when you copy the SSL clone URL it looks something like this: git@github.com:{account name}/{repository name}.git. So let’s say it’s a repository from the corporate account in my sample config file then I’d have to modify the URL as follows: git@github-corporate:{account name}/{repository name}.git so that the correct host name and RSA file can be used.

SSH Success

Resources

Development, Lego, Mindstorms, EV3 comments edit

I had my eye on the new Lego Mindstorms set for a while. Finally I decided to order it from Amazon. It’s still a bit pricey but I think it’s worth it. I read nice review of EV3 here which also includes comparisons to the previous generation of the Mindstorms kit.

Lego Mindstorms EV3

Programming EV3: The Official Way

Programming the kit is very easy using Lego’s official graphical programming tool6. You just have to drag and drop the components and fiddle with the parameters. Check out the following very basic application:

Lego Mindstorms EV3

It powers the motors connected to ports B and C. It keeps doing that in a loop as long as the value read from Infrared Sensor is larger than 20. If there is an object closer to 20 centimetres it breaks and ends the program. And here’s the output in action:

###Programming EV3: The .NET Way The .NET API is an open-source project and code can be found on CodePlex. I recommend watching the introductory video which shows the basics. It shows how to move the robot by sending direct commands to turn the motors and how to read values from the sensors. The following part the test application shows the event handlers for the direction buttons and setup code to connect to the brick.

Lego Mindstorms EV3

I also added a similar implementation of the NXT-G program above. It’s a while loop which breaks when the value from the IR sensor is lower than 10.

Lego Mindstorms EV3

And voila! Here is a clever robot that senses the object in front of it and avoids the collision by stopping!

You can find the source code for the test application on GitHub

Tips & Tricks

  • When I first implemented the NXT-G equivalent version the sensor value wasn’t updating properly. I checked the discussion forums in the CodePlex project page and found out that other people were having a similar issue. A workaround was adding the Thread.Sleep(10) line. After that I could read the updated sensor values without any problems. Although it doesn’t feel like the right solution it works fine as a temporary workaround.

  • During the testing and debugging I managed to crash the Lego brick a few times. First I feared I actually “bricked” the brick but luckily a reset resolved it. Resetting the brick is not obvious though, I had to check the manual for that. So in case you need to reset it you have to hold down Back, Center and Left buttons. Then release the Back button when the screen goes blank and release the other two when the screen says “Starting” as shown in EV3 User Guide

Resources

  1. Lego EV3 review and comparison to NXT 2.0
  2. API code on CodePlex
  3. Channel9 video on programming EV3 with .NET API
  4. Sample Track3r robot project and building instructions
  5. EV3 User Guide
  6. Lego software
  7. Codeplex discussion on IR sensor not updating values
  8. Source code for the test application

Development comments edit

My previous post was about grid-based clustering. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) takes another approach called density-based clustering. It grows regions with high density (above threshold provided) into clusters and discovers clusters of arbitrary shape.

DBSCAN Implementation

First off the data is loaded and a distance matrix is calculated based on the data points.

DBSCAN 1

The algorithm visits every data point and finds its neighbours. If the neighbours are dense enough than the cluster is expanded to include those points as well.

DBSCAN 3

So data points that are close enough to each other are included in the same cluster.

DBSCAN 2

Joining dense clusters is a similar approach taken in grid clustering. The difference is this way the clusters can have arbitrary shapes.

Resources

fitness, game comments edit

I’ve been playing this game for about 6 months. It’s an interesting game in a number of ways.

Ingress Basics

I discussed the basics of the game in my previous post. If you want to find out more about the rules and game play you can check out that post here. For me the definite guide for the game is a site called decodeingress.me. You can find the latest news and updates about the game. It is updated quite frequently so it pays off to keep an eye on the latest changes in the game. Anyway, what I want to log in this post is mainly my experiences.

What’s It All About?

Feeling proud of achievements in a virtual world is a bit silly but I guess it goes for all sorts of games. Here’s a snapshot of the fields I created a while back:

Ingress Field

In theory the goal is to create and try to maintain those fields so that you capture “mind units”. The team who has more mind units wins (actually the game never ends so I guess there are no winners or losers. The worst case of scenario is your team falls behind). Once you start playing it’s all about levelling up but after a certain level you seek more. Here’s my view of pros and cons of the game:

Pros

  • Exercise: This is the main reason I started the game. It motivates you walk more. I stopped taking the bus from town centre and started walking home because there are some portals that need to be captured along the way!</li>
  • Get to know the city better: Before this game, I knew where my house was, where the nearest train station was and where the nearest Sainsbury’s store was. Now I know all the churches, parks and many other public buildings around my neighbourhood and around the city.
  • New friends: You can play the game alone with no problems and can have fun at first. But after a while it becomes a bit boring because you do the same activities repeatedly. By joining the local community you get to know the people playing around you.

Cons

  • Repetition: The portals have a mitigation cap meaning that you cannot keep enforcing the portals. So it can be taken down by a single player.After some time it becomes quite repetitive. You capture the portals and create fields. A few hours later someone blows them up and the next morning you have to do the same thing all over again. This might get the fun away especially If you are playing alone.
  • Can be time-consuming: Because of the above fact, portals require maintenance especially in crowded areas. If you become obsessed about keeping the portals yours you might end up spending a lot of time.

For me pros of the game far outweigh the cons. I can control the time I spend on the game and play with other people so it becomes a social event.

Some Tips

Just off the top of my head here are a few tips for beginners:

  • Deploy resonators as far as possible from the center so that they don’t all get hit hard at the same time.
  • When firing XM bursters, long press fire button and release it when XM is accumulated in the center. It allows you to get bonus firepower up to %20
  • Links make portals stronger so try to create as many links and fields you can.

Resources

Development comments edit

CLIQUE (CLustering In QUEst) algorithm is a grid-based clustering algorithm that partitions each dimension of the dataset as a grid.

CLIQUE Implementation

The algorithm starts by assigning each data point to the closest grid in the matrix.

CLIQUE 1

After loading the data it looks something like this:

CLIQUE 2

In order for a grid cell to be considered as “dense” it has to have data points more than or equal to the threshold. In this sample the threshold was provided as 2. So after the adjacent dense cells are joined together we get 3 clusters:

CLIQUE 3

Resources

Development comments edit

A rough set is an approximation of a set that gives the lower and upper borders of the set.

Rough set implementation

The sample implementation starts off with reading the data set and the configuration file. The object and attribute index lists are acquired from the configuration. The items that have indices specified in the object list are considered as the set of objects and the rest of the items are the complement of the data set.

RoughSet 1

If a given index in attribute index list equal in both object list and complementary object list are not equal the object is considered to belong to the negative border, if not it belongs to the positive border.

Sample Output

RoughSet Sample Output

Resources

Development comments edit

Genetic algorithm sounds fascinating to me as in a way it mimics evolution. It’s quite easy to implement and understand.

Genetic Algorithm

The algorithm has 3 stages:

  • Selection: Every generation is subjected to a selection process which eliminates the unfit ones.
  • Cross-over: This happens randomly among the selected generation and some genomes swap genes.
  • Mutation: Some genes are mutated randomly based on a given threshold value for mutation.

Implementation

The implementation starts with generating random bit sequences to represent genomes. The length of the genome and the number of genomes in a generation are specified in the configuration.

Genetic Algorithm 1

First generation is calculated randomly and their fitness is calculated based on the fitness function the system uses. After the groundwork has been done we have a genome pool each with their own fitness values and selection ranges. The width of the selection range is determined by the fitness of the genome. So that the larger the range the more likely they are selected for the next generation.

Genetic Algorithm 2

In selection process, a random value is generated and the genome whose selection range includes that value is selected. Same genome can be selected more than once.

Genetic Algorithm 3

After the next generation is selected and their new fitness values are calculated cross-over starts. In cross-over 2 random genomes are picked and they swap genes. The point in the genome string is specified in the configuration (3 in this example)

Genetic Algorithm 4

The final phase of the algorithm is mutation. In this step a number of genomes are picked and some bits is flipped randomly.

Genetic Algorithm 5

The most important thing about this algorithm is having a good fitness function that can accurately compute the fitness values for a given feature.

Resources

Development comments edit

Another data mining algorithm: AGNES (Agglomerative Nesting)

AGNES Algorithm

AGNES takes a different approach than k-means which was the subject of my previous post. Initially it considers all data points as a separate cluster.

AGNES 1

Then finds the minimum distance between clusters and merges the closest clusters:

AGNES 2

The resulting cluster is added to the all cluster list the merged clusters are removed as they are no longer valid.

Resources

Development comments edit

I’m keeping on reviving my old projects. This is the second data mining algorithm implementation. It is another clustering algorithm called k-means.

k-means Algorithm

Algorithm groups and creates k clusters from n data points. First the cluster centres are picked randomly from the data points. Then the entire dataset is iterated and all points are assigned to their closest cluster. Closest cluster is determined by measuring the distance of the data point to the centroid of the clusters. This process is repeated until there is no change in the dataset and all points are assigned to the closest ones.

K-means Results

Implementation

The project contains 6 libraries:

  • VP.KMeans.Core: Core library including the algorithm implementation
  • VP.KMeansClient.GUI: User interface for entering the parameters and plotting the clusters
  • VP.KMeansClient.Console: Console user interface. No fancy plots, just an output file is generated
  • VP.KMeans.DataGenerator.Core: Library to generate test data
  • VP.KMeans.DataGenerator.Console: Console application to feed the core library to generate test data
  • CPI.Plot3D: External library to plot the results

Resources

Development comments edit

I was digging through some old projects and found out the Data Mining and Machine Learning projects I implemented. Instead of letting them gather dust(!) in the hard disk I decided to review and publish the source code. This will also give me a chance to revise my data mining knowledge. Let’s start with Apriori algorithm.

Apriori Algorithm

It is an algorithm to determine to most frequent items in a given collection. The term “frequent” is defined by a given threshold or “minimum support count”.

Implementation

The project is implemented in Java. It uses a Hashtable to keep count of itemsets in the database. The algorithm starts with finding the most frequent 1-item sets first. Then using the previous most frequent item list of item size k, it generates candidate item list of size k+1.

For example, for the sample data below and a threshold 4, 1,2,3,4,5 are all frequent 1-itemsets. From this list we generate a 2-item candidate list (all 10 combinations) and check if the subsets are also frequent. For 1-itemsets they are all frequent so they all pass pruning. Then we count the occurences of these candidates. Only 7 of them are equal to or greater than the threshold. From this list we generate our 3-item candidates. Such as 1,2 and 1,4 combined to generate 1,2,4. Then we count the occurrence of 3-itemsets and prune the results by checking all of its subsets.

Apriori Check Subsets

The idea of pruning is if the there are some infrequent subsets inside an itemset then the larger set cannot be frequent so it is removed from the candidate list. (1,2,3 is a candidate but as 2,3 is not a frequent 2-itemset it is removed from the candidate list) This process helps improve the performance of the algorithm as it reduced the number of iterations.

Output

1,3,4,5

1,2,4,5

1,2,4

1,3

3

1,5

1

3

1,3,5

4

1,2,4

2

1,2

3,4

3,5

1,3,4

2

3,5

1,2,3,4

Results:

Apriori Results

Source Code

I created a public repository for the source code. You can check it out here if it tickles your fancy!

Resources

review, gadget comments edit

Fitbit Flex

I bought this about 2 months ago. I wore it every single day since then and I just loved it! It is basically a motion sensor that detects and keeps track of your daily movements. You can set your own daily goals steps you walked, distance you took or calories you burnt.

I can keep track of distance by using my Garmin ForeRunner 10 (which I reviewed here) but this one is easier to use because it does everything in the background. Garmin takes a few minutes to start because it needs to find your GPS coordinates but that’s not the case for Flex.

Flex lets you keep track of distance, active minutes, calories and steps.

Fitbit Goals

Also you can log your weight, other exercises and food intake so that you can calculate the net calories throughout a time period.

Another great feature about it is tracking your sleep quality. You can use this data in conjunction with your daily activities.

Fitbit Sleep

And here is my favourite feature: Alarm! It turns out if something on your wrist start to vibrate you wake up. Instantly! Of course I keep my phone’s alarm still running as a fallback method but this one works pretty good.

Conclusion

I bought it for £68 and as of now it is listed as £83 on Amazon. Apparently the price fluctuates a bit but I think £70 – 80 price range is good for this product. I charge once every 3 days or so. Other than that I completely forget about it while it does its job in the background. It motivates you to reach your goals and be more active in general and the silent alarm is absolutely fantastic. I’d recommend this to anyone who would like to have more exercise.

Resources

review, gadget comments edit

It’s been 5 months since I’ve gone to a metal show. The last one was Bloodstock Festival and Lamb of God was headlining that. The playlist on this show had no surprises (Just the way we like it). O2 Academy Brixton twitter account kindly tweeted the playlist:

Lamb of God Playlist

“Open up the dance floor!”

Apparently this was a sold out show. It was quite crowded and people went nuts the second LoG hit the stage. Normally there is a small area for moshpit in front of the stage right behind the first 2-3 rows or people. But from where I stood it looked like the half of the floor was a giant moshpit area. Check this short clip to get a sense of the atmosphere (sound quality is awful so make sure you don’t start it at full volume).

Notes

  • Number of girls seemed to be a lot more than a usual metal concert. Number of girls in the moshpit was definitely a lot more than a usual metal concert!
  • Guitarist Mark Morton didn’t take the stage. At the beginning of the show Randy introduced Paul Waggoner as his temporary replacement and explained Mark’s absence as “He had to attend to a family business and family always comes first in this band”
  • Beer prices were ridiculous (or maybe I should say tragic): £4.75 for a Carlsberg. I only had two. The feeling of being ripped off ruins the fun of it but mainly the long queues in the front of the bars were a huge turn off.

Lamb of God Image 1

Lamb of God Image 2

Lamb of God Image 3

Lamb of God Image 4

Lamb of God Image 5

Lamb of God Image 6

productivity comments edit

I’ll keep rambling about my opinions, observations and experiences about productivity. Here goes:

01. Don’t try to cut back sleep to gain time

We all fee like we need more time to accomplish things but sleep is not the right item for savings. I tried it in the past. Tried to get away with sleeping 3-4 hours every night. At first it felt like the days were longer but towards the end of the week I was feeling so tired that I couldn’t perform well during the day. So I was up for more hours but my productivity had reduced significantly. I strongly suggest to have a good night’s sleep around 6 – 8 hours.

02. Two pomodoros of reading a day makes you (feel) smarter

In my last post I talked a bit about the Pomodoro technique. I love using it to divide small tasks into short work units. The idea of spending two pomodoros for reading is similar to saving money in a piggy bank. It’s easy to use 50 minutes a day for reading no matter how busy your day is. You wouldn’t notice and it wouldn’t become a huge burden once you make it a habit. But over time it would amount to a bunch finished books that would be gathering dust on your desk otherwise (or iPad or Kindle or phablet or.. anyway, you get the point)

03. Review your processes often

Setting goals for the day and week is a good practice but to make it work you have to review the outcomes periodically. In the excellent book called 30 days of Getting Results, J.D.Meier names is as “Friday Reflection”. Every week the status of the tasks that were targeted to be completed must be reviewed and more importantly the process must be updated the following week based on lessons learned from the previous week.

04. Keep learning

Everyday we are wasting a tremendous amount of time doing chores, commuting etc. At this day and age we have access to so many different means of accessing technology. We have small computers in our podcasts that are almost always connected to Internet. It’s a great chance we have to make use of the idle time by watching a PluralSight video for example. There are similar online courses of course but it just happens to be my favourite with apps for all mobile platforms. Another way it listening to a podcast. There are a lot of podcasts about technology and development. A few of my favourites are:

  • .NET Rocks
  • Run As Radio
  • Hanselminutes
  • Yet Another Podcast by Jesse Liberty

I’m planning to compile a list of my favourite shows so I hope this list will look more comprehensive.

The reason I added this item is using these learning mechanisms help you keep sharp and on track. Generally it’s not possible to really learn something by just listening to it but the idea here is to keep an eye of the latest to research about it.

05. Don’t multitask!

Trying to do multiple things at the same time creates an illusion of going fast and accomplishing a lot but I don’t believe that’s the case. Of course I’m not talking about doing something meaningful along with a chore. Like as in #4, it’s perfectly acceptable to watch a technical video on your way to work. The type of multitasking I’m against is doing it while coding for example. My idea is divide such tasks into pomodoros, focus on one thing at a time and nothing else. This would improve the quality of that unit of work dramatically and will eventually lead to accomplishing more.

Resources

review, book comments edit

Framework Design Guidelines

Fitbit Flex

Having a common framework is quite important to reduce code reuse. But designing that framework properly and consistently is a huge challenge. Even though we are living in a RESTful world now, I think having a framework or a set of common libraries for personal or commercial projects is still relevant. A well-designed well-tested framework would significantly improve any application built on top of it.

I had referred to this book partially before but this time decided to read it from cover to cover and make sure I “digest” all. It contains countless gems that every developer should know. Anyone developing even small libraries can benefit from this book a lot. You don’t need to design the .NET framework (like the authors). Also it comes with a DVD full of presentations of authors.

Companion DVD

Unfortunately I lost my DVD. Probably it’s inside one of the many CD cake boxes. I was hoping to check it out as I went along the book. But luckily I found out that it is freely available from the publisher. Check you the download link in the resources section. One thing to beware about the download is that you can come across another link in the Brad Abrams’s blog here. The download works fine but one of the presentations inside it is corrupted so I suggest you download each section separately from the site in the resources.

Some notes

The book is full of gems and very useful tips. Here are just a few:

  • Keep it simple: “You can always add, you cannot ever remove”
  • There is no perfect design: You always have to make some sacrifices and consider the trade-offs
  • Well-designed frameworks are consistent
  • Scenario-driven design: Imagine scenarios to visualize the real-world use of the API When in doubt leave the feature out, add it later. Conduct usability studies to get developers’ opinions
  • Keep type initialization as simple as possible. It lowers the barrier of entry.
  • Throw exceptions to communicate the proper usage of API. It makes the API self-documenting and supports the learning-by-doing approach.
  • Going overboard with abstractions may deteriorate the performance and usability of the framework.

Even though it’s been a few years since this books was released it is still a very helpful resource.

Resources

fitness, game comments edit

Yet another horrible term coined by inaptly fusion of two words! I was going to call dibs on it but a quick Google search showed other people used it before. Someone even registered the .com domain for it! Anyway, obviously the term is an abomination of words “Ingress” and “Exercise”.

What is Ingress

Ingress Logo

Ingress is an augmented reality game developed by Google (Well, actually by a company called Niantic Labs that is owned by Google). To play the game you need to install the free app on an Android device. (I saw iOS support is coming in 2014 here)

For me the beauty of this game is having exercise without even noticing it. Even having a desire to do it. Last Sunday, for example, I walked around at least 30 minutes hacking a bunch of portals. To an outsider I might have looked like I was enjoying a nice walk in the park but actually I was trying to keep to world free from the unreliable Shapers! I didn’t even notice I walked for that long.

The story

An alien race called Shapers use Portals to teleport to earth. There are two teams: The Resistance and The Enlightened. The resistance are opposed to Shapers’ whereas Enlightened believe they should be embraced.

The goal

The ultimate goal is to control as many portals as you can. You can create links between the portals your team controls and the triangular area between the portals forming the link is called “control field” and any player located inside this area is considered to be under your team’s control. They are called “mind units”. That’s essentially how the current control rates on the intel page is calculated. (As a proud Resistance agent it just saddened me to see we are down to 43% of global control)

The gameplay

Portals can be found anywhere. They can be controlled by either of the teams or they can be unclaimed. To claim a portal at least one resonator must be placed. To capture a portal controlled by the opponent team their resonators must be destroyed by firing XMP. You progress to higher levels by collecting AP (Action Point). After you start hacking the portals you will notice some items in your inventory.

Basic strategy

Okay here’s a guide to the absolute beginner: Just walk or run around and hack every portal you see. Don’t attack powerful enemy portals, you will just end up using all your weapons and energy. If you find unclaimed portals (the gray ones) jump on them instantly. You will gain a lot of APs by deploying resonators and linking them to other portals. Before attacking enemy portals check out the deployment and mod status. Once you have enough weapons in your arsenal and move up a few levels try to find fairly weak portals and zap the resonators. You can earn some sweet APs by creating a control field and don’t let enemy portals surround you because links cannot intersect and if they have a link going through your portals you cannot establish a link between them. So in that sense best defense is offense. Just pick your fights carefully.

Glossary

  • Portal: Places where Shapers can teleport to earth. The main goal is finding and hacking them.

  • Hacking: You hack the portals to get Action Points and add items to your inventory. You can hack any portal but you only get AP by hacking other people’s portals. You get more items by hacking your own portals.

  • Resonator: Every portal has 8 resonators. The resonators deplete over time and eventually the portal becomes unclaimed unless they are recharged. This helps your team to keep control of the portal for a longer period of time.

  • Portal Key: When you hack a portal you might get portal key (there is 15% chance you will). They are very important because you need them to create links between portals.

  • Mod: There are various mod items that let you upgrade a portal. Every portal have 4 mod slots. The mods are portal shields (to defends portals against XMP attacks), force amplifier (increases force of the portal), link amplifier (increases the portal link range), multi-hack (increases the number of times a player can hack a portal), heat sink (decreases the duration between hacks) and turret (increases the portal attack rate).

  • XMP Burster: Weapons to fire XMP on enemy portals. You can destroy resonators, links and control fields with them

  • Power Cube: Used to store XM so you can recharge your XM reserves.

Conclusion

It’s an enjoyable game for me but I can see myself getting bored of it pretty soon. But I have a great motivation to keep playing. It makes exercise just go by. I cannot wait for spring when I can start using it while running which would allow me to hack so many more portals that I can by walking.

Resources

Amazon Web Services, Cloud Computing comments edit

Amazon S3

I have two AWS accounts and I made a mistake by using mixing the usage of services. More specifically, I hosted an application on one account but used S3 on the other. So I perpetually had to switch back and forth between accounts to access all services I used. First I thought fixing it would be a non-issue but it proved to be a rather daunting task.

Bucket naming in S3

In S3, all buckets must have unique names. You cannot use a name if it’s already taken (much like domain names). Since I was using the bucket already, creating the same bucket in the other account and copying its contents was not an option. The second idea was to create the target bucket with a temporary name, copy the contents, delete the first one and rename the target bucket. Well, guess what? You cannot rename a bucket either! Another problem is when you delete a bucket you can create a new one with the same name right away. I’m guessing this is because of the redundancy S3 provides. It takes time to propagate the operation to all the nodes. My tests showed that I could re-create the bucket in the other account only after 45 – 50 minutes.

To develop or not to develop

My initial instinct was to develop a tool to handle this operation but I decided to check out the what’s already available. I was occasionally using Cloudberry but wanted to check its competitors hoping one of the tools would support the functionality I need.

Cloudberry Explorer for Amazon S3

I find this tool quite handy. It has lots of functions and a nice intuitive. It comes in flavours: Free and Pro version. I used free version so far and unless you are a big enterprise it seems sufficient. It allows you to manage multiple AWS account. It allows copying objects among accounts but not moving a bucket (actually after my findings above I wasn’t very hopeful anyway)

AmazonS3 CloudBerry Main

As you can see in the menu bar, it supports lots of features.

S3 Browser

This one comes with a free version too as well as a paid version. The free version is limited to 2 accounts and you can only see one account at a time.

S3 Browser

I tried to copy a file and paste to another but it got an Access Denied error. I could do the same thing with Cloudberry in seconds by simply dragging and dropping to the target folder.

Bucket Explorer

Third candidate only has a 30-day trial version as opposed to a free one. The second I installed it I knew it was a loser for me because it doesn’t support multiple accounts. Also as you can see below the UI is hideous so this is not a tool for me.

Bucket explorer

..and the winner is

Cloudberry won by a landslide! It looks much more superior than both of the other tools combined.

Operation Bucket Migration

So I backed up everything locally and deleted the source bucket so that I could create the same one in the new account. After periodically checking for 45 minutes I finally created the bucket and uploaded the files. Set the permissions and the operation was completed without any casualties.. Well, at least I thought that was the case..

Nobody is perfect!

After I uploaded the images I reloaded my blog. The first image re-appeared and I was ready for the celebrations which were abruptly interrupted by the missing images in the second post. The images were nowhere to be found locally in none of the two backups I took. I think Cloudberry has a bug when handling filenames with hyphens. I’m still not certain that is the case but that’s the only characteristic that differs from the other files. Anyway, the moral of the story is triple-check everything before you’re initiating a destructive process and don’t trust external tools blindly.

Resources

book review comments edit

SLAAAAYERRR!

This book had been on my queue for quite a long time. Luckily Slayer never gets old so this book is still relevant! It’s been 30 years since they released their debut album (Show No Mercy). I’ve listened to all of their albums countless times yet whenever I need something fast and heavy the name Slayer comes to mind first. Now that Jeff Hanneman is dead and Dave Lombardo has left the band (yet again), they are close to the end of their careers at least in terms of releasing new material. So I’m glad I got to finish the book before they disbanded!

The Book

The Bloody Reign of Slayer

This book was written in 2008 and it covers all but the last album (World Painted Blood). Honestly I finished the book just because it is about one of my favourite bands, not because it was a page-turner. In the beginning, the sections about the early years of the band are rather interesting but later on it doesn’t offer much.

It’s like a collection of album reviews. I liked the album review parts, analyzing all songs helps understand them better. But I think there is just too much boring stuff. The stuff that might make sense if you read it in an article about a recent article. But pointless and irrelevant after the album is released 15 years ago.

It was also a bit disappointment to see how boring the lives of the band members are! They are just 4 regular guys that come together to make the records and then live their separate lives. Of course, as the author mentions, this is not an official book endorsed by the band. So maybe it’s coming across boring because of the lack of information released by the members.

Anyway, I’d not recommend this book unless you are a diehard Slayer fan.  Even in that case I’d recommend to buy it just because it would look cool on your bookshelf. It’s a nice-looking hardcover book but the contents are just too mundane. There are not much interesting notes I remember from the book but here’s some trivia that I found enjoyable:

  • Tom Araya’s birthday is on June 6 (Sixth day of the sixth month!)
  • In the early years they stole lights from nearby houses to use them in their shows
  • The producers of the debut album are Brian SLAgel and Bill MetoYER (get it?)
  • Lombardo recorded Show No Mercy without cymbals because of the studio’s inadequacy. Later he overdubbed the cymbals.
  • Slayer name was owned by a San Antonio-based band. They even supported LA Slayer one night.
  • Kerry King played for Dave Mustaine’s Megadeth in some shows. Mustaine tried to convince him to join his band permanently (but failed in the end obviously)
  • Dave Lombardo keft the band in 1986 because he wasn’t allowed to bring his wife on tour. A few months later he joined back only to quit again in 1992.
  • They toured with Megadeth, Anthrax, Suicial Tendencies in Tge Big Four of Thrash and had personal issues with Dave Mustaine. Turns out Mustaine had issues with everybody else too in the industry!
  • The idea of Wall of Blood (Sprinkler for fake blood at the beginning of Raining Blood) came up to Hanneman while watching the movie Blade

Resources

show review comments edit

Ed Byrne is one of my favourite comedians. I like his jokes and his especially his style. He doesn’t offend people with his jokes. Being offensive is not necessarily a bad comedy in my opinion. I do like Jim Jefferies, for example, and he is offensive as it gets. But most of them don’t have his ability to present their jokes in a funny and not-so-annoying way. Anyway, back to Ed Byrne..

Row X!

Unfortunately, my seat was way back in the stalls so I can’t say I clearly saw the stage. That was a bummer but maybe it’s not as important in comedy shows as it’s more about listening. There is not much action on the stage anyway.

Ed Byrne Ticket

The Show

The show had a very interesting setup. First, Ed Byrne came to stage at 20:10 to warm up the crowd for his support act! The Ben Norris came and warmed us up for another 20 minutes. Then there was an intermission followed by a 5 minute set by Ben Norris. Finally, the one-hour long main attraction started. I liked Ben Norris and will try to keep an eye on his tours. Especially his jokes about the Post Office were killer, probably because we could relate to them! There is not much to say about Ed Byrne. I saw his DVDs before and this show was very much like them in general. About half of the show was about marriage jokes which were quite funny even though I didn’t relate.

The Venue

It’s right across the Hammersmith tube station so it’s hard to miss. It’s generally a nice place but he seat was not too comfortable. As the seat being an important factor I’m afraid I’ll have to lower its grade in my eyes.

Resources

game, minecraft comments edit

I recently decided to give Minecraft a shot as there are a few people playing it already (and by a few I mean 12,995,53 as of now, according to its official site). The problem is I’m a casual gamer and I don’t want to spend too much time on it. Anyway, we’ll see what I can do with the amount of time I can spare.

Enter Minecraft

As it is a game encouraging creativity (well, I hope it does at least) when you first start it looks like just a blank slate. There is just some wood and grass around you. The problem is I didn’t have any goals! After spending a few minutes wandering around pointlessly I closed it and forgot about it some time. Until tonight!

Youtube to the rescue

Sometimes I forget that little thing called the Internet. I found a great tutorial in Youtube called “How to survive your first night in Minecraft”. First I thought it was referring to the actual night you are playing the game. And I thought by surviving it meant getting the hang of it and keep playing instead of being overwhelmed by it and leaving it for good as I was almost doing. But turns out there are days and nights in the game! And at night time some creatures emerge and they attack and kill you if you are not prepared for it. The tutorial I watched was only 15 minute long but it gave great insight about the game. As you can see it even gathered a lot of resources and built some tools using them already.

Surviving the Minecraft - Inventory view

Didn’t make it through the night :-(

Well, apparently pausing the game doesn’t stop time. When I was watching the tutorial in paused state the night fell and I was surrounded with all sorts of creatures like skeletons and zombies. Out in the open, without any tools I didn’t stand a chance!

Conclusion

Even though I couldn’t survive the night a learned a lot about the game. It looks like a lot of fun. So maybe I should invest some time on it during the weekends. I know if I play at all nights I will be addicted very quickly. So I guess if I just limit it to weekends from the get-go I can stay loyal to my not spending too much time on games rule. Also I know there is an interest in Raspberry Pi community about running Minecraft servers on RPis. It sounds like it would be a nice use case for my idle Pi.

Resources

Amazon Web Services, Cloud Computing, System Administration comments edit

Amazon Web Services (AWS) Auto-Scaling

Auto-scaling has always been a feature of Amazon Web Services (AWS). Until today, it could be done in 2 ways:

  • Using command line tool (See resources section for the link)
  • Using Elastic BeanStalk to deploy your application

Yesterday (10/12/2013) they announced they added Auto-Scaling support to AWS console. I was planning to create auto-scaling my blog anyway so I cannot think of a better time to apply this.

Auto-scaling using AWS Management Console

Step 01: Launch Configuration

First we tell AWS what we want to launch. This step is a lot like creating a new EC2 instance. First you select an AMI. So before I started I created an AMI of my current blog and selected that one for the launch configuration. Then we select the instance properties. In this wizard we have the option for using spot instances. They are not suitable for Internet-facing applications so I’ll skip that part.

Step 02: Auto Scaling Group

At the end of Launch Configuration wizard we can select create auto-scaling group with that launch configuration and jump right into Step 2. First we specify the name and the initial instance count for the group. Also we need to choose at least 1 availability zone. I always select all of them, I’m not sure if there is any trade-off with narrowing down your selection.

An important point to pay attention here is to expand Advanced Details section because it contains the load balancer selection. For web applications auto-scaling makes sense when the instances are behind a load-balancer. Otherwise new instances could not be reached anyway. Once you create the auto-scaling group you cannot associate it with an ELB so make sure you select your load balancer at this step.

Create Auto Scaling Group

After comes another important step: Specifying scaling policies. Basically, telling AWS the action to take when it needs to scale up or down and when to do it. “When” is defined by CloudWatch alarms. For scaling up, I added an alarm for average CPU utilization over 80% for 5 minutes and for scaling up CPU utilization under 20% for 5 minutes. When high CPU alarm goes off it will take the action we select, which in my case is adding 1 more instance. And scaling down is just the opposite: remove 1 instance from the existing machine farm.

Create Auto Scaling Group

On next step we define the notifications we want to receive when an AS event is triggered. I would definitely would like to know everything that happens to my machines so I requested an email for all events.

Create Auto Scaling Group

That’s all it takes to create an AS group using the wizard.

Testing the scaling

The easiest way to test auto-scaling group is to terminate the instance it just launched. As you can see below once I killed the instance it immediately launched another one to match the minimum number requirement of AS group. So auto-scaling group is working but how can I be sure that it will launch a new instance when I need it most. Time to make it sweat a little! But first we have to setup an environment to create load on the system:

Installing Siege

The easiest and simplest load testing tool I know is a Linux-based one called Siege. To prepare my simple load testing environment I quickly downloaded siege:

wget http://www.joedog.org/pub/siege/siege-latest.tar.gz

tar -xzvf siege-latest.tar.gz

It requires a C compiler which doesn’t come out-of-the-box with an Amazon Linux AMI. So first we need to install that:

yum install gcc*

And configure it by

./configure

At the end of the configuration it instructs us to run the following commands:

Siege configuration

So after running make Siege is ready to go. We can check the configuration by

/usr/local/bin/siege -C

It should display the current version and other details about the tool.

Siege Configuration

Ready to go

Now, we have a micro instance running Siege and a small instance launched by auto-scaling.

AWS Instances

The auto-scaling is supposed to launch another instance and add it to load balancer if the CPU usage is too high on the existing one. Let’s see if it’s really working.

Under Siege!

I first created a URL file from my sitemap so that the load can be more realistic. I fired up 20 threads and it started to bombard my site:

Siege in Action

When I try to load my site it was incredibly slow. The CPU usage kept rising on the single instance until the CloudWatch alarm went off. It triggered auto-scale to launch a new instance.

AWS Instances

Now, I had 2 instances to share the load but that could only happen if the new instance was added to the Elastic Load Balancer (ELB) automatically. After a few minutes it passed the health checks and went in service.

Auto-scaling using AWS Management Console - Elastic Load Balancer Overview

At this point I had 2 instances and when I tried to load posts from my blog I noticed it was quite fast again. The CPU usage graph below tells how it all went down:

Auto-scaling using AWS Management Console - CPU utilization

My first instance (orange) was running silently and peacefully until it was attacked by Siege. After a few minutes of hard times the cavalry came to rescue (blue instance) and started getting its fair share of the load. Then ELB distributed load as evenly as possible making the system running smoothly again. OK, so the system can withstand a spike and scale itself but it costs money. What’s going to happen after the storm. So I stopped Siege and sure enough, as we’d expect, after a few minutes Low CPU alarm kicked off and set the instance count back to 1 by terminating one of the instances.

AWS Instances

Also, I was notified in every step of this process. So that I could be able to keep track of my instances at all times.

Auto-scaling using AWS Management Console - Notifications

Architecture of the system

So at this point the architecture of the system looks like this:

Auto-scaling using AWS Management Console - System Architecture

I’m planning to cover some basics (EC2, RDS, S3) in more detail in a later post. Also I’ll try to add more AWS services and enhance this architecture as I go along.

Final Words

  • If you are planning to use auto-scaling in production environment make sure to backup all your stuff externally. Also create snapshots for all the volumes.
  • Even though network traffic is cheap it still costs. So for extended tests I suggest you keep an eye on your billing statement
  • In Amazon Linux AMI Apache and MySQL don’t start automatically so you may need to update your configuration like I did. I used the script I found here.

Resources