Tip of the Day, Tips & Tricks comments edit

Recently I had this problem and was surprised by this limitation. The problem is wildcard SSL certificates only support one level of subdomains, i.e *.a.com matches foo.a.com but not bar.foo.a.com (taken from RFC2818, see link below). To support all subdomains beyond foo you would need a wildcard certificate for *.foo.a.com

Resources

Tip of the Day, Tips & Tricks comments edit

To capitalize all words in a string you can use the function built-in the .NET framework ToTitleCase function in System.Globalization.TextInfo class

Here is a sample (taken from MSDN link below):

using System;
using System.Globalization;

public class Example
{
   public static void Main()
   {
      string[] values = { "a tale of two cities", "gROWL to the rescue",
                          "inside the US government", "sports and MLB baseball",
                          "The Return of Sherlock Holmes", "UNICEF and children"};

      TextInfo ti = CultureInfo.CurrentCulture.TextInfo;
      foreach (var value in values)
         Console.WriteLine("{0} --> {1}", value, ti.ToTitleCase(value));
   }
}
// The example displays the following output: 
//    a tale of two cities --> A Tale Of Two Cities 
//    gROWL to the rescue --> Growl To The Rescue 
//    inside the US government --> Inside The US Government 
//    sports and MLB baseball --> Sports And MLB Baseball 
//    The Return of Sherlock Holmes --> The Return Of Sherlock Holmes 
//    UNICEF and children --> UNICEF And Children

Resources

Tip of the Day, Tips & Tricks comments edit

AWS is a very critical service for businesses and enabling a 2nd factor authentication would be very helpful to increase security.

Step 0: Install Google Authenticator or a similar OTP generator tool on your mobile device.

Step 1: Logon to your account and select Security Credentials from the menu on the upper right corner. Then click on Activate MFA button. AWS MFA Step 1

Here you have two options. You can choose to use a hardware device if you choose to purchase one from Amazon. At the time of this writing the price on Amazon was $12.99 so it’s also an affordable option. I decided to use Google Authenticator because it was already installed and active on my phone and I didn’t want to manage another piece of hardware.

Step 2: It will display a QR code which you can scan using Google Authenticator. Once the scan is completed you will need to enter 2 consecutive numbers the device generates. Once the device is synchronized it will be ready to use. You can verify it by checking its status:

AWS MFA Step 2

Step 3: Log out and log back in to see the results. After the password screen you should see an additional screen asking for the authentication code.

AWS MFA Step 3

Resources

Tip of the Day, Tips & Tricks comments edit

If you are working with large files, transferring them to S3 correctly might be an issue. To ensure data integrity you can send the MD5 hash of the file in Content-MD5 header.

After the transfer S3 compares the MD5 value of the transferred file with the value you provided. If they don’t match it returns an error. This is an easy and great way to ensure the file has been uploaded safe and sound.

Resources

Tip of the Day, Tips & Tricks comments edit

The feature exists but a bit buried away. To create insert statements for table(s):

  1. Right click on the database and go to Tasks -> Generate Scripts
  2. Select the tables that you want to generate the script
  3. In “Set scripting options” page click on the Advanced button
  4. Under the General category, locate “Types of data to script”. It contains 3 options: Data Only, Schema and Data, and Schema Only. Default is Schema only,if you select an option with data in it you can get the data with the insert statements.

Generating Insert Scripts With SSMS

Automation, System Administration comments edit

When moving to a new machine, re-installing all the applications is sounds a very daunting task. Linux distros have been enjoying the ease of package repositories for quite a long time and users can install almost everything they need by package manager applications like apt-get and yum. Chocolatey aims to bring the same convenience to Windows.

Easy Install Indeed!

Installation is a breeze as shown in their website. By using some Powershell and NuGet magic you download a script which then downloads Chocolatey package from the repository. You may have noticed the URL is very similar to nuget.org’s package path. That’s simply because it uses NuGet under the hood. Even the user interface is just a skinned version of NuGet Gallery (which is open source and can be downloaded from GitHub).

Chocolatey-Install

Automate all installations

After Chocolatey is installed all you have to do is browse the packages and select the ones you want to install with an easy command such as:

choco install atom

And after a bunch of colourful messages you’re good to go.

Chocolatey-Install

Are we done?

Most core tools for development and everyday use (SysInternals, Fiddler, Chrome, FireFox, NodeJS, Python, VLC, Paint.NET etc.) can be installed automatically using Chocolatey. The major applications such as Visual Studio and SQL Server still need some love and your caring hands but I think automating installation of tens of applications in such an easy way is priceless.

Resources

software review comments edit

Many times I used file hashes to verify that it’s been correctly downloaded or uploaded. It’s especially useful with large files where things are more likely to go wrong. That’s why most MD5 or SHA1 hashes provided with many large file downloads such Linux distros.

Ubuntu MD5 Hashes

Best shell extension ever: HashTab!

This is a just quick plug about one of my favourite tools in Windows: HashTab. The best thing about it is that it’s a shell extension. So to find out the hash values of a file is simple as right-clicking and selecting File Hashes tab. So, using the MD5 has example above, if you have downloaded ubuntu-14.04.1-server-amd64.iso all you have to do is check it’s properties.

File Hashes

HahsTab also monitors the clipboard. So if you copy the MD5 value from Ubuntu’s site it automatically compares it with the calculated hash values and confirms file integrity with a large shiny green check!

Out of the box it displays most common 3 hash algorithms but if you need more supported algorithms.

HashTab Settings

Resources

Development comments edit

Topshelf

In almost all projects I’ve worked there was always a need for scheduled background jobs. Windows Services are well-suited for the job but when it comes to deployment and debugging they fall short. This is where TopShelf comes to rescue!

TopShelf in action

With TopShelf all you need to do is create a Console Application as your Windows Service and initialize TopShelf in the Main method:

Topshelf

Configuring it is quite easy and straightforward. It also has a nice comprehensive documentation on its official site. You can define all the parameters like the account that will run the service, the recovery policy and start-up type.

What about scheduling

Scheduling is also a crucial feature in background services. TopShelf doesn’t help with that but that’s easy to fix with FluentScheduler.

Topshelf

FluentScheduler is yet another great library available at NuGet which makes scheduling a breeze. No need to fiddle with timers, it can be defined very easily.

And here is the sample code:

Resources

Development comments edit

If you have multiple accounts for git providers (i.e multiple accounts on Github and/or Bitbucket) then you’d need to update your SSH configuration to be able to access all your repositories seamlessly. Of course you can use HTTPS but then you’d have to enter username and password every time.

If you don’t specify which key to use for each account SSH agent will try the default key if there is one (id_rsa) and will most likely fail if you didn’t grant access to that key in your git provider settings.

SSH Access Denied

To resolve the issue you need to create a config file under .ssh folder that looks like this:

SSH Config

If you are just using one account per provider you don’t need to create multiple keys, you can just use id_rsa for both accounts. But if you have multiple accounts for a provider you’d need a key for each account. In the example configuration above I used a new key for BitBucket anyway. After creating the config file and adding the keys to your accounts you can start cloning repositories from various sources.

Final step to accomplish this is to use the hostname you set in the config file when cloning the repository. For example when you copy the SSL clone URL it looks something like this: git@github.com:{account name}/{repository name}.git. So let’s say it’s a repository from the corporate account in my sample config file then I’d have to modify the URL as follows: git@github-corporate:{account name}/{repository name}.git so that the correct host name and RSA file can be used.

SSH Success

Resources

Development, Lego, Mindstorms, EV3 comments edit

I had my eye on the new Lego Mindstorms set for a while. Finally I decided to order it from Amazon. It’s still a bit pricey but I think it’s worth it. I read nice review of EV3 here which also includes comparisons to the previous generation of the Mindstorms kit.

Lego Mindstorms EV3

Programming EV3: The Official Way

Programming the kit is very easy using Lego’s official graphical programming tool6. You just have to drag and drop the components and fiddle with the parameters. Check out the following very basic application:

Lego Mindstorms EV3

It powers the motors connected to ports B and C. It keeps doing that in a loop as long as the value read from Infrared Sensor is larger than 20. If there is an object closer to 20 centimetres it breaks and ends the program. And here’s the output in action:

Programming EV3: The .NET Way

The .NET API is an open-source project and code can be found on CodePlex. I recommend watching the introductory video which shows the basics. It shows how to move the robot by sending direct commands to turn the motors and how to read values from the sensors. The following part the test application shows the event handlers for the direction buttons and setup code to connect to the brick.

Lego Mindstorms EV3

I also added a similar implementation of the NXT-G program above. It’s a while loop which breaks when the value from the IR sensor is lower than 10.

Lego Mindstorms EV3

And voila! Here is a clever robot that senses the object in front of it and avoids the collision by stopping!

You can find the source code for the test application on GitHub

Tips & Tricks

  • When I first implemented the NXT-G equivalent version the sensor value wasn’t updating properly. I checked the discussion forums in the CodePlex project page and found out that other people were having a similar issue. A workaround was adding the Thread.Sleep(10) line. After that I could read the updated sensor values without any problems. Although it doesn’t feel like the right solution it works fine as a temporary workaround.

  • During the testing and debugging I managed to crash the Lego brick a few times. First I feared I actually “bricked” the brick but luckily a reset resolved it. Resetting the brick is not obvious though, I had to check the manual for that. So in case you need to reset it you have to hold down Back, Center and Left buttons. Then release the Back button when the screen goes blank and release the other two when the screen says “Starting” as shown in EV3 User Guide

Resources

  1. Lego EV3 review and comparison to NXT 2.0
  2. API code on CodePlex
  3. Channel9 video on programming EV3 with .NET API
  4. Sample Track3r robot project and building instructions
  5. EV3 User Guide
  6. Lego software
  7. Codeplex discussion on IR sensor not updating values
  8. Source code for the test application

Development comments edit

My previous post was about grid-based clustering. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) takes another approach called density-based clustering. It grows regions with high density (above threshold provided) into clusters and discovers clusters of arbitrary shape.

DBSCAN Implementation

First off the data is loaded and a distance matrix is calculated based on the data points.

DBSCAN 1

The algorithm visits every data point and finds its neighbours. If the neighbours are dense enough than the cluster is expanded to include those points as well.

DBSCAN 3

So data points that are close enough to each other are included in the same cluster.

DBSCAN 2

Joining dense clusters is a similar approach taken in grid clustering. The difference is this way the clusters can have arbitrary shapes.

Resources

fitness, game comments edit

I’ve been playing this game for about 6 months. It’s an interesting game in a number of ways.

Ingress Basics

I discussed the basics of the game in my previous post. If you want to find out more about the rules and game play you can check out that post here. For me the definite guide for the game is a site called decodeingress.me. You can find the latest news and updates about the game. It is updated quite frequently so it pays off to keep an eye on the latest changes in the game. Anyway, what I want to log in this post is mainly my experiences.

What’s It All About?

Feeling proud of achievements in a virtual world is a bit silly but I guess it goes for all sorts of games. Here’s a snapshot of the fields I created a while back:

Ingress Field

In theory the goal is to create and try to maintain those fields so that you capture “mind units”. The team who has more mind units wins (actually the game never ends so I guess there are no winners or losers. The worst case of scenario is your team falls behind). Once you start playing it’s all about levelling up but after a certain level you seek more. Here’s my view of pros and cons of the game:

Pros

  • Exercise: This is the main reason I started the game. It motivates you walk more. I stopped taking the bus from town centre and started walking home because there are some portals that need to be captured along the way!</li>
  • Get to know the city better: Before this game, I knew where my house was, where the nearest train station was and where the nearest Sainsbury’s store was. Now I know all the churches, parks and many other public buildings around my neighbourhood and around the city.
  • New friends: You can play the game alone with no problems and can have fun at first. But after a while it becomes a bit boring because you do the same activities repeatedly. By joining the local community you get to know the people playing around you.

Cons

  • Repetition: The portals have a mitigation cap meaning that you cannot keep enforcing the portals. So it can be taken down by a single player.After some time it becomes quite repetitive. You capture the portals and create fields. A few hours later someone blows them up and the next morning you have to do the same thing all over again. This might get the fun away especially If you are playing alone.
  • Can be time-consuming: Because of the above fact, portals require maintenance especially in crowded areas. If you become obsessed about keeping the portals yours you might end up spending a lot of time.

For me pros of the game far outweigh the cons. I can control the time I spend on the game and play with other people so it becomes a social event.

Some Tips

Just off the top of my head here are a few tips for beginners:

  • Deploy resonators as far as possible from the center so that they don’t all get hit hard at the same time.
  • When firing XM bursters, long press fire button and release it when XM is accumulated in the center. It allows you to get bonus firepower up to %20
  • Links make portals stronger so try to create as many links and fields you can.

Resources

Development comments edit

CLIQUE (CLustering In QUEst) algorithm is a grid-based clustering algorithm that partitions each dimension of the dataset as a grid.

CLIQUE Implementation

The algorithm starts by assigning each data point to the closest grid in the matrix.

CLIQUE 1

After loading the data it looks something like this:

CLIQUE 2

In order for a grid cell to be considered as “dense” it has to have data points more than or equal to the threshold. In this sample the threshold was provided as 2. So after the adjacent dense cells are joined together we get 3 clusters:

CLIQUE 3

Resources

Development comments edit

A rough set is an approximation of a set that gives the lower and upper borders of the set.

Rough set implementation

The sample implementation starts off with reading the data set and the configuration file. The object and attribute index lists are acquired from the configuration. The items that have indices specified in the object list are considered as the set of objects and the rest of the items are the complement of the data set.

RoughSet 1

If a given index in attribute index list equal in both object list and complementary object list are not equal the object is considered to belong to the negative border, if not it belongs to the positive border.

Sample Output

RoughSet Sample Output

Resources

Development comments edit

Genetic algorithm sounds fascinating to me as in a way it mimics evolution. It’s quite easy to implement and understand.

Genetic Algorithm

The algorithm has 3 stages:

  • Selection: Every generation is subjected to a selection process which eliminates the unfit ones.
  • Cross-over: This happens randomly among the selected generation and some genomes swap genes.
  • Mutation: Some genes are mutated randomly based on a given threshold value for mutation.

Implementation

The implementation starts with generating random bit sequences to represent genomes. The length of the genome and the number of genomes in a generation are specified in the configuration.

Genetic Algorithm 1

First generation is calculated randomly and their fitness is calculated based on the fitness function the system uses. After the groundwork has been done we have a genome pool each with their own fitness values and selection ranges. The width of the selection range is determined by the fitness of the genome. So that the larger the range the more likely they are selected for the next generation.

Genetic Algorithm 2

In selection process, a random value is generated and the genome whose selection range includes that value is selected. Same genome can be selected more than once.

Genetic Algorithm 3

After the next generation is selected and their new fitness values are calculated cross-over starts. In cross-over 2 random genomes are picked and they swap genes. The point in the genome string is specified in the configuration (3 in this example)

Genetic Algorithm 4

The final phase of the algorithm is mutation. In this step a number of genomes are picked and some bits is flipped randomly.

Genetic Algorithm 5

The most important thing about this algorithm is having a good fitness function that can accurately compute the fitness values for a given feature.

Resources

Development comments edit

Another data mining algorithm: AGNES (Agglomerative Nesting)

AGNES Algorithm

AGNES takes a different approach than k-means which was the subject of my previous post. Initially it considers all data points as a separate cluster.

AGNES 1

Then finds the minimum distance between clusters and merges the closest clusters:

AGNES 2

The resulting cluster is added to the all cluster list the merged clusters are removed as they are no longer valid.

Resources

Development comments edit

I’m keeping on reviving my old projects. This is the second data mining algorithm implementation. It is another clustering algorithm called k-means.

k-means Algorithm

Algorithm groups and creates k clusters from n data points. First the cluster centres are picked randomly from the data points. Then the entire dataset is iterated and all points are assigned to their closest cluster. Closest cluster is determined by measuring the distance of the data point to the centroid of the clusters. This process is repeated until there is no change in the dataset and all points are assigned to the closest ones.

K-means Results

Implementation

The project contains 6 libraries:

  • VP.KMeans.Core: Core library including the algorithm implementation
  • VP.KMeansClient.GUI: User interface for entering the parameters and plotting the clusters
  • VP.KMeansClient.Console: Console user interface. No fancy plots, just an output file is generated
  • VP.KMeans.DataGenerator.Core: Library to generate test data
  • VP.KMeans.DataGenerator.Console: Console application to feed the core library to generate test data
  • CPI.Plot3D: External library to plot the results

Resources

Development comments edit

I was digging through some old projects and found out the Data Mining and Machine Learning projects I implemented. Instead of letting them gather dust(!) in the hard disk I decided to review and publish the source code. This will also give me a chance to revise my data mining knowledge. Let’s start with Apriori algorithm.

Apriori Algorithm

It is an algorithm to determine to most frequent items in a given collection. The term “frequent” is defined by a given threshold or “minimum support count”.

Implementation

The project is implemented in Java. It uses a Hashtable to keep count of itemsets in the database. The algorithm starts with finding the most frequent 1-item sets first. Then using the previous most frequent item list of item size k, it generates candidate item list of size k+1.

For example, for the sample data below and a threshold 4, 1,2,3,4,5 are all frequent 1-itemsets. From this list we generate a 2-item candidate list (all 10 combinations) and check if the subsets are also frequent. For 1-itemsets they are all frequent so they all pass pruning. Then we count the occurences of these candidates. Only 7 of them are equal to or greater than the threshold. From this list we generate our 3-item candidates. Such as 1,2 and 1,4 combined to generate 1,2,4. Then we count the occurrence of 3-itemsets and prune the results by checking all of its subsets.

Apriori Check Subsets

The idea of pruning is if the there are some infrequent subsets inside an itemset then the larger set cannot be frequent so it is removed from the candidate list. (1,2,3 is a candidate but as 2,3 is not a frequent 2-itemset it is removed from the candidate list) This process helps improve the performance of the algorithm as it reduced the number of iterations.

Output

1,3,4,5

1,2,4,5

1,2,4

1,3

3

1,5

1

3

1,3,5

4

1,2,4

2

1,2

3,4

3,5

1,3,4

2

3,5

1,2,3,4

Results:

Apriori Results

Source Code

I created a public repository for the source code. You can check it out here if it tickles your fancy!

Resources

review, gadget comments edit

Fitbit Flex

I bought this about 2 months ago. I wore it every single day since then and I just loved it! It is basically a motion sensor that detects and keeps track of your daily movements. You can set your own daily goals steps you walked, distance you took or calories you burnt.

I can keep track of distance by using my Garmin ForeRunner 10 (which I reviewed here) but this one is easier to use because it does everything in the background. Garmin takes a few minutes to start because it needs to find your GPS coordinates but that’s not the case for Flex.

Flex lets you keep track of distance, active minutes, calories and steps.

Fitbit Goals

Also you can log your weight, other exercises and food intake so that you can calculate the net calories throughout a time period.

Another great feature about it is tracking your sleep quality. You can use this data in conjunction with your daily activities.

Fitbit Sleep

And here is my favourite feature: Alarm! It turns out if something on your wrist start to vibrate you wake up. Instantly! Of course I keep my phone’s alarm still running as a fallback method but this one works pretty good.

Conclusion

I bought it for £68 and as of now it is listed as £83 on Amazon. Apparently the price fluctuates a bit but I think £70 – 80 price range is good for this product. I charge once every 3 days or so. Other than that I completely forget about it while it does its job in the background. It motivates you to reach your goals and be more active in general and the silent alarm is absolutely fantastic. I’d recommend this to anyone who would like to have more exercise.

Resources

review, gadget comments edit

It’s been 5 months since I’ve gone to a metal show. The last one was Bloodstock Festival and Lamb of God was headlining that. The playlist on this show had no surprises (Just the way we like it). O2 Academy Brixton twitter account kindly tweeted the playlist:

Lamb of God Playlist

“Open up the dance floor!”

Apparently this was a sold out show. It was quite crowded and people went nuts the second LoG hit the stage. Normally there is a small area for moshpit in front of the stage right behind the first 2-3 rows or people. But from where I stood it looked like the half of the floor was a giant moshpit area. Check this short clip to get a sense of the atmosphere (sound quality is awful so make sure you don’t start it at full volume).

Notes

  • Number of girls seemed to be a lot more than a usual metal concert. Number of girls in the moshpit was definitely a lot more than a usual metal concert!
  • Guitarist Mark Morton didn’t take the stage. At the beginning of the show Randy introduced Paul Waggoner as his temporary replacement and explained Mark’s absence as “He had to attend to a family business and family always comes first in this band”
  • Beer prices were ridiculous (or maybe I should say tragic): £4.75 for a Carlsberg. I only had two. The feeling of being ripped off ruins the fun of it but mainly the long queues in the front of the bars were a huge turn off.

Lamb of God Image 1

Lamb of God Image 2

Lamb of God Image 3

Lamb of God Image 4

Lamb of God Image 5

Lamb of God Image 6