design, development, design patterns, csharp comments edit

A few days ago I published a post discussing Factory Method pattern. This article is about the other factory design pattern: Abstract Factory.

Use case: Switching between configuration sources easily

Imagine in a C# application you accessed ConfigurationManager.AppSettings whenever you needed a value from the configuration. This would essentially be hardcoding the configuration source and it would be hard to change if you needed to switch to another configuration source (database, web service, etc). A nicer way would be to “outsource” the creation of configuration source to another class.

What is Abstract Factory?

Here’s the official definition from GoF:

Provide an interface for creating families of related or dependent objects without specifying their concrete classes.

Implementation

The application first composes the main class (ArticleFeedGenerator) with the services it will use and starts the process.

static void Main(string[] args)
{
    IConfigurationFactory configFactory = new AppConfigConfigurationFactory();

    IApiSettings apiSettings = configFactory.GetApiSettings();
    IFeedSettings feedSettings = configFactory.GetFeedSettings();
    IFeedServiceSettings feedServiceSettings = configFactory.GetFeedServiceSettings();
    IS3PublisherSettings s3PublishSettings = configFactory.GetS3PublisherSettings();
    IOfflineClientSettings offlineClientSettings = configFactory.GetOfflineClientSettings();

    var rareburgClient = new OfflineRareburgClient(offlineClientSettings);
    var rareburgArticleFeedService = new RareburgArticleFeedService(feedServiceSettings);
    var publishService = new S3PublishService(s3PublishSettings, feedSettings);
    var feedGenerator = new ArticleFeedGenerator(rareburgClient, rareburgArticleFeedService, publishService, feedSettings);

    feedGenerator.Run();
}

This version uses AppConfigConfigurationFactory to get the values from the App.config. When I need to switch to DynamoDB which I used in this example all I have to do is replace one line of code in the application:

var configFactory = new DynamoDBConfigurationFactory();

With this change alone we are essentially replacing a whole family of related classes.

On the factory floor

The abstract factory and the concrete factories implement it are shown below:

Concrete configuration factories create the classes that deal with specific configuration values (concrete products). For instance AppConfigConfigurationFactory looks like this (simplified for brevity):

public class AppConfigConfigurationFactory : IConfigurationFactory
{
    public IApiSettings GetApiSettings()
    {
        return new AppConfigApiSettings();
    }

    public IFeedServiceSettings GetFeedServiceSettings()
    {
        return new AppConfigFeedServiceSettings();
    }
}

Similarly, DynamoDBConfigurationFactory is responsible for creating concrete classes that access DynamoDB values:

public class DynamoDBConfigurationFactory : IConfigurationFactory
{
    protected Table _configTable;
    
    public DynamoDBConfigurationFactory()
    {
        AmazonDynamoDBClient dynmamoClient = new AmazonDynamoDBClient("accessKey", "secretKey", RegionEndpoint.EUWest1);
        _configTable = Table.LoadTable(dynmamoClient, "tableName");
    }
    
    public IApiSettings GetApiSettings()
    {
        return new DynamoDBApiSettings(_configTable);
    }

    public IFeedServiceSettings GetFeedServiceSettings()
    {
        return new DynamoDBFeedServiceSettings(_configTable);
    }
}

Notice all the “concrete products” implement the same “abstract product” interface and hence they are interchangable. With the product classes in the picture the diagram now looks like this:

Finally let’s have a look at the concrete objects that carry out the actual job. For example the IApiSettings exposes 2 string properties:

public interface IApiSettings
{
    string ApiKey { get; }
    string ApiEndPoint { get; }
}

If we want to read these values from App.config it’s very straightforward:

public class AppConfigApiSettings : IApiSettings
{
    public string ApiKey
    {
        get { return ConfigurationManager.AppSettings["Rareburg.ApiKey"]; }
    }

    public string ApiEndPoint
    {
        get { return ConfigurationManager.AppSettings["Rareburg.ApiEndPoint"]; }
    }
}

The DynamoDB version is fairly more complex but it makes no difference from the consumer’s point of view. Here GetValue is a method in the base class that returns the value from the encapsulated Amazon.DynamoDBv2.DocumentModel.Table object.

public class DynamoDBApiSettings : DynamoDBSettingsBase, IApiSettings
{
    public DynamoDBApiSettings(Table configTable)
        : base (configTable)
    {
    }

    public string ApiKey
    {
        get { return GetValue("Rareburg.ApiKey"); }
    }

    public string ApiEndPoint
    {
        get { return GetValue("Rareburg.ApiEndPoint"); }
    }
}

The concrete factory is responsible for creating the concrete classes it uses. So the client is completely oblivious to the classes such as DynamoDBApiSettings or AppConfigApiSettings. This means we can add a whole new set of configuration classes (i.e. a web service) and all we have to change in the client code will be one line where we instantiate the concrete factory.

This approach also allows us to be more flexible with the concerete class implementations. For example DynamoDB config class family requires a Table object in their constructors. To avoid code repetition I derived them all from a base class and moved the table to the base but the that doesn’t change anything in the client code.

Resources

design, development, design patterns, csharp comments edit

I recently developed a toy project called Rareburg article feed generator. It basically gets the articles from Rareburg.com (a marketplace for collectors) and creates an RSS feed. One challenge I had was picking the feed formatter class (RSS vs Atom). I utilised Factory Method design pattern to solve that and this post discusses the usage of the pattern over that use case.

Use case: Creating RSS feed formatter

I wanted my feed generator to support both RSS and Atom feeds based on the value specified in the configuration. In order to pass the XML validations they needed to modified a bit and I wanted these details hidden from the client code.

What is Factory Method?

Here’s the official definition from GoF:

Define an interface for creating an object, but let subclasses decide which class to instantiate. Factory Method lets a class defer instantiation to subclasses

Factory Method pattern is one of the creational patterns which make it easy to separate object creation from the actual system. The actual business logic doesn’t need to care about these object creation details.

Implementation

To see the implementation at a glance let’s have a look at the class diagram:

The base abstract class is called ArticleFeedGenerator (it corresponds to Creator in the GoF book). Here’s a shortened version:

public abstract class ArticleFeedGenerator
{
    private SyndicationFeedFormatter _feedFormatter;

    // Factory method
    public abstract SyndicationFeedFormatter CreateFeedFormatter();

    public void Run()
    {
        var allArticles = _feedDataClient.GetAllArticles();
        _feed = _feedService.GetFeed(allArticles);

        _feedFormatter = CreateFeedFormatter();

        _publishService.Publish(_feedFormatter);
    }
}

ArticleFeedGenerator does all the work except creating a conccrete implementation of SyndicationFeedFormatter. It delegates it to the derived class who provide the implementation for CreateFeedFormatter abstract method.

And here comes the concrete implementations (which correspond to ConcreteCreator)

public class AtomFeedGenerator : ArticleFeedGenerator
{
    public AtomFeedGenerator(
        IFeedDataClient feedDataClient,
        IFeedService feedService,
        IPublishService publishService,
        IFeedSettings feedSettings)
        : base (feedDataClient, feedService, publishService, feedSettings)
    {
    }

    public override SyndicationFeedFormatter CreateFeedFormatter()
    {
        return new Atom10FeedFormatter(_feed);
    }
}

In order to pass RSS validations RSS formatter needed more “love” than the Atom version above:

public class RssFeedGenerator : ArticleFeedGenerator
{
    public RssFeedGenerator(
        IFeedDataClient feedDataClient,
        IFeedService feedService,
        IPublishService publishService,
        IFeedSettings feedSettings)
        : base (feedDataClient, feedService, publishService, feedSettings)
    {
    }
    
    public override SyndicationFeedFormatter CreateFeedFormatter()
    {
        var formatter = new Rss20FeedFormatter(_feed);
        formatter.SerializeExtensionsAsAtom = false;
        XNamespace atom = "http://www.w3.org/2005/Atom";
        _feed.AttributeExtensions.Add(new XmlQualifiedName("atom", XNamespace.Xmlns.NamespaceName), atom.NamespaceName);
        _feed.ElementExtensions.Add(new XElement(atom + "link", new XAttribute("href", _feedSettings.FeedUrl), new XAttribute("rel", "self"), new XAttribute("type", "application/rss+xml")));
        return formatter;
    }
}

Concerete creators create concrete products but the factory method returns the abstract product which ArticleFeedGenerator (abstract creator) works with.

To avoid hard-coding the class name, a helper method called CreateFeedGenerator is added. The client code calls this method to get either a AtomFeedGenerator or a RssFeedGenerator based on configuration value.

class Program
{
    static void Main(string[] args)
    {
        var feedSettings = new AppConfigFeedSettings();
        ArticleFeedGenerator feedGenerator = CreateFeedGenerator(feedSettings);
        feedGenerator.Run();
    }
    
    private static ArticleFeedGenerator CreateFeedGenerator(IFeedSettings feedSettings)
    {
        string feedFormat = feedSettings.FeedFormat;
        switch (feedFormat.ToLower())
        {
            case "atom": return new AtomFeedGenerator(new RareburgClient(), new RareburgArticleFeedService(), new S3PublishService(), feedSettings);
            case "rss": return new RssFeedGenerator(new RareburgClient(), new RareburgArticleFeedService(), new S3PublishService(), feedSettings);
            default: throw new ArgumentException("Unknown feed format");
        }
    }
}

Resources

design, development, design patterns comments edit

Often times we need to change some algorithm with another without changing the client code that’s consuming it. In this post I want to show a use case I came across and utilised Strategy pattern.

What is it?

Here’s the official definition of the pattern from GoF book:

Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

Use case: Get external IP address

In an application I was working on I needed to get the external IP address of the computer the application is running on. There are various ways to achieve that. This looked like a good opprtunity to use the Strategy pattern as I wanted to be able switch between the different methods easily.

Implementation

The interface is quite simple:

public interface IIpCheckStrategy
{
    string GetExternalIp();
}

Some services return their data in JSON format, some have extra text in it. But encapsulating the algorithms in their own classes this way, the clint code doesn’t have to worry about parsing these various return values. It’s handled in each class. If one service changes it’s output and breaks the implementation I can recover just by changing the code instantiates the class.

The concrete implementations of the interface are below. They implement the IIpCheckStrategy and are responsible for getting the data and return parsed IP address as string.

AWS IP Checker:

public class AwsIPCheckStrategy : IIpCheckStrategy
{
    public string GetExternalIp()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://checkip.amazonaws.com/");
            string result = client.GetStringAsync("").Result;
            return result.TrimEnd('\n');
        }
    }
}

DynDns IP Checker:

public class DynDnsIPCheckStrategy : IIpCheckStrategy
{
    public string GetExternalIp()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://checkip.dyndns.org/");
            HttpResponseMessage response = client.GetAsync("").Result;
            return HelperMethods.ExtractIPAddress(response.Content.ReadAsStringAsync().Result);
        }
    }
}

Custom IP Checker:

public class CustomIpCheckStrategy : IIpCheckStrategy
{
    public string GetExternalIp()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://check-ip.herokuapp.com/");
            client.DefaultRequestHeaders.Accept.Clear();
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

            HttpResponseMessage response = client.GetAsync("").Result;
            string json = response.Content.ReadAsStringAsync().Result;
            dynamic ip = Newtonsoft.Json.JsonConvert.DeserializeObject(json);
            string result = ip.ipAddress;
            return result;
        }
    }
}

Choosing the algorithm

The consumer of the algorithm can pick any class that implements IIpCheckStrategy and switch between them. For example:

class StrategyClient1
{
    public void Execute()
    {
        IIpCheckStrategy ipChecker;

        ipChecker = new DynDnsIPCheckStrategy();
        Console.WriteLine(ipChecker.GetExternalIp());

        ipChecker = new AwsIPCheckStrategy();
        Console.WriteLine(ipChecker.GetExternalIp());

        ipChecker = new CustomIpCheckStrategy();
        Console.WriteLine(ipChecker.GetExternalIp());

        Console.ReadKey();
    }
}

Also the class name to be used can be stored in the configuration in some cases so that it can be changed at runtime without recompiling the application. For instance:

class StrategyClient2
{
    public void Execute()
    {
        string ipcheckerTypeName = ConfigurationManager.AppSettings["IPChecker"];

        IIpCheckStrategy ipchecker = Assembly.GetExecutingAssembly().CreateInstance(ipcheckerTypeName) as IIpCheckStrategy;

        Console.WriteLine(ipchecker.GetExternalIp());
    }
}

and the appSettings in the configuration would look like this:

<appSettings>
	<add key="IPChecker" value="Strategy.AwsIPCheckStrategy"/>
</appSettings>

Resources

development, aws, lambda comments edit

Just two days ago I was jumping through a lot of hoops (maintain state on S3, subscribe to public SNS topics) just to schedule a Lambda function, as explained here. If I knew all these problems were to be solved in a day, I would just wait! In re-Invent 2015 event AWS announced scheduled event support for Lambda functions.

After tweaking around a little bit with the sample I set my schedule like this which will run first day of every month at 10:00:

cron(0 10 1 * ? *)

That’s all there is to it for a monthly schedule. Reliable and no need for my state management workarounds. So the final code becomes as simple as sending an emali when invoked:

Resources

development, aws, lambda, s3, sns comments edit

UPDATE: Yesterday (October 8th, 2015) Amazon announced official support for scheduled events so I updated my function to use this feature. For the most up-to-date version of this project please visit the updated version

I have a great accountant but he has one flaw: I have to ask for the invoice every month! While waiting for him to automate the process, I decided to automate what I can on my end. There are many ways to skin a cat, as the saying goes, the way I picked for this task was developing an AWS Lambda funciton and trigger it by subscribing to a public SNS topic.

Step 1: Prepare a function to send emails

Developing a simple node.js function that sends E-mails was simple. First I needed the install two modules:

npm install nodemailer
npm install nodemailer-smtp-transport

And the function is straightforward:

var transporter = nodemailer.createTransport(smtpTransport({
    host: 'email-smtp.eu-west-1.amazonaws.com',
    port: 587,
    auth: {
        user: '{ACCESS KEY}',
        pass: '{SECRET KEY}'
    }
}));

var text = 'Hi, Invoice! Thanks!';

var mailOptions = {
    from: 'from@me.net',
    to: 'to@someone.net',
    bcc: 'me2@me.com',
    subject: 'Invoice',
    text: text 
};

transporter.sendMail(mailOptions, function(error, info){
      if(error){
		console.log(error);
      }else{
		console.log('Message sent');
      }
  });

The challenge was deployment as the script had some dependencies. If you choose Edit Inline and just paste the script you would get an error like this

"errorMessage": "Cannot find module 'nodemailer'",

But it’s very easy to deploy a full package with dependencies. Just zip everything in the folder (wihtout the folder itself) and upload the zip file. The downside of this method is you can no longer edit the code inline. So even just for fixing a trivial typo you need to re-zip and re-upload.

Step 2: Schedule the process

One simple method to schedule the process is to invoke the method using Powershell and schedule a task to run the script:

Invoke-LMFunction -FunctionName automatedEmails -AccessKey accessKey -SecretKey secretKey -Region eu-west-1

But I don’t want a dependency on any machine (local or EC2 instance). Otherwise I could write a few lines of code in C# to do the same job anyway. The idea of using Lambda is to avoid maintenance and let everything run on infrastructure that’s maintained by AWS.

Unreliable Town Clock

Unfortunately AWS doesn’t provide an easy method to schedule Lambda function invocations. For the sake of simplicity I decided to use Unreliable Town Clock (UTC) which is essentially a public SNS topic that sends “chime” messages every 15 minutes.

Since all I need is one email, I don’t care if it skips a beat or two as long as it chimes at least once throughout the day.

State management

Of course to avoid bombarding my accountant with emails I have to maintain a state so that I would only send one email per month. But Lambda functions must be stateless. Some alternatives are using AWS S3 or DynamoDB. Since all I need is one simple integer value I decided to store in a text file on S3. So first I download the log file and check the last sent email month:

function downloadLog(next) {
	s3.getObject({
			Bucket: bucketName,
			Key: fileName
		},
		next);

function checkDate(response, next) {
	var currentDay = parseInt(event.Records[0].Sns.Message.day);
	currentMonth = parseInt(event.Records[0].Sns.Message.month);
	var lastMailMonth = parseInt(response.Body.toString());
	if (isNaN(lastMailMonth)) {
		lastMailMonth = currentMonth - 1;
	}
	if ((currentDay == targetDayOfMonth) && (currentMonth > lastMailMonth)) {
		next();
	}
}

Putting it together

So putting it all together the final code is:

Let’s see if it’s going to help me get my invoices automatically!

Conclusion

  • A better approach would be to check emails for the invoice and send only if it wasn’t received already. Also a copule of reminders after the initial email would be nice. But as my new resolution is to progress in small, incremental steps I’ll call it version 1.0 and leave the remaining tasks for a later version.

  • My main goal was to achieve this task without having to worry about the infrastructure. I still don’t but that’s only because a nice guy (namely Eric Hammond) decided to setup a public service for the rest of us.

  • During my research I came across a few references saying that the same task can be done using AWS Simple Workflow (SWF). I haven’t used this service before. Looked complicated and felt like there is a steep learning curve to go through. In Version 2 I should look into SWF which would…

    • allow me to handle a complex workflow
    • make dependency to public SNS topic redundant
    • handle state properly

Resources

csharp, development, WPF comments edit

I like scanning all my documents and keep a digital version as well as the dead-tree version just in case. But storing the documents as individual image files is too messy. There are various solutions to merge image files into a single PDF but I don’t want to upload my sensitive documents to unknown parties so I rolled my own: Image2PDF

Implementation

I decided to go with a WPF application rathen than a web-based solution because it would be unnecessary to upload a bunch of image files to cloud and download the PDF back. It’s a lot of network traffic for something that can be handled locally much faster.

Image2PDF a simple WPF application developed in C#. I’ve used iTextSharp library to create the PDFs. It’s a great open-source library with lots of capabilities.

I’ve also started using SyncFusion WPF control library recently. The entire control suite is free of charge (which is always a great price!). It has a Wizard control which I decided to go with. I think using a wizard and is cleaner UI instead of dumping every control on one window.

Usage

As you might expect you provide the list of input images, re-order them if needed, choose the path for the output folder and go!

Step 1: Select input

Page 1: Specify input image files

Step 2: Select output

Page 2: Specify output PDF path

Step 3: Go!

Page 3: Run!

Simples!

Conclusion

In the future I’m planning to add various PDF operarions and enhance the functionality but since shipping is a feature I’ll limit the scope for now. After all, this was the main feature I needed anyway.

Resources

csharp, neo4j, cypher, json, development comments edit

I have a simple JSON file that contained a bunch of users with their followers which looked like this:

  {
    "user-id": 2,
    "username": "user_2",
    "avatar": "URL",
    "following": [
      "user_10",
      "user_6",
      "user_1"
    ],
    "followers": [
      "user_10"
    ]
  }

It felt like a good exercise would be to import that data into a graph database as it wasn’t something I had done before.

As my go-to language is C# and I had some experience with Cypher before, my initial instinct was to develop a tool to generate Cypher statements from the JSON.

First I create the nodes:

private void GenerateNodesCypher()
{
    string filename = @"..\..\output\nodes.cql";
    var output = new StringBuilder();

    foreach (var user in _userList)
    {
        string s = $"CREATE ({user.username}:User {{ userid: {user.userid} , username: '{user.username}', avatar: '{user.avatar}'  }} ); ";
        output.AppendLine(s);
    }

    File.WriteAllText(filename, output.ToString());
}

and then the relationships:

private void GenerateRelationshipsCypher()
{
    string filename = @"..\..\output\relationships.cql";
    var output = new StringBuilder();
    int n = 0;
    foreach (var user in _userList)
    {
        foreach (var following in user.following)
        {
            string s = $"MATCH (a), (b) WHERE a.username = '{user.username}' AND b.username = '{following}' CREATE (a)-[:FOLLOWING]->(b); ";
            output.AppendLine(s);
            n++;
        }
    }

    File.WriteAllText(filename, output.ToString());
}

So I ended up with two cql files that looked like this

CREATE (user_1:User { userid: 1 , username: 'user_1', avatar: 'URL' }); 
CREATE (user_2:User { userid: 2 , username: 'user_2', avatar: 'URL' }); 
CREATE (user_3:User { userid: 3 , username: 'user_3', avatar: 'URL' }); 

and

MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_26' CREATE (a)-[:FOLLOWING]->(b); 
MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_53' CREATE (a)-[:FOLLOWING]->(b); 
MATCH (a), (b) WHERE a.username = 'user_2' AND b.username = 'user_6' CREATE (a)-[:FOLLOWING]->(b); 

and I used neo4j-shell to execute the files and import the data. But it was no bed of roses. I’ll list the problems I faced along the way and how I got around them so that this experience might be helpful for other people as well:

Issues along the way and lessons learned

Running multiple statements on Neo4J browser

First I tried to run the create statements using the Neo4J browser which turned out to be problematic because it cannot run multiple statements that end with semi-colons. So I removed the semi-colons but then it started giving me this error

WITH is required between CREATE and MATCH

I found a workaround for that on SO. So the following works:

MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_14' CREATE (a)-[:FOLLOWING]->(b); 
WITH 1 as dummy
MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_22' CREATE (a)-[:FOLLOWING]->(b); 

Now the problem was, if the data was a bit dirty, for instance if user_14 didn’t exist, it stopped executing the rest and no other relationships were created). I had a few nodes like that so this method didn’t work for me after all.

Starting the shell was not as easy as I’d imagined

I installed Neo4J using the default settings and to start the shell just navigated to that directory and ran the batch file. Got an error instead of my shell:

C:\Program Files\Neo4j Community\bin>Neo4jShell.bat

The system cannot find the path specified.

Error: Could not find or load main class org.neo4j.shell.StartClient

Apparently the way to run the shell is using the Neo4J server application and clicking Options -> Command Prompt

Neo4J command prompt

This launches the Neo4J Command Prompt then the rest is easy:

neo4jshell -file nodes.cql

neo4jshell -file relationships.cql

“Exiting with unterminated multi-line input”

Final issue was the statements in a sql file must be terminated with a semi-colon in order for the shell to execute them. Otherwise it gives the above warning and quits

So after all this hassle my data is in the Neo4J database ready to be queried to death!:

Next I’ll investigate converting the data into CSV and achieve the same results by using LOAD CSV command and look into visualization of this data.

Resources

csharp, development, asp.net, mvc comments edit

In ASP.NET, TempData is one of the mechanisms used to pass data from controller to the view. In this post I’ll dive into its source code to investigate its behaviour.

What is TempData

TempData is an instance of TempDataDictionary that is used to pass data from the controller to the view.

Lifespan

The lifespan of TempData is rather unusual: It lives for one request only. In order to achieve this it maintains two HashSets to manage keys as well as the data dictionary:

private Dictionary<string, object> _data;
private HashSet<string> _initialKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
private HashSet<string> _retainedKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

When we read some data using an indexer or TryGetValue method it removes that key from _initalKeys collection.

public bool TryGetValue(string key, out object value)
{
    _initialKeys.Remove(key);
    return _data.TryGetValue(key, out value);
}

The actual dictionary that holds the data is intact at this point. That’s why we can read same data consecutively without any issues. It only removes the key from _initialKeys collection, basically marking it to be deleted when the data is persisted.

Peek and keep

If we want the values in TempData last longer we can use Peek and Keep methods. What Peek does is return the value without removing it from the _initialKeys:

public object Peek(string key)
{
    object value;
    _data.TryGetValue(key, out value);
    return value;
}

Alternatively, we can call Keep method. Similarly it doesn’t manipulate the data directly but just marks the key to be persisted by adding it to the _retainedKeys collection.

public void Keep(string key)
{
    _retainedKeys.Add(key);
}

Parameterless overload of Keep method add all keys in the _data dictionary to _retainedKeys.

Which keys to persist

So as seen above when we get values and call Peek/Keep methods, operations are carried out on _initialKeys and _retainedKeys collections and nothing happens to the actual data. These operations take effect when the _data is actually saved:

public void Save(ControllerContext controllerContext, ITempDataProvider tempDataProvider)
{
    _data.RemoveFromDictionary((KeyValuePair<string, object> entry, TempDataDictionary tempData) =>
        {
            string key = entry.Key;
            return !tempData._initialKeys.Contains(key) 
                && !tempData._retainedKeys.Contains(key);
        }, this);

    tempDataProvider.SaveTempData(controllerContext, _data);
}

Before the data is passed on to the provider it’s pruned. The keys that don’t exist in the _retainedKeys (the keys we explicitly told to keep) and _initialKeys (the keys that have not been touched so far or accessed through Peek method) collections are removed.

Providers

By default, TempData uses session variables to persist data from one request to the next. Serializing and deserializing data is carried out via an object implementing ITempDataProvider. By default SessionStateTempDataProvider class is used to provide this functionality. It occurs in the CreateTempDataProvider method in Controller.cs class:

protected virtual ITempDataProvider CreateTempDataProvider()
{
    return Resolver.GetService<ITempDataProvider>() ?? new SessionStateTempDataProvider();
}

This also means we can replace the provider with our own custom class. For demonstration purposes I wrote my own provider which uses a simple text file to persist TempData:

public class TextFileTempDataProvider : ITempDataProvider
{
    internal readonly string FileName = Path.Combine(HttpContext.Current.Server.MapPath(@"~/App_Data"), @"TempData.txt");

    public IDictionary<string, object> LoadTempData(ControllerContext controllerContext)
    {
        if (File.Exists(FileName))
        {
            string json = File.ReadAllText(FileName);
            return Newtonsoft.Json.JsonConvert.DeserializeObject<Dictionary<string, object>>(json);
        }

        return new Dictionary<string, object>(StringComparer.OrdinalIgnoreCase);
    }
    
    public void SaveTempData(ControllerContext controllerContext, IDictionary<string, object> values)
    {
        string json = Newtonsoft.Json.JsonConvert.SerializeObject(values);
        File.WriteAllText(FileName, json);
    }
}

In order to use this class it needs to be assigned to TempDataProvider in the controller’s constructor

public FirstController()
{
    TempDataProvider = new TextFileTempDataProvider();
}

Of course it’s not a bright idea to use disk for such operations, this is just for demonstration purposes and makes it easier to observe the behaviour.

Conclusion

Often times I’ve found knowledge about the internals of a construct useful. Applications and frameworks are getting more complex each day, adding more layers and hiding the complexity from the consumers. It’s great because we can focus on the actual business logic and application we are building but when we get stuck it takes quite a while to figure out what’s going on. Having in-depth knowledge on the internals can save a lot if time.

Resources

tfs, team foundation server, ci, continuous integration, alm, application lifecycle management comments edit

Setting up continuous integration environment with TFS is quite easy and free. In this post I’ll go over the details of setting up a CI environment. The tools I will use are:

  • Visual Studio 2015 Community Edition
  • TFS 2015 Express
  • AWS EC2 instance (To install TFS)
  • AWS SES (To send notification mails)

Step 01: Download and install TFS Express 2015

I always prefer the offline installer so download the whole ISO just in case (it’s 891MB so it might be a good time for a coffee break!)

Installation is standard next-next-finish so nothing particular about it. Accepting all defaults yields a working TFS instance.

Step 02: Connect to TFS

From the Team menu select Manage Connections. This will open Team Explorer on which we can choose the TFS instance.

It can connect to TFS Online or Github. For this example I will use the hosted TFS we have just installed. We first need to add it to TFS server list.

If you are using EC2 like I did don’t forget to allow inbound traffic on port 8080 beforehand.

Note that we don’t have a Team Project yet, what we have connected is the Project Collection. A collection is an additional abstraction layer used to group related projects. Using Default Collection generally works out just fine for me.

Step 03: Configure workspace

Since a local copy needs to be stored locally we have to map it to the project

Just pick a local empty folder. Leaving “$/” means the entire collection will be mapped to this folder.

Click Map & Get and you will have a blank workspace.

Step 04: Create team project

Now we can create a team project on TFS. click on Home -> Projects and My Teams then New Team Project.

Just following the wizard selects Scrum methodology and Team Foundation Version Control as the source control system. Starting with TFS 2013, Git is also supported now.

After several minutes later, our team project is ready.

Step 05: Clone and start developing

I chose Git as the version control system. If you use TFSVC the terminology you’ll see is a bit different but since the main focus of this post is establishing continuous integration it doesn’t matter much as long you as you can check-in the source code.

So now that we have a blank canvas let’s start painting! I added a class library with a single class like the one below:

public class ComplexMathClass
{
    public int Add(int x, int y)
    {
        return x + y;
    }

    public double Divide(int x, int y)
    {
        return x / y;
    }
}

and 3 unit test projects (NUnit, XUnit and MSTest).

NUnit tests:

public class ComplexMathClassTests
{
    [TestCase(1, 2, ExpectedResult = 3)]
    [TestCase(0, 5, ExpectedResult = 5)]
    [TestCase(-1, 1, ExpectedResult = 0)]
    public int Add_TwoIntegers_ShouldCalculateCorrectly(int x, int y)
    {
        var cmc = new ComplexMathClass();
        return cmc.Add(x, y);
    }

    [Test]
    // [ExpectedException(typeof(DivideByZeroException))]
    public void Divide_DenominatorZero_ShouldThrowDivideByZeroException()
    {
        var cmc = new ComplexMathClass();
        double result = cmc.Divide(5, 0);
    }
}

XUnit tests:

public class ComplexMathClassTests
{
    [Theory]
    [InlineData(1, 2, 3)]
    [InlineData(0, 5, 5)]
    [InlineData(-1, 1, 0)]
    public void Add_TwoIntegers_ShouldCalculateCorrectly(int x, int y, int expectedResult)
    {
        var cmc = new ComplexMathClass();
        int actualResult = cmc.Add(x, y);
        Assert.Equal<int>(expectedResult, actualResult);
    }

    [Fact]
    public void Divide_DenominatorZero_ShouldThrowDivideByZeroException()
    {
        var cmc = new ComplexMathClass();
        cmc.Divide(5, 0);
    //  Assert.Throws<DivideByZeroException>(() => cmc.Divide(5, 0));
    }
}

MSTest tests:

[TestClass]
public class ComplexMathClassTests
{
    [TestMethod]
    // [ExpectedException(typeof(DivideByZeroException))]
    public void Divide_DenominatorZero_ShouldThrowDivideByZeroException()
    {
        var cmc = new ComplexMathClass();
        cmc.Divide(5, 0);
    }
}

Unfortunately MSTest still doesn’t support parametrized tests which is a shame IMHO. That’s why I was never a big fan of it but added to this project for the sake of completeness.

I commented out the lines that expect exception in all tests to fail the tests. So now the setup is complete: We have a working project with a failing test. Our goal is to get alert notifications about the failing test whenever we check in our code. Let’s proceed to the next steps to accomplish this.

Step 06: Building on the server

As of TFS 2015 they renamed the Build Configuration to XAML Build Configuration for some reason and moved it under Additional Tools and Components node but everything else seems to be the same.

Default configuration installs one build controller and one agent for the Default Collection so for this project we don’t have to do add or change anything.

Each build controller is dedicated to a team project collection. Controller performs lightweight tasks such as determining the name and reporting the status of the build. Agents are the heavy-lifters and carry out processor-intensive work of the build process. Each agent is controlled by a single controller.

In order to build our project on the build server we need a build definition first. We can create one by selecting Build -> New build Definition

  • One important setting is the trigger: By default the build is not triggered automatically which means soon enough it will wither and die all forgotten! To automate the process we have to select Continuous Integration option.

  • In order to automatically trigger the build, the branch that is to be built must be added to monitored branches list in the Source Settings tab.

  • Last required setting is failing the build when a test fails. It’s a bit buried so you have to go 3.1.1 Test Source and set “Fail build on test failure” to true.

Step 07: Notifications

At this point, our build definition is triggered automatically but we don’t get any notifications if the build fails (due to a compilation error for example). TFS comes with an application called Build Notifications. It pops up an alert but it requires to be installed on the developer machine so I don’t like this solution.

A better approach is enabling E-Mail notifications. In order to do that first we need to set up Email Alert Settings for TFS. In the Team Foundation Server Express Administration Console select Application Tier and scroll down to the “Email Alert Settings” section. enter the SMTP credentials and server info here.

You can also send a test email to verify your settings.

Second and final stage is to enable the project-based alerts. In Visual Studio Team Explorer window select Home -> Settings -> Project Alerts. This will pop up a browser and redirect to alert management page. Here select “Advanced Alert Management Page”. In the advanced settings page there are a few pre-set notification conditions and the build failure is at the top of the list!

I intentionally broke the build and checked in my code and in a minute, as expected I received the following email:

With that we have automated the notifications. We already set the build to fail upon test failure in the previous step. Final step is to run the tests on the build server to complete the circuit.

Step 08: Run tests on build server

Running tests on the server is very simple. For NUnit all we have to do is install the NUnitTestAdapter package:

Install-Package NUnitTestAdapter

After I committed and pushed my code with the failing test I got the notification in a minute:

Uncommented the ExpectedException line and the build succeeded.

For xUnit the solution is similar, just install the xUnit runner NuGet package and checkin the code

Install-Package xunit.runner.visualstudio

For MSTest it works out of the box so you don’t have to install anything.

In the previous version of TFS I had to install Visual Studio on the build server as advised in this MSDN article. Seems like in TFS 2015 you don’t have to do that. The only benefit of using MSTest (as far as I can see at least) is that it’s baked in so you don’t have to install extra packages. You create a Unit Test project and all your tests are immediately visible in the test explorer and automatically run on the server.

Wrapping up

Continuous Integration is a crucial part of the development process. Team Foundation Server is a powerful tool with lots of features. In this post I tried to cover basics from installation to setting up a basic CI environment. I’ll try to cover more features in the future but my first priority will be new AWS services CodeDeploy, CodeCommit and CodePipeline. As I’m trying to migrate everything to AWS having all projects hosted and managed on AWS would make more sense in my case.

Resources

aws, route 53, csharp, development comments edit

If you don’t have a static IP and you need to access your home network for some reason you may find Dynamic DNS services useful. I was using No-IP until its limitations became offputting. Since I love DIY stuff and AWS I thought I could build the exact same service on my own using Route 53.

Enter DynDns53

It’s a simple application. What it does is basically:

  • Get current external IP
  • For each subdomain in the configuration, call AWS API and get domain info
  • Check if the recorded IP is different than the current one
  • If so, call AWS API to update it with the new one, if not do nothing.

In a different post on setting up Calibre on a Raspberry Pi I developed a script that did the same thing:

import boto.route53
import urllib2
currentIP = urllib2.urlopen("http://checkip.amazonaws.com/").read()

conn = boto.connect_route53()
zone = conn.get_zone("volki.info.")
change_set = boto.route53.record.ResourceRecordSets(conn, '{HOSTED_ZONE_ID}')

for rrset in conn.get_all_rrsets(zone.id):
    if rrset.name == 'calibre.volki.info.':
        u = change_set.add_change("UPSERT", rrset.name, rrset.type, ttl=60)
        rrset.resource_records[0] = currentIP
        u.add_value(rrset.resource_records[0])
        results = change_set.commit()

The difference of DynDns53 is that it can run as a background service and it has a cute logo! The C# equivalent of above code is like this:

public void Update()
{
    var config = _configHandler.GetConfig();
    string currentExternalIp = _ipChecker.GetExternalIp();

    foreach (var domain in config.DomainList)
    {
        string subdomain = domain.DomainName;
        string zoneId = domain.ZoneId;

        var listResourceRecordSetsResponse = _amazonClient.ListResourceRecordSets(new ListResourceRecordSetsRequest() { HostedZoneId = zoneId });
        var resourceRecordSet = listResourceRecordSetsResponse.ResourceRecordSets.First(recordset => recordset.Name == subdomain);
        var resourceRecord = resourceRecordSet.ResourceRecords.First();

        if (resourceRecord.Value != currentExternalIp)
        {
            resourceRecord.Value = currentExternalIp;

            _amazonClient.ChangeResourceRecordSets(new ChangeResourceRecordSetsRequest()
            {
                HostedZoneId = zoneId,
                ChangeBatch = new ChangeBatch()
                {
                    Changes = new List<Change>() 
                    { 
                        new Change(ChangeAction.UPSERT, resourceRecordSet)
                    }
                }
            });
        }
    }
}

That’s it. The rest is just user interface, a Windows service and some basic uni tests.

Usage

The project has a WPF user interface and a Windows service. Both are using the same library (DynDns53.Core) so the functionality is the same. Using the user interface is pretty straightforward.

Just add your AWS keys that have access privileges to your domains. You can set it to run at system start so you don’t have to start it manually.

I built the Windows service using Topshelf. To install it, build the application and on an elevated command prompt just run

DynDns53.Service.exe install

Similarly to uninstall:

DynDns53.Service.exe uninstall

What’s missing

  • Landing page: I’ll update the project details on the site generated by GitHub Pages.
  • Mac/Linux support: As the worlds are colliding I’d like to discover what I can do with the new .NET Core runtime. Since this is a simple project I think it’s a good place to start

Resources

csharp, development, aws, route53 comments edit

Part 3: Consuming the APIs

So far in the series:

In this installment it’s time to implement the actual application that consumes the TLD API as well as AWS Route 53 API. I decided to turn this into a little project with a name, logo and everything. I’ll keep updating the project. Currently it just has the core library and a console application. I know it’s not enough but you have to start somewhere!

Core functionality

Basically what it does is:

  1. Get the supported TLD list

  2. For each TLD send a CheckDomainAvailability request to AWS

  3. Return the resultset

When I first started out I was meaning to make these requests run in parallel. But they got throttled and I ended up using all my allowance. So I changed the code to make a call per second:

The TldProvider I developed can be used in two different ways:

  1. As a NuGet Package

I uploaded my package to NuGet.org so you can install it by running

Install-Package TldProvider.Core.dll

and the usage would be:

private List<Tld> GetTldListFromLibrary()
{
    string url = ConfigurationManager.AppSettings["Aws.Route53.DocPageUrl"];
    var tldListProvider = new TldListProvider();
    var supportedTLDlist = tldListProvider.GetSupportedTldList(url);
    return supportedTLDlist;
}
  1. As an API the usage is a bit more complex because of the JSON to list conversion:
private List<Tld> GetTldListFromApi()
{
    string apiUrl = ConfigurationManager.AppSettings["TldProvider.Api.EndPoint"];
    var httpClient = new HttpClient();
    var request = new HttpRequestMessage() { Method = new HttpMethod("GET"), RequestUri = new Uri(apiUrl) };
    var httpResponse = httpClient.SendAsync(request).Result;
    var apiResponse = httpResponse.Content.ReadAsStringAsync().Result;
    dynamic resp = Newtonsoft.Json.JsonConvert.DeserializeObject(apiResponse);
    var resultAsStringList = JToken.Parse(apiResponse)["tldList"].ToObject<List<string>>(); ;
    var resultAsTldList = resultAsStringList.Select(t => new Tld() { Name = t }).ToList();
    return resultAsTldList;
}

Actually this method should not return List as it would require a reference to the library. The idea of using the API is to get rid of that dependency in the first place but I just wanted to add both and use them interchangably.

So finally tha main course: The check method that loops through the TLD list and gets the availability from AWS Route 53 API

public List<DomainCheckResult> Check(string domainName)
{
    string ACCESS_KEY = ConfigurationManager.AppSettings["Aws.Route53.AccessKey"];
    string SECRET_KEY = ConfigurationManager.AppSettings["Aws.Route53.SecretKey"];

    var results = new List<DomainCheckResult>();

    var credentials = new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY);
    var config = new AmazonRoute53DomainsConfig()
    {
        RegionEndpoint = RegionEndpoint.USEast1
    };

    var amazonRoute53DomainsClient = new AmazonRoute53DomainsClient(credentials, config);

    // var supportedTLDlist = GetTldListFromLibrary();
    var supportedTLDlist = GetTldListFromApi();

    foreach (var tld in supportedTLDlist)
    {
        var request = new CheckDomainAvailabilityRequest() { DomainName =  $"{domainName}.{tld.Name}" };

        var response = amazonRoute53DomainsClient.CheckDomainAvailabilityAsync(request);
        var result = response.Result;
        Console.WriteLine($"{request.DomainName} --> {result.Availability}");
        results.Add(new DomainCheckResult()
        {
            Domain = domainName,
            Tld = tld.Name,
            Availability = result.Availability,
            CheckDate = DateTime.UtcNow
        });

        System.Threading.Thread.Sleep(1000);
    }

    return results;
}

TroubleShooting

I had a few problems while implementing. Learned a few things while fixing them. So here they are:

  • I created an IAM account that has access to Route53. But that wasn’t enough. There is a separate action called route53domains. I had to grant access to this as well to use domains API.

  • I use EUWest1 region since it’s the closest one. Normally Route53 doesn’t require any region setting but weirdly I had to set region to US East1 in order to access Route53Domains API.

The problems above makes me think domains didn’t fit very well with the current Route53. It feels like it has not integrated seamlessly yet.

First client: Console application

This used to be the test application but since I don’t have any user interface currently this will be my first client. It just accepts the domain name from the user, checks the availability and saves the results in JSON format. This is all the code of the console application:

static void Main(string[] args)
{
    if (args.Length == 0)
    {
        Console.WriteLine("Usage: DomChk53.UI.Console {domain name}");
        return;
    }

    string domain = args[0];
    var checker = new DomainChecker();
    var results = checker.Check(domain);

    string outputFilename = $"results-{domain}-{DateTime.UtcNow.ToString("yyyyMMdd-HHmmss")}.json";
    string json = Newtonsoft.Json.JsonConvert.SerializeObject(results);
    File.WriteAllText(outputFilename, json);
    Console.WriteLine($"Saved the results to {outputFilename}");
}

It also displays the results of each domain on the console you can have keep track of what it’s doing

It saves the results with a timestamp so a history can be maintained:

  {
    "Domain": "volkan",
    "Tld": "cool",
    "Availability": "AVAILABLE",
    "CheckDate": "2015-08-05T12:31:58.40081Z"
  },

What’s missing

It’s not much but at least it delivers what it promises: You give it a domain name and it checks the availability of all supported TLDs by AWS Route 53 and returns the results. But there’s a lot to do to improve this project. Some items in my list:

  • A web-based user interface that allows viewing these results online.
  • Get the price list for each tLD as well and display it along with the name
  • Registering selected domains: It would be nice if you could buy the available domains using the same interface
  • A landing page at domchk53.com so that it can have links to the source code, online checker etc.

Conclusion

The reason I started this project was to overcome the limitations of AWS Route 53 domains UI and be able to search all supported TLDs all at once. To consider the project complete I will prepare a single-page website about the project and develop a nice user-interface. But in terms of basic functionality I can get what I set out for so I can think of it as a small success I guess.

Resources

csharp, development, javascript, aws, route53, lambda, Amazon API gateway, nodejs comments edit

Part 2: Converting TLD Provider into an API

In all fairness, this part is not entirely necessary for the project. I already found a way to get the list of TLDs from Amazon documentation so I could easily integrate it with my client applications. But I recently heard about Amazon API Gateway which has been introduced a few weeks back so I thought it would be cool to access my JavaScript method via a web API hosted on AWS. After all, the point of many projects I develop is to learn new stuff! In light of that I’ll try something new with the C# version as well and use a self-hosted service so that I can use a Windows Service instead of IIS.

API #1: Amazon API Gateway

There are a couple of new technologies here that I haven’t used before so this was a good chance for me to play with them.

Amazon API Gateway looks like a great service to create hosted APIs. Also it’s integrated with AWS Lambda. You can bind an endpoint directly to lambda function. There are other binding options but in this project I will use Lambda as it was also in my to-learn list.

Setup Amazon API Gateway

API Gateway interface is very intuitive. Within minutes I was able to create a GET method calling my Lambda function. First you create the API by clicking the “Create API” button!

Then you add your resource by clicking “Create Resource”. By default it comes with the root resource (“/”) so you can just use that one as well to add methods.

I created a resource called tldlist. All I needed was a GET method so I created it by “Create Method”.

You select the region and enter the full ARN of your Lambda function. In the UI it just says “function name” but it requires full ARN (i.e.: arn:aws:lambda:eu-west-1:1234567890:function:getTldList)

…and Lambda

The function is a bit different from the previous version. In Node.js I wasn’t able to use XMLHttpRequest object and turns out your use the http module to make web requests so I modified the code a bit. Here’s the final version of my Lambda function:

console.log('Loading function');

var docHost = 'docs.aws.amazon.com';
var docPath = '/Route53/latest/DeveloperGuide/registrar-tld-list.html';
var supportedTldPage = docHost + docPath;
var http = require('http');

exports.handler = function(event, context) {
    var options = {
      host: docHost,
      path: docPath,
      port: 80
    };
    
    http.get(options, function(res) {
        var body = '';
        res.on('data', function(chunk) {
            body += chunk;
        });
        
        res.on('end', function() {
            console.log(body);
            var result = parseHtml(body);
            context.succeed(result);
        });
    }).on('error', function(e) {
        console.log("Got error: " + e.message);
    });     

};

function parseHtml(pageHtml) {
    var pattern = /<a class="xref" href="registrar-tld-list.html#.+?">[.](.+?)<\/a>/g;
    var regEx = new RegExp(pattern);

    var result = {};
    result.url = 'http://' + supportedTldPage;
    result.date = new Date().toUTCString();
    result.tldList = [];

    while ((match = regEx.exec(pageHtml)) !== null) {
        result.tldList.push(match[1]);
    }

    return result;
}

In order to return a value from Lambda function you have to call succeed, fail or done:

context.succeed (Object result);
context.fail (Error error);
context.done (Error error, Object result);

Succeed and fail are self-explanatory. done is like a combination of both. If error is non-null it treats it as failure. Even if you call fail or done with an error the HTTP response code is always 200. what changes is the message body. For example, I played around with a few possibilities to test various results:

Method call: context.succeed(result);  
Output: Full JSON results

Method call: context.done(null, result);  
Output: Full JSON results

Method call: context.fail("FAIL!!!");  
Output: {"errorMessage":"FAIL!!!"}

Method call: context.done("FAIL!!!", results);  
Output: {"errorMessage":"FAIL!!!"}

As you can see, if error parameter is not null it ignores the actual results. Also I removed the JSON.stringify call from the parseHtml method because API gateway automatically converts it to JSON.

Tying things together

Deployment is also a breeze, just like creating the API, resource and the methods all it takes is a few button clicks. You click Deploy API and create an environment such as staging or prod. And that’s it! You’re good to go!

Since this will be a public-facing API with no authentication I also added a CloudWatch alarm:

This way if some mental decides to abuse I will be aware of it. The good thing is it’s very cheap. It costs $3.5 per million API calls which is about £2.25. I don’t think it will break the bank but for serious applications authorization is a must so I will need to investigate that feature in the future anyway.

At this point, I have a Lambda function called from the API hosted by AWS. I don’t have to worry about anything regarding the maintenance and scaling which feels great!

API #2: C# & Self-hosting on a Windows Service

Speaking of maintenance, here comes the Windows version! I don’t intend to deploy it on production but I was meaning to learn self-hosting APIs with Web API to avoid IIS and here’s chance to do so.

Found this nice concise article showing how to run a Web API inside a console application. I tweaked the code a little bit to suit my needs. Basically it takes 4 simple steps:

Step 01. Install OWIN Self-Host NuGet package:

Install-Package Microsoft.AspNet.WebApi.OwinSelfHost

Step 02. Setup routing

public class Startup
{
    public void Configuration(IAppBuilder appBuilder)
    {
        HttpConfiguration config = new HttpConfiguration();
        config.Routes.MapHttpRoute(
            name: "DefaultApi",
            routeTemplate: "api/{controller}/{id}",
            defaults: new { id = RouteParameter.Optional }
        );

        appBuilder.UseWebApi(config);
    }
}

Step 03. Add the controller

public class TldController : ApiController
{
    public HttpResponseMessage Get()
    {
        string supportedTLDPage = ConfigurationManager.AppSettings["AWS-URL"];
        var tldProvider = new TldListProvider();
        var tldList = tldProvider.GetSupportedTldList(supportedTLDPage);
        var output = new {
            url = supportedTLDPage, 
            date = DateTime.UtcNow.ToString(),
            tldList = tldList.Select(tld => tld.Name)
        };
        return this.Request.CreateResponse(HttpStatusCode.OK, output);
    }
}

Step 04. Start OWIN WebApp

public void Start()
{
    string baseAddress = ConfigurationManager.AppSettings["BaseAddress"];
    WebApp.Start<Startup>(url: baseAddress);
    Console.WriteLine("Service started");
}

Final step is installation. As I used TopShelf all I had to do was running a command prompt with administrator privileges run this command:

TldProvider.Service.exe install

Now that my service is running in the background and accepting HTTP requests let’s take it out for a spin:

Brilliant!

What’s next?

So now I have an API that returns me the supported TLD list. In the next post I’ll work on a basic client that consumes that API and AWS Route53 to get the availability results finally.

Resources

csharp, development, javascript, aws, route53 comments edit

Part 1: Getting supported TLD List

Around this time last year Amazon announced the Route 53 update which allowed domain registrations.

Domain search in Route 53

Unfortunately, the AWS Management Console UI doesn’t allow searching all TLDs at once for a given domain. So I set out to write a console application in C# that retrieved the TLDs from AWS API in order to “enhance” AWS a bit. As all of my projects, the scope got out of hand very quickly so I decided to break this adventure into smaller parts. In this post I’ll talk about how to get the supported TLD list.

AWS: Everything is API-based, right? Not quite!

The first step to a domain checker application is to acquire a list of TLDs to check. So I started exploring AWS API to get the supported TLD list. To my surprise there wasn’t any! I asked the question in AWS support forums and the response is it’s not supported. There is a webpage that has the list but that’s it! So until they make it available I decided to do a little scraping to generate the list myself. Obviously this is not the proper way of doing it and it’s prone to breaking very easily but they gave me no choice!

First method: C# Library

It’s just one method and it was easy to implement (apart from fiddling with regular expressions!)

public List<Tld> GetSupportedTldList()
{
    Uri supportedTldPage = new Uri("http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/registrar-tld-list.html");
    string pageHtml = string.Empty;
    using (var webclient = new WebClient())
    {
        pageHtml = webclient.DownloadString(supportedTldPage);
    }

    string pattern = "<a class=\"xref\" href=\"registrar-tld-list.html#.+?\">[.](.+?)</a>";
    var matches = Regex.Matches(pageHtml, pattern);

    var result = new List<Tld>();
    foreach (Match m in matches)
    {
        result.Add(new Tld() { Name = m.Groups[1].Value });
    }

    return result;
}

It first downloads the raw HTML from the AWS documentation page. The TLDs on the page have a link in this format:

<a class="xref" href="registrar-tld-list.html#cab">.cab</a> 

The regular expression pattern retrieves all anchors in this format. Only the ones that contain a dot in the value are retrieved to eliminate irrelevant links. To extract the actual TLD from this matching string I used parenthesis to create a group. By default the whole matched string is the first group. Additional groups can be created by parenthesis so that we can only get the values we are interested in. For example the group values for a match look like this

m.Groups[0].Value: "<a class=\"xref\" href=\"registrar-tld-list.html#academy\">.academy</a>"
m.Groups[1].Value: "academy"

Adds these extracted values to a list (for now Tld class only contains Name property, more info like type of TLD or description can be added in the future)

Second method: JavaScript

It’s not very complicated with C# to accomplish this task so I thought it would be even easier with JavaScript as I already tackled the regular expression bit. I couldn’t be wronger!

I created a simple AJAX call with jQuery and got the following error:

Apparently you cannot get external resources willy nilly using jQuery! It took some time but I found a neat workaround here It’s essentially a jQuery plugin sending the request to query.yahooapis.com instead of the URL we request. Apparently Yahoo API returns the results to a callback method we provide. I hadn’t heard about YQL before. The pitch on their site is:

"The YQL (Yahoo! Query Language) platform enables you to query, filter, and combine data across the web through a single interface." 

so a bit explains why they allow Cross-domain queries.

In the test page I added the plugin script

<script-- src="jquery.xdomainajax.js"></script>

Then the main function using jQuery worked just fine:

var supportedTldPage = "http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/registrar-tld-list.html";

function getTldList(callback) {
    $.ajax({
        url: supportedTldPage,
        type: 'GET',
        success: function (res) {
            var pageHtml = res.responseText;
            var jsonResult = parseHtml(pageHtml);
            callback(jsonResult);
        }
    });
}

Another thing I tried was using pure JavaScript:

function getTldList(callback) {
    var request = new XMLHttpRequest();
    request.open("GET", supportedTldPage, true);
    request.onreadystatechange = function () {
        if (request.readyState != 4 || request.status != 200) return;

        var pageHtml = request.responseText;
        var jsonResult = parseHtml(pageHtml);
        callback(jsonResult);
    };
    request.send();
}

This version doesn’t work by default because of the same CORS restrictions. But apparently there’s an extension for that. I installed it and it worked like a charm. Wondering what was happening behind the scenes I captured the request with Fiddler and it looked like this:

Host: docs.aws.amazon.com
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36
Origin: http://evil.com/
Accept: */*
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,tr;q=0.6
If-None-Match: "19ce8-51b1a64cfe880-gzip"
If-Modified-Since: Fri, 17 Jul 2015 23:17:38 GMT

Origin was “http://evil.com”! Works great as long as it’s not null! In the sample code I’ll leave both versions. Of course all this hassle is to bypass browser restrictions. Depending on your use case, you can always fire the HTTP request above in Fiddler and get the results. I guess it’s a nice mechanism if you need to implement a closed system and want to ensure that resources are not consumed from the outside. Of course for public web pages like this which are meant to be accessed from anywhere in the world there are no restrictions on the server side.

So that now we can get the raw HTML from AWS documentation, the rest is smooth sailing. The part that handles finding the TLDs in the HTML is very similar:

function parseHtml(pageHtml) {
    var pattern = /<a class="xref" href="registrar-tld-list.html#.+?">[.](.+?)<\/a>/g;
    var regEx = new RegExp(pattern);

    var result = {};
    result.url = supportedTldPage;
    result.date = new Date().toUTCString();
    result.tldList = [];

    while ((match = regEx.exec(pageHtml)) !== null) {
        result.tldList.push(match[1]);
    }

    var myString = JSON.stringify(result);
    return myString;
}

The regular expression pattern is the same. The only difference is .exec method doesn’t return mutliple matches so you have to loop through unless there are no matches.

So finally the result in JSON format is:

{
  "url": "http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/registrar-tld-list.html",
  "date": "Tue, 29 Jul 2015 20:26:02 GMT",
  "tldList": [
    "academy",
    "agency",
    "bargains",
    "bike",
    "biz",
    "blue",
    "boutique",
    "-- removed for brevity --",
    "eu",
    "fi",
    "fr",
    "it",
    "me",
    "me.uk",
    "nl",
    "org.uk",
    "ruhr",
    "se",
    "wien"
  ]
}

What’s next?

Now I have the supported TLD list. I can easily use it in my application but where’s the fun that? I’d rather convert this code into an API so that we can get the new TLDs whenever they are added to that page. I just they don’t change their HTML markup anytime soon! :-)

So in the next post I’ll work on the API…

Resources

dotnet, configuration, github comments edit

I hate configuration files! They contain highly sensitive information and there is always a risk of them leaking out. Especially when I read horror stories like this I remind myself to be more careful about it. It is particularly important when starting off with a private repository and converting it to open-source.

There are a few approaches and I will try to make pros and cons list for each of them which hopefully would make it easier for me to make a decision.

Method 1: Ignore config files completely

Simply add to gitignore before checking in the config files. This is not really a viable option but to cover all basis I wanted to add this one as well.

Pros

  • Very easy to implement
  • Easy to be consistent (use the same gitignore everywhere)

Cons

  • When someone checks out the project initially it wouldn’t compile because of the missing config file.
  • For a new user there is no way to figure out what the actual config should look like
  • For internal applications, you have to maintain local copies and handle distribution among developers manually.

Method 2: Check in configuration with placeholders

Check in the initial file then run the following command:

git update-index --assume-unchanged <file>

After this you can add the actual values and use the config as usual. But those values will be ignored. This way when someone checks out they can at least compile the project.

Pros

  • Less disruptive as the project can be compiled

Cons

  • When you make a change to the config file (e.g. add/rename keys) you can’t check in those changes as well

Method 3: Ignore config file, check in a sample file instead

This is essentially a merger of Method 1 and 2. Maintain two files (e.g. App.config and App.config.sample). Ignore the app.config from the getgo and only check in .sample file. Structurally it will be exactly the same as app.config without the confidential values.

Pros

  • Won’t compile by default, extra step for the people who check out (small one but still)

Cons

  • Both files need to be kept in sync manually
  • Still no online copy of the actual config file available

Method 4: Reference to another configuration file

A slight variation of the previous method. .NET allows cascading and linking with configuration files. For example say this is my app.config file:

<configuration>
	<appSettings file="Actual.config">
		<add key="Twilio.AccountSID" value="SAMPLE SID" />
		<add key="Twilio.AuthToken" value="SAMPLE TOKEN" />
		<add key="Non-sensitive-info" value="some value" />
	</appSettings>
</configuration>

I can link the appSettings section to another file named actual.config which would look like this:

<appSettings>
  <add key="Twilio.AccountSID" value="REAL 123456" />
  <add key="Twilio.AuthToken" value="REAL Token" />
  <add key="Non-sensitive-info" value="some value" />
</appSettings>

Actual.config never goes into source control. When the application cannot find the actual.config it just uses the values in the app.config. When the actual.config is present, those values override the sample values.

Pros

  • Only keys with sensitive values can be hidden

Cons

  • Doesn’t work with custom config sections
  • Still no online copy of the actual config file available

The idea is to maintain a private repository that contains the config files and add it as a submodule:

git submodule add git@github.com:volkanx/private-config.git

And link to that configuration file in the app.config as in Method 4.

Pros

  • Actual values are online and accessible

Cons

  • Requires a paid account for private repositories
  • More complicated

Verdict

As I already have a paid GitHub account the con for Method 5 is not quite valid for me so I think I will go with that one. Otherwise the actual values will be stored locally on the disk only and will eventually get lost. Looks like there is no convenient and easy way of doing it after all. If you have any suggestions I’m all ears.

As I said, I hate configuration files!

Resources

csharp, development comments edit

Generally we don’t need to understand how every single line of code is compiled but sometimes it helps to have a look at what’s going on under the hood. The code we write doesn’t always translate directly to IL (Intermediate Language) by the compiler (assuming optimizations come into play). It also helps what really happens when we use shorthand notations in the language. So I decided to explore the IL and see how simple constructs are compiled. Some of them are very simple and commonly known but some of them were new to me so compiled all of them in this post.

Toolkit

I used a disassembler and decompilers to analyze the output. When we compile a .NET program, it’s converted into IL code. Using a disassembler we can view the IL. Decompiling is the process of regenerating C# code. It’s hard to read IL so getting the C# code back helps to analyse easier. I used the following tools to view the IL and C# code:

  • ILDASM: Main tool I used is ILDASM (IL Disassembler). It’s installed with Visual Studio so doesn’t require anything special. It’s hard to make sense of IL but generally helpful to get an idea about what’s going on.
  • Telerik JustDecompile: Decompiler takes an assembly and re-generates .NET code from it. It may not fully return the original source code and sometimes it optimizes stuff so it may skew the results a little bit. But it’s generally helpful when you need to see a cleaner representation
  • ILSpy: Same functionality as JustDecompile. I used it to compare the decompiled results.

So without further ado, here’s how some language basics we use are compiled:

01. Constructor

When you create a new class the template in Visual Studio creates a simple blank class like this:

namespace DecompileWorkout
{
    class TestClass
    {
    }
}

and we can create instances of this class as usual:

var newInstance = new TestClass();

It works even though we don’t have a constructor in our source code. The reason it works is that the compiler adds the default constructor for us. When we decompile the executable we can see it’s in there:

But when we add a constructor that accepts parameters the default constructor is removed:

It makes sense otherwise all classes would have parameterless constructor no matter what you do.

02. Properties

Properties have been supported since forever and they are extremely handy. In the olden days we used to create a field and create getter and setter methods that access that field such as:

private string _name;
public void SetName(string name)
{
    _name = name;
}
public string GetName()
{
    return _name;
}

Now we can get the same results by

public string Name { get; set; }

What happens under the hood is the compiler generates the backing field and getter/setter methods. For example if we disassemble the class with the above we’d get something like this:

Apart from the weird naming (I guess it’s to avoid naming collisions) it’s exactly the same code.

The backing field is only generated when this shorthand form is used. If we implement the getter/setter like this

public string Name 
{
    get
    {
        throw new NotImplementedException();
    }
    set
    {
        throw new NotImplementedException();
    }
}

Then the generated IL only contains the methods and not the backing field

One last thing to note is that we cannot have fields on interfaces but we are allowed to have properties. It’s because when used on interfaces the compiler again generates only the getter and setter methods.

03. Using statement

A handy shortcut to use when dealing with IDisposable objects is the using statement. To see what happens when we use using I came up with this simple code:

After decompiling:

So basically it just surrounds the code with a try-finally block and calls Dispose method of the object if it’s not null. Instead of writing this block of code it’s very handy to use the using statement. But at the end there is nothing magic about it, it’s just a shortcut to dispose objects securely.

04. var keyword and anonymous types

var keyword is used to implicitly specify a type. The compiler infers the type from the context and generates the code accordingly. Variables defined with var keyword are still strongly typed because of this feature. Initially I resisted using it as a shortcut because it’s main purpose is not to provide the luxury not to create implicitly typed variables. It’s introduced at the same time as anonymous types which makes sense because without having such a keyword you cannot store anonymous objects. But I find myself using it more and more for implicitly specifying the types. Looking around I see that I’m not the only one so if it’s just laziness at least I’m not the only one to blame!

So let’s check out how this simple code with one var and an anonymous type:

public class UsingVarAndAnonymousTypes
{
    public UsingVarAndAnonymousTypes()
    {
        var someType = new SomeType();
        var someAnonymousType = new { id = 1, name = "whatevs" };
    }
}

Decompiling the code doesn’t show the anonymous type side of the story but we can see that var keyword is simply replaced with the class type.

To see what happens with anonymous types let’s have a look at the IL:

For the anonymous type the compiler created a class for us with a very user-friendly name: <>f__AnonymousType0`2. It also generated readonly private fields and only getters that means they are immutable and we cannot set those values once the object is initialized.

06. Async and await

Async/await helps asynchronous programming much easier for us. It’s hiding a ton of complexity and I think it makes sense to invest some time into investigating how it works to use it properly. First of all, marking a method async doesn’t make everything asynchronous automagically. If you don’t use await inside the function it will run synchronously (the compiler will generate a warning about it). When you call an async method the execution continues without blocking the main thread. If it’s not awaited it returns a Task instead of the actual expected result. This means it's an ongoing process and it will run in the background without blocking the main flow. This is great so you can fire up a bunch of tasks you need and then wait for the results whenever you need all of them. For example:

public async void DownloadImageAsync(string imageUrl, string localPath)
{
    using (var webClient = new WebClient())
    {
        Task task = webClient.DownloadFileTaskAsync(imageUrl, localPath);

        var hashCalculator = new SynchronousHashCalculator();

        // Do some more work while download is running

        await task;
        using (var fs = new FileStream(localPath, FileMode.Open, FileAccess.Read))
        {
            byte[] testData = new byte[new FileInfo(localPath).Length];
            int bytesRead = await fs.ReadAsync(testData, 0, testData.Length);
            
            // Do something with testData
        }
    }
}

In this example, a file is downloaded asynchronously. The execution continues after the call has been made so we can do other unrelated work while the download is in progress. Only when we need the actual value, to open the file in this instance, we await the task. If we didn’t await we would try to access the file before it’s completely written and closed resulting in an exception.

One confusing aspect maybe calling await on the same line as in fs.ReadAsync in the above example. It looks like it’s synchronous as we are still waiting on the same line but the difference is the main thread is not blocked. If it’s a GUI-based application for exmaple it would still be responsive.

So let’s decompile the assembly to see how it works behind the scenes. This is one of the times I like having redundancy. Because Just Decompile failed to generate C# code for the class using async/await so I had to switch to ILSpy.

It generates a struct implementing IAsyncStateMachine interface. The “meaty” part is the MoveNext method:

It’s very hard to read compiler-generated code but I think this part is interesting:

switch (this.<>1__state)
{
case 0:
	taskAwaiter = this.<>u__$awaiter7;
	this.<>u__$awaiter7 = default(TaskAwaiter);
	this.<>1__state = -1;
	break;
case 1:
	goto IL_102;
default:
	this.<task>5__2 = this.<webClient>5__1.DownloadFileTaskAsync(this.imageUrl, this.localPath);
	this.<hashCalculator>5__3 = new SynchronousHashCalculator();
	taskAwaiter = this.<task>5__2.GetAwaiter();
	if (!taskAwaiter.IsCompleted)
	{
		this.<>1__state = 0;
		this.<>u__$awaiter7 = taskAwaiter;
		this.<>t__builder.AwaitUnsafeOnCompleted<TaskAwaiter, AsyncMethods.<DownloadImageAsync>d__0>(ref taskAwaiter, ref this);
		flag = false;
		return;
	}
	break;
}
taskAwaiter.GetResult();
taskAwaiter = default(TaskAwaiter);

If the task has not been completed it sets the state to 0 which then comes back to default so it goes back and forth until taskAwaiter.IsCompleted is true.

07. New C# 6.0 Features

C# 6.0 has just been released a few days ago. There aren’t many groundbreaking features as the team at Microsoft stated. Mostly the new features are shothand notations to make the code easier to read and write. I decided to have a look at the decompiled code of some of the new structs

Null-conditional operators

This is a great remedy to ease the neverending null-checks. For example in the code below

var people = new List<Person>();
var name = people.FirstOrDefault()?.FullName;

If the list is empty name will be null. If we didn’t have this operator it would throw an exception therefore we would have to do the null checks on our own. The decompiled code for the above block is like this:

private static void Main(string[] args)
{
    string fullName;
    Program.Person person = (new List<Program.Person>()).FirstOrDefault<Program.Person>();
    if (person != null)
    {
        fullName = person.FullName;
    }
    else
    {
        fullName = null;
    }
}

As we can see, the generated code is graciously carrying out the null check for us!

String interpolation

This is one of my favourites: Now we can simply place put the parameters directly inside a string instead of placeholders. Expanding on the example above, imagine the Person class is like this:

class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string FullName => $"{FirstName} {LastName}";
}

The decompiled code looks familiar though:

public string FullName
{
    get
    {
        return string.Format("{0} {1}", this.FirstName, this.LastName);
    }
}

This is exactly how I would do it if we didn’t have this new feature.

Auto-property initializers

We can set the default value for a property on the declaration line now, like this:

public string FirstName { get; set; } = "Volkan";

And the IL generated for the code is like this:

Decompiled code varies. Just Decompile doesn’t just displays the exact same statement as above. ILSpy on the other hand is closer to the IL:

public Person()
{
	this.<FirstName>k__BackingField = "Volkan";
	this.<BirthDate>k__BackingField = new DateTime(1966, 6, 6);
	base..ctor();
}

So what it does is, in the constructor it sets the private backing field with this default value we assign.

Expression bodied members

We can now provide the body of the function as an expression following the declaration like this:

public int GetAge() => (int)((DateTime.Now - BirthDate).TotalDays / 365);

The decompiled code is not very fancy:

public int GetAge()
{
    TimeSpan now = DateTime.Now - this.BirthDate;
    return (int)(now.TotalDays / 365);
}

It simply puts the expression back in the body.

Index initializers

It’s a shorthand for index initialization such as

Dictionary<int, string> dict = new Dictionary<int, string>
{
    [1] = "string1",
    [2] = "string2",
    [3] = "string3" 
};

And the decompiled code is no different than what it is now:

Dictionary<int, string> dictionary = new Dictionary<int, string>();
dictionary[1] = "string1";
dictionary[2] = "string2";
dictionary[3] = "string3";

As I mentioned there is nothing fancy about the new features in terms of the generated IL but some of them will help save a lot of time for sure.

08. Compiler code optimization

It is interesting to see how decompilers use optimization on their own. Before I went into this I thought they would just reverse engineer whatever is in the assembly but turns out that’s not the case.

First, let’s have a look at the C# compiler’s behaviour. In Visual Studio, Optimize code flag can be found under Project Properties -> Debug menu

In order to test code optimization I created a dummy method like this:

public class UnusedVariables
{
    public void VeryImportantMethod()
    {
        var testClass = new TestClass("some value");
		TestClass testClass2 = null;
        var x = "xyz";
        var n = 123;
    }
}

First I compiled it by default values:

Then disassembled the output:

I’m no expert in reading IL code but it’s obvious in the locals section there are 2 TestClass instances, a string and a integer.

When I compiled it with the optimization option (/o):

I got the following when disassembled:

There’s no trace of the value types (string and the int). Also the TestClass instance with null value is gone. But the TestClass instance, which is created in the heap rather than the stack, is still there even though there is nothing referencing it. It’s interesting to see that difference.

When I decompiled both versions with ILSpy and JustDecompile unused local variables were removed all the time. Apparently they do their own optimization.

Conclusion

It was a fun exercise to investigate and see the actual generated code. It was helpful to understand better complex constructs like async/await and also helpful to see there’s nothing to fear about the new features of C# 6.0!

Resources

csharp, development, aws, s3, rareburg, rss comments edit

Recently I got burned by another Windows update and my favorite podcatcher on the desktop, Miro, stopped functioning. I was already planning to develop something on my own so that I wouldn’t have to manually backup OPMLs (I’m sure there must be neat solutions already but again couldn’t resist the temptation of DIY!). So started exploring RSS feeds and the ways to produce and consume them. My first toy project is a feed generator for Rareburg, one of my favourite sites, Knowledge Base. Word on the street is the official feed for articles is on its way. Before it’s out there I decided play around with their API and generate my own feed.

Implementation

I developed a console application in C# that can be scheduled to generate the feed and upload it to AWS S3. Source code is here

Apparently .NET Framework has a System.ServiceModel.Syndication namespace since version 3.5 that contains all the tools need to consume and create an RSS feed with a few lines of code. The core part of the application is the part that generates the actual feed:

public SyndicationFeed GetFeed(List<Article> articles)
{
    SyndicationFeed feed = new SyndicationFeed(_feedServiceSettings.Title, _feedServiceSettings.Description, new Uri(_feedServiceSettings.BaseUri));
    feed.Title = new TextSyndicationContent(_feedServiceSettings.Title);
    feed.Description = new TextSyndicationContent(_feedServiceSettings.Description);
    feed.BaseUri = new Uri(_feedServiceSettings.BaseUri);
    feed.Categories.Add(new SyndicationCategory(_feedServiceSettings.Category));

    var items = new List<SyndicationItem>();
    foreach (var article in articles
        .Where(a => a.ispublished)
        .Where(a => a.ispubliclyvisible)
        .OrderByDescending(a => a.publisheddate))
    {
        var item = new SyndicationItem(article.title, 
            article.bodysnippet,
            new Uri (string.Format(_feedServiceSettings.ArticleUrl, article.slug)),
            article.articleid.ToString(),
            article.publisheddate);
        
        item.Authors.Add(new SyndicationPerson("", article.authorname, string.Format(_feedServiceSettings.UserUrl, article.user.username)));
        items.Add(item);
    }
    
    feed.Items = items;
    return feed;
}

The feed itself is independent from the format (RSS or Atom). The classes that do the actual formatting are derived from the abstract SyndicationFeedFormatter class: Rss20FeedFormatter and Atom10FeedFormatter. The format is read from the config file so the application supports both formats.

public SyndicationFeedFormatter CreateFeedFormatter(SyndicationFeed feed)
{
    string feedFormat = _feedSettings.FeedFormat;
    switch (feedFormat.ToLower())
    {
        case "atom": return new Atom10FeedFormatter(feed);
        case "rss": return new Rss20FeedFormatter(feed);
        default: throw new ArgumentException("Unknown feed format");
    }
}

and in the publisher service it gets the output feed:

var memStream = new MemoryStream();
var settings = new XmlWriterSettings(){ Encoding = Encoding.UTF8 };
using (var writer = XmlWriter.Create(memStream, settings))
{
    feedFormatter.WriteTo(writer);
}

I added the output of the API call as a JSON file under samples. Also implemented a fake API client called OfflineRareburgClient. It reads the response from a file instead of actually making an API call. It comes in handy if you don’t have an Internet connection or a valid API key. To use it in offline mode you have to change the the line that creates the client from this

var rareburgClient = new RareburgClient(configFactory.GetApiSettings());

to this

var rareburgClient = new OfflineRareburgClient(configFactory.GetOfflineClientSettings());

Lessons learned / Implementation Notes

So since the motivation behind the project is to learn more about manipulating RSS by myself here’s a few things that I’ve learned and used:

  • I created an IAM account that has write-only access to the bucket the feed will be stored in. It works with default permissions which is private access. But since RSS reader service will need access to the feed I had to upload the file with public access, Apparently changing ACL requires different permissions, namely s3:PutObjectAcl. Weird thing is just replacing s3:PutObject with s3:PutObjectAcl didn’t work either. They had to be both allowed. So after a few retries the final policy shaped up to be like this:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1418647210000",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::vp-public-temp/*"
            ]
        }
    ]
}
  • In this implementation I used Visual Studio’s neat feature Paste JSON As Classes.

Edit -> Paste Special -> Paste JSON As Classes

First I captured the API response with Fiddler. Then created a blank .cs file and using this option created the class to deserialize the response. Using strongly typed objects can easily be a daunting task if you are wrapping a whole API so I’d prefer to use dynamic objects like this:

var response = client.Execute<dynamic>(request);
var latestArticles = ((IEnumerable) response.Data.payload.articles).Cast<dynamic>()
                        .OrderByDescending(a => a.publisheddate)
                        .Select(a => new 
                        { 
                            slug = a.slug,
                            id = a.articleid,
                            title = a.title,
                            publishedDate = a.publisheddate,
                            ispublished = a.ispublished,
                            isvisible = a.ispubliclyvisile
                        });

This works fine but the problem in this case was the hyphens in some of the JSON property names which are not supported in C#. I can get around it if I use the strongly typed objects and specify the property name explicitly, such as:

[JsonProperty("is-published?")]
public bool ispublished { get; set; }

But I cannot do it in the dynamic version. I’ll put a pin into it and move on for now but have a feeling it will come back and haunt me in the future!

  • Default output of the RSS feed passes the validation but get 3 warnings. I’m sure they can be safely ignored but just of curiosity researched a little bit to see if I could pass with flying colors. Two of the three warnings were

    line 1, column 39: Avoid Namespace Prefix: a10 [help] <rss xmlns:a10=”http://www.w3.org/200 …

    line 12, column 5302: Missing atom:link with rel=”self” [help] … encompassing the Ch</description></item></channel></rss>

I found the solution on StackOverflow (not surprisingly!)

I made a few changes in the formatter factory

case "rss":
{
    var formatter = new Rss20FeedFormatter(feed);
    formatter.SerializeExtensionsAsAtom = false;
    XNamespace atom = "http://www.w3.org/2005/Atom";
    feed.AttributeExtensions.Add(new XmlQualifiedName("atom", XNamespace.Xmlns.NamespaceName), atom.NamespaceName);
    feed.ElementExtensions.Add(new XElement(atom + "link", new XAttribute("href", _feedSettings.FeedUrl), new XAttribute("rel", "self"), new XAttribute("type", "application/rss+xml")));
    return formatter;
}

I liked the fact I only had to make changes in one place so the factory could return a customized formatter instead of the default one and the rest of the application didn’t care at all. But unfortunately the fix required the publish URL or the feed. I got around it by adding to FeedSettings in the config but now the S3 settings and Feed settings need to be changed at the same time.

My idea was to make it like a pipeline so that the feed generator didn’t have to care how and where it’s published but this fix contradicted with that approach a little bit. Unfortunately it doesn’t look possible to use variables in the config files so that I could generate Feed.Url using the other settings.

  • The 3rd warning was encoding-related. If don’t explicitly specify the API uses ISO-8859-1 charset. I tried playing around a with a few headers to get the response in UTF-8 but the solution came from a friend: Accept-Charset header. So adding the header fixed that issue as well:
request.AddHeader("Accept-Charset", "UTF-8");

Conclusion

The genereated Atom feed doesn’t pass the validation but I will handle it later on. Since Atom is a newer format I think I’ll go with that in the future but so far it’s good to know that it’s fairly easy to play with RSS/Atom feeds with C# so it was a fun experiment after all…

Resources

personal, leisure, comedy, travel comments edit

I may not go out too often but I couldn’t miss this one for the world! Jim Jefferies is my favourite comedian and I finally had a chance to see a live show.

Jim Jefferies Ticket

When I saw the terrace option I went for it. Hoping paying extra would warrant better seats. I couldn’t be wronger! Even the MC made a lot of jokes about rich people sitting miles away from the stage while regular ticket owners had very close places.

There were 3 support acts which were about 10-15 minutes each. Jim took the stage around 20:40 and stayed until 22:00.

The first half was mostly about his newborn son and his relation with his girlfriend. Second half is mostly about religion, guns and sex. Overall it was hilarious as I anticipated. I especially loved how he encouraged people to overpower the security and have free beers :-)

I’ve seen all his shows before so seeing all new material was quite refreshing. Even though the stage was in galaxy far far away and the sound was not too loud still it was very nice to watch Jim Jefferies in open air.

Resources

xml, programming, csharp comments edit

Even though it’s a noisy data format it’s still commonly used and I happen to end up in situations that I need to use .NET to serialize and deserialize to and from XML documents. Here are a few problems that I had to tackle in the past. All the sample source code can be found in a GitHub repository.

01. Skip serializing unassigned values

Let’s assume we have a hypothetical Player class that has a few fields that looks like this:

public class Player
{
    public int Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int TotalGoalsScored { get; set; }
    public double AverageGoalsPerGame { get; set; }
    public Team Team { get; set; }
}

public class Team
{
    public string Name { get; set; }
    public int YearEstablished { get; set; }
}

So it contains both value and reference fields.

Now let’s create two random players and serialize them:

XmlSerializer serializer = new XmlSerializer(typeof(List<Player>));
Player player1 = new Player() { Id = 1, FirstName = "John", LastName = "Smith", TotalGoalsScored = 50, AverageGoalsPerGame = 0.7, Team = new Team() { Name = "Arsenal" } };
Player player2 = new Player() { Id = 2, FirstName = "Jack" };
using (StringWriter writer = new StringWriter())
{
    serializer.Serialize(writer, new List<Player>() { player1, player2 });
    Console.WriteLine(writer.ToString());
}

This code yields the following XML:

<?xml version="1.0" encoding="utf-16"?>
<ArrayOfPlayer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="
http://www.w3.org/2001/XMLSchema">
  <Player>
    <Id>1</Id>
    <FirstName>John</FirstName>
    <LastName>Smith</LastName>
    <TotalGoalsScored>50</TotalGoalsScored>
    <AverageGoalsPerGame>0.7</AverageGoalsPerGame>
    <Team>
      <Name>Arsenal</Name>
      <YearEstablished>0</YearEstablished>
    </Team>
  </Player>
  <Player>
    <Id>2</Id>
    <FirstName>Jack</FirstName>
    <TotalGoalsScored>0</TotalGoalsScored>
    <AverageGoalsPerGame>0</AverageGoalsPerGame>
  </Player>
</ArrayOfPlayer>

The thing to note here is that reference types were not serialized when they were unassigned, i.e Team object and LastName field in Player2. But same didn’t go for values fields. TotalScore, AverageScorePerGame and YearEstablished fields were serialized as 0. Of course this might be the desired outcome depending on your business requirements but in my case I didn’t want this because it might mislead the client consuming this data. At the very least I find it inconsistent as some unassigned values are serialized and some aren’t.

So to change the behaviour all we have to do is set the DefaultValue attribute for the numeric values like this:

public class Player
{
    public int Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    [DefaultValue(0)]
    public int TotalGoalsScored { get; set; }

    [DefaultValue(0)]
    public double AverageGoalsPerGame { get; set; }
    
    public Team Team { get; set; }
}

public class Team
{
    public string Name { get; set; }

    [DefaultValue(0)]
    public int YearEstablished { get; set; }
}

With this change the output becomes:

<?xml version="1.0" encoding="utf-16"?>
<ArrayOfPlayer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="
http://www.w3.org/2001/XMLSchema">
  <Player>
    <Id>1</Id>
    <FirstName>John</FirstName>
    <LastName>Smith</LastName>
    <TotalGoalsScored>50</TotalGoalsScored>
    <AverageGoalsPerGame>0.7</AverageGoalsPerGame>
    <Team>
      <Name>Arsenal</Name>
    </Team>
  </Player>
  <Player>
    <Id>2</Id>
    <FirstName>Jack</FirstName>
  </Player>
</ArrayOfPlayer>

So as the int and double values defaulted to 0 and we explicitly set the default value they won’t be serialized unless they are assigned a value other than zero.

In case you are wondering making the int and double nullable doesn’t produce the same result. In that case they are serialized with null values:

<TotalScore xsi:nil="true" />
<AverageScorePerGame xsi:nil="true" />

I think this if-you-set-it-you-get-it-back approach is consistent and makes most sense to me. I created a test project to fiddle with these classes. It has all 3 versions of the classes under different names and a console application displaying the outputs. If you want to play around you can get the source code here

02. Deserialize straight to list

Sometimes what you get in an XML document is just a list of items and the list is just a container so that the XML is well-formed. For example in the following example we have a list of players:

<PlayerList>
  <Player>
    <Id>1</Id>
    <FirstName>John</FirstName>
    <LastName>Smith</LastName>
    <TotalGoalsScored>50</TotalGoalsScored>
    <AverageGoalsPerGame>0.7</AverageGoalsPerGame>
    <Team>
      <Name>Arsenal</Name>
      <YearEstablished>0</YearEstablished>
    </Team>
  </Player>
  <Player>
    <Id>2</Id>
    <FirstName>Jack</FirstName>
    <TotalGoalsScored>0</TotalGoalsScored>
    <AverageGoalsPerGame>0</AverageGoalsPerGame>
  </Player>
</PlayerList>

The sole purpose of PlayerList tag is to act as root and contain multiple objects. Other than that it has no function. When we deserialize this to C# objects we would normally need 2 objects like this:

public class Player
{
    public int Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int TotalGoalsScored { get; set; }
    public double AverageGoalsPerGame { get; set; }
}

and

[XmlRoot]
public class PlayerList
{
    [XmlElement("Player")]
    public List<Player> Players { get; set; }
}

and we can get the list of objects by:

string inputXmlPath1 = @".\InputXml.xml";
using (StreamReader reader = new StreamReader(inputXmlPath1))
{
    XmlSerializer playerListSerializer = new XmlSerializer(typeof(PlayerList));
    PlayerList playerList = (PlayerList)playerListSerializer.Deserialize(reader);
}

In such cases I generally tend to eliminate the “middle man”. I don’t want a container class which only holds a List. So I’d like to deserialize this XML directly into List.

What I want to do is actually this:

using (StreamReader reader = new StreamReader(inputXmlPath1))
{
    XmlSerializer playerListSerializer = new XmlSerializer(typeof(List<FinalPlayer>));
    List<FinalPlayer> playerList = (List<FinalPlayer>)playerListSerializer.Deserialize(reader);
}

But without any modifications to our classes it throws an exception:

By eliminating PlayerList class we actually stopped providing XmlRoot info to the serializer. But that can quickly be remedied by using a constructor overload of XmlSerializer:

using (StreamReader reader = new StreamReader(inputXmlPath1))
{
    XmlSerializer playerListSerializer = new XmlSerializer(typeof(List<FinalPlayer>), new XmlRootAttribute("PlayerList"));
    List<FinalPlayer> playerList = (List<FinalPlayer>)playerListSerializer.Deserialize(reader);
}     

This works when the class name matches the XmlElement name. If you need to customize your class’s name (like FinalPlayer in my example) you need to decorate the class with XmlType and supply the element name so that the serializer can do the mapping.

[XmlType("Player")]
public class FinalPlayer
{
    public int Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int TotalGoalsScored { get; set; }
    public double AverageGoalsPerGame { get; set; }
}

So now we can have any class name mapping to corresponding elements and deserialized straight to a List.

03. Remove namespaces

I know using namespaces is considered a good practice as it helps avoid name conflicts but honestly I never suffered from such a problem before so I don’t mind removing them from my XML documents and clean the clutter. XML is already a noisy data format no need to bloat it any further. I think it might help when you are working with data from different sources but if you are only working with your own classes and data structures name conflict is generally not something to worry about (assuming you name your objects properly)

So say you have a simple Player class and you serialize it with a out-of-the-box XmlSerializer:

Player player = new Player() { Id = 102, FirstName = "Danny", LastName = "TopScorer", AverageGoalsPerGame = 3.5, TotalGoalsScored = 150 };
XmlSerializer serializer = new XmlSerializer(typeof(Player));
XmlWriterSettings settings = new XmlWriterSettings() { OmitXmlDeclaration = true, Indent = true, Encoding = Encoding.UTF8 };
StringBuilder output = new StringBuilder();
XmlWriter writer = XmlWriter.Create(output, settings);
serializer.Serialize(writer, player);
Console.WriteLine(output.ToString());

This yields the following output:

<Player xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://
www.w3.org/2001/XMLSchema">
  <Id>102</Id>
  <FirstName>Danny</FirstName>
  <LastName>TopScorer</LastName>
  <TotalGoalsScored>150</TotalGoalsScored>
  <AverageGoalsPerGame>3.5</AverageGoalsPerGame>
</Player>

So in order to get rid of namespaces we have to specify our custom namespaces, which is empty in this case and use the overloaded XmlSerializer constructor to pass it in:

XmlSerializerNamespaces xns = new XmlSerializerNamespaces();
xns.Add(string.Empty, string.Empty);
serializer.Serialize(writer, player, xns);

to get the XML without any namespaces:

<Player>
  <Id>102</Id>
  <FirstName>Danny</FirstName>
  <LastName>TopScorer</LastName>
  <TotalGoalsScored>150</TotalGoalsScored>
  <AverageGoalsPerGame>3.5</AverageGoalsPerGame>
</Player>

04. Change output encoding

When you serialize with the default options and with the XML declaration the encoding is UTF-16. Oddly enough there is no option to specify the output encoding. In order to achieve that and sometimes you may need to change it. For example a 3rd party was expecting UTF-8 in my case so the default value didn’t cut it for me.

So using the same Player class from the last example the following code produces an output with UTF-16

Player player = new Player() { Id = 102, FirstName = "Danny", LastName = "TopScorer", AverageGoalsPerGame = 3.5, TotalGoalsScored = 150 };
XmlSerializer serializer = new XmlSerializer(typeof(Player));
XmlWriterSettings settings = new XmlWriterSettings() { OmitXmlDeclaration = false, Indent = true, Encoding = Encoding.UTF8 };
StringBuilder output = new StringBuilder();
XmlWriter writer = XmlWriter.Create(output, settings);
XmlSerializerNamespaces xns = new XmlSerializerNamespaces();
xns.Add(string.Empty, string.Empty);
serializer.Serialize(writer, player, xns);
Console.WriteLine(output.ToString());

Output:

<?xml version="1.0" encoding="utf-16"?>
<Player>
  <Id>102</Id>
  <FirstName>Danny</FirstName>
  <LastName>TopScorer</LastName>
  <TotalGoalsScored>150</TotalGoalsScored>
  <AverageGoalsPerGame>3.5</AverageGoalsPerGame>
</Player>

I found the solution here on StackOverflow.

So the solution is to extend from StringWriter and add a new constructor that accepts Encoding. StringWriter already has an Encoding property as shown below, but unfortunately it doesn’t have a public setter so we need a subclass to fiddle with it.

StringWriterWithEncoding simply overrides th encoding field:

public sealed class StringWriterWithEncoding : StringWriter
{
    private readonly Encoding encoding;

    public StringWriterWithEncoding(Encoding encoding)
    {
        this.encoding = encoding;
    }

    public override Encoding Encoding
    {
        get { return encoding; }
    }
}

By using the new class the following code produces the desired outcome:

StringWriterWithEncoding utf8StringWriter = new StringWriterWithEncoding(Encoding.UTF8);
Player player = new Player() { Id = 102, FirstName = "Danny", LastName = "TopScorer", AverageGoalsPerGame = 3.5, TotalGoalsScored = 150 };
XmlSerializer serializer = new XmlSerializer(typeof(Player));
XmlWriterSettings settings = new XmlWriterSettings() { OmitXmlDeclaration = false, Indent = true, Encoding = Encoding.UTF8 };
XmlWriter writer = XmlWriter.Create(utf8StringWriter, settings);
XmlSerializerNamespaces xns = new XmlSerializerNamespaces();
xns.Add(string.Empty, string.Empty);
serializer.Serialize(writer, player, xns);
Console.WriteLine(utf8StringWriter.ToString());
Console.ReadLine();

Output:

<?xml version="1.0" encoding="utf-8"?>
<Player>
  <Id>102</Id>
  <FirstName>Danny</FirstName>
  <LastName>TopScorer</LastName>
  <TotalGoalsScored>150</TotalGoalsScored>
  <AverageGoalsPerGame>3.5</AverageGoalsPerGame>
</Player>

05. Fail deserialization when unexpected elements are encountered

By default XMLSerializer is very fault tolerant. It just salvages whatever it can and leaves alone the unmatching values. Sometimes you may need to be stricter. For example I had a case when the external source returned a whole different XML when there was an error on its end. So when that happened I wanted to be notified about it instead of getting null objects quietly.

For example, assume we have the usual PlayerList that we are deserializing to List. If for some reason we get a weird Player list like this:

<PlayerList>
  <Customer>
    <Id>1</Id>
    <FirstName>John</FirstName>
    <LastName>Smith</LastName>
    <TotalGoalsScored>50</TotalGoalsScored>
    <AverageGoalsPerGame>0.7</AverageGoalsPerGame>
  </Customer>
</PlayerList>

When we deserialize it with the following code block

try
{
    using (MemoryStream memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(rawXml)))
    {
        XmlSerializer serializer = new XmlSerializer(typeof(List<Player>), new XmlRootAttribute("PlayerList"));
        var result = (List<Player>)serializer.Deserialize(memoryStream);
    }
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
    if (ex.InnerException != null)
    {
        Console.WriteLine(ex.InnerException.Message);    
    }
}

XmlSerializer doesn’t complain at all. Instead it just returns an empty list because it cannot find any Player objects. In order to change this behaviour we can use the events it exposes such as UnknownNode, UnknownElement, UnknownAttribute. UnknownNode is just the combination of the first two events. In my case I didn’t want to be too strict so I didn’t want an exception in case of a missing attribute but hooked into the UnknownElement event:

try
{
    using (MemoryStream memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(rawXml)))
    {
        XmlSerializer serializer = new XmlSerializer(typeof(List<Player>), new XmlRootAttribute("PlayerList"));
        serializer.UnknownElement += serializer_UnknownElement;
        var result = (List<Player>)serializer.Deserialize(memoryStream);
    }
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
    if (ex.InnerException != null)
    {
        Console.WriteLine(ex.InnerException.Message);
    }
}

and added the event handler:

void serializer_UnknownElement(object sender, XmlElementEventArgs e)
{
    throw new ArgumentException(string.Format("Unknown element: {0}", e.Element.LocalName));
}

So now at least I can distinguish a weird list from a really empty list.

Resources

programming, debug comments edit

Even though APIs are RESTful these days sometimes you might need to interact with a SOAP-based web service.

I had to consume an SOAP XML web service the other day and encountered a strange problem while debugging my test application. The application was working fine but when I tried to debug it was closing silently. So as a first step I opened Debug –> Exceptions and checked all to let the application break upon any type of exception it gets.

Break when any exception is thrown

After running the application with this setting at least I was able to see what the fuss was all about:

SOAP BindingFailure exception

There are various approaches to resolve this issue and finally I found the correct one on a Stackoverflow answer: Go to Project Properties and in the Build tab turn on Generate Serialization Assembly setting.

Turn on Generate Serialization Assembly setting

When this setting it turned on, it generates an assembly named {Your Application Name}.XmlSerializers.dll.

Out of curiosity I peeked into the assembly with ILSpy and it looks like this:

XmlSerializers.dll in ILSpy

Basically it just generates an abstract class deriving from XmlSerializer (named XmlSerializer1) and generates a bunch of sealed child classes deriving from that class.

I’ve never been a fan of auto-generated code anyway and looks like I’m not going to need it in my code but it’s used in the background by the framework. I added links to a few Stackoverflow answers related to that assembly though. I gather the advantage of turning it on is reduced startup time and being able to debug in this case. The disadvantage is increased deployment size which I think is negligible in this day and age so I’ll just keep it on and debug happily ever after!

Resources

events comments edit

Yesterday I attended AWS Summit event and wanted to post my impressions and notes from the event. As you can see in the image below there were quite a few people there:

Overview during lunch break

Keynote

Keynote was given by Amazon.com’s CTO Werners Vogel. Some notes from the keynote:

  • It was quite a big event: 3000+ attendees
  • Intel was introduced as platinum partner. Last month I attended to AWSome Day and they mentioned Intel have manufactured a special chip for EC2 only (C4-type instances). It’s designed specifically to address heat considerations.
  • They have now more than 1 minnion active customers
  • According to Gartner, AWS has 5-times compute capacity than 14 closest competitors combined (which also explains the specific requirements to reduce heat in chips)

  • There were guest speakers who emphasized different features of AWS
    • GoSquared: Experimentation, how it is easier to try out new technologies in a cost-effective way
    • iTv: Microservices as opposed to monolithic applications
    • OmniFone: Scalability
    • Just Giving: Distributed regions and data analysis capabilities
    • AWS VP of public sector: Usage in public sector. It started after 12 and everyone started leaving.

  • New features introduced

    • Elastic File System: SSD-based, automatically replicated, auto-resized file system. Currently in preview mode, will be released this summer
    • Machine Learning: Makes it easier to analyze big data for search intent, demand estimation etc.
    • Lambda: Event-driven, fully-managed compute service
    • EC2 Container Service: There is a big shift towards Docker and containers. He said: “The container is now the output of your development process”

  • Generally microservices approach is favored: Building large solutions with smaller blocks allows faster, cost-effective solutions which can be adapted easier to changes
  • Security and their compliance with major certification requirements is emphasized. But he didn’t mention shared-responsibility principle which AWS adopts. Just because you AWS doesn’t mean you’re compliant to all the regulations as well.
  • They have support for hybrid solutions but according to AWS’s vision it’s not the destination, just a transition
  • He made an analogy that fighting the cloud is like “fighting the gravity”: It’s a fight you cannot win!

Track sessions

After the lunch break there were a lot of sessions about various AWS features. I picked Technical Track 1 which included EC2 Container Services, Lambda, CloudFormation and CodeDeploy

EC2 Container Service

I know using containers is a big deal nowadays but still haven’t the chance to try it out myself. I was hoping from this session to find out more about it but didn’t benefit much from it as it didn’t cover the very basics. But in the light of keynote, it’s obvious there’s a huge demand on containers so will be the first service I’ll try next.

Lambda

This is a very cool new service. Instead of running every small job in small services in small EC2 instances now we can get rid of all the maintenance and costs and just let AWS run our code whenever an event is triggered.

  • It currently supports a wide variety of event sources such as objects put to S3 buckets, DynamoDB table updates, SNS messages etc.
  • SQS support is coming soon
  • Currently it runs node.js but it can be used to launch a Java application but native Java support is coming soon so it can directly execute Java code.
  • It even allows access to underlying processes, threads, file system and sockets
  • It’s charged per invocation so you don’t pay anything for idle times.

Infrastructure as code

The focus of the session was CloudFormation but a client of AWS showed how they used Eclipse to deploy with a single click so it can be done in several ways. (That’s why the title of the talk wasn’t CloudFormation)

This is also a great tool to automatically launch a whole infrastructure based on ERB configuration files. I don’t have much experience in this one but was a nice session to see its capabilities in action.

CodeDeploy

This is yet another cool new feature just gone out of preview mode. You can automatically deploy new code based on your rules. For example, you can deploy one-at-a-time. It verifies every deployment and moves to next one. Or you can deploy new version on half of the instances meaning that half of your system will be available even if the deployment fails. Or if you like some adrenaline rush you can deploy to all instances at once :-)

You can specify pre and post scripts that handle the clean-up tasks and verifying the deployment.

CodeDeploy has been GA (General Availability) but 2 more services were introduced yesterday: CodePipeline and CodeCommit

The idea is to fully-automate the whole process from source code checking to deployment which according to the speaker is a very time consuming task in their own applications.

Conclusion

It was a nice big event and I’m glad I had the chance to attend to it. The content was rich to cover every aspect of AWS. I decided to attend to the sessions instead of labs as I can do them online here. It’s also a bit overwhelming to see there’s so much to learn about but that’s always a challenge in this industry anyway. As Werner Vogels said in his keynote: “Innovation is continuous!”

Resources