awsdev route53, csharp

If you don’t have a static IP and you need to access your home network for some reason you may find Dynamic DNS services useful. I was using No-IP until its limitations became offputting. Since I love DIY stuff and AWS I thought I could build the exact same service on my own using Route 53.

Enter DynDns53

It’s a simple application. What it does is basically:

  • Get current external IP
  • For each subdomain in the configuration, call AWS API and get domain info
  • Check if the recorded IP is different than the current one
  • If so, call AWS API to update it with the new one, if not do nothing.

In a different post on setting up Calibre on a Raspberry Pi I developed a script that did the same thing:

import boto.route53
import urllib2
currentIP = urllib2.urlopen("http://checkip.amazonaws.com/").read()

conn = boto.connect_route53()
zone = conn.get_zone("volki.info.")
change_set = boto.route53.record.ResourceRecordSets(conn, '{HOSTED_ZONE_ID}')

for rrset in conn.get_all_rrsets(zone.id):
    if rrset.name == 'calibre.volki.info.':
        u = change_set.add_change("UPSERT", rrset.name, rrset.type, ttl=60)
        rrset.resource_records[0] = currentIP
        u.add_value(rrset.resource_records[0])
        results = change_set.commit()

The difference of DynDns53 is that it can run as a background service and it has a cute logo! The C# equivalent of above code is like this:

public void Update()
{
    var config = _configHandler.GetConfig();
    string currentExternalIp = _ipChecker.GetExternalIp();

    foreach (var domain in config.DomainList)
    {
        string subdomain = domain.DomainName;
        string zoneId = domain.ZoneId;

        var listResourceRecordSetsResponse = _amazonClient.ListResourceRecordSets(new ListResourceRecordSetsRequest() { HostedZoneId = zoneId });
        var resourceRecordSet = listResourceRecordSetsResponse.ResourceRecordSets.First(recordset => recordset.Name == subdomain);
        var resourceRecord = resourceRecordSet.ResourceRecords.First();

        if (resourceRecord.Value != currentExternalIp)
        {
            resourceRecord.Value = currentExternalIp;

            _amazonClient.ChangeResourceRecordSets(new ChangeResourceRecordSetsRequest()
            {
                HostedZoneId = zoneId,
                ChangeBatch = new ChangeBatch()
                {
                    Changes = new List<Change>() 
                    { 
                        new Change(ChangeAction.UPSERT, resourceRecordSet)
                    }
                }
            });
        }
    }
}

That’s it. The rest is just user interface, a Windows service and some basic uni tests.

Usage

The project has a WPF user interface and a Windows service. Both are using the same library (DynDns53.Core) so the functionality is the same. Using the user interface is pretty straightforward.

Just add your AWS keys that have access privileges to your domains. You can set it to run at system start so you don’t have to start it manually.

I built the Windows service using Topshelf. To install it, build the application and on an elevated command prompt just run

DynDns53.Service.exe install

Similarly to uninstall:

DynDns53.Service.exe uninstall

What’s missing

  • Landing page: I’ll update the project details on the site generated by GitHub Pages.
  • Mac/Linux support: As the worlds are colliding I’d like to discover what I can do with the new .NET Core runtime. Since this is a simple project I think it’s a good place to start

Resources

devaws csharp, route53

Part 3: Consuming the APIs

So far in the series:

In this installment it’s time to implement the actual application that consumes the TLD API as well as AWS Route 53 API. I decided to turn this into a little project with a name, logo and everything. I’ll keep updating the project. Currently it just has the core library and a console application. I know it’s not enough but you have to start somewhere!

Core functionality

Basically what it does is:

  1. Get the supported TLD list

  2. For each TLD send a CheckDomainAvailability request to AWS

  3. Return the resultset

When I first started out I was meaning to make these requests run in parallel. But they got throttled and I ended up using all my allowance. So I changed the code to make a call per second:

The TldProvider I developed can be used in two different ways:

  1. As a NuGet Package

I uploaded my package to NuGet.org so you can install it by running

Install-Package TldProvider.Core.dll

and the usage would be:

private List<Tld> GetTldListFromLibrary()
{
    string url = ConfigurationManager.AppSettings["Aws.Route53.DocPageUrl"];
    var tldListProvider = new TldListProvider();
    var supportedTLDlist = tldListProvider.GetSupportedTldList(url);
    return supportedTLDlist;
}
  1. As an API the usage is a bit more complex because of the JSON to list conversion:
private List<Tld> GetTldListFromApi()
{
    string apiUrl = ConfigurationManager.AppSettings["TldProvider.Api.EndPoint"];
    var httpClient = new HttpClient();
    var request = new HttpRequestMessage() { Method = new HttpMethod("GET"), RequestUri = new Uri(apiUrl) };
    var httpResponse = httpClient.SendAsync(request).Result;
    var apiResponse = httpResponse.Content.ReadAsStringAsync().Result;
    dynamic resp = Newtonsoft.Json.JsonConvert.DeserializeObject(apiResponse);
    var resultAsStringList = JToken.Parse(apiResponse)["tldList"].ToObject<List<string>>(); ;
    var resultAsTldList = resultAsStringList.Select(t => new Tld() { Name = t }).ToList();
    return resultAsTldList;
}

Actually this method should not return List as it would require a reference to the library. The idea of using the API is to get rid of that dependency in the first place but I just wanted to add both and use them interchangably.

So finally tha main course: The check method that loops through the TLD list and gets the availability from AWS Route 53 API

public List<DomainCheckResult> Check(string domainName)
{
    string ACCESS_KEY = ConfigurationManager.AppSettings["Aws.Route53.AccessKey"];
    string SECRET_KEY = ConfigurationManager.AppSettings["Aws.Route53.SecretKey"];

    var results = new List<DomainCheckResult>();

    var credentials = new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY);
    var config = new AmazonRoute53DomainsConfig()
    {
        RegionEndpoint = RegionEndpoint.USEast1
    };

    var amazonRoute53DomainsClient = new AmazonRoute53DomainsClient(credentials, config);

    // var supportedTLDlist = GetTldListFromLibrary();
    var supportedTLDlist = GetTldListFromApi();

    foreach (var tld in supportedTLDlist)
    {
        var request = new CheckDomainAvailabilityRequest() { DomainName =  $"{domainName}.{tld.Name}" };

        var response = amazonRoute53DomainsClient.CheckDomainAvailabilityAsync(request);
        var result = response.Result;
        Console.WriteLine($"{request.DomainName} --> {result.Availability}");
        results.Add(new DomainCheckResult()
        {
            Domain = domainName,
            Tld = tld.Name,
            Availability = result.Availability,
            CheckDate = DateTime.UtcNow
        });

        System.Threading.Thread.Sleep(1000);
    }

    return results;
}

TroubleShooting

I had a few problems while implementing. Learned a few things while fixing them. So here they are:

  • I created an IAM account that has access to Route53. But that wasn’t enough. There is a separate action called route53domains. I had to grant access to this as well to use domains API.

  • I use EUWest1 region since it’s the closest one. Normally Route53 doesn’t require any region setting but weirdly I had to set region to US East1 in order to access Route53Domains API.

The problems above makes me think domains didn’t fit very well with the current Route53. It feels like it has not integrated seamlessly yet.

First client: Console application

This used to be the test application but since I don’t have any user interface currently this will be my first client. It just accepts the domain name from the user, checks the availability and saves the results in JSON format. This is all the code of the console application:

static void Main(string[] args)
{
    if (args.Length == 0)
    {
        Console.WriteLine("Usage: DomChk53.UI.Console {domain name}");
        return;
    }

    string domain = args[0];
    var checker = new DomainChecker();
    var results = checker.Check(domain);

    string outputFilename = $"results-{domain}-{DateTime.UtcNow.ToString("yyyyMMdd-HHmmss")}.json";
    string json = Newtonsoft.Json.JsonConvert.SerializeObject(results);
    File.WriteAllText(outputFilename, json);
    Console.WriteLine($"Saved the results to {outputFilename}");
}

It also displays the results of each domain on the console you can have keep track of what it’s doing

It saves the results with a timestamp so a history can be maintained:

  {
    "Domain": "volkan",
    "Tld": "cool",
    "Availability": "AVAILABLE",
    "CheckDate": "2015-08-05T12:31:58.40081Z"
  },

What’s missing

It’s not much but at least it delivers what it promises: You give it a domain name and it checks the availability of all supported TLDs by AWS Route 53 and returns the results. But there’s a lot to do to improve this project. Some items in my list:

  • A web-based user interface that allows viewing these results online.
  • Get the price list for each tLD as well and display it along with the name
  • Registering selected domains: It would be nice if you could buy the available domains using the same interface
  • A landing page at domchk53.com so that it can have links to the source code, online checker etc.

Conclusion

The reason I started this project was to overcome the limitations of AWS Route 53 domains UI and be able to search all supported TLDs all at once. To consider the project complete I will prepare a single-page website about the project and develop a nice user-interface. But in terms of basic functionality I can get what I set out for so I can think of it as a small success I guess.

Resources

devaws csharp, javascript, route53, lambda, amazon_api_gateway, nodejs

Part 2: Converting TLD Provider into an API

In all fairness, this part is not entirely necessary for the project. I already found a way to get the list of TLDs from Amazon documentation so I could easily integrate it with my client applications. But I recently heard about Amazon API Gateway which has been introduced a few weeks back so I thought it would be cool to access my JavaScript method via a web API hosted on AWS. After all, the point of many projects I develop is to learn new stuff! In light of that I’ll try something new with the C# version as well and use a self-hosted service so that I can use a Windows Service instead of IIS.

API #1: Amazon API Gateway

There are a couple of new technologies here that I haven’t used before so this was a good chance for me to play with them.

Amazon API Gateway looks like a great service to create hosted APIs. Also it’s integrated with AWS Lambda. You can bind an endpoint directly to lambda function. There are other binding options but in this project I will use Lambda as it was also in my to-learn list.

Setup Amazon API Gateway

API Gateway interface is very intuitive. Within minutes I was able to create a GET method calling my Lambda function. First you create the API by clicking the “Create API” button!

Then you add your resource by clicking “Create Resource”. By default it comes with the root resource (“/”) so you can just use that one as well to add methods.

I created a resource called tldlist. All I needed was a GET method so I created it by “Create Method”.

You select the region and enter the full ARN of your Lambda function. In the UI it just says “function name” but it requires full ARN (i.e.: arn:aws:lambda:eu-west-1:1234567890:function:getTldList)

…and Lambda

The function is a bit different from the previous version. In Node.js I wasn’t able to use XMLHttpRequest object and turns out your use the http module to make web requests so I modified the code a bit. Here’s the final version of my Lambda function:

console.log('Loading function');

var docHost = 'docs.aws.amazon.com';
var docPath = '/Route53/latest/DeveloperGuide/registrar-tld-list.html';
var supportedTldPage = docHost + docPath;
var http = require('http');

exports.handler = function(event, context) {
    var options = {
      host: docHost,
      path: docPath,
      port: 80
    };
    
    http.get(options, function(res) {
        var body = '';
        res.on('data', function(chunk) {
            body += chunk;
        });
        
        res.on('end', function() {
            console.log(body);
            var result = parseHtml(body);
            context.succeed(result);
        });
    }).on('error', function(e) {
        console.log("Got error: " + e.message);
    });     

};

function parseHtml(pageHtml) {
    var pattern = /<a class="xref" href="registrar-tld-list.html#.+?">[.](.+?)<\/a>/g;
    var regEx = new RegExp(pattern);

    var result = {};
    result.url = 'http://' + supportedTldPage;
    result.date = new Date().toUTCString();
    result.tldList = [];

    while ((match = regEx.exec(pageHtml)) !== null) {
        result.tldList.push(match[1]);
    }

    return result;
}

In order to return a value from Lambda function you have to call succeed, fail or done:

context.succeed (Object result);
context.fail (Error error);
context.done (Error error, Object result);

Succeed and fail are self-explanatory. done is like a combination of both. If error is non-null it treats it as failure. Even if you call fail or done with an error the HTTP response code is always 200. what changes is the message body. For example, I played around with a few possibilities to test various results:

Method call: context.succeed(result);  
Output: Full JSON results

Method call: context.done(null, result);  
Output: Full JSON results

Method call: context.fail("FAIL!!!");  
Output: {"errorMessage":"FAIL!!!"}

Method call: context.done("FAIL!!!", results);  
Output: {"errorMessage":"FAIL!!!"}

As you can see, if error parameter is not null it ignores the actual results. Also I removed the JSON.stringify call from the parseHtml method because API gateway automatically converts it to JSON.

Tying things together

Deployment is also a breeze, just like creating the API, resource and the methods all it takes is a few button clicks. You click Deploy API and create an environment such as staging or prod. And that’s it! You’re good to go!

Since this will be a public-facing API with no authentication I also added a CloudWatch alarm:

This way if some mental decides to abuse I will be aware of it. The good thing is it’s very cheap. It costs $3.5 per million API calls which is about £2.25. I don’t think it will break the bank but for serious applications authorization is a must so I will need to investigate that feature in the future anyway.

At this point, I have a Lambda function called from the API hosted by AWS. I don’t have to worry about anything regarding the maintenance and scaling which feels great!

API #2: C# & Self-hosting on a Windows Service

Speaking of maintenance, here comes the Windows version! I don’t intend to deploy it on production but I was meaning to learn self-hosting APIs with Web API to avoid IIS and here’s chance to do so.

Found this nice concise article showing how to run a Web API inside a console application. I tweaked the code a little bit to suit my needs. Basically it takes 4 simple steps:

Step 01. Install OWIN Self-Host NuGet package:

Install-Package Microsoft.AspNet.WebApi.OwinSelfHost

Step 02. Setup routing

public class Startup
{
    public void Configuration(IAppBuilder appBuilder)
    {
        HttpConfiguration config = new HttpConfiguration();
        config.Routes.MapHttpRoute(
            name: "DefaultApi",
            routeTemplate: "api/{controller}/{id}",
            defaults: new { id = RouteParameter.Optional }
        );

        appBuilder.UseWebApi(config);
    }
}

Step 03. Add the controller

public class TldController : ApiController
{
    public HttpResponseMessage Get()
    {
        string supportedTLDPage = ConfigurationManager.AppSettings["AWS-URL"];
        var tldProvider = new TldListProvider();
        var tldList = tldProvider.GetSupportedTldList(supportedTLDPage);
        var output = new {
            url = supportedTLDPage, 
            date = DateTime.UtcNow.ToString(),
            tldList = tldList.Select(tld => tld.Name)
        };
        return this.Request.CreateResponse(HttpStatusCode.OK, output);
    }
}

Step 04. Start OWIN WebApp

public void Start()
{
    string baseAddress = ConfigurationManager.AppSettings["BaseAddress"];
    WebApp.Start<Startup>(url: baseAddress);
    Console.WriteLine("Service started");
}

Final step is installation. As I used TopShelf all I had to do was running a command prompt with administrator privileges run this command:

TldProvider.Service.exe install

Now that my service is running in the background and accepting HTTP requests let’s take it out for a spin:

Brilliant!

What’s next?

So now I have an API that returns me the supported TLD list. In the next post I’ll work on a basic client that consumes that API and AWS Route53 to get the availability results finally.

Resources