developmentawslambdas3sns

UPDATE: Yesterday (October 8th, 2015) Amazon announced official support for scheduled events so I updated my function to use this feature. For the most up-to-date version of this project please visit the updated version

I have a great accountant but he has one flaw: I have to ask for the invoice every month! While waiting for him to automate the process, I decided to automate what I can on my end. There are many ways to skin a cat, as the saying goes, the way I picked for this task was developing an AWS Lambda funciton and trigger it by subscribing to a public SNS topic.

Step 1: Prepare a function to send emails

Developing a simple node.js function that sends E-mails was simple. First I needed the install two modules:

npm install nodemailer
npm install nodemailer-smtp-transport

And the function is straightforward:

var transporter = nodemailer.createTransport(smtpTransport({
    host: 'email-smtp.eu-west-1.amazonaws.com',
    port: 587,
    auth: {
        user: '{ACCESS KEY}',
        pass: '{SECRET KEY}'
    }
}));

var text = 'Hi, Invoice! Thanks!';

var mailOptions = {
    from: 'from@me.net',
    to: 'to@someone.net',
    bcc: 'me2@me.com',
    subject: 'Invoice',
    text: text 
};

transporter.sendMail(mailOptions, function(error, info){
      if(error){
		console.log(error);
      }else{
		console.log('Message sent');
      }
  });

The challenge was deployment as the script had some dependencies. If you choose Edit Inline and just paste the script you would get an error like this

"errorMessage": "Cannot find module 'nodemailer'",

But it’s very easy to deploy a full package with dependencies. Just zip everything in the folder (wihtout the folder itself) and upload the zip file. The downside of this method is you can no longer edit the code inline. So even just for fixing a trivial typo you need to re-zip and re-upload.

Step 2: Schedule the process

One simple method to schedule the process is to invoke the method using Powershell and schedule a task to run the script:

Invoke-LMFunction -FunctionName automatedEmails -AccessKey accessKey -SecretKey secretKey -Region eu-west-1

But I don’t want a dependency on any machine (local or EC2 instance). Otherwise I could write a few lines of code in C# to do the same job anyway. The idea of using Lambda is to avoid maintenance and let everything run on infrastructure that’s maintained by AWS.

Unreliable Town Clock

Unfortunately AWS doesn’t provide an easy method to schedule Lambda function invocations. For the sake of simplicity I decided to use Unreliable Town Clock (UTC) which is essentially a public SNS topic that sends “chime” messages every 15 minutes.

Since all I need is one email, I don’t care if it skips a beat or two as long as it chimes at least once throughout the day.

State management

Of course to avoid bombarding my accountant with emails I have to maintain a state so that I would only send one email per month. But Lambda functions must be stateless. Some alternatives are using AWS S3 or DynamoDB. Since all I need is one simple integer value I decided to store in a text file on S3. So first I download the log file and check the last sent email month:

function downloadLog(next) {
	s3.getObject({
			Bucket: bucketName,
			Key: fileName
		},
		next);

function checkDate(response, next) {
	var currentDay = parseInt(event.Records[0].Sns.Message.day);
	currentMonth = parseInt(event.Records[0].Sns.Message.month);
	var lastMailMonth = parseInt(response.Body.toString());
	if (isNaN(lastMailMonth)) {
		lastMailMonth = currentMonth - 1;
	}
	if ((currentDay == targetDayOfMonth) && (currentMonth > lastMailMonth)) {
		next();
	}
}

Putting it together

So putting it all together the final code is:

Let’s see if it’s going to help me get my invoices automatically!

Conclusion

  • A better approach would be to check emails for the invoice and send only if it wasn’t received already. Also a copule of reminders after the initial email would be nice. But as my new resolution is to progress in small, incremental steps I’ll call it version 1.0 and leave the remaining tasks for a later version.

  • My main goal was to achieve this task without having to worry about the infrastructure. I still don’t but that’s only because a nice guy (namely Eric Hammond) decided to setup a public service for the rest of us.

  • During my research I came across a few references saying that the same task can be done using AWS Simple Workflow (SWF). I haven’t used this service before. Looked complicated and felt like there is a steep learning curve to go through. In Version 2 I should look into SWF which would…

    • allow me to handle a complex workflow
    • make dependency to public SNS topic redundant
    • handle state properly

Resources

csharpdevelopmentWPF

I like scanning all my documents and keep a digital version as well as the dead-tree version just in case. But storing the documents as individual image files is too messy. There are various solutions to merge image files into a single PDF but I don’t want to upload my sensitive documents to unknown parties so I rolled my own: Image2PDF

Implementation

I decided to go with a WPF application rathen than a web-based solution because it would be unnecessary to upload a bunch of image files to cloud and download the PDF back. It’s a lot of network traffic for something that can be handled locally much faster.

Image2PDF a simple WPF application developed in C#. I’ve used iTextSharp library to create the PDFs. It’s a great open-source library with lots of capabilities.

I’ve also started using SyncFusion WPF control library recently. The entire control suite is free of charge (which is always a great price!). It has a Wizard control which I decided to go with. I think using a wizard and is cleaner UI instead of dumping every control on one window.

Usage

As you might expect you provide the list of input images, re-order them if needed, choose the path for the output folder and go!

Step 1: Select input

Page 1: Specify input image files

Step 2: Select output

Page 2: Specify output PDF path

Step 3: Go!

Page 3: Run!

Simples!

Conclusion

In the future I’m planning to add various PDF operarions and enhance the functionality but since shipping is a feature I’ll limit the scope for now. After all, this was the main feature I needed anyway.

Resources

csharpneo4jcypherjsondevelopment

I have a simple JSON file that contained a bunch of users with their followers which looked like this:

  {
    "user-id": 2,
    "username": "user_2",
    "avatar": "URL",
    "following": [
      "user_10",
      "user_6",
      "user_1"
    ],
    "followers": [
      "user_10"
    ]
  }

It felt like a good exercise would be to import that data into a graph database as it wasn’t something I had done before.

As my go-to language is C# and I had some experience with Cypher before, my initial instinct was to develop a tool to generate Cypher statements from the JSON.

First I create the nodes:

private void GenerateNodesCypher()
{
    string filename = @"..\..\output\nodes.cql";
    var output = new StringBuilder();

    foreach (var user in _userList)
    {
        string s = $"CREATE ({user.username}:User {{ userid: {user.userid} , username: '{user.username}', avatar: '{user.avatar}'  }} ); ";
        output.AppendLine(s);
    }

    File.WriteAllText(filename, output.ToString());
}

and then the relationships:

private void GenerateRelationshipsCypher()
{
    string filename = @"..\..\output\relationships.cql";
    var output = new StringBuilder();
    int n = 0;
    foreach (var user in _userList)
    {
        foreach (var following in user.following)
        {
            string s = $"MATCH (a), (b) WHERE a.username = '{user.username}' AND b.username = '{following}' CREATE (a)-[:FOLLOWING]->(b); ";
            output.AppendLine(s);
            n++;
        }
    }

    File.WriteAllText(filename, output.ToString());
}

So I ended up with two cql files that looked like this

CREATE (user_1:User { userid: 1 , username: 'user_1', avatar: 'URL' }); 
CREATE (user_2:User { userid: 2 , username: 'user_2', avatar: 'URL' }); 
CREATE (user_3:User { userid: 3 , username: 'user_3', avatar: 'URL' }); 

and

MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_26' CREATE (a)-[:FOLLOWING]->(b); 
MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_53' CREATE (a)-[:FOLLOWING]->(b); 
MATCH (a), (b) WHERE a.username = 'user_2' AND b.username = 'user_6' CREATE (a)-[:FOLLOWING]->(b); 

and I used neo4j-shell to execute the files and import the data. But it was no bed of roses. I’ll list the problems I faced along the way and how I got around them so that this experience might be helpful for other people as well:

Issues along the way and lessons learned

Running multiple statements on Neo4J browser

First I tried to run the create statements using the Neo4J browser which turned out to be problematic because it cannot run multiple statements that end with semi-colons. So I removed the semi-colons but then it started giving me this error

WITH is required between CREATE and MATCH

I found a workaround for that on SO. So the following works:

MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_14' CREATE (a)-[:FOLLOWING]->(b); 
WITH 1 as dummy
MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_22' CREATE (a)-[:FOLLOWING]->(b); 

Now the problem was, if the data was a bit dirty, for instance if user_14 didn’t exist, it stopped executing the rest and no other relationships were created). I had a few nodes like that so this method didn’t work for me after all.

Starting the shell was not as easy as I’d imagined

I installed Neo4J using the default settings and to start the shell just navigated to that directory and ran the batch file. Got an error instead of my shell:

C:\Program Files\Neo4j Community\bin>Neo4jShell.bat

The system cannot find the path specified.

Error: Could not find or load main class org.neo4j.shell.StartClient

Apparently the way to run the shell is using the Neo4J server application and clicking Options -> Command Prompt

Neo4J command prompt

This launches the Neo4J Command Prompt then the rest is easy:

neo4jshell -file nodes.cql

neo4jshell -file relationships.cql

“Exiting with unterminated multi-line input”

Final issue was the statements in a sql file must be terminated with a semi-colon in order for the shell to execute them. Otherwise it gives the above warning and quits

So after all this hassle my data is in the Neo4J database ready to be queried to death!:

Next I’ll investigate converting the data into CSV and achieve the same results by using LOAD CSV command and look into visualization of this data.

Resources