aws s3, powershell

Cloud computing is a relatively new concept, especially when compared to FTP which dates back to 70s (History of FTP server). So not every device supports S3 uploads. If you cannot force a device to upload directly to S3 and have control over the FTP server machine (and assuming it’s running Windows) you can create a simple PowerShell script to upload files to S3.

FTP to S3

First you need to install AWS Tools for Windows. I tested on a couple of machines and the results were dramatically different. My main development machine is running Windows 8.1 and has PowerShell v4 on it. I had no issues with using AWS commandlets in this environment. The VM I tested has PS v2 on it and I had to make some changes first.

v2 vs. v4

The problem is AWS module is not loaded and you have to do it yourself with this command

import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"

After this command you can use AWS commandlets but when you close and open another shell it will be completely oblivious and will deny knowing what you’re talking about! To automate this process you need to add it to your profile. AWS documentation tells you to edit your profile straight away but the profile did not exist in my case. So first check if the profile exists:

Test-Path $profile

If you get “False” like I did then you need to create a new profile first.

To create the profile run the following command:

New-Item -path $profile -type file –force

then you can edit the profile by simply running

notepad $profile

Add the import-module command above and save the file. From now on every time you run PowerShell it will be ready to run AWS commandlets.

Time to upload

Now that we got over with troubleshooting we can finally upload our files. The commandlet we need is called Write-S3Object. The parameters it requires are the target bucket name, source file, target path, and the credentials.

Write-S3Object -BucketName my-bucket -File file.txt -Key subfolder/remote-file.txt -CannedACLName Private -AccessKey accesskey -SecretKey secretKey

Most likely you would like to upload a bunch of files under a folder. In order to accomplish that you can create a simple PowerShell script like this one:

$results = Get-ChildItem .\path\to\files -Recurse -Include "*.pdf" 
foreach ($path in $results) {
	Write-Host $path
	$filename = [System.IO.Path]::GetFileName($path)
	Write-S3Object -BucketName my-bucket -File $path -Key subfolder/$filename -CannedACLName Private -AccessKey accessKey -SecretKey secretKey
}

Resources

misc jekyll

As you probably know I’ve migrated to GitHub Pages from WordPress as I blogged here.

It was a fairly easy migration but migrating the actual content proved to be trickier. There are lots of resources on using Jekyll’s importers. I found this one useful. Just export everything to an XML and run the converter to get the posts in markdown. The problem is the YAML Front Matter it generates is a bit messy:

---
layout: post
title: Blind Password Masking
date: 2011-06-14 03:46:18.000000000 +00:00
categories:
- Off the Top of My Head
tags: []
status: publish
type: post
published: true
meta:
  _edit_last: '1'
author:
  login: blogadmin
  email: admin@myvirtualhome.net
  display_name: Volkan
  first_name: ''
  last_name: ''
---

I don’t want or need most of this stuff anyway!. Also I had two main issues:

  • Images didn’t work as it didn’t get the full path. I use S3 to host all images but the imported posts were converted to use a local assets folder. There may be a configuration setting for that but in my case I decided to convert all my posts to markdown from HTML anyway (which was a great way to practice Markdown)
  • Main issue was with Disqus. It’s not like people are racing to submit comments to my ramblings but still I’d like to have Disqus enabled on all my posts. Apparently to enable comments you need to specify it in the front matter like this:
comments: true

Manual vs. programmatical

First I resisted the temptation to write a small application to convert the layouts but manual conversion soon proved to be very time consuming even with 100+ posts. So I developed the simple console application below. It scans the folder you specify (filters *.markdown files) and reads the existing layout and converts it to the format I wanted:

---
layout: post
title: @TITLE
date: @DATE
categories: [@CATEGORIES]
comments: true

I wanted to keep it simple and clean. Also as all my posts are now in pure markdown I can easily loop through and update the elements (like converting H3 to H2 or adding tags to layout etc)

Source Code

Usage

It probably won’t apply to most situations but helped me out so why not publish it just in case, right? To use it you have to change the ROOT_FOLDER value in Program class at the bottom (Do NOT forget to backup your posts first!)

As I wanted to revisit all my posts I wanted to mark them instead of replacing automatically with the original one. So when you run the program it deletes the original post and creates the updated one with “.output” appended. So you can easily find which files are modified by checking the extension. If you want it to replace the original post you can uncomment this line at the end of the ConvertFile method

 // File.Move(outputFilePath, inputFilePath);

Resources

misc events

On Wednesday I attended this meetup. It was fairly informative. There is a bit uncertainty about the new features of Windows 10, Visual Studio 2015 and C# 6.0 but it was still good to cover most of them in a few hours.

It was divided into two main sections: Windows 10 and VS 2015/C# 6.0. I’m planning to review these myself and blog about them separately but here are the highlights of the events:

Windows 10

When they first announced the name I immediately thought about Winamp skipping version 4. We all know how that story ended so I hope Windows 10 fares better than that!

A few items discussed in the event about Windows 10:

  • First demonstration was sending a feedback which was painfully slow and it took forever to send a simple feedback message.
  • Some enterprise features are coming like like having store and custom company portals
  • There was a rather long discussion about what the 4 device from the left was in the image below: One Windows

I don’t think it makes any difference though but the idea is having one code base and one store to deploy apps to various devices with different form factors. Sounds cool but I’m a bit cautious about it for the time being. If it sounds to good to be true, it generally isn’t!

  • Looks like they are going to use hamburger icons everywhere which they initially opposed to.
  • There seems to be back buttons in a few screens which is a bit unusual so they may have to deal with some lash back from people like they did for their brilliant(!) charms invention.

Visual Studio 2015

Preparing for Windows 10 Event

  • There will be a free version of VS 2015 and there is already a free version for VS 2013 called “Community Edition”. It’s said to be Professional equivalent so it sounds cool to have it for free.
  • Like all Microsoft products versioning is a bit off and confusing here as well. There was a discussion about the version number vs. release year. So VS 2013 actually is version 12 but VS 2015 is version 14 even though there is no other VS in the middle!
  • A research project called Pex used to generate unit tests made its way to VS 2015 under Smart Unit Tests. It should help to create a lot boilerplate test code.

C# 6.0

  • Not a C# feature but one of the most impressive developments nowadays is that .NET Framework has become open source.
  • New version is coming with a much faster 64-bit compiler called .net RyuJIT
  • .net Native Project N is coming which is supposedly make the applications run faster
  • The compiler has been completely rewritten and its name is now Roslyn. It’s open sources and it exposes APIs that can be consumed by any application. So the compiler doesn’t have be a black box anymore.

Conclusion

A lot of exciting and revolutionary developments are going on in the .NET world these days. Taking a more open-source and multi-platform based approach will definitely help the platform in the future. It’s thrilling to be a developer and experience them first hand as these news come along. I have my virtual machine running Windows 10 and VS 2015 so I’ll play around more and blog about these specific features in detail in the near future.

Resources