Tuesday, October 28, 2014

Open Source + Me

Hi guys,

it's been a while - but as you see I've kept my 'letter-style' of writing. Just kidding.

Recently I've came out of my "comfort" zone at work, therefore I've made a big shift from the Microsoft/.NET/C# to open source - Node/Angular/Mongo DB technologies. Adopting to this new for me world was quite challenging and knowledge rewarding. Although not feeling very confident, I wanted to start contributing somehow to the Open Source community. Therefore, I've decided to extract 2 very simple and generic projects from my personal stuff.

Minimalistic: Node JS + Angular JS + Require JS

For the past months, I've been working on a great project, involving the so-called MEAN (Mongo DB + Express JS + Angular JS + Node JS) stack. Bootstrapping the application in the beginning seemed like the hardest part and involved a steep learning curve. What I needed and wanted:

  • Web Application with Angular JS
  • With Require JS to wire up the external dependencies
  • With a minimalistic Node JS server
  • All this as simple as possible, without any overhead
  • Run in 2-3 commands from the command line

What I found out there were many seed projects, with HTML 5 boilerplates, styles, heavy Node JS server implementations and etc. So, I've setup a minimalistic seed project, which has all the points above covered - nothing more, nothing less.

The project is located at GitHub here: https://github.com/yyosifov/angular-require-seed and is truly minimalistic and simple. 

Yet, this is an Angular JS web app wired up with Require JS and served by Node. I'll definitely use it whenever I need to setup an Angular JS web app. What can be added - Grunt/Gulp make files, sample tests with Karma and a little heavier Angular JS app with a Service, Module and one more route.

Arduino: Temperature Measurer

Another minimalistic project - a sample Temperature Measurer class, implemented in Node JS. What it does is measure a Temperature retrieving the data from an Arduino device and emit event with the gathered data. It can be easily plugged in any home/office automation project, as long as you've already have the required hardware (Arduino + Temp sensor). The sample application + temperature measurer is again hosted on GitHub: https://github.com/yyosifov/temperature-measurer. As simple as it gets, I've been using this for a while and haven't noticed any side effects, yet.


The open source world reveals a whole new range of opportunities. It's awesome that you can learn and get inspired from the code of so many smart guys. Now I'll just have to find out how some Open Source have become viral and popular, whereas others have never been downloaded/forked...

In a bottom line, what came to my mind is the following wordplay:

Some things never change and some things always change (like technology and trends).

Tuesday, April 8, 2014

Upstart - Syntax error: Bad fd number upstart

Debugging upstart scripts is hard and cumbersome. I've been writing some very simple and straight forward upstart configuration files. The problems began when I tried to start my service using one of them.

Undescriptive error messages

I've experienced two problems I didn't find anything on the internet about:
  • Errors in the service standard error output: : not foundfd/9: 3: /proc/self/fd/9:
  • Errors in /var/log/upstart/{service-name}/logSyntax error: Bad fd number upstart
What happened to be the problem wasn't the actual configuration files, but their formatting. I've been preparing the .conf files on Windows and have uploaded them on some Linux machines. What I haven't thought of was these hidden symbols, which are not visible by default - the line endings. 

Line endings

Windows uses '\r\n' or CRLF at the end of each line and Linux uses only '\n' or LF (Newline explained in Wikipedia). So,  my configuraton files were all with CRLF at the end of each line, which wasn't accepted by the Linux and Upstart respectively. Upstart failed to start my services and didn't give any descriptive errors. Changing CRLF to LF in the .conf file fixed these issues.

TIP: You can use vim -b filename to open a file name in binary mode, which shows all special characters in the file.

I hope this saves some hours someone out there.

Friday, January 24, 2014

Windows Powershell - Extract Json property values

Hi guys,

I've had to extract json property values, from a CSV document. Sounds bizarre, right? We've got the CSV file exported from a database, where a JSON was stored in one of the columns. As the JSON is not any close to native to relational database, we couldn't query the JSON column to extract data from it.

The problem

Can be restated as "How can you extract JSON property values from any text file" or even generalized "How to extract anything from a large file". Our requirement would be 'do this fast and easy in Windows environment and be able to process any text file'.

Sample data

We'll be using this short and simple json file for our example:

        "firstName": "A",
        "clientId": 2
        "firstName": "B",
        "clientId": 14
        "firstName": "C",
        "clientId": 15
        "firstName": "D",
        "clientId": 3


Let's decompose the problem into few simple steps:

  • Searching through text is a complex operation. The most common approach would be to use Regular Expressions as they provide flexibility and support in practically any programming language.
    • The json language complies to several simple rules, so extracting from it would be easy. A regex for extracting the client id is: '"clientId": (?(\d+))'. It matches the "clientId" property name, then a colon :, then a new  named group starts, which expects a number (\d+).
  • Instant coding and execution leads us to some scripting language. The scripting language does not need compilation, creating new projects or anything alike which could slow us down.
    • We'll be using Powershell to execute the regular expression, as it's a common language for executing such activities on Windows.


# set our client id regex, which can extract client id fields from the text
$clientIdRegex = '"clientId": (?(\d+))'

# search through the file with "select-string", providing our regex as a pattern and find all matches
# pipe the matches to a foreach, which traverses them and then pipe each match to a foreach which extracts
# the group clientid and get's its value.
select-string -Path .\clients.js -Pattern $clientIdRegex -AllMatches | ForEach-Object {$_.Matches } | ForEach-Object { $_.groups["clientid"].Value }

And that's the whole script - just two lines!

# If we need to output the values in a file, we need just to pipe the output into a file like this: | ForEach-Object { $_.groups["clientid"].Value } > output.txt

Note: In most powershell scripts, you would see the usage of the alias "%" instead of ForEach-Object. Here it's used to illustrate the idea.

In conclusion

Knowing script languages is powerful tool in our utility belt. They can facitilate actions which should be done only once and there's no need for a maintainable and robust solution. Plus, they are easily edited and executed.