Step by step with Step

As a part of a statistics gathering script in Node.js, the final step is to dump the information into a relation database that can be queried in an standard fashion.
In my case I’m using MySQL, where I store the different hits my pixel has recorded. The script first receives the request through an nginx proxy, stores the data onto Redis, and finally a 3rd script reads the information and dumps it to MySQL.
I’ve found that reading data from one source and passing it to another can be a little tricky in Node.js since the single feature that makes it awesome gets in your way: asynchronous calls. Keeping a clear trace of which record you’re fetching, which one recording, which one failed, etc. can become very cumbersome and prone to error.
In comes to help this very nice module called Step. Straight from the README:

“A simple control-flow library for node.JS that makes parallel execution, serial execution, and error handling painless.”

This way I can write good old fashioned blocking code.

The code

The explanation:

  • Fetch data from Redis, establish a temporary working set to avoid collision with write operations.
  • Read each record obtained, build and execute a MySQL query.
  • Clear the temporary working set.
  • Set the next running loop.

The documentation in Step is pretty straight forward, so reading it will give you a clear example of what it does and how to use it. Each funcion defined within the “Step(” portion will be executed sequentially and will receive the result of the previous async. execution.

I haven’t made use of parallel execution yet. If anyone has some real life examples, I’d be happy to check them out.

Advertisements

One thought on “Step by step with Step

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s