As a part of a statistics gathering script in Node.js, the final step is to dump the information into a relation database that can be queried in an standard fashion.
In my case I’m using MySQL, where I store the different hits my pixel has recorded. The script first receives the request through an nginx proxy, stores the data onto Redis, and finally a 3rd script reads the information and dumps it to MySQL.
I’ve found that reading data from one source and passing it to another can be a little tricky in Node.js since the single feature that makes it awesome gets in your way: asynchronous calls. Keeping a clear trace of which record you’re fetching, which one recording, which one failed, etc. can become very cumbersome and prone to error.
In comes to help this very nice module called Step. Straight from the README:
“A simple control-flow library for node.JS that makes parallel execution, serial execution, and error handling painless.”
This way I can write good old fashioned blocking code.
- Fetch data from Redis, establish a temporary working set to avoid collision with write operations.
- Read each record obtained, build and execute a MySQL query.
- Clear the temporary working set.
- Set the next running loop.
The documentation in Step is pretty straight forward, so reading it will give you a clear example of what it does and how to use it. Each funcion defined within the “Step(” portion will be executed sequentially and will receive the result of the previous async. execution.
I haven’t made use of parallel execution yet. If anyone has some real life examples, I’d be happy to check them out.