The file looked somthing like this:
I wanted to produce a file of the form
Ignoring the movies that don't include both genres.
I inmediately thought of using hadoop, even if the file was not huge, the map reduce algorithm seems a good fit for the problem.
I have done some small work in hadoop with Java, but in this case my project was Ruby based and I wanted to keep on using Ruby even for my hadoop job, so I used the streaming API of hadoop to solve the problem.
I needed to develop a fast and easy solution. What follows is the code:
Consulting some bibliography and the Web I came to a very easy solution.
map.rb reduce.rb hadoop_run.sh
I consider the main differences between the streaming API and the Java API are:
1. The streaming API works everything in the stdin and stdout between the scripts, (take a look at the ARGF and puts use in both map.rb and reduce.rb) like when we use the pipe in the command line between commands
2. In the Java API the results from the mapper phase are grouped together for example in our case we would actually receive directly on the reduce phase the line (movie-a [Romance,Comedy]). In the streaming API on the other hand, the grouping needs to be done manually, what we get is an ordered list by key (so all the movie-a would be next to each other).
So there we have a small and functional map reduce job in Ruby with Hadoop.