Joining files by Pig

In the previous post, we used scala to merge two files. The interesting feature is that scala bypasses Mapreduce. Pig uses Mapreduce. If we undertake the same example, we will see a serious performance difference.

fooOriginal = LOAD '/user/prut/foo' USING PigStorage('|')
AS (id :long, foo:long);
barOriginal = LOAD '/user/prut/bar' USING PigStorage('|')
AS (id :long, bar:long);
joinedValues = JOIN fooOriginal by id, barOriginal by id;
store joinedValues into '/user/pig' USING PigStorage('|');

The end result will be written to a HDFS file that is stored in the directory /user/pig.
Of course, the results are the same; on the other hand, we see a serious performance difference. Reason being that under water, mapreduce is used.