Towards a Statistical Evaluation of PigLatin Joins

Abstract

There are different scalable data management solutions which can take advantage of cloud features making them more attractive for a deployment in such environments. One of the most critical operations in data processing is joining large data sets. This is one of the most expensive and hardest operations to optimize. We are mainly  concerned here with join operations expressed in PigLatin, an abstract query language for a high-level platform, called Pig, which creates MapReduce programs with Hadoop. In this work we explore statistical methods, namely multiple regression analysis, in order to predict three distinct join types execution times, comparing them with actual running times.