When you start compiling true big data sets, having redundant records is a massive headache. Read on to learn how to avoid redundancy