Pergunta de entrevista da empresa TrialPay

How would you design an incoming reference based aggregation system for TrialPay back end server so that info about the incoming traffic can be stored and used to improve the system? Duplicate files in a directory. Files are very large, so can't read the whole file into the memory.