Pergunta de entrevista da empresa Genpact

Explain the ETL (Extract, Transform, Load) process and its importance in data engineering. What are the key differences between batch and streaming data processing? When would you use each approach? How do you ensure data quality and consistency in a data pipeline? What is the purpose of data warehousing, and how does it differ from a data lake? Can you describe the components of a typical data pipeline architecture? What is the role of Apache Hadoop in data engineering, and how does it work? Explain the concept of data partitioning in distributed databases. Why is it important? What is schema-on-read and schema-on-write, and when would you use each approach in data storage?