I went through some pieces of gathering the list and storing it. Having a number of agents to scrape the URLs, using some pub/sub to decouple the response and another worker to process them. Also talked about spreading the list to region workers and how to not parse the same url twice