Use cases

We'll scope the problem to handle only the following use cases

Service crawls a list of urls:

Generates reverse index of words to pages containing the search terms

Generates titles and snippets for pages

Title and snippets are static, they do not change based on search query

User inputs a search term and sees a list of relevant pages with titles and snippets the crawler generated

Only sketch high level components and interactions for this use case, no need to go into depth

Service has high availability

Out of scope

Search analytics

Personalized search results

Page rank

https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/web_crawler/README.md