Often called spidering.

Automatic browsing of World Wide Web.

Breadth-First crawling

Prioritize explore a website width before going deep.

Start by crawling all links on the seed page, then move to the links on those pages.

Get a broad overview of a website’s structure and content.

Depth-First Crawling

Prioritize depth over breadth.

Usefull for finding specific content or reaching deep into a website’s structure.

robots.txt