Control how deep the crawler will follow links from the starting URL.
Limit the no. of unique URLs to crawl from the website. Max: 10,000.
Only follow URLs that match this pattern. All URLs included if left empty.
Scrape underlying URLs for new content every 7 days
Remove URL documents if corresponding URL no longer exists