137271 Building spider for crawling

Build a spider that crawls the internet looking for pages based on specific keyword terms that are given to the spider. We might also want to supply a list of specific websites to crawl. Use regular expressions to determine if the site we've found is consistent with the category of item that we're crawling. Crawl the rest of the site to find additional items that might also match our crawling goals. Use regular expressions to pull specific elements of the page into a table. Make sure to abide by [login to view URL] & meta-tag robots rules.