Web Address Finder

We need a program that will scan a number of directories (i.e. google, dmoz etc) and pull out all the domain names in each section and put them into a database - we do not need the full domain listings, just the actually [[login to view URL]][1] part. The program will have to dedupe whilst running and not put duplicate domains into the database either. The program will have to run pretty much independently, scanning each section and subsection then moving onto the next one. We would need to be able to leave the program running for a period of time to collect maximum domains. The idea is that over a small period of time we will have a list of a few hundred thousand domain names that are listed and active in a directory.... for the next stage of our project!! Please ask any questions Thanks ## Deliverables 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables): a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment. b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request. 3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement). ## Platform windows