Build a information gathering tool in php or similar

Cerrado Publicado Apr 25, 2015 Pagado a la entrega
Cerrado Pagado a la entrega

I have a MySQL database with a close to 5000 URL that I need tested, I need a PHP (or similar, please state in your bid what kind of development language you will be using) tool that can access all those urls and fetch a lot of information for me and save it to a new database.

I have attatched the git repository for wappalyzer, can be found here [url removed, login to view]

I have also attatched some tools I got developed here earlier, but the coder is not a member here anymore, not sure how those codes work, but it was supposed to check a lot of urls for a http response code.. If it is just as easy to build from skratch then please dont mind that file. :)

So to sum it up, I wish a tool that I can operate from a webpage (hidden behind a login form of course) to feed it a list of url from a mysql database, and it then visits all those urls and saves ALL the information that wappalyzer finds on a page.

In addition to the information wappalyzer finds I also wish it to save the http status code and the ip address of the page, and the following html tags: title tag, the keywords, the description tag.

And if it is not hard to do, I would also wish to save the email address if it finds a @ on the front page.

If it gets a time out or something, I wish that that entry gets put to the end of the queue tried again a second time, before it is put on a manual list for review so that we dont get a never ending loop.

Please state in your bid how many days you will use to solve this, and what development languages you will be using.

I want the code to be commented, so that I can learn from it (am starting to learn php and java and html myself, but it is a long way to go, so I want to have comments so I can understand it in the future.)

I want this installed at my webpage, I have whm, cpanel and ftp access to it.

HTML MySQL PHP Python Ruby on Rails

Nº del proyecto: #7553273

Sobre el proyecto

13 propuestas Proyecto remoto Activo Jun 1, 2015

13 freelancers están ofertando un promedio de $346 por este trabajo

jfreak53

Good day and thank you for reading my proposal. I would be creating this system for you in PHP with CURL. However, as you mentioned, it would be 1 million times easier and faster to start from scratch than to use the a Más

$444 USD en 7 días
(55 comentarios)
6.8
programerpk

A proposal has not yet been provided

$333 USD en 5 días
(4 comentarios)
4.1
Taiwaninja

Sounds like a very doable project, I'd be happy to do it in Python. Either python 2 or 3, Looking forward to talking with you on PM Thanks in advance.

$200 USD en 3 días
(12 comentarios)
3.8
eitan1195

A proposal has not yet been provided

$444 USD en 2 días
(11 comentarios)
3.8
wordpressbaker

A proposal has not yet been provided

$333 USD en 15 días
(21 comentarios)
3.4
hemantmahalle

Hi sir have you mentioned

$311 USD en 20 días
(2 comentarios)
2.6
RafaAguilar1987

Hello Buddy, I can do this, i plan to use Python, a very nice language to learn, i can do it on Java instead (if you want) but it will take some days longer, this is the general plan: 1. Install Wappalyzer and le Más

$333 USD en 9 días
(3 comentarios)
2.4
zkutch

Hello. More 20 years programming experience. I need more details to set real time and price - for example: do you accept perl? Regards. ------------------------------------------------------------------------------ Más

$305 USD en 3 días
(9 comentarios)
3.8
milanmartinjuk

Hi, Your project sounds very interesting and simple enough and I believe that I'll be able to perform the task perfectly. As for my experience, you can see for yourself that I've worked here mainly as a content a Más

$200 USD en 3 días
(3 comentarios)
2.8
projexan

Please find below my short experience summary specific to this project. * Several years experience developing Text Mining and Information Extraction for web crawling, scraping, extraction and aggregation from unstru Más

$340 USD en 7 días
(0 comentarios)
0.0