Find Jobs
Hire Freelancers

Django/web2py price comparison website with scrapy scrapper

$250-750 USD

Cerrado
Publicado hace alrededor de 11 años

$250-750 USD

Pagado a la entrega
Hello I need someone to develop a website in Django/web2py for a price comparison site using Scrapy (or something better) & Selenium - code must be documented in English. It should allow Scrapy/alternative to crawl a "variable" number of separate sites (using a number of "spiders") that can pull out product details such as Product ID, Title, Price, Vendor, Description, Image, URL and Stock Position etc. This information should then be placed in a PostgrSQL database to be displayed using Web2py/Django. There should also be a way of the URL to the products be changed to affiliate links. This is an easy project for someone who has done this before, if you have examples of previous work this will go in your favour so please reference them. Additionally if you have advice on the a better architecture/solution I am open to ideas. Expected Features: a) The Products Table in the server database to be automatically populated by the scraper. The required fields are Product ID, Title, Price, Vendor, Stock Position, Payment Options, Delivery Time b) Easy extensibility (with some python coding) to add more sites in future. c) To meet the above, the scraper to be implemented as two modules. The "Scraper Module" and the "Parameter Module". d) The "Scraper Module" would do the actual scraping of multiple sites (based on parameters read from the Parameters Module), and also automatically populate the Products Table in the database server. For sites with content rendered in JavaScript, Scrapy to be used with Selenium for effective scraping. e) The "Parameters Module" would include a Form through which scrape parameters such as the primary URL, scraping rules for each field to be scraped, format of data to be extracted, and whether to use simple crawl (for sites without JavaScript) or complex crawl (for sites with content rendered in JavaScript). These parameters would be stored in a table, and accessed by the "Scraper Module" at run time. f) The scraped URLs (referred by the primary URL) to be saved in a Database Table with "processed flag", so that these can be skipped if scraping needs to be resumed after interruption. g) Primary URLs also to be saved with the date of last successful scraping, to enable scheduling of periodic repeat scrapings. h) While executing scraping, only those fields that have changed since last scrape are to be extracted and the original table entry for the product to be "updated", as required. In case of new products, the details to be "inserted" as a new row in the Products Table. i) Scrapy to be used with Selenium for effective scraping of sites with heavy JavaScript content. j) Performance must be adequate to enable scraping of the sites in order to generate the Products database k) There should also be a way of the URL to the comparison products within the website to be updated changed to affiliate links. Expected Skills: Web Scraping, Scrapy, Selenium, Python, Data Mining, Javascript, MySQL Budget: USD 200 to USD 300
ID del proyecto: 4437217

Información sobre el proyecto

7 propuestas
Proyecto remoto
Activo hace 11 años

¿Buscas ganar dinero?

Beneficios de presentar ofertas en Freelancer

Fija tu plazo y presupuesto
Cobra por tu trabajo
Describe tu propuesta
Es gratis registrarse y presentar ofertas en los trabajos
7 freelancers están ofertando un promedio de $489 USD por este trabajo
Avatar del usuario
I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.
$450 USD en 10 días
4,7 (86 comentarios)
7,0
7,0
Avatar del usuario
I can deliver the project
$630 USD en 12 días
5,0 (47 comentarios)
6,5
6,5
Avatar del usuario
i can do it
$300 USD en 30 días
5,0 (46 comentarios)
5,0
5,0
Avatar del usuario
Hello, I can do this work for you and I'm ready to start. Please see pmb for more details. Regards Raul
$250 USD en 10 días
4,9 (13 comentarios)
4,8
4,8
Avatar del usuario
Hi, Veltrod Software services is a global software consulting company specialized in providing Mobile applications, Social media frameworks and eCommerce solutions. Leveraging best-in-class people, processes, and technologies, Veltrod provides high-quality software development and BPO services to independent software vendors and enterprises. We are specialized in developing Mobile applications, Search Engine optimizations, Social Network frameworks, Software Testing and ERP applications. We are pleased to take care of this project, if it is offered to us. We assure that we will work towards 100% client satisfaction. Thanks Vinoth
$721 USD en 25 días
5,0 (1 comentario)
0,0
0,0
Avatar del usuario
Please check our PM
$525 USD en 3 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
Um a Python Developer and have a sound knowledge on Django , Google App Engine. I can deliver the project in a good coding standard and a quality product
$550 USD en 12 días
0,0 (0 comentarios)
0,0
0,0

Sobre este cliente

Bandera de UNITED KINGDOM
Croydon, United Kingdom
0,0
0
Miembro desde abr 6, 2012

Verificación del cliente

¡Gracias! Te hemos enviado un enlace para reclamar tu crédito gratuito.
Algo salió mal al enviar tu correo electrónico. Por favor, intenta de nuevo.
Usuarios registrados Total de empleos publicados
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Cargando visualización previa
Permiso concedido para Geolocalización.
Tu sesión de acceso ha expirado y has sido desconectado. Por favor, inica sesión nuevamente.