Stagecoach Software Pty Ltd requires extensions to the operation of two functioning nokogiri scrapers. Currently the scrapers extract certain specified data from a single website on a daily basis and store the data in a MySQL database. The scrapers currently run for several minutes each day on a daily cron job.
The extensions required are:
(1) to the geographical spread of the searches undertaken by the scrapers on the one website;
(2) to the depth of the data, with at least 3 extra columns of data to be populated for each item; and
(3) to obtain an additional type of data available from the same website.
Some other minor sundry changes to the code and database will be required. Full access to the existing code will be provided. The freelancer must upload code changes to GitHub and provide his own development environment ensure that the database and existing scrapers continue to operate without interference throughout the project. The additional data obtained must be automatically parsed into the database and normalised. Some normalisation code is already functioning and may be re-used.
This project follows freelancer Project ID’s 5129765, 5066285, and 4975478. Further projects similar to this one will be offered to the successful bidder if the project is completed successfully. Those projects will be followed by the creation of a RoR website that will present graphs showing aggregate and individual data obtained from the database and allowing user searches.
Preference will be given to freelancers interested in and qualified to carry on work in the further back projects and subsequent building of the website.
Bidders without demonstrable experience in Ruby, MySQL and Web Scraping will be rejected. Bids placed within minutes of the project being posted will also be rejected as indicative of no thought being applied to the project requirements.
Full written instructions will be provided. Bidders are free to revise their bids upon studying the instructions or receiving further information.
Hello, I am professional rails developer and graduated software engineer. I do quick and high quality work. You can check my profile reviews for more details. Thanks
Hi,
I'm new as a freelancer here, but I'm a web scraper expert (please check my reviews: 5 stars with 100% completion rate - all from scraping projects. Also just finished 3 bigger projects and I'm just waiting for the clients to check the results)..
I have computer science degree (2007) + specialization in web engineering (2008) and over 14 years' experience with web development (almost 10 with web scraping)..
If you're interested, please provide me more details..
Thanks for your time
Hi David, thanks for the invite. I've done a LOT of web scraping so I'd be interested to see how your system works. In the past I've looked into Nokogiri but moved away from it since it can't handle AJAX-loaded data as well as other tools can (e.g. a headless browser like Phantom). But it's an awesome tool if you don't need JS interaction, and I'm sure it'll be straight forward to extend. I'm also very experienced building apps in Rails and doing all kinds of data visualization, so I'd be more than capable of taking on your follow-up projects. Let me know if you want to set up a time to Skype; I'm on Central time and my skype username is paul.hiatt. If you can share your Github repo ahead of time it would be helpful for me to review it (I signed the NDA as part of the bid), my handle is hiattp. I look forward to hearing from you! -Paul
David,
Thanks for inviting me to bid on this project. I have signed the NDA and would love to get further details for the bid and estimation.
Do you have a deadline date in mind?
Look forward to working with you.
Pragnesh