Convert ruby cron job scrapers to run as background jobs in a queue

Hello, Since last 9+ years I've been working as Full Stack Web Developer and from last 6 years as a Ruby on Rails(ROR) Developer. Ruby & ROR is my passion, my skill but apart from it, I have a good expertise in other technologies like Java(1.5+), Databases(Oracle, MySQL, PostgreSQL, Mongodb), Search engine(Solr, Thinking Sphinx, Elastic Search), JS frameworks(Angular, Bootstrap, jQuery etc.), SEO, REST API, Social media integration, Web Scrapping, Web Services, Amazon S3, AWS setup, Web servers(Apache, passenger), HTML5, CSS3, Redis, Rspec, CoffeeScript and many others. I think you’ll find I have the skills you’re looking for. There are many gems for Background job in Ruby language. like Delayed Job, Resque worker, Sidekiq etc. I have used all these in my projects. I would like to suggest you to use Sidekiq for background its easy to integrate with app and its performance is better than other. You can use Redis with Sidekiq if job require to store temp data for process. Thanks KrunaL

Hello. I hope scrapers written in Ruby? I plan use Sidekiq (together Redis) for to achieve goals. Sidekiq is a mature solution that we used a lot of time in their projects.

Hi, thanks for invite me to this project. I'm placing this bid but I'm not 100% sure on what do you need to be done. I'm assuming that you already have these scrappers on ruby and that they're already working. In my proposal, I will convert these 22 scrappers to be triggered by delayed_jobs (or sidekiq), install and configure delayed_jobs, and run tests on your server to make sure that's all working as expected. I've experience with scrappers, rails and linux servers - I have myself a rails classified aggregator website that runs more than 20 scrappers daily. Also, I provide a 2 months "insurance" against bugs on my works. If you find some issue on it in this meantime, I'll fix it for free.

My extensive experience with Rails include building end to end RoR backed websites, offer community solutions to critical problems, provide framework level security fixes for Ruby on Rails. It is Rails that helped me to bring up websites in a very competitive timelines. I'm planning to use ActiveJob along with Backburner, as it has Job level priority & timeout configuration. Amazon AWS offers few solutions like SWF (Simple Workflow Service) which guarantees the job execution, if we can leverage.