Scrapy Jobs

I need a script that visits the California State License Board site, drills down into every contractor record, and captures every publicly visible field—license number, classifications, bonding, personnel, status history, contact info, everything. The scraper must handle pagination, search-form quirks, and any soft-rate limiting the site imposes so the run finishes without manual stops or captchas blocking the flow. Once gathered, the data should be written to a single CSV file, overwriting the previous file on each run so I always have a fresh snapshot. The entire process has to trigger automatically every 24 hours (cron, systemd timer, Cloud Scheduler—whatever you prefer) and run headless on a Linux VPS I’ll provide. I am fine with Python (requests, BeautifulSoup, Sc...

I have a list of niche websites and need every relevant piece of text and its accompanying images extracted, organised, and delivered in a clean, ready-to-use format. I’ll share the URLs and the exact data points once we start, but expect a mix of article-style pages and media galleries. Scope • Capture both written content (headings, paragraphs, metadata) and all on-page images. • Provide the text in CSV or JSON and store images in clearly named folders that map back to the records. • Preserve basic structure—so each text record includes the image file name or path. • Respect and rate limits; the scrape must be discreet and repeatable. What I’d like to see in your proposal Please outline your end-to-end approach: preferred language or f...

AI Enabled Data Scraping Engineer – Junior / Mid Level Experience: 1 to 4 Years Location: Remote (Work from Home) / Bangalore / India Mode of Engagement: Full-time No of Positions: 3 Educational Qualification: B.E / / MCA / Computer Science / IT Industry: AI / Data Engineering / Automation / SaaS Notice Period: Immediate / 15 Days Preferred What We Are Looking For: 1–4 years of experience in Python-based web scraping, browser automation, and data extraction projects. Hands-on experience with Scrapy, Selenium, Playwright, Requests, BeautifulSoup, or similar scraping frameworks. Basic to intermediate understanding of AI/LLM-powered automation workflows using ChatGPT, OpenAI APIs, Claude, Gemini, or LangChain. Experience handling dynamic websites, login sessions, cookies, browse...

I need a straightforward Python script that reliably pulls text content from websites I will specify once the project starts. The focus is strictly on text—no images or media files—so the routine should locate, extract, and save headings, paragraphs, or other copy in a clean, structured format such as CSV or JSON. Because the target sites may vary, the code must be written with easy-to-tweak selectors and robust error handling. Please use widely supported libraries (requests, BeautifulSoup, or Scrapy if you prefer) and include polite scraping practices such as user-agent rotation, rate limiting, and graceful retries on common HTTP errors. Deliverables: • A well-commented .py file ready to run from the command line • A brief README explaining any required packages...

I need a reliable script that can harvest contact details from a set of company websites I will provide. The information I’m after is any publicly listed email address, phone number and physical address that appears on those sites. Please build the scraper in a way that I can rerun it easily—Python with BeautifulSoup, Scrapy or Selenium is perfect as long as the code is well-commented and the output is clean. I prefer a single CSV (or JSON if you feel it suits the data better) with one row per company and distinct columns for each field you capture. Accuracy and completeness matter more than sheer speed, so let me know how long you’ll need once you see the list of domains. If you have questions about structure or formatting, ask before you dive in.

I have roughly ten thousand ISBN-13 codes and I need a production-ready Python pipeline that can take those codes, pull the corresponding book details from , , and a small set of external APIs, then push the cleaned results straight into a Google Sheets workbook. The pipeline must • survive Amazon’s throttling, bot checks, and page format changes without manual babysitting, • finish a full run on 10 k titles in a single session without crashing or silently skipping rows, and • give me fields that are already matched and normalised so downstream staff can link them to our catalogue instantly. Architecture is up to you: Scrapy, Playwright, headless Chrome, rotating residential proxies, Selenium, or a custom HTTP solution—whichever mix keeps the request foot...

I have roughly ten thousand ISBN-13 codes and I need a production-ready Python pipeline that can take those codes, pull the corresponding book details from , , and a small set of external APIs, then push the cleaned results straight into a Google Sheets workbook. The pipeline must • survive Amazon’s throttling, bot checks, and page format changes without manual babysitting, • finish a full run on 10 k titles in a single session without crashing or silently skipping rows, and • give me fields that are already matched and normalised so downstream staff can link them to our catalogue instantly. Architecture is up to you: Scrapy, Playwright, headless Chrome, rotating residential proxies, Selenium, or a custom HTTP solution—whichever mix keeps the request foot...