build scraper for [login to view URL]

This project will require you to use Perl/mechanize to scrape [url removed, login to view], which is an ASP.NET site.

Deliverable is a perl script.

You may use outside libraries like mechanize and your favorite html parser. I use mechanize but you can also use selenium if that works too (though I'd prefer mechanize, which would require you to sift through the javascript functions)

The script should basically continue what I've begun below. It should be able to plug in cities and dates from the home page, submit that form, then be able to extract the dates/times/prices. ( I will add code to insert that data into a database)

-----

Here's what I have so far... it doesn't work at all. I think the big thing is knowing the form elements, but there is also what appears to be a session id or something that changes each time you load the page.

#!/usr/bin/perl

use WWW::Mechanize;

use HTTP::Cookies;

use LWP::Simple;

my $url = '[url removed, login to view]';

$outpath = 'C:\Documents and Settings\My Documents';

my $mech = WWW::Mechanize->new();

$mech->cookie_jar(HTTP::Cookies->new());

$mech->get($url);

$mech->submit_form(

form_name => 'ctl01',

fields => {

'SearchAndBuy1_txtPassengers' => 1,

'SearchAndBuy1_ddlLeavingFrom' => 37,

'SearchAndBuy1_ddlTravellingTo' => 41,

'SearchAndBuy1_txtOutboundDate' => 4305

}

);

sleep 1;

$html = $mech->content();

$out = ">" . $outpath . '\\' . '[url removed, login to view]';

open(OUT, $out);

print OUT $html;

close(OUT);

print "done";

ASP Perl Extracción de datos web

Nº del proyecto: #1000549

Sobre el proyecto