Entry

How can I download a webpage and store it in a variable?

May 10th, 2002 17:10
John B, Per M Knutsen,

Use the LWP::Simple module from CPAN.
# Example
use LWP::Simple;
my $URL     = "http://domain.tld/somepage.html";
my $webpage = get($URL) || die "Cannot connect to $URL: $!\n";
print $webpage;
####################################################
That code does not always work.  I have found the following to be more 
reliable:
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Headers;
$header = HTTP::Headers->new('content-type' => 'text/html');
# If you are requesting a user/only page, fill out and uncomment 
# the following line 
#$header->authorization_basic("username", "password");
# If you need to pass data to the page you are requesting (ie, script),
# uncomment out the next line
#$content = "var1=blah&var2=alksjd";
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new
(POST,"http://domain.tld/page.pl",$header,$content);
$resp = $ua->request($req);
$lines = $resp->as_string();
# the contents of the page are now in the $lines variable