libwww::lwptut (3)
Leading comments
Automatically generated by Pod::Man 4.09 (Pod::Simple 3.35) Standard preamble: ========================================================================
NAME
lwptut -- An LWP Tutorial
DESCRIPTION
LWP (short for "Library for WWW in Perl") is a very popular group of Perl modules for accessing data on the Web. Like most Perl module-distributions, each of LWP's component modules comes with documentation that is a complete reference to its interface. However, there are so many modules in LWP that it's hard to know where to start looking for information on how to do even the simplest most common things.
Really introducing you to using LWP would require a whole book -- a book that just happens to exist, called Perl & LWP. But this article should give you a taste of how you can go about some common tasks with LWP.
Getting documents with LWP::Simple
If you just want to get what's at a particular URL, the simplest way to do it is LWP::Simple's functions.
In a Perl program, you can call its "get($url)" function. It will try getting that URL's content. If it works, then it'll return the content; but if there's some error, it'll return undef.
my $url = 'http://www.npr.org/programs/fa/?todayDate=current';
# Just an example: the URL for the most recent /Fresh Air/ show
use LWP::Simple;
my $content = get $url;
die "Couldn't get $url" unless defined $content;
# Then go do things with $content, like this:
if($content =~ m/jazz/i) {
print "They're talking about jazz today on Fresh Air!\n";
}
else {
print "Fresh Air is apparently jazzless today.\n";
}
The handiest variant on "get" is "getprint", which is useful in Perl one-liners. If it can get the page whose URL you provide, it sends it to STDOUT; otherwise it complains to STDERR.
% perl -MLWP::Simple -e "getprint 'http://www.cpan.org/RECENT'"
That is the URL of a plain text file that lists new files in CPAN in the past two weeks. You can easily make it part of a tidy little shell command, like this one that mails you the list of new "Acme::" modules:
% perl -MLWP::Simple -e "getprint 'http://www.cpan.org/RECENT'" \
| grep "/by-module/Acme" | mail -s "New Acme modules! Joy!" $USER
There are other useful functions in LWP::Simple, including one function for running a HEAD request on a URL (useful for checking links, or getting the last-revised time of a URL), and two functions for saving/mirroring a URL to a local file. See the LWP::Simple documentation for the full details, or chapter 2 of Perl & LWP for more examples.
The Basics of the LWP Class Model
LWP::Simple's functions are handy for simple cases, but its functions don't support cookies or authorization, don't support setting header lines in the HTTP request, generally don't support reading header lines in the HTTP response (notably the full HTTP error message, in case of an error). To get at all those features, you'll have to use the full LWP class model.
While LWP consists of dozens of classes, the main two that you have to understand are LWP::UserAgent and HTTP::Response. LWP::UserAgent is a class for "virtual browsers" which you use for performing requests, and HTTP::Response is a class for the responses (or error messages) that you get back from those requests.
The basic idiom is "$response = $browser->get($url)", or more fully illustrated:
# Early in your program:
use LWP 5.64; # Loads all important LWP classes, and makes
# sure your version is reasonably recent.
my $browser = LWP::UserAgent->new;
...
# Then later, whenever you need to make a get request:
my $url = 'http://www.npr.org/programs/fa/?todayDate=current';
my $response = $browser->get( $url );
die "Can't get $url -- ", $response->status_line
unless $response->is_success;
die "Hey, I was expecting HTML, not ", $response->content_type
unless $response->content_type eq 'text/html';
# or whatever content-type you're equipped to deal with
# Otherwise, process the content somehow:
if($response->decoded_content =~ m/jazz/i) {
print "They're talking about jazz today on Fresh Air!\n";
}
else {
print "Fresh Air is apparently jazzless today.\n";
}
There are two objects involved: $browser, which holds an object of class LWP::UserAgent, and then the $response object, which is of class HTTP::Response. You really need only one browser object per program; but every time you make a request, you get back a new HTTP::Response object, which will have some interesting attributes:
o A status code indicating success or failure (which you can test with "$response->is_success").
o An HTTP status line that is hopefully informative if there's failure (which you can see with "$response->status_line", returning something like "404 Not Found").
o A MIME content-type like "text/html", "image/gif", "application/xml", etc., which you can see with "$response->content_type"
o The actual content of the response, in "$response->decoded_content". If the response is HTML, that's where the HTML source will be; if it's a GIF, then "$response->decoded_content" will be the binary GIF data.
o And dozens of other convenient and more specific methods that are documented in the docs for HTTP::Response, and its superclasses HTTP::Message and HTTP::Headers.
Adding Other HTTP Request Headers
The most commonly used syntax for requests is "$response = $browser->get($url)", but in truth, you can add extra HTTP header lines to the request by adding a list of key-value pairs after the URL, like so:
$response = $browser->get( $url, $key1, $value1, $key2, $value2, ... );
For example, here's how to send some commonly used headers, in case you're dealing with a site that would otherwise reject your request:
my @ns_headers = (
'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',
'Accept-Charset' => 'iso-8859-1,*,utf-8',
'Accept-Language' => 'en-US',
);
...
$response = $browser->get($url, @ns_headers);
If you weren't reusing that array, you could just go ahead and do this:
$response = $browser->get($url,
'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',
'Accept-Charset' => 'iso-8859-1,*,utf-8',
'Accept-Language' => 'en-US',
);
If you were only ever changing the 'User-Agent' line, you could just change the $browser object's default line from "libwww-perl/5.65" (or the like) to whatever you like, using the LWP::UserAgent "agent" method:
$browser->agent('Mozilla/4.76 [en] (Win98; U)');
Enabling Cookies
A default LWP::UserAgent object acts like a browser with its cookies support turned off. There are various ways of turning it on, by setting its "cookie_jar" attribute. A "cookie jar" is an object representing a little database of all the HTTP cookies that a browser knows about. It can correspond to a file on disk or an in-memory object that starts out empty, and whose collection of cookies will disappear once the program is finished running.
To give a browser an in-memory empty cookie jar, you set its "cookie_jar" attribute like so:
use HTTP::CookieJar::LWP;
$browser->cookie_jar( HTTP::CookieJar::LWP->new );
To save a cookie jar to disk, see "dump_cookies" in HTTP::CookieJar. To load cookies from disk into a jar, see "load_cookies" in HTTP::CookieJar.
Posting Form Data
Many HTML forms send data to their server using an HTTP POST request, which you can send with this syntax:
$response = $browser->post( $url,
[
formkey1 => value1,
formkey2 => value2,
...
],
);
Or if you need to send HTTP headers:
$response = $browser->post( $url,
[
formkey1 => value1,
formkey2 => value2,
...
],
headerkey1 => value1,
headerkey2 => value2,
);
For example, the following program makes a search request to AltaVista (by sending some form data via an HTTP POST request), and extracts from the HTML the report of the number of matches:
use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;
my $word = 'tarragon';
my $url = 'http://search.yahoo.com/yhs/search';
my $response = $browser->post( $url,
[ 'q' => $word, # the Altavista query string
'fr' => 'altavista', 'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',
]
);
die "$url error: ", $response->status_line
unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type
unless $response->content_is_html;
if( $response->decoded_content =~ m{([0-9,]+)(?:<.*?>)? results for} ) {
# The substring will be like "996,000 results for"
print "$word: $1\n";
}
else {
print "Couldn't find the match-string in the response\n";
}
Sending GET Form Data
Some HTML forms convey their form data not by sending the data in an HTTP POST request, but by making a normal GET request with the data stuck on the end of the URL. For example, if you went to "www.imdb.com" and ran a search on "Blade Runner", the URL you'd see in your browser window would be:
http://www.imdb.com/find?s=all&q=Blade+Runner
To run the same search with LWP, you'd use this idiom, which involves the URI class:
use URI;
my $url = URI->new( 'http://www.imdb.com/find' );
# makes an object representing the URL
$url->query_form( # And here the form data pairs:
'q' => 'Blade Runner',
's' => 'all',
);
my $response = $browser->get($url);
See chapter 5 of Perl & LWP for a longer discussion of HTML forms and of form data, and chapters 6 through 9 for a longer discussion of extracting data from HTML.
Absolutizing URLs
The URI class that we just mentioned above provides all sorts of methods for accessing and modifying parts of URLs (such as asking sort of URL it is with "$url->scheme", and asking what host it refers to with "$url->host", and so on, as described in the docs for the URI class. However, the methods of most immediate interest are the "query_form" method seen above, and now the "new_abs" method for taking a probably-relative URL string (like "../foo.html") and getting back an absolute URL (like "http://www.perl.com/stuff/foo.html"), as shown here:
use URI;
$abs = URI->new_abs($maybe_relative, $base);
For example, consider this program that matches URLs in the HTML list of new modules in CPAN:
use strict;
use warnings;
use LWP;
my $browser = LWP::UserAgent->new;
my $url = 'http://www.cpan.org/RECENT.html';
my $response = $browser->get($url);
die "Can't get $url -- ", $response->status_line
unless $response->is_success;
my $html = $response->decoded_content;