Introduction to writing your very own Web Clients
Welcome to this brief tutorial. This tutorial will outline the creation of simple perl scripts which have the capability of requesting HTML data from the internet, storing it in a file, mirroring it, printing it, and searching it.
Perl: You've learned how to write scripts, parse text, blah blah... now what? Sure, its nice to save a text file to your hard drive... rename it... and... uh- save another text file after that. But surely there has to be somthing else- beside cgi- that perl can be used for. Here's an answer to that question- just one of the thousands found at www.cpan.org. Within five minutes you'll be writing perl scripts capable of retreaving webpages. Cool, eh?
Let's begin.
The first step to writing a www aware perl script is to download the libwww-perl module. This module features a non-object oriented package, and a object oriented package. For your conveniance, I'll cover the non-object oriented (it's faster and does the same stuff). To download this perl module, simply go here:

 http://www.cpan.org/authors/id/GAAS/libwww-perl-5.63.tar.gz


For those wishing to do this on the Windows platform, you can find this at the ActiveState website (www.ActiveState.com, I believe)- if not, search either google for libwww-perl, or www.cpan.org, for a windows version. This tutorial was written on a linux machine, thus, I wont truly be able to refer you to the windows download location.
Once you've downloaded it, un tar/zip it to a new directory. Enter the shell/DOS, change directory to the directory you unzipped/tarred to, and type the following:



perl Maketest.PL
-wait for this program to complete. If it cannot find this this file, type dir on dos, or ls on unix to view the content of the directory. Type perl and whatever file you have that ends in .PL (capitals)



Next, type

make


-wait for this to finish, then type

make test


-and then


make install



once you type all four of these commands(perl Makefile.pl -> make -> make test -> make install), your copy of Perl will have been patched to recognize the wwwlib-perl module.

Next, open up your favorite perl editor, and create a new file. In this file, you must say- "Hey, Perl-- I want to use the wwwlib-perl thingie I just installed, so stick it in my code" Which can be roughly translated into perl-speak by typing


use LWP::Simple;


So, not much of a program yet, eh? What LWP stands for is libwww-perl (Duh)... which should help you remember it. What next? Well, you've got a program that knows this modules there- but... how do you use it?

Simply! This package is called "Simple" for just that reason. Go figure. 
What you'll find is that your copy of Perl has suddenly been expanded to house several brand spanking new functions- thats right, simple, old fashioned, functions.




1. get($url);

2. getstore($url, $filename);

3. getprint($url);

4. mirror($url, $filename);


Oohh, ahhh! I'm sure you guys could memorize those right now. Let me tell you what each of those do, and how your expected to use them.
First, you've got get($url). Simply take a scalar variable- lets say $html, and assign it to get("http://get_this_url/"). You can replace the string inside the get function with the url of whatever website you'd like to get. SO, This is an example of doing just that:



use LWP::Simple;

$html = get("http://www.megathink.com");

print $html;


A program that gets a webpage's html and crams it into a scalar
Don't forget the use LWP::Simple; at the top! That can fudge everything up.
Let's say you wanna do this real quick- illiminate the need for a variable at all. Well- you've got the getprint($url); function to do that! Simply type getprint("http://www.whatever.com/");, and the program will automatically print the html of that website.
What if you wanna store the html to a file on your harddrive- for backup. Or, let's say you want to start your own cache of websites. That's just as easy! Type getstore("http://url", "stored.htm");, and badda bing, badda boom- you've got a new .htm file in your directory, loaded to the brim with whatever URL you requested. A working example, you say? Sure!

use LWP::Simple;

getstore("http://www.megathink.com", "temp.htm");

Well- thats cool, huh? No? You want to be able to only store a website when it has changed-- like google does, for example? That's no big deal-- change the function in the last example ("getstore"), to mirror- leaving the parameters as they are- and the program will only store the html file if it has changed from the version already stored on disk. Cool? Yup!
At this point, you have a ton of things you can do. Let's say you want to check for dead links. Simple get the html into a variable ($html = get("http://www.google.com/");), and use a couple search strings and splits on it until you find all the <.a href.> tags-- next, find the src= parameter, and add each URL to an array. Create a loop (foreach $i(@array_of_links){}) to cycle through each, and attempt to connect to them.If the link is bad, get() will return a false string ("") [there is nothing between the quotes]. Otherwise, the get will return the html. I don't want to ruin this for you- since I'm sure you'd love to try it on your own [yeaaaa, riiight]. Another "creative" idea is to write a proxy. If you run Apache on your machine (or IIS for you windowers), you can now use these functions in cgi programs. YUP! Think of the bucks you can make for writing a proxy to get past that god forsaken netnanny, or bess, software your school/home/office forces onto you. Simply write a script to accept a query_string of a URL, and use getprint() to display it. Cha-ching!
I hope you enjoyed reading this tutorial, and continue to contribute to the free Perl community. It was my pleasure writing this lil' file.
Please post feedback so I know whether or not I'm actually helping.
Happy New Year!