Before web services came along, screen scraping was a popular technique for grabbing the output from another application by examining the text it displays on the screen. For web applications, this meant making a request to a URL and examining the HTML the server returns. You could then parse the HTML to grab the latest news headlines or stock quotes from a news site, or the price of a book on amazon.com. With RSS, XML, and Web Services, the need to screen scrape has diminished, but is not extinct. in this article we will examine a few methods to grab the HTML from another URL and for display in your own page. HttpServerUtility If the page you need to fetch is part of the current web application, you can use the execute method on the Server object of the current page The Server object is of type HttpServerUtility, which also includes the well-known methods Transfer and MapPath Using execute is straightforward:.. TextWriter textWriter = new StringWriter ();
Server.execute ("My PortPage.aspx", TextWriter;
Response.Output.write (TextWriter.toString ());
You can use Server.Execute to add content to frames, or devise print friendly pages. We generally would not want to write the entire contents of the resulting string into the response as we have in this sample, but instead would parse select content from myOtherPage .aspx. of course, we are not always so lucky to have the resource we need inside of the same web application, and this is where classes from the System.Net namespace come into play. WebClient The WebClient class presents the simplest API possible for Retrieving Content From A URL, As Seen Below. Using (WebClient WebClient = New WebClient ())
{
Byte [] response = WebClient.downloaddata (Tur); response.outputstream.write (response, 0, response.length);
}
We need only three lines of code, but this time instead of passing the name of an ASPX page inside of our application, we can pass the URL to a remote resource, like http://www.OdeToCode.com/default.aspx. The next hurdle you might face is retrieving content from a web site requiring forms authentication. Forms authentication usually requires a user to enter credentials into a form and press a submit button. Pressing submit will cause the browser to perform an HTTP "POST" and send the form values, such as the username and password, in the message body to the server (for more information on GET and POST see the resource section at the bottom of the article). As an example, consider the source code for the following login Form: