Screen Scraping, ViewState, And Authentication Using ASP.NET

xiaoxiao2021-03-06  72

Before web services came along, screen scraping was a popular technique for grabbing the output from another application by examining the text it displays on the screen. For web applications, this meant making a request to a URL and examining the HTML the server returns. You could then parse the HTML to grab the latest news headlines or stock quotes from a news site, or the price of a book on amazon.com. With RSS, XML, and Web Services, the need to screen scrape has diminished, but is not extinct. in this article we will examine a few methods to grab the HTML from another URL and for display in your own page. HttpServerUtility If the page you need to fetch is part of the current web application, you can use the execute method on the Server object of the current page The Server object is of type HttpServerUtility, which also includes the well-known methods Transfer and MapPath Using execute is straightforward:.. TextWriter textWriter = new StringWriter ();

Server.execute ("My PortPage.aspx", TextWriter;

Response.Output.write (TextWriter.toString ());

You can use Server.Execute to add content to frames, or devise print friendly pages. We generally would not want to write the entire contents of the resulting string into the response as we have in this sample, but instead would parse select content from myOtherPage .aspx. of course, we are not always so lucky to have the resource we need inside of the same web application, and this is where classes from the System.Net namespace come into play. WebClient The WebClient class presents the simplest API possible for Retrieving Content From A URL, As Seen Below. Using (WebClient WebClient = New WebClient ())

{

Byte [] response = WebClient.downloaddata (Tur); response.outputstream.write (response, 0, response.length);

}

We need only three lines of code, but this time instead of passing the name of an ASPX page inside of our application, we can pass the URL to a remote resource, like http://www.OdeToCode.com/default.aspx. The next hurdle you might face is retrieving content from a web site requiring forms authentication. Forms authentication usually requires a user to enter credentials into a form and press a submit button. Pressing submit will cause the browser to perform an HTTP "POST" and send the form values, such as the username and password, in the message body to the server (for more information on GET and POST see the resource section at the bottom of the article). As an example, consider the source code for the following login Form:

username

password

In The Message Body of The Browser Post, The Form Values ​​Could Appear Like So: UsernametextBox = Scott & PasswordTextBox = Scott & LoginButton = Login

When this payload arrives at the server, the code will know the user entered 'scott' into the username textbox, 'scott' in the password text box, and posted the form using the Login button. We can use the WebClient class to simulate a Post for this form with the folline code. WebClient WebClient = New WebClient (); WebClient.Headers.Add ("Content-Type", "Application / X-WWW-FORM-URLENCODED");

Byte [] response = WebClient.uploaddata

Login_url, "post", encoding.ascii.getbytes (postdata)

);

However, trying to POST to an ASP.NET page will usually involve one more obstacle:. The Viewstate We will not be covering Viewstate in this article (see resources below), except we need to know how to correctly POST the Viewstate back to the server. ASP.NET sends Viewstate to the client in a hidden form field, and we must parse out the correct value in order to submit the login form programmatically. If we view the source for a login web form like the form above in ASP. Net, We will see the folload appear just after the opening form:

Value = "DDWTMZG4MDA0NZA7OZ5C3QUCJNFEAIFSJCEZK8NDLKR4YA ==" />

You might be asking what else might appear in a form, and what is the easiest way to see what the browser sends to the server? If you are going to do any nontrivial screen-scraping, sooner or later you will need to answer this question and debug problems. The easiest way to debug is to use a tool like Fiddler, which will show you every request and response between your machine and a web server. you can inspect the headers and message content, and watch exactly what happens when your browser performs a POST, then try to replicate the behavior programmatically. In order to send the correct Viewstate value to the server, we will first need to request the form from the server, parse the Viewstate, and then POST the form back. Let's try this In ouxt example. Byte [] response; WebClient WebClient = New WebClient ();

Response = WebClient.downloaddata (login_url);

String ViewState = ExtractViewState

Encoding.ascii.getstring (Response)

);

String postdata = string.format

"__Viewstate = {0} & usernametextbox = {1} & passwordtextbox = {2} & loginbutton = login",

ViewState, UserName, Password;

WebClient.Headers.Add ("Content-Type", "Application / X-WWWW-Form-Urlencoded");

Response = WebClient.uploadData

Login_url, "post", encoding.ascii.getbytes (postdata)

);

Now we have a lot more activity happening. First, we request the login form, then we parse out the Viewstate value (more on this coming up). Once we have the Viewstate, we can create a string (postData) with the form values . We have not mentioned the reason for adding the Content-Type header, but if you use the Fiddler tool this will be one of those small details you might notice as a difference between your programmatic POST and the browser POST, and is required for POST to work. We can parse out the Viewstate value with some string manipulation. First, we will find the location of the identifier __VIEWSTATE, then identify the string after the identifier and between the double quotes of the value attribute. private string ExtractViewState (string s ) {

String ViewStatenameDelimiter = "__viewstate";

String valueDelimiter = "value = /" ";

INT ViewStateNamePosition = S.Indexof (ViewStateNameDelimiter);

INT viewStateValuePosition = S.indexof

Valuedelimiter, ViewStateNamePosition

);

INT ViewStateStartPosition = ViewStateValuePosition

Valuedelimiter.Length;

INT ViewStateEndPosition = S.Indexof ("/", viewstatestartPosition;

Return httputility.urlencodeunicate (

S.SUBString

ViewStatestArtPosition,

ViewStateEndPosition - ViewStatestArtPosition

)

);

}

Notice the use of URL encoding to make sure the server misinterprets no characters with a special meaning (like the equal sign). If you are familiar with forms authentication in ASP.NET you'll know the runtime issues a cookie to the browser when a user has successfully authenticated themselves. On subsequent requests, the browser needs to pass along the cookie value to reach protected resources. Unfortunately, I have not found an easy way for the WebClient to work with cookie values, so we will try a more advanced API with the HttpWebRequest class. HttpWebRequest The code using HttpWebRequest will look a bit different than what we have seen with WebClient. HttpWebRequest uses streams to write form values ​​into the request and read the response. We also need to add some code to handle the forms authentication Cookie. This Final Code Example Will Success, Login To A Website and Pull The HTML from a protected resource. private void button5_click (object sender, system.eventargs e) {

// first, Request the login form to get the viewstate value

HttpWebrequest WebRequest = WebRequest.create (login_url) AS HTTPWEBREQUEST

StreamReader Responsereader = New StreamReader

WebRequest.getResponse (). getResponseSteream ()

);

String responseData = responsereader.readtoend ();

ResponseReader.close ();

// Extract the viewstate value and build out post data

String viewState = extractViewState (ResponseData);

String postData =

String.Format

"__Viewstate = {0} & usernametextbox = {1} & passwordtextbox = {2} & loginbutton = login",

ViewState, Username, Password

);

// Have a cookie Container Ready to Receive The Forms Auth Cookie

CookieContainer cookies = new cookiecontainer ();

// Now post to the login formwebrequest = WebRequest.create (login_url) AS HTTPWEBREQUEST

WebRequest.method = "post";

WebRequest.contentType = "Application / X-WWW-FORM-URLENCODED";

WebRequest.cookieContainer = cookies;

// Write the Form Values ​​Into The Request Message

Streetwriter Requestwriter = New StreamWriter (WebRequest.getRequestStream ());

RequestWriter.write (PostData);

RequestWriter.Close ();

// We don't need the contents of the response, Just the cookie it isssues

WebRequest.getResponse (). Close ();

// Now We can send out cookie along with a request for the protected page

WebRequest = WebRequest.create (Secret_page_url) AS HTTPWEBREQUEST;

WebRequest.cookieContainer = cookies;

ResponseReader = New StreamReader (). getResponseSponse ());

// and read the response

ResponseData = responseReader.ReadToend ();

ResponseReader.close ();

Response.write (responseData);

}

转载请注明原文地址:https://www.9cbs.com/read-121441.html

New Post(0)