Screen Scraping, ViewState, And Authentication Using ASP.NET

username

password

In The Message Body of The Browser Post, The Form Values ​​Could Appear Like So:

Usernametextbox = Scott & PasswordTextBox = Scott & loginbutton = login

When this payload arrives at the server, the code will know the user entered 'scott' into the username textbox, 'scott' in the password text box, and posted the form using the Login button. We can use the WebClient class to simulate a POST for this form with the stocking code.WebClient WebClient = New WebClient ();

WebClient.Headers.Add ("Content-Type", "Application / X-WWWW-Form-Urlencoded");

Byte [] response = WebClient.uploaddata

Login_url, "post", encoding.ascii.getbytes (postdata)

);

However, trying to POST to an ASP.NET page will usually involve one more obstacle:. The Viewstate We will not be covering Viewstate in this article (see resources below), except we need to know how to correctly POST the Viewstate back to the server. ASP.NET sends Viewstate to the client in a hidden form field, and we must parse out the correct value in order to submit the login form programmatically. If we view the source for a login web form like the form above in ASP. NET, We will see The Following Appear Just After The Opening Form Tag:

Value = "DDWTMZG4MDA0NZA7OZ5C3QUCJNFEAIFSJCEZK8NDLKR4YA ==" />

You might be asking what else might appear in a form, and what is the easiest way to see what the browser sends to the server? If you are going to do any nontrivial screen-scraping, sooner or later you will need to answer this question and debug problems. The easiest way to debug is to use a tool like Fiddler, which will show you every request and response between your machine and a web server. you can inspect the headers and message content, and watch exactly what happens when your browser performs a POST, then try to replicate the behavior programmatically.In order to send the correct Viewstate value to the server, we will first need to request the form from the server, parse the Viewstate, and then POST the form back. Let's try this IN Our next esample.

Byte [] response

WebClient WebClient = New WebClient ();

Response = WebClient.downloaddata (login_url);

String ViewState = ExtractViewState

Encoding.ascii.getstring (Response)

);

String postdata = string.format

"__Viewstate = {0} & usernametextbox = {1} & passwordtextbox = {2} & loginbutton = login",

ViewState, UserName, Password;

WebClient.Headers.Add ("Content-Type", "Application / X-WWWW-Form-Urlencoded");

Response = WebClient.uploadData

Login_url, "post", encoding.ascii.getbytes (postdata)

);

Now we have a lot more activity happening. First, we request the login form, then we parse out the Viewstate value (more on this coming up). Once we have the Viewstate, we can create a string (postData) with the form values . We have not mentioned the reason for adding the Content-Type header, but if you use the Fiddler tool this will be one of those small details you might notice as a difference between your programmatic POST and the browser POST, and is required for POST to work.We can parse out the Viewstate value with some string manipulation. First, we will find the location of the identifier __VIEWSTATE, then identify the string after the identifier and between the double quotes of the value attribute.

Private string extractViewState (String S)

{

String ViewStatenameDelimiter = "__viewstate";

String valueDelimiter = "value = /" ";

INT ViewStateNamePosition = S.Indexof (ViewStateNameDelimiter);

INT viewStateValuePosition = S.indexof

Valuedelimiter, ViewStateNamePosition

);

INT ViewStateStartPosition = ViewStateValuePosition

Valuedelimiter.Length;

INT ViewStateEndPosition = S.Indexof ("/", viewstatestartPosition;

Return httputility.urlencodeunicate (

S.SUBString

ViewStatestArtPosition,

ViewStateEndPosition - ViewStatestArtPosition

)

);

}

NOTICE The Use of Url Encoding to make Sure The Server Misinterprets No Characters with a Special Meaning (Like The Equal Sign).

If you are familiar with forms authentication in ASP.NET you'll know the runtime issues a cookie to the browser when a user has successfully authenticated themselves. On subsequent requests, the browser needs to pass along the cookie value to reach protected resources. Unfortunately , I have not found an easy way for the WebClient to Work with cookie values, sowe Will Try A More Advanced API with the httpwebrequest class.httpWebRequest

The code using HttpWebRequest will look a bit different than what we have seen with WebClient. HttpWebRequest uses streams to write form values ​​into the request and read the response. We also need to add some code to handle the forms authentication cookie. This final code example Will SuccessFully Login to a Website and Pull The HTML from a protected.

Private void button5_click (Object Sender, System.Eventargs E)

{

// first, Request the login form to get the viewstate value

HttpWebrequest WebRequest = WebRequest.create (login_url) AS HTTPWEBREQUEST

StreamReader Responsereader = New StreamReader

WebRequest.getResponse (). getResponseSteream ()

);

String responseData = responsereader.readtoend ();

ResponseReader.close ();

// Extract the viewstate value and build out post data

String viewState = extractViewState (ResponseData);

String postdata =

String.Format

"__Viewstate = {0} & usernametextbox = {1} & passwordtextbox = {2} & loginbutton = login",

ViewState, Username, Password

);

// Have a cookie Container Ready to Receive The Forms Auth Cookie

CookieContainer cookies = new cookiecontainer ();

// Now post to the login form

WebRequest = WebRequest.create (login_url) AS HTTPWEBREQUEST;

WebRequest.method = "post"; WebRequest.contentType = "Application / X-www-form-urlencoded";

WebRequest.cookieContainer = cookies;

// Write the Form Values ​​Into The Request Message

Streetwriter Requestwriter = New StreamWriter (WebRequest.getRequestStream ());

RequestWriter.write (PostData);

RequestWriter.Close ();

// We don't need the contents of the response, Just the cookie it isssues

WebRequest.getResponse (). Close ();

// Now We can send out cookie along with a request for the protected page

WebRequest = WebRequest.create (Secret_page_url) AS HTTPWEBREQUEST;

WebRequest.cookieContainer = cookies;

ResponseReader = New StreamReader (). getResponseSponse ());

// and read the response

ResponseData = responseReader.ReadToend ();

ResponseReader.close ();

Response.write (responseData);

}

If you have been following along, the above code should make some sense, even though the HttpWebRequest class requires us to do a more work. For instance, instead of using the UploadData method of WebClient to POST and have a response, we need to get the request stream, write into the request stream, get the response stream, and read from the response stream. Notice the use of the CookieContainer class to keep the authentication ticket alive in our request.

Conclusion

Until every single web site on the Internet offers a web service to programmatically retrieve data, screen scraping will be around. It's good to know a few tricks to fetch content from the web using code and classes in the .NET framework.

BY K. Scott Allen

Additional Resources