Scraping, or Programatically Accessing, a Secure Webpage

home
c#
view story

21 Kicks

Kick It.

Drop Kick

Scraping, or Programatically Accessing, a Secure Webpage (ryanfarley.com)

published 1 year, 5 months ago, submitted by StanJackson(395) 1 year, 5 months ago

There are many secure websites out there that provide useful information but do not have a public API to access it's data. A prime example of this is the LinkedIn website. You might love to gather some info from LinkedIn, but their promise to deliver a public API has yet to come to fruition. The problem is, the pages with all the good data are secure, requiring the user to log in before accessing these pages. Let's say we want to scrape this data from these pages programatically? We need to authenticate to access these pages. We can do that by reusing the authentication cookie from the site that we receive when we log in with a browser.

add a comment | category: C# | Views: 305