Scraping recalcitrant web sites with Python & Selenium
6 likes9,081 views
This document discusses using Selenium and Python to scrape websites that are difficult to scrape due to their use of dynamic content, JavaScript, and other obfuscation techniques. It describes how Selenium can automate a browser to interact with such sites as a user would, allowing all content to load properly before scraping. The document provides a general recipe for using Selenium with Python, an example Selenium test case exported to Python, and discusses using PIL and MD5 hashes to scrape an image-based keypad on one such difficult site.
![Example from Selenium IDE
class Ingdirect2(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait( 30)
self.base_url = "https://www.ingdirect.com.au"
self.verificationErrors = []
def test_ingdirect2(self):
driver = self.driver
But what about
driver.get( self.base_url + "/client/index.aspx")
that dang
driver.switch_to_frame( 'body') # Had to add this keypad? ...
driver.find_element_by_id( "txtCIF").clear()
driver.find_element_by_id( "txtCIF").send_keys( "12345678")
driver.find_element_by_id( "objKeypad_B1").click()
driver.find_element_by_id( "objKeypad_B2").click()
driver.find_element_by_id( "objKeypad_B3").click()
driver.find_element_by_id( "objKeypad_B4").click()
driver.find_element_by_id( "btnLogin").click()
self.assertTrue( self.is_element_present(By.ID, "ctl2_lblBalance"))](https://image.slidesharecdn.com/scrapingrecalcitrantwebsiteswithpythonselenium-120705192849-phpapp02/85/Scraping-recalcitrant-web-sites-with-Python-Selenium-10-320.jpg)
![PIL saves the day
# Get screenshot for extraction of button images
screenshot = driver.get_screenshot_as_base64()
im = Image.open(StringIO.StringIO(base64.decodestring(screenshot)))
table = driver.find_element_by_xpath( '//*[@id="objKeypad_divShowAll"]/table')
all_buttons = table.find_elements_by_tag_name( "input")
# Determine md5sum of each button by cropping based on element positions
for button in all_buttons:
button_image = im.crop(getcropbox(button))
hexid = hashlib.md5(button_image.tostring()).hexdigest()
button_mapping[hexid] = button.get_attribute( "id")
# Now we know which button is which ( based on previous lookup), enter the PIN
for char in self.pin:
driver.find_element_by_id(button_mapping[hex_mapping[char]]).click()
driver.find_element_by_id( "btnLogin").click()
# We're in!!!11one](https://image.slidesharecdn.com/scrapingrecalcitrantwebsiteswithpythonselenium-120705192849-phpapp02/85/Scraping-recalcitrant-web-sites-with-Python-Selenium-11-320.jpg)
Scraping recalcitrant web sites with Python & Selenium
- 1. Scraping recalcitrant web sites with Python & Selenium Roger Barnes SyPy July 2012
- 3. Some sites suck - "for your own good" For security reasons, each button is an image, dynamically generated by a hash wrapped in a mess of javascript, randomly placed
- 4. ...but they work in a web browser! Let's use the web browser to scrape them
- 5. Enter Selenium Selenium automates browsers That's it
- 6. Selenium can... ● navigate (windows, frames, links) ● find elements and parse attributes ● interact and trigger events (click, type, ...) ● capture screenshots ● run javascript ● let the browser take care of the hard stuff (cookies, javascript, sessions, profiles, DOM) Comes with various components and bindings ... including python
- 7. General Recipe Ingredients: ● firefox (or chrome) ● firebug (or chrome dev tools) ● Selenium IDE ○ record a session, write less code ● python and its batteries ● python-selenium ● xvfb and pyvirtualdisplay (optional) ● other libraries to taste ○ eg image manipulation, database access, DOM parsing, OCR
- 8. General Recipe Method: ● Install requirements (apt-get, pip etc) ○ sudo apt-get install xvfb firefox ○ pip install selenium pyvirtualdisplay ● Start up Firefox and Selenium IDE ● Record a "test" run through site ○ Add in some assertions along the way ● Export test as Python script ● Hack from there ○ Loops ○ Image/data extraction ○ Wrangling data into a database
- 10. Example from Selenium IDE class Ingdirect2(unittest.TestCase): def setUp(self): self.driver = webdriver.Firefox() self.driver.implicitly_wait( 30) self.base_url = "https://www.ingdirect.com.au" self.verificationErrors = [] def test_ingdirect2(self): driver = self.driver But what about driver.get( self.base_url + "/client/index.aspx") that dang driver.switch_to_frame( 'body') # Had to add this keypad? ... driver.find_element_by_id( "txtCIF").clear() driver.find_element_by_id( "txtCIF").send_keys( "12345678") driver.find_element_by_id( "objKeypad_B1").click() driver.find_element_by_id( "objKeypad_B2").click() driver.find_element_by_id( "objKeypad_B3").click() driver.find_element_by_id( "objKeypad_B4").click() driver.find_element_by_id( "btnLogin").click() self.assertTrue( self.is_element_present(By.ID, "ctl2_lblBalance"))
- 11. PIL saves the day # Get screenshot for extraction of button images screenshot = driver.get_screenshot_as_base64() im = Image.open(StringIO.StringIO(base64.decodestring(screenshot))) table = driver.find_element_by_xpath( '//*[@id="objKeypad_divShowAll"]/table') all_buttons = table.find_elements_by_tag_name( "input") # Determine md5sum of each button by cropping based on element positions for button in all_buttons: button_image = im.crop(getcropbox(button)) hexid = hashlib.md5(button_image.tostring()).hexdigest() button_mapping[hexid] = button.get_attribute( "id") # Now we know which button is which ( based on previous lookup), enter the PIN for char in self.pin: driver.find_element_by_id(button_mapping[hex_mapping[char]]).click() driver.find_element_by_id( "btnLogin").click() # We're in!!!11one
- 12. But why do all this? It's my data! ... and I'll graph if i want to * Actual results may vary. Graph indicates open inodes, not high-roller gambling problem
- 13. That's all folks Slides ● http://bit.ly/scrapium Code ● https://gist.github.com/3015852 Me ● https://twitter.com/mindsocket ● https://github.com/mindsocket ● [email protected]