SAVIA
0 likes•160 views
This document provides an overview of using the Mechanize gem to programmatically interact with and parse web pages in Ruby. It demonstrates how to navigate pages, submit forms, select and manipulate page elements, and parse content from the DOM. Key capabilities covered include making HTTP requests, submitting forms, interacting with forms and fields, selecting elements, and parsing returned pages.
![The Agent Accessing page elements Forms
# The current page # Submitting a form without a button:
require 'rubygems' form.submit
agent.page
require 'mechanize'
# The HTML page content # Submitting a form with the default button
agent = WWW::Mechanize.new page.body form.click_button()
# disable keep_alive, when running into # forms, links, frames # Submitting a form with a specific button
# problems with session timeouts or getting an page.forms, page.links, frames form.click_button(form.button_with(:name => 'OK')
# EOFError
# Selecting by criteria follows the pattern: # Form elements
agent.keep_alive = false form.fields, form.buttons, form.file_uploads, form.radio_buttons,
# page.element(s)_with(:criteria => value)
# The plural form (.elements) returns an form.checkboxes
# Setting user agent: # array, the singular form (.element) the
agent.user_agent = 'Friendly Mechanize Script" # first matching element or nil. Criteria # Form elements can be selected just like page elements
# is an attribute symbol and value may be # form.element(s)_with(:criteria => value)
# Using one of the predefined user agents: # a string or a regular expression. If no # e.g.:
# 'Mechanize', 'Mac Mozilla', 'Linux Mozilla'. # criteria attributr is given, :name will form.field_with(:name => 'password')
# be used. e.g.: form.field_with('password')
# 'Windows IE 6', 'iPhone', 'Linux Konqueror',
page.form_with(:name => 'formName') form.checkboxes(:value => /view_.*/)
# 'Windows IE 7', 'Mac FireFox', 'Mac Safari',
page.form_with('formName')
# 'Windows Mozilla' # Field values can also be selected directly by their name
page.links_with(:text => /[0-9]*/
agent.user_agent_alias = 'Mac Safari' form.password = 'secret'
# To verify server certificates: # Setting field values
# (A collection of certificates is available # field : .value = 'something'
# here: http://curl.haxx.se/ca/ ) # checkbox : .(un)check / .checked = true|false
# radio_button: .(un)check / .checked = true|false
agent.ca_file = 'cacert.pem'
# file_upload : .file_name = '/tmp/upload.dat'
Ruby / Mechanize
# e.g.:
# Don't follow HTTP redirects form.field_with('foo').value = 'something'
agent.redirect_ok = false form.checkbox_with(:value => 'blue').uncheck
http://mechanize.rubyforge.org/mechanize/ form.radio_buttons[3].check
# Follow refresh in meta tags
agent.follow_meta_refresh = true Nokogiri # Select lists / drop down fields:
form.field_with('color').option[2].select
http://nokogiri.org/ form.field_with('color').options.find{|o| o.value == 'red'}.select
# Enable logging
form.field_with('color').select_none
require 'logger' form.field_with('color').select_all
agent.log = Logger.new('mechanize.log')
Parsing the page content
# Selecting elements from the documents DOM
Hello Mechanize! nodes = agent.page.search('expression')
nodes = agent.page / 'expression'
require 'rubygems'
require 'mechanize'
Navigation/History # Selecting the first matching element or nil
agent = WWW::Mechanize.new node = agent.page.at('expression')
# load a page agent.get('http://rubyforge.org/')
agent.get('http://the.internet.net') # 'expression' might be an XPath or CSS selector
agent.page.forms.first.words = 'mechanize' nodes = agent.page.search('//h2/a[@class="title"]')
# Go back to the last page: agent.page.forms.first.click_button nodes = agent.page.search('.h2 a.title')
agent.back
agent.page.link_with(:text => /WWW::Mechanize/).click
agent.page.link_with(:text => 'Files').click
# navigating the document tree:
# Follow a link by its text
agent.link_with(:text => 'click me').click node.parent
links = agent.page / 'strong/a' node.children
# Backup history, execute block and version = links.find do |link|
# restore history link['href'] =~ /shownotes.*release_id/ # node content and attributes
agent.transact do end.text node.text
... node.inner_html
end puts "Hello Mechanize #{version}!"
node.attributes['width']
# found nodes, can be searched the same way
rows = agent.page / 'table/tr'
Version 2010-01-30 Creative Commons License value = rows[0].at('td[@class="value"]').text
(c) 2010 Tobias Grimm http://creativecommons.org/licenses/by/3.0](https://image.slidesharecdn.com/expofinal-121001205759-phpapp02/85/SAVIA-1-320.jpg)
SAVIA
- 1. The Agent Accessing page elements Forms # The current page # Submitting a form without a button: require 'rubygems' form.submit agent.page require 'mechanize' # The HTML page content # Submitting a form with the default button agent = WWW::Mechanize.new page.body form.click_button() # disable keep_alive, when running into # forms, links, frames # Submitting a form with a specific button # problems with session timeouts or getting an page.forms, page.links, frames form.click_button(form.button_with(:name => 'OK') # EOFError # Selecting by criteria follows the pattern: # Form elements agent.keep_alive = false form.fields, form.buttons, form.file_uploads, form.radio_buttons, # page.element(s)_with(:criteria => value) # The plural form (.elements) returns an form.checkboxes # Setting user agent: # array, the singular form (.element) the agent.user_agent = 'Friendly Mechanize Script" # first matching element or nil. Criteria # Form elements can be selected just like page elements # is an attribute symbol and value may be # form.element(s)_with(:criteria => value) # Using one of the predefined user agents: # a string or a regular expression. If no # e.g.: # 'Mechanize', 'Mac Mozilla', 'Linux Mozilla'. # criteria attributr is given, :name will form.field_with(:name => 'password') # be used. e.g.: form.field_with('password') # 'Windows IE 6', 'iPhone', 'Linux Konqueror', page.form_with(:name => 'formName') form.checkboxes(:value => /view_.*/) # 'Windows IE 7', 'Mac FireFox', 'Mac Safari', page.form_with('formName') # 'Windows Mozilla' # Field values can also be selected directly by their name page.links_with(:text => /[0-9]*/ agent.user_agent_alias = 'Mac Safari' form.password = 'secret' # To verify server certificates: # Setting field values # (A collection of certificates is available # field : .value = 'something' # here: http://curl.haxx.se/ca/ ) # checkbox : .(un)check / .checked = true|false # radio_button: .(un)check / .checked = true|false agent.ca_file = 'cacert.pem' # file_upload : .file_name = '/tmp/upload.dat' Ruby / Mechanize # e.g.: # Don't follow HTTP redirects form.field_with('foo').value = 'something' agent.redirect_ok = false form.checkbox_with(:value => 'blue').uncheck http://mechanize.rubyforge.org/mechanize/ form.radio_buttons[3].check # Follow refresh in meta tags agent.follow_meta_refresh = true Nokogiri # Select lists / drop down fields: form.field_with('color').option[2].select http://nokogiri.org/ form.field_with('color').options.find{|o| o.value == 'red'}.select # Enable logging form.field_with('color').select_none require 'logger' form.field_with('color').select_all agent.log = Logger.new('mechanize.log') Parsing the page content # Selecting elements from the documents DOM Hello Mechanize! nodes = agent.page.search('expression') nodes = agent.page / 'expression' require 'rubygems' require 'mechanize' Navigation/History # Selecting the first matching element or nil agent = WWW::Mechanize.new node = agent.page.at('expression') # load a page agent.get('http://rubyforge.org/') agent.get('http://the.internet.net') # 'expression' might be an XPath or CSS selector agent.page.forms.first.words = 'mechanize' nodes = agent.page.search('//h2/a[@class="title"]') # Go back to the last page: agent.page.forms.first.click_button nodes = agent.page.search('.h2 a.title') agent.back agent.page.link_with(:text => /WWW::Mechanize/).click agent.page.link_with(:text => 'Files').click # navigating the document tree: # Follow a link by its text agent.link_with(:text => 'click me').click node.parent links = agent.page / 'strong/a' node.children # Backup history, execute block and version = links.find do |link| # restore history link['href'] =~ /shownotes.*release_id/ # node content and attributes agent.transact do end.text node.text ... node.inner_html end puts "Hello Mechanize #{version}!" node.attributes['width'] # found nodes, can be searched the same way rows = agent.page / 'table/tr' Version 2010-01-30 Creative Commons License value = rows[0].at('td[@class="value"]').text (c) 2010 Tobias Grimm http://creativecommons.org/licenses/by/3.0