Innoplexia DevTools to Crawl Webpages

DevTools

09.05.12 @chrschneider 2

DevTools

… Apache … toolset of low level Java components
focused on HTTP and associated protocols.“

● HttpComponents Core
… is a set of low level HTTP transport components

● HttpComponents Client
… provides reusable components for client-side ... HTTP connection
management.

● HttpComponents AsyncClient (DEV)
… ability to handle a great number of concurrent connections ... more ...
performance in terms of a raw data throughput.

● Commons HttpClient (Legacy)
… All users of Commons HttpClient 3.x are strongly encouraged to upgrade to
HttpClient 4.1.


DevTools

HttpComponents Client

Example Components

● Get, Post, Delete, … Request Objects

● Cookie Manager

● SSL

● Content Encoding Aware

● HTTP Authentication (Basic, Digest, ...)


DevTools

HttpComponents Client Example

public final static void main(final String[] args) throws Exception
{

final HttpClient httpclient = new DefaultHttpClient();
try
{
final HttpGet httpget = new HttpGet("http://www.google.com/");

System.out.println("executing request " + httpget.getURI());

// Create a response handler
final ResponseHandler<String> responseHandler = new BasicResponseHandler();
final String responseBody = httpclient.execute(httpget, responseHandler);
System.out.println("----------------------------------------");
System.out.println(responseBody);
System.out.println("----------------------------------------");

}
finally
{
httpclient.getConnectionManager().shutdown();
}
}

http://hc.apache.org/httpcomponents-client-ga/examples.html


DevTools

HttpComponents Client

Demo


DevTools

… is an asynchronous event-driven network application framework for rapid
development of maintainable high performance protocol servers & clients.

See: http://netty.io/


DevTools

… is a "GUI-Less browser for Java programs"

Features (extraction):
● Support for the HTTP and HTTPS protocols

● Support for cookies

● Ability to specify whether failing responses from the server should throw exceptions

or should be returned as pages of the appropriate type (based on content type)
● Ability to customize the request headers being sent to the server

● Support for HTML responses

● Support for submitting forms
● Support for clicking links
● Support for walking the DOM model of the HTML document
● JavaScript support


DevTools


@Test
public void homePage() throws Exception
{
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("http://htmlunit.sourceforge.net");

System.out.println(page.getTitleText());

assertEquals("Welcome to HtmlUnit", page.getTitleText());

final String pageAsXml = page.asXml();
assertTrue(pageAsXml.contains("<body class="composite">"));

final String pageAsText = page.asText();
assertTrue(pageAsText.contains("Support for the HTTP and HTTPS protocols"));

webClient.closeAllWindows();
}

http://htmlunit.sourceforge.net/gettingStarted.html


DevTools


@Test
public void getElements() throws Exception
{
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("http://some_url");
final HtmlDivision div = page.getHtmlElementById("some_div_id");
final HtmlAnchor anchor = page.getAnchorByName("anchor_name");

webClient.closeAllWindows();
}

Luxus :)

Note: Also html tables are supported. They wrote easy wrapper classes to walk though them. … Handy!
http://htmlunit.sourceforge.net/table-howto.html

http://htmlunit.sourceforge.net/gettingStarted.html


DevTools

… automates browsers. That's it.

Selenium-WebDriver supports the following browsers along with the
operating systems these browsers are compatible with.
● Google Chrome 12.0.712.0+
● Internet Explorer 6, 7, 8, 9 - 32 and 64-bit where applicable
● Firefox 3.0, 3.5, 3.6, 4.0, 5.0, 6, 7
● Opera 11.5+
● HtmlUnit 2.9
● Android – 2.3+ for phones and tablets (devices & emulators)
● iOS 3+ for phones (devices & emulators) and 3.2+ for tablets (devices
& emulators)


DevTools


The Selenium Family

Selenium IDE

Also c#, Phython, Ruby, ...
Selenium WebDriver

Also on Windows and Mac

Selenium Grid


DevTools


The Selenium Family

… create quick bug reproduction scripts
Selenium IDE
… create scripts to aid in automation-aided
exploratory testing

Selenium WebDriver … create robust, browser-based regression
automation

… scale and distribute scripts across many
environments
Selenium Grid

http://seleniumhq.org/


DevTools

Requirements for Selenium WebDriver with Firefox
(and HtmlUnit)

Dependencies Browser Binaries
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>2.21.0</version>
</dependency>

<dependency>
<artifactId>selenium-htmlunit-driver</artifactId>
</dependency>

<dependency>

it.
<artifactId>selenium-firefox-driver</artifactId>

's

at
Th
</dependency>


DevTools

Basic Selenium example

@Test
public void testSeleniumWithFirefox() throws InterruptedException
{
final WebDriver webDriver = new FirefoxDriver();

webDriver.get("http://www.majug.de");

final WebElement veranstaltungenLink = webDriver.findElement(By.linkText("Veranstaltungen"));

veranstaltungenLink.click();

// Close the browser
Thread.sleep(5000);
webDriver.quit();
}


DevTools

Selenium WebDriver Locator Strategies

It's also possible to call findElements(...) to get a List<> of WebElements.:

List<WebElement> hits = webDriver.findElements(By.tagName("a"));


DevTools

Selenium WebDriver Interactions

If you got a webElement, you can...

● webElement.click() it

● webElement.sendKeys(...) to it

● webElement.submit() on it.

It is also possible to perform “Actions“ like DoubleClick, DragAndDrop, ClickAndHold, …
with the “Actions“ class.


DevTools

Selenium WebDriver

Demo


DevTools

Selenium WebDriver Pitfalls

Newbie Pitfalls:

● Selenium doesn't wait until the hole site is loaded (Keyword: Implicit wait)
● webElement.xPath(“@// ...“) starts from root of the DOM (use “.//...“ instead)
● Google brings up “Selenium RC“ solutions. This is the old Selenium project.
● A reference to a WebElement will become invalid if the driver “moves“ to
another page.
● Firefox doesn't run on our CI because it is a headless system (try Xvfb)
● New XPath 2.0 functions (like ends-with(...)) are failing. This is because Selenium
uses the driver's native Xpath engine. For Firefox this means it is Xpath 1.0 today.


Noch Fragen?
Vielen Dank für Ihre Aufmerksamkeit!

Innoplexia DevTools to Crawl Webpages

More Related Content

What's hot (17)

Viewers also liked (16)

Similar to Innoplexia DevTools to Crawl Webpages (20)

Recently uploaded (20)

Innoplexia DevTools to Crawl Webpages