JavaOne 2014: Taming the Cloud Database with jclouds

1
Taming the Cloud Database
with
Apache jclouds
http://rack.to/jo14db

Before we begin: Setup
Taming the Cloud Database with Apache jclouds http://rack.
to/jo14db
2
Virtual Machine pre-setup provided
http://rack.to/jo14vm
Avoid downloading during this presentation

to/jo14db
3
Alternatively you need
Oracle Java 7
http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-
ubuntu-via.html
maven
http://maven.apache.org/download.cgi
git config --global core.autocrlf false

to/jo14db
4
Testing on your machine (no subscriptions)
Devstack
http://devstack.org/
http://blog.phymata.com/2014/04/18/devstack-icehouse-on-the-
rackspace-cloud/
Providers: Rackspace, HP, others
http://www.openstack.org/marketplace/public-clouds/

to/jo14db
5
git clone https://github.com/zack-shoylev/jclouds-
developer-examples.git
Multiple branches
git clone https://github.com/rackerlabs/jclouds-
examples.git
Branch trove-example-only

Introductions
to/jo14db
6
➢ Zack Shoylev
➢ irc: zacksh twitter: @zackshoylev
freenode #jclouds
Also email
@rackspace.com

Introductions
to/jo14db
7
Developer Experience
https://developer.rackspace.com/support/

Goals
to/jo14db
8
➢ jclouds
➢ Create a database in the cloud
➢ Abstractions and how to contribute
➢ How to add support for an API

The Cloud
to/jo14db
9
➢ Networked and distributed computing
➢ Variety of services
➢ Compute (servers)
➢ Storage (files)
➢ Databases
➢ Email
➢ …

The Cloud
to/jo14db
10
➢ Can’t someone else do it?

The Cloud
to/jo14db
11

The Cloud
to/jo14db
12
➢ Public cloud (external provider)
➢ Private cloud (internal deployment)
➢ Hybrid cloud
➢ Public + Private
Rackspace
example

The Cloud
to/jo14db
13
Your
System
S
D
K
Cloud
AP
I
VM
VM
VM
➢ This can be an application server
➢ Or your home machine
➢ Or belong to one of your end users
➢ Or a cloud VM
➢ Or a smartphone
jclouds

The Cloud
to/jo14db
14
➢ Advantages
➢ Metered pricing (pay as you go)
➢ Project scalability (unlimited)
➢ Safer (offsite/redundant)
➢ Economies of scale
➢ Expertise
➢ Support

The Cloud
to/jo14db
15
➢ Disadvantages
➢ Less hardware control
➢ Provider-controlled downtime
➢ Virtualization (efficiency)
➢ Provider lock-in
➢ Provider-specific apis/sdks/features
➢ Expensive to switch clouds or deploy locally
➢ jclouds minimizes this disadvantage

The Cloud
to/jo14db
16
➢ Database specific advantages
➢ Optimized by provider
➢ Settings
➢ Container virtualization
➢ Automated software updates
➢ Redundant data storage
➢ Backup
➢ Migration
➢ Choice
➢ Clustering (upcoming)
➢ Support

OpenStack
to/jo14db
17
➢ Cloud OS
➢ Open-sourced by Rackspace and NASA
(2010)
➢ Free
➢ Supported
➢ AT&T, HP, IBM, Red Hat, Rackspace, Dell, Cisco,
Intel, VMware, …

OpenStack
to/jo14db
18
➢ http://stackalytics.com/

jclouds
to/jo14db
19
➢ Cloud SDK
➢ Apache project
➢ Java (and Clojure)
➢ Easy
➢ Portable
➢ Cloud-agnostic
➢ Community
➢ Open source

jclouds
to/jo14db
20
➢ https://jclouds.apache.org/

jclouds
to/jo14db
21
➢ HTTP requests, responses, retries
➢ Authentication and re-authentication
➢ Pagination
➢ Polling
➢ Rate limits
➢ Retries
➢ Abstractions
➢ Logging
➢ Less code

jclouds
to/jo14db
22

jclouds
to/jo14db
23
➢ Services
➢ Storage
➢ Compute
➢ VM Images
➢ Load Balancers
➢ DNS
➢ Databases
➢ …
➢ Providers
➢ Openstack
➢ Rackspace
➢ HP
➢ Amazon
➢ Azure
➢ …

jclouds
to/jo14db
24
AWS aws-ec2 US-VA,US-CA,IE,SG
Bluelock bluelock-vcloud-zone01 US-IN
CloudSigma US cloudsigma-lvs US-NV
CloudSigma CH cloudsigma-zrh CH-ZH
CloudSigma DC cloudsigma-wdc US-DC
DigitalOcean digitalocean
ElasticHosts GB elastichosts-lon-b GB-LND
ElasticHosts GB elastichosts-lon-p GB-LND
ElasticHosts US elastichosts-sat-p US-TX
Go2Cloud go2cloud-jhb1 ZA-GP
GoGrid gogrid US-CA,US-VA
Green House Data greenhousedata-element-
vcloud
US-WY
HP hpcloud-compute US-NV
Ninefold ninefold-compute AU-NSW
OpenHosting openhosting-east1 US-VA
Rackspace UK (First Gen) cloudservers-uk GB-SLG
Rackspace US (First Gen) cloudservers-us US-IL,US-TX
Rackspace UK (Next Gen) rackspace-cloudservers-uk GB-SLG
Rackspace US (Next Gen) rackspace-cloudservers-us US-IL,US-TX
SeverLove serverlove-z1-man GB-MAN
SkaliCloud skalicloud-sdg-my MY-10
SoftLayer softlayer
Compute
Providers

jclouds
to/jo14db
25
➢ Showcase database code
➢ Best practices
➢ Compare with compute code
➢ Gotchas and workarounds
➢ Abstractions

Requirements
to/jo14db
26
➢ Maven 3
➢ Java 7+
➢ jclouds
➢ Windows or Linux [etc..] (thanks Java!)

Java Project
to/jo14db
27
Skeleton
Project
Create User
Create Instance
Initialize API
Create Database
Test Connection
Cleanup

Java Project
to/jo14db
28
➢ Maven Dependencies
➢ Selective subset
➢ Versioning
➢ jclouds-labs

POM
to/jo14db
29
https://github.com/jclouds/jclouds-examples/blob/master/rackspace/pom.xml
<dependency>
<groupId>org.apache.jclouds.provider</groupId>
<artifactId>rackspace-clouddatabases-us</artifactId>
<version>${jclouds.version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.25</version>
</dependency>

POM
to/jo14db
30
mvn dependency:copy-dependencies "-DoutputDirectory=./lib"

Logging
to/jo14db
31
// This module is responsible for enabling logging
Iterable<Module> modules = ImmutableSet.<Module> of(new SLF4JLoggingModule());
ComputeServiceContext context = ContextBuilder.newBuilder(provider)
.credentials(username, apiKey)
.modules(modules) // don't forget to add the modules to your context!
.buildView(ComputeServiceContext.class);
logback.xml
<configuration scan="false">
…
<appender name="WIREFILE" class="ch.qos.logback.core.FileAppender">
<file>target/test-data/jclouds-wire.log</file>
<encoder>
<Pattern>%d %-5p [%c] [%thread] %m%n</Pattern>
</encoder>
</appender>

Java Project
to/jo14db
32
Skeleton
Project
Create User
Create Instance
Initialize API
Create Database
Test Connection
Cleanup

Initialize
to/jo14db
33
private final TroveApi troveApi;
troveApi = ContextBuilder
.newBuilder("rackspace-clouddatabases-us")
.buildApi(TroveApi.class);

Architecture
to/jo14db
34
Cloud Databases
Trove
jclouds-database
Increasing level of
abstraction
Provider
settings:
Endpoint, etc.
Implementation
code
Abstraction
s

Architecture
to/jo14db
35
Cloud Databases
Trove
jclouds-database
Increasing level of
abstraction
Provider
settings:
Endpoint, etc.
Implementation
code
Abstraction
s
Not implemented

Initialize
to/jo14db
36
private final FlavorApi flavorApi;
flavorApi = troveApi.getFlavorApiForZone(ZONE);

APIs
to/jo14db
37
➢ TroveApi
➢ FlavorApi
➢ InstanceApi
➢ DatabaseApi
➢ UserApi
➢ Utils

Java Project
to/jo14db
38
Skeleton
Project
Create User
Create Instance
Initialize API
Create Database
Test Connection
Cleanup

Create Instance
to/jo14db
39
Flavor flavor = Iterables.getFirst(flavorApi.list(), null);
…
Instance instance = instanceApi.create(flavorId, size, name);
flavor.getId()
Volume size in
GB

Create Instance
to/jo14db
40
Instance updatedInstance = awaitAvailable(instance, instanceApi);

Create Instance
to/jo14db
41
Polls status, waits until ready. Will not retry by
itself!

Actual Code
to/jo14db
42
InstanceApi instanceApi = api.getInstanceApiForZone(zone);
for (int retries = 0; retries < 10; retries++) {
Instance instance = null;
try {
instance = instanceApi.create(flavorId, size, name);
} catch (Exception e) {
Uninterruptibles.sleepUninterruptibly(15, TimeUnit.SECONDS);
logger.error(Arrays.toString(e.getStackTrace()));
continue;
}
if (updatedInstance != null) {
return updatedInstance;
}
instanceApi.delete(instance.getId());
InstancePredicates.awaitDeleted(instanceApi).apply(instance);
}
return null;

Actual Code
to/jo14db
43
Utils.
getWorkingInstance(
String zone,
String name,
String flavorId,
int size)

Java Project
to/jo14db
44
Skeleton
Project
Create User
Create Instance
Initialize API
Create Database
Test Connection
Cleanup

Create Database
to/jo14db
45
troveApi = ContextBuilder.newBuilder(PROVIDER)
.buildApi(TroveApi.class);
instanceApi = troveApi.getInstanceApiForZone(ZONE);
databaseApi = troveApi
.getDatabaseApiForZoneAndInstance(ZONE,getInstance().getId());
getSomeApiForXandYandZ – fairly common in jclouds

Create Database
to/jo14db
46
boolean result = databaseApi.create(NAME);

Java Project
to/jo14db
47
Skeleton
Project
Create User
Create Instance
Initialize API
Create Database
Test Connection
Cleanup

Create User
to/jo14db
48
userApi = troveApi
.getUserApiForZoneAndInstance(ZONE, instance.getId());
boolean result = userApi.create(UNAME, PASSWORD, DBNAME);
User name
User password
Database
name

Java Project
to/jo14db
49
Skeleton
Project
Create User
Create Instance
Initialize API
Create Database
Test Connection
Cleanup

Test Connection
to/jo14db
50
Load balancer needed!
CreateLoadBalancer createLB = CreateLoadBalancer.
builder()
.name(NAME)
.protocol("MYSQL")
.port(3306)
.algorithm(LoadBalancer.Algorithm.RANDOM)
.nodes(addNodes)
.virtualIPType(VirtualIP.Type.PUBLIC)
.build();

Test Connection
to/jo14db
51
loadBalancer = lbApi.create(createLB);

Test Connection
to/jo14db
52
StringBuilder connString = new StringBuilder();
connString.append("jdbc:mysql://");
connString
.append(getVirtualIPv4(getLb().getVirtualIPs()));
connString.append("/");
connString.append(DBNAME); connString.append("?
user=");
connString.append(UNAME);
connString.append("&password=");
connString.append(PASSWORD);

Test Connection
to/jo14db
53
Connection conn =
DriverManager.getConnection(connString.toString());

Test Connection
to/jo14db
54
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT 3+5");
rs.first();
System.out.format(" 3+5 is %s%n", rs.getInt(1));

Going forward
to/jo14db
55
➢ jclouds examples
➢ https://github.com/jclouds/jclouds-examples
➢ More docs
➢ http://jclouds.incubator.apache.org/documentation/
➢ http://javadocs.jclouds.cloudbees.net/
➢ Maven alternatives?
➢ http://jclouds.incubator.apache.
org/documentation/userguide/installation-guide/
➢ Contribute!
➢ https://wiki.apache.org/jclouds/How%20to%
20Contribute

Java Project
to/jo14db
56
➢ Linux
➢ Compile
➢ javac -classpath "lib/*:src/main/java/:src/main/resources/"
src/main/java/org/jclouds/examples/rackspace/*.java
➢ Run
➢ java -classpath "lib/*:src/main/java/:src/main/resources/" org.jclouds.examples.
rackspace.clouddatabases.CreateInstance username apikey

Java Project
to/jo14db
57
➢ Windows
➢ Compile
➢ javac -classpath "lib/*;src/main/java/;src/main/resources/"
src/main/java/org/jclouds/examples/rackspace/*.java
➢ Run
➢ java -classpath "lib/*;src/main/java/;src/main/resources/" org.jclouds.examples.
rackspace.clouddatabases.CreateInstance username apikey

Going forward
to/jo14db
58
➢ TroveApi
➢ Backup Extension
➢ Settings Extension
➢ Clustering
➢ Abstraction layer

Going forward
to/jo14db
59
➢ TroveApi
Rackspac
e
Microsof
t
Amazo
n
CouchDB
Salesforc
e

Going forward
to/jo14db
60
➢ TroveApi
Rackspac
e
Microsof
t
Amazo
n
CouchDB
Salesforc
e

Going forward
to/jo14db
61
Collaborate!
http://jclouds.apache.org/
http://jclouds.apache.org/community/

Implementing an API
to/jo14db
62
Fork <link to my fork>
Alternatively, fork jclouds-labs-openstack (for
openstack).

Implementing an API
to/jo14db
63
On to the IDE

to/jo14db
64
Thank you!
http://developer.rackspace.com
sdk-support@rackspace.com
Zack Shoylev
Software Developer
zacksh #jclouds
@zackshoylev

JavaOne 2014: Taming the Cloud Database with jclouds

More Related Content

What's hot (19)

Viewers also liked (15)

Similar to JavaOne 2014: Taming the Cloud Database with jclouds (20)

Recently uploaded (20)

JavaOne 2014: Taming the Cloud Database with jclouds

Editor's Notes