2016 07 - CloudBridge Python library (XSEDE16)

CloudBridge
A Simple Cross-Cloud Python Library
Nuwan Goonasekera, Andrew Lonie, James Taylor, Enis Afgan

Outline
● Background and motivation
● Evaluation of available options
● CloudBridge
○ Design philosophy
○ Testing philosophy
○ Structure
○ Code Samples
● Demo
○ CloudLaunch
● Using
● Contributing
● Questions

Why?
● We needed to make the Genomics Virtual Lab/Galaxy CloudMan
cloud independent
○ Original version written against Amazon’s EC2 APIs
○ Ran on OpenStack using OpenStack’s EC2 Compatibility Layer
● OpenStack EC2 compatibility layer had limitations
○ Stability issues
○ Lacked features: Cannot name instances, cannot specify root volume sizes etc.
etc.
○ Deprecated - being spun off as separate project
● Go Native! - native APIs were
○ More stable
○ Cutting edge
○ Not likely to be dropped

The Available Options
● Apache Libcloud
○ The obvious candidate
○ Supports more than 30 providers
○ Supports Python 2.5 - 3
● HTTP abstraction layers
○ e.g. Apache DeltaCloud, Dasein Cloud
○ Need to run a separate HTTP service
○ Service would abstract away the specific clouds
● Build your own abstraction
○ NIH syndrome?
○ Differences between clouds can be complicated
○ How do you get access to so many clouds anyway?

Apache Libcloud Design
● Offers a lowest-common-denominator approach
○ Pros
■ Widest possible compatibility
■ Over 30 providers
■ Supports many versions of Python
■ Basic abstraction standardised
○ Cons
■ Everything else exposed as a provider specific extension (“ex_”)
■ Lots of methods/properties are “ex_”:
● ex_list_availability_zones
● def create_node(ex_keyname, ex_userdata..)
● ex_create_security_group
● ex_create_floating_ip etc.
○ In practice, requires special-cased code, unless requirements are very modest.

Apache Libcloud Design (contd…)
● Wraps libraries at a ReST/HTTP level, instead of using native SDKs
○ Pros
■ Minimises dependencies
■ Gives control over client-side implementation
○ Cons
■ Slower to incorporate features over native SDKs
■ Significant duplication of work
● Boto is over 77,000 lines of code
● Libcloud weighing in at 250,000+

● Testing is largely provider specific
○ This is because the exposed methods tend to be provider specific
○ Does not guarantee a write-once-run-anywhere experience
○ Makes it difficult to support a large number of clouds in practice
■ How do you test against many clouds yourself?
■ How do you even get access to so many clouds?

Does not offer a way to determine provider feature sets/profiles at runtime
Makes it difficult to write cross-cloud code

Meanwhile...
Mature clouds are converging in terms of functionality and scope
Making a single, uniform interface possible

Desirable requirements
● Need a write-once, run-anywhere experience
● Should not need to special-case code on a per cloud basis
○ Cannot obtain access to clouds easily
○ Don’t have the time to test them on each
● Simple and Pythonic
Tradeoffs
● Skip on widest possible compatibility
● Fine-grained feature determination

CloudBridge
A Simple Cross-Cloud Python Library

CloudBridge Design Goals
1. Offer a uniform API irrespective of the underlying provider
a. No special casing of application code
b. Simpler code
2. Provide a set of conformance tests for all supported clouds
a. No need to test against each cloud
b. Goal of “write-once-run-anywhere”
3. Focus on mature clouds with a required minimal set of features
a. Enables 1 and 2
4. Be as thin as possible
a. Keep the code-simple and reuse existing work ⟶ greater reliability, reduced development time

CloudBridge Testing Goals
1. Write one set of tests that all provider implementations must pass
2. Make the tests a ‘conformance’ test suite
● Validate that each implementation correctly implements the CloudBridge specification
3. No provider specific testing required
● A provider which passes the test suite can be used safely by an application with no additional
provider specific testing
4. Tests must pass against real infrastructure
● Mocks used to speed-up feedback
● But same tests must pass on real infrastructure
5. Aim for 100% code coverage

Automated testing using Travis
Testing py27, py35,
pypy
On OpenStack
(DevStack) and Mock
AWS (moto)

CloudBridge Design
API revolves around 3 concepts
a. Providers
An entry point for a service, encapsulating a connection
b. Services
Exposes provider functionality/feature sets
c. Resources
A remote cloud resource, such a Machine Instance, Volume etc.

Provider Service Resource Detail

Service name list get find create
InstanceService yes yes yes yes
VolumeService yes yes yes yes
SnapshotService yes yes yes yes
ImageService yes yes yes no
NetworkService yes yes yes yes
SubnetService yes yes no yes
ObjectStoreService yes yes yes yes
KeyPairService yes yes yes yes
SecurityGroupService yes yes yes yes
InstanceTypeService yes yes yes N/A
RegionService yes yes no N/A
Service feature set
Services have consistent interfaces:
<ServiceName>.[list|get|find|create]()
Resource feature
set
Resources have a common set of basic
properties: id, name
Plus fields and methods appropriate for
that resource type

Sample code: provider setup
$ pip install cloudbridge
1. from cloudbridge.cloud.factory import CloudProviderFactory
2. from cloudbridge.cloud.factory import ProviderList
3. aws_config = {'aws_access_key': 'a_key', 'aws_secret_key': 's_key')
4. provider = CloudProviderFactory().create_provider(ProviderList.AWS, aws_config)
5. os_config = {'os_username': 'username',
'os_password': 'pwd',
'os_tenant_name': 'tenant',
'os_auth_url': 'url',
'os_region_name': 'region'}
6. provider = CloudProviderFactory().create_provider(ProviderList.OPENSTACK, os_config)
OpenStack
AWS

Sample code: launch an instance
1. kp = provider.security.key_pairs.create('cloudbridge_intro')
2. with open('cloudbridge_intro.pem', 'w') as f:
3. f.write(kp.material)
4. sg = provider.security.security_groups.create(
5. 'cloudbridge_intro', 'A security group used by CloudBridge')
6. sg.add_rule('tcp', 22, 22, '0.0.0.0/0')
7. img = provider.compute.images.get(image_id)
8. inst_type = sorted([t for t in provider.compute.instance_types.list() if t.vcpus >= 2 and t.ram >= 4],
key=lambda x: x.vcpus*x.ram)[0]
9. inst = provider.compute.instances.create(
name='CloudBridge-intro', image=img, instance_type=inst_type,
key_pair=kp, security_groups=[sg])
10. # Wait until ready
11. inst.wait_till_ready()
12. # Show instance state
13. inst.state
14. # 'running'
15. inst.public_ips
16. # [u'54.166.125.219']
Create a key pair
Create a security group
Launch an instance

cloudbridge.readthedocs.org
Get started guide
Usage guide
Contributor guide
API reference

A quick demo: CloudLaunch
CloudLaunch is a general-purpose cloud launcher
○ Multi-cloud
○ ReST API driven with Angular 2 front end
○ Built on CloudBridge and Django Rest Framework
http://beta.launch.usegalaxy.org/

Using/Contributing/Source Code
https://github.com/gvlproject/cloudbridge
http://cloudbridge.readthedocs.org/

2016 07 - CloudBridge Python library (XSEDE16)

More Related Content

What's hot (20)

Similar to 2016 07 - CloudBridge Python library (XSEDE16) (20)

More from Enis Afgan (17)

Recently uploaded (20)

2016 07 - CloudBridge Python library (XSEDE16)