JCache data store for Apache Gora

GSoC with Apache
JCache Data store for Apache
Gora
Kevin Ratnasekera, Software Engineer, WSO2


Software Engineer for WSO2 ( kevin@wso2.com )

Working as member of Integration technologies team

Interests for Distributed systems

Open source Fan

Not related to Google or Hazelcast.
[1] http://wso2.com

About myself


GSoC and Apache contribution.

Apache Gora project.

JCache data store for Apache Gora

JCache API.

Roadmap for Apache Gora.

Conclusion.

Agenda


How does GSoC work?

GSoC statistics for 2016 program
1,206 students
178 open source organizations
85.6% overall success rate

ASF contribution
~50 students
37 completed final evaluation
[1] https://developers.google.com/open-source/gsoc/resources/stats

Google Summer of code


175 committees managing 294 community based
projects

59 incubating podlings

Active repos for ASF
870 active repos maintained at github
314 active Apache members at github
[1] https://projects.apache.org/
[2] https://github.com/apache
[3] https://people.apache.org/committer-index.html

Apache software foundation


ASF as GSoC mentoring organization

Considering 2010-2016 statistics

Accepted students ~50 for each year

Assigned mentors ~75 for each year

One of the largest mentoring organizations
[1] www.slideshare.net/smarru/google-summer-of-code-at-apache-software-
foundation


Benefits to community.

New contributors to the project.

Long term contributors ( committers/PMC members )

New features/improvements/bug fixes to project.


Data Persistence
Abstract persistent layer for NoSQL, In memory data model, Persistence for
Big data, Object to data store, Data store specific mappings

Data Access
Abstract Datastore API, Common interface for retrieval, alteration and query,
Hide details on specific persistent data store implementation.

MapReduce support
Out of the box to run MR jobs over the Gora input data store, store results
over the output data stores ( Recently introduced Spark backend )

Apache Gora Project


Define persistent bean definition using Apache AVRO
JSON schema.

Compile the schema using Gora compiler.

Create mapping file which maps between persistent
bean to physical data store.

Configure gora.properties to reflect data store
properties.

Create data store using DataStoreFactory
[1]https://gora.apache.org/current/tutorial.html

Typical Gora usage


Writing a dataStore for Apache Gora.

Implementation for 3 Abstract classes.
DataStoreBase<K, T>
QueryBase<K, T>
ResultBase<K, T>
[1]https://cwiki.apache.org/confluence/display/GORA/Writing+a+new+DataStore
+for+Gora+HOW_TO


Limitations of Gora secret in memory store – MemStore

Static ConcurrentSkipList map restricted to single
instance per JVM, MemStore cannot be shared across
JVMs ( distributed )

Reduce latency in persistent bean creation/retrieval
from back-end database ( repetitive reads )

Caching layer irrespective backend persistent data
store implementation ( decoupled )
[1] http://events.linuxfoundation.org/sites/events/files/slides/deploying_gora_as_query_broker.pdf

The need for Cache data store


Standardize Caching API for Java platform. No more
proprietary API’s.

Common mechanism to create, access, update and
remove data from caches.

Doesn’t say anything about data distribution, network
topology and wire level protocol etc.

Implementation by different vendors,
Ehcache, Infinispan, Hazelcast

JCache API


Portability between different Vendor implementations

Developer productivity – learning curve is smaller.

Why JCache?


Fundamental differences
java.util.Map javax.cache.Cache
Key Value based API Key Value based API
Support Atomic updates Support Atomic updates
Entries don’t get Expired/Evicted Entries get Expired/Evicted
Entries stored on-heap Entries stored anywhere
Store-By-Reference Store-By-Value/ Store-by
reference
Integration with Loaders/writers
Observation with Entry Listeners
Statistics
[1] http://www.slideshare.net/DavidBrimley/jcache-its-finally-here

Fundamental differences
Fundamental differences


JCache Cache Loader/Writer

Integration with external resources.

Handles Read through and write through caching for
external resources.

Register Loader/Writer and Read/Write through enabled
at cache configuration.


JCache Cache Entry Listener

Receives events related to cache entries
( create,expiry, update, remove )

Useful in distributed caches.

Register at cache configuration.


Apache license compliance

Rich vendor specific additions such as
Asynchronous operations
Eviction
Near cache
Data distribution/partitioning exposed over vendor
specific API

Hazelcast as JCache provider


Implement cache as another data store exposing the
same data store interface

Cache data Store act as wrapper to persisting store
delegating operations

Make Persistent bean serializable.

Basic Design


Configuring persistent data store to expose over
caching data store

gora.properties

Configuration for caching data store


Creating persistent data store instances which are
exposed over the caching data store


Hazelcast as cache provider.

Maintain data beans in serialized form inside caches.

Need to preserve dirty state bytes as well as data.

Two Approaches
Using pure JAVA serialization, writing custom
serializers.

Making Persistent data beans serializable


Utf8, ByteBuffer and GenericData.Array are not in it s
serializable form

AVRO SpecificRecord class level fields instances
Either should be declared as transient or implement
serializable

Rather not depend on another 3 rd party dependency
for serialization.

Custom serialiazer have freedom get extended from
pluggable serializers from variety of methods

Pure Java Vs. Custom AVRO serializers


Pure Java Vs. Custom AVRO serializers


Caching performance heavily depend on
serialization/deserialization performance. Experiment
with different serialization methods.

Remove vendor specific Hazelcast JCache
implementation ( Eg :- Eviction policy – Not included
JCache specification ) from JCache data store.

Ability to dynamically take any JCache provider.
[1] http://blog.hazelcast.com/comparing-serialization-methods

Possible improvements

●
DistributedLogManager sample.
●
Demonstrates standalone/distributed caching for data
stores.
[1] https://issues.apache.org/jira/browse/GORA-484
[2] http://github.com/apache/gora/blob/master/gora-
tutorial/src/main/java/org/apache/gora/tutorial/log/DistributedLogManager.java
[3] http://gora.apache.org/current/tutorial.html#jcache-caching-datastore

Sample/Tutorial for JCache data store


JCache store implementation [1]

Documentation for project [2][3]
[3] http://gora.apache.org/current/gora-jcache.html

References for project


REST API exposing data store functionalities. [1]

Improve data store support.
Eg:- Apache Kudu

Different serialization frameworks other than AVRO. [2]
Eg:- Apache thrift, Protocol buffers

Different execution engine support. [3]
Eg:- Apache Flink

Roadmap for Apache Gora


Contribute to Apache Gora

Check Roadmap, Mailing lists, JIRA issues

Join Apache GSoC effort

Higher project acceptance/slot count for GSoC 2017
[1] https://issues.apache.org/jira/browse/gora
[2] http://gora.apache.org/mailing_lists.html
[3] https://developers.google.com/open-source/gsoc/timeline

Conclusion

JCache data store for Apache Gora

More Related Content

What's hot (20)

Similar to JCache data store for Apache Gora (20)

Recently uploaded (20)

JCache data store for Apache Gora