ROMA User-Customizable NoSQL Database in Ruby
0 likes1,075 views
Rakuten, Inc. developed ROMA, a user-customizable NoSQL database in Ruby. ROMA provides high scalability, availability, and fault tolerance using techniques like consistent hashing and virtual nodes. It has a plug-in architecture that allows users to extend ROMA's functionality through custom commands. The plug-ins use a domain specific language for easily defining commands to perform operations on structured data stored in ROMA, such as lists and maps.
![Message Protocol
• Extended Memcached protocol over TCP
– Used between clients and ROMA nodes
• Memcached client libraries are also available.
– Without distribution concern
• User can access any node and ROMA forwards it later.
– Users can use telnet.
$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escapecharacter is ‘^]’.
set foo 0 0 3
bar
User can set/get data in ROMA STORED
with telnet like memcached
get foo
VALUE foo 0 3
bar
END
25](https://image.slidesharecdn.com/20110611-mori-v003-120123195150-phpapp02/85/ROMA-User-Customizable-NoSQL-Database-in-Ruby-25-320.jpg)
![Case Study: Commands for List Operations
# alist_insert <key> <index> <bytes> [forward]¥r¥n
# <data block>¥r¥n
#
# (STORED|NOT_STORED|SERVER_ERROR <error message>)¥r¥n
def ev_alist_insert(s)
hname, k, d, vn, nodes = calc_hash(s[1])
data = read_bytes(s[3].to_i); read_bytes(2)
return forward2(nodes[0], s, data) if nodes[0] != @nid
Method declaration ddata = @storages[hname].get(vn, k, d)
for list command v = [[], []] unless ddata
named “alist_insert
alist_insert”
alist_insert v = Marshal.load(ddata) if ddata
v[0].insert(s[2].to_i, data)
v[1].insert(s[2].to_i, Time.now.to_i)
expt = 0x7fffffff
ret = @storages[hname].set(vn, k, d, expt, Marshal.dump(v))
@stats.write_count += 1
As for lines, like this. if ret
redundant(nodes[1..-1], hname, k, d, ret[2], expt, ret[4])
User can use also send_data("STORED¥r¥n")
Telnet. end
send_data("NOT_STORED¥r¥n") unless ret
end 29](https://image.slidesharecdn.com/20110611-mori-v003-120123195150-phpapp02/85/ROMA-User-Customizable-NoSQL-Database-in-Ruby-29-320.jpg)
![DSL for Command Plug-in
• DSL enables users to simply declare commands
– Without distribution concern (data replication, data
partitioning)
– def_write_command_with_key_value
• Allows easily defining commands for storing structured data
in ROMA
• For example,
– User can declare a alist_insert command with DSL
# alist_insert <key> <index> <bytes> [forward]¥r¥n
# <data block>¥r¥n
#
# (STORED|NOT_STORED|SERVER_ERROR <error message>)¥r¥n
def_write_command_with_key_value :alist_insert, 3 do |ctx|
v = [[], []]
v = Marshal.load(ctx.stored.value) if ctx.stored
v[0].insert(ctx.argv[2].to_i, ctx.params.value
v[1].insert(ctx.argv[2].to_i, Time.now.to_i)
expt = 0x7fffffff
[0, expt, Marshal.dump(v), :write, 'STORED']
end 31](https://image.slidesharecdn.com/20110611-mori-v003-120123195150-phpapp02/85/ROMA-User-Customizable-NoSQL-Database-in-Ruby-31-320.jpg)
![Commands for Map Operations with DSL
• Another example: User can declare map_set command
with DSL
– User can store map data in ROMA as value of specified key
# map_set <key> <mapkey> <flags> <expt> <bytes>¥r¥n
# <data block>¥r¥n
#
# (STORED|NOT_STORED|SERVER_ERROR <error message>)¥r¥n
def_write_command_with_key_value :map_set, 5 do |ctx|
v = {}
v = Marshal.load(ctx.stored.value) if ctx.stored
v[ctx.argv[2]] = ctx.params.value
expt = ctx.argv[4].to_i
if expt == 0
expt = 0x7fffffff
elsif expt < 2592000
expt += Time.now.to_i
end
[0, expt, Marshal.dump(v), :write, 'STORED']
end
32](https://image.slidesharecdn.com/20110611-mori-v003-120123195150-phpapp02/85/ROMA-User-Customizable-NoSQL-Database-in-Ruby-32-320.jpg)
![DSL Mechanism
• def_write_command_with_key_value is declared as method in Ruby
def def_write_command_with_key_value(cmd, idx_of_val_len, forward = :one_line, &block)
define_method "ev_#{cmd}" do |s|
params = CommandParams.new
params.key, params.hash_name = s[1].split("¥e")
params.digest = Digest::SHA1.hexdigest(params.key).hex % @rttable.hbits
params.vn = @rttable.get_vnode_id(params.digest)
params.nodes = @rttable.search_nodes_for_write(params.vn)
params.value = read_bytes(s[idx_of_val_len].to_i)
read_bytes(2)
stored = StoredData.new
stored.vn, stored.last, stored.clk, stored.expt, stored.value =
@storages[params.hash_name].get_raw(params.vn, params.key, params.digest)
stored = nil if stored.vn == nil || Time.now.to_i > stored.expt
ctx = CommandContext.new(s, params, stored)
ret = instance_exec(ctx, &block)
if ret
redundant(ctx.params.nodes[1..-1], ctx.params.hash_name,
ctx.params.key, ctx.params.digest, ret[2], expt, ret[4])
send_data("#{msg}¥r¥n")
end
send_data("NOT_#{msg}¥r¥n") unless ret
end
33](https://image.slidesharecdn.com/20110611-mori-v003-120123195150-phpapp02/85/ROMA-User-Customizable-NoSQL-Database-in-Ruby-33-320.jpg)
ROMA User-Customizable NoSQL Database in Ruby
- 1. ROMA User-Customizable NoSQL Database in Ruby Rakuten, Inc., Rakuten Institute of Technology | Masaya Mori 1
- 2. Introduction • 森 正弥 （もり まさや） まさや） • 楽天株式会社 執行役員 • 楽天技術研究所 所長 • 職掌 Masaya Mori – 開発部署のマネジメント Twitter: @emasha – 研究開発の推進・統括 2
- 3. Rakuten Institute of Technology Strategic R&D organization for Rakuten group Concept Tokyo & NY More Than Web - Your great reality through emerging technologies - Mission Turning emerging and growing new technology seeds into new business/service opportunities to enrich the internet life (& real life) all over the world 3
- 4. Global R&D has begun. 30 in Tokyo & 10 in NY Tokyo New York 4
- 5. Issues and our R&D area The internet has been growing to be diverse, huge, complicated and high-valued. • On basis of that, we progress three following R&D area to provide solution in the near future. Power Intelligence Reality ・Distributed computing ・Knowledge mining ・Multimedia Processing ・High performance computing ・NLP / Recommender ・Ubiquitous / next UI 5
- 6. Unite, Contribution to Academia Data Ecosystem R&D symposium Post / publish Promote academic researchers to exploit Rakuten’s web public data Expected results ・share R&D with external researchers ・increase data & service awareness of people 6
- 7. Agenda • Background • Features of ROMA • Overall architecture of ROMA • Plug-in architecture and its domain specific language • Conclusion 7
- 8. Internet Service • User-driven service – Release is just a beginning. – As per user’s request, always Improve, always Advance. • Software runs on Sever side – Can change any time – Big gap between package software and server side application • Flexibility, Speed > Perfect – Lightweight Language • Ex. Ruby, Perl, Python, etc. – A.R.C. – Schema-less • CouchDB, MongoDB – Virtualization, Cloud 8
- 9. Internet technology • Advantage of Open Source – Risk of vendor rock-on – Easy to start – Collective Inteligence, Collective Development • Simple & Loose – HTTP – REST + JSON > SOAP or EBXML or EJB – MySQL, memcached • Scalability – Load balancing – Cache – Distributed Cache – Disk I/O is always too slow 9
- 10. High Performance • Importance of good performance – Amazon : 0.1s of latency -> down 1% sales – Google : 0.5s extra rending time -> drop 20% traffic – Generally, 1sec delay means .. • Down 11% Page Views • Down 7% Conversions • Down 16% Customer Satisfaction – Lost tens of millions yen by 5 minutes service down • To achieve high performance website – Reduce size of HTML, Javascript – Local Cache, Ajax – On memory solution with consistency – SSD > HDD – Less latency – GPGPU 10
- 11. High availability • 24/7 – Downtime = Lose profit • Redundancy – Power – Network – Load balancer – BGP – RAID – Replication and Backup – Data Center – Operator • BCP for disaster – Big earchquake 11
- 12. Big Data, the direction • New value from big data – Data mining – Suggestion, Recommendation – Personalization • Technology to deal with big data – Cassandra – Hadoop – Lucene + Solr Cloud – It’s referred to as NoSQL roughly. • Processing ex. MapReduce • Storing ex. KVS • Search – Answer within 0.1s from over 70 million items • BI – Can provide analyzed data to sales & marketing – Also provide tools reflecting knowledge of statics 12
- 13. Storing: NoSQL databases • Needs for NoSQL databases are on the increase in order to easily store surging data • Basic features of several NoSQL databases are – High scalability – High availability – High throughput – Other features vary by databases • ROMA – Started development with Matz in 2007 – Open sourced in Oct. 2009 • See http://github.com/roma/roma/ 13
- 14. ROMA Usage in Rakuten • ROMA is used in various Rakuten services – ROMA runs on dozens of servers – Various types of data are stored • E.g. session data, personal page view history, etc • Rakuten, Inc. – It provides many e-commerce platforms • E.g. Rakuten Ichiba, Rakuten Travel, Rakuten Books, etc – Rakuten has over 70 millions of users 14
- 15. Application-Specific Needs • Many specific needs from application-side – These specific needs come from actual service development sites • For example, users say – “How can we store structured data?” • Not value, but…. Map? List? – “How can we easily process stored data on DWH or Hadoop?” – “How can we delete null character that was stored somewhere by error?” – “How can we delete duplicate data that were accidentally stored somewhere, maybe by bugs of app?” • We want to respond to all of these needs – We are trying to solve these problems, one by one. – Thereby, we hope we focus on what really matter on site. 15
- 16. Case Study: List Operation in Rakuten Travel • “How can we easily access list data stored in ROMA?” – Request from Rakuten Travel to apply page view history using ROMA – Page view history function is useful function for users 16
- 17. Case Study: List Operation in Rakuten Travel • The application stores list data for users in database – Key: user ID, Value: a list of pages that the user viewed • In such case as memcached, to delete list data stored in NoSQL database, application… 1. Gets binary data of specified key 2. Deserializes it as list data 3. delete the list data according to user requirement 4. Serializes the list data to binary data 5. Set it to database ① get binary data “delete 2nd elm into list” req ⑤ put binary data ② deserialize binary as list data Memcached cluster ③ delete 2nd elm into list ④ serialize list data to binary 17
- 18. ROMA • User-customizable NoSQL database in Ruby • Features – Key-value model – High scalability – High availability – Fault-tolerance – Better throughput – And… • To meet application-specific needs, ROMA provides – Plug-in architecture – Domain specific language (DSL) for Plug-in • ROMA enables meeting the above need in Rakuten Travel 18
- 19. Overall Architecture • ROMA integrates several well-known techniques to achieve scalability, availability and fault-tolerance – For example, consistent hashing, virtual nodes, chain replication-like mechanism, lamport clocks, etc • ROMA node consists of 4 modules – Network IO module: Receiving data from clients and other ROMA nodes – Command exec module: Creating and executing commands – Routing module: Maintaining ring information – Storage module: Storing data Network IO module Command Execution Routing module module Storage module 19
- 20. Data Partitioning • Consistent hashing and virtual nodes – ROMA consists of several nodes that run on servers – Many virtual nodes are allocated on 1-dimensional hash space of 32-bits • Each virtual node has a 32-bits ID – To determine which ROMA node to store key in. – SHA-1 hash E.g. ROMA consists of three nodes ROMA 20
- 21. Data Accessing • In getting value of specified key from ROMA 1. User accesses to ROMA nodes 2. The node determines others that are responsible for value of specified key 3. The node gets the value from the other node 4. The node returns it to user ① E.g. ROMA consists of three nodes ② ROMA 21
- 22. Sharing routing table • Each node maintains and periodically shares routing table with others – Routing tables: range of hashs, machines, port – If several versions of routing table exist, node updates the latest version. – Lamport clocks and Merkle hash tree • ROMA node multicasts with others to share routing tables – We use multicast though being aware of scalability. – Current version of ROMA doesn’t use gossip-based protocol. E.g. ROMA consists of three nodes ROMA Sharing ring information 22
- 23. Client has cache of routing table • ROMA client enables direct accessing to data – Client has a cache of routing table. – It checks to see if routing table is updated or not every 3 sec. • In getting value of specified key, ROMA client 1. Determines nodes according to cache 2. Gets the value from the node directly E.g. ROMA consists of three nodes ROMA 23
- 24. Data Replication • Automatic data replication – Client waits until data replication finishes successfully. – If data replication failed, the data is push to asynchronous queue in node, which it will retry replication. – Eventual consistency • Each data is replicated at N nodes – N: this parameter is configured in advance E.g. ROMA consists of three nodes ROMA 24
- 25. Message Protocol • Extended Memcached protocol over TCP – Used between clients and ROMA nodes • Memcached client libraries are also available. – Without distribution concern • User can access any node and ROMA forwards it later. – Users can use telnet. $ telnet localhost 11211 Trying 127.0.0.1... Connected to localhost. Escapecharacter is ‘^]’. set foo 0 0 3 bar User can set/get data in ROMA STORED with telnet like memcached get foo VALUE foo 0 3 bar END 25
- 26. Failure Detection • Heartbeat detection – Each node multicasts periodic heartbeat with others – Heartbeat is flooded every 1 sec. • If heartbeat is missed continuously, the node is declared as failed – Failover – Removal of the node E.g. ROMA consists of three nodes ROMA Sending heartbeat 26
- 27. Plug-in Architecture • Plug-ins allow users to extend behavior of ROMA • For example, – Command plug-ins enable to change behavior of command module – Users can append user-defined commands to ROMA – Current version provides plug-ins for command module only • Plug-ins for other modules coming soon • For example, storage-plug-in. Network IO module Command Execution Command Plug-ins Routing module module Storage module 27
- 28. Case Study: Commands for List Operations • ROMA allows defining commands for list operations as plug-ins – Users can atomically access list data stored in ROMA as value ① get binary data “delete 2nd elm into list” req ⑤ put binary data ② de-serialize binary data ③ delete 2nd elm into list Memcached cluster ④ serialize list data delete 2nd elm into list data “delete 2nd elm into list” req ROMA 28
- 29. Case Study: Commands for List Operations # alist_insert <key> <index> <bytes> [forward]¥r¥n # <data block>¥r¥n # # (STORED|NOT_STORED|SERVER_ERROR <error message>)¥r¥n def ev_alist_insert(s) hname, k, d, vn, nodes = calc_hash(s[1]) data = read_bytes(s[3].to_i); read_bytes(2) return forward2(nodes[0], s, data) if nodes[0] != @nid Method declaration ddata = @storages[hname].get(vn, k, d) for list command v = [[], []] unless ddata named “alist_insert alist_insert” alist_insert v = Marshal.load(ddata) if ddata v[0].insert(s[2].to_i, data) v[1].insert(s[2].to_i, Time.now.to_i) expt = 0x7fffffff ret = @storages[hname].set(vn, k, d, expt, Marshal.dump(v)) @stats.write_count += 1 As for lines, like this. if ret redundant(nodes[1..-1], hname, k, d, ret[2], expt, ret[4]) User can use also send_data("STORED¥r¥n") Telnet. end send_data("NOT_STORED¥r¥n") unless ret end 29
- 30. Command Plug-in Mechanism • Command exec module – Loads plug-in and registers plug-in method at startup • Plug-in method has method name starting with ev_ • For example, ev_alist_insert is plug-in method • Ruby allows adding new methods to classes dynamically – In calling plug-in method 1. Receives data from network IO module and creates new command 2. Finds registered plug-in method responding to the command 3. Calls plug-in method • Uses send method provided by Ruby. Network IO module Command Execution Command Plug-ins Routing module module Storage module 30
- 31. DSL for Command Plug-in • DSL enables users to simply declare commands – Without distribution concern (data replication, data partitioning) – def_write_command_with_key_value • Allows easily defining commands for storing structured data in ROMA • For example, – User can declare a alist_insert command with DSL # alist_insert <key> <index> <bytes> [forward]¥r¥n # <data block>¥r¥n # # (STORED|NOT_STORED|SERVER_ERROR <error message>)¥r¥n def_write_command_with_key_value :alist_insert, 3 do |ctx| v = [[], []] v = Marshal.load(ctx.stored.value) if ctx.stored v[0].insert(ctx.argv[2].to_i, ctx.params.value v[1].insert(ctx.argv[2].to_i, Time.now.to_i) expt = 0x7fffffff [0, expt, Marshal.dump(v), :write, 'STORED'] end 31
- 32. Commands for Map Operations with DSL • Another example: User can declare map_set command with DSL – User can store map data in ROMA as value of specified key # map_set <key> <mapkey> <flags> <expt> <bytes>¥r¥n # <data block>¥r¥n # # (STORED|NOT_STORED|SERVER_ERROR <error message>)¥r¥n def_write_command_with_key_value :map_set, 5 do |ctx| v = {} v = Marshal.load(ctx.stored.value) if ctx.stored v[ctx.argv[2]] = ctx.params.value expt = ctx.argv[4].to_i if expt == 0 expt = 0x7fffffff elsif expt < 2592000 expt += Time.now.to_i end [0, expt, Marshal.dump(v), :write, 'STORED'] end 32
- 33. DSL Mechanism • def_write_command_with_key_value is declared as method in Ruby def def_write_command_with_key_value(cmd, idx_of_val_len, forward = :one_line, &block) define_method "ev_#{cmd}" do |s| params = CommandParams.new params.key, params.hash_name = s[1].split("¥e") params.digest = Digest::SHA1.hexdigest(params.key).hex % @rttable.hbits params.vn = @rttable.get_vnode_id(params.digest) params.nodes = @rttable.search_nodes_for_write(params.vn) params.value = read_bytes(s[idx_of_val_len].to_i) read_bytes(2) stored = StoredData.new stored.vn, stored.last, stored.clk, stored.expt, stored.value = @storages[params.hash_name].get_raw(params.vn, params.key, params.digest) stored = nil if stored.vn == nil || Time.now.to_i > stored.expt ctx = CommandContext.new(s, params, stored) ret = instance_exec(ctx, &block) if ret redundant(ctx.params.nodes[1..-1], ctx.params.hash_name, ctx.params.key, ctx.params.digest, ret[2], expt, ret[4]) send_data("#{msg}¥r¥n") end send_data("NOT_#{msg}¥r¥n") unless ret end 33
- 34. Conclusion • Background – Spread of NoSQL databases – application-specific needs • Features of ROMA – To respond to specific needs, ROMA provides user-customizable interfaces – Plug-in architecture and its domain specific language • Overall architecture of ROMA – It integrates several well-known techniques • Consistent hashing, chain replication-like mechanism, lamport clocks, etc • Plug-in architecture and DSL – Plug-ins allow enhancing behavior of ROMA easily 34
- 35. Thank you for kindly attention 35