Real-World Functional Programming @ Incubaid

About Incubaid
Functional Programming @ Incubaid
OCaml
Conclusion

Real-World Functional Programming @
Incubaid

Romain Slootmaekers Nicolas Trangez

Incubaid BVBA
{romain,nicolas}@incubaid.com
Twitter: @incubaid
Team Blog: blog.incubaid.com

Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid

About Incubaid
OCaml
Conclusion

1 About Incubaid

2 Functional Programming @ Incubaid
Why FP?
Current Projects

3 OCaml
OCaml vs. Haskell
Code Teaser

4 Conclusion


About Incubaid
OCaml
Conclusion

Outline

1 About Incubaid

Why FP?
Current Projects

3 OCaml
OCaml vs. Haskell
Code Teaser

4 Conclusion


About Incubaid
OCaml
Conclusion

Company Overview

Technology incubator
Creating & providing services to startups
Not a VC, collaboration with external parties
Main activities: Datacenter & Cloud Computing


About Incubaid
OCaml
Conclusion

History

Dedigate (2000)
acquired by Terremark (2005)
Hostbasket (2000)
acquired by Telenet (2008)
DataCenter Technologies (2002)
acquired by Veritas/Symantec (2005)
Q-Layer (2007)
acquired by Sun Microsystems (2009)


About Incubaid
Functional Programming @ Incubaid Why FP?
OCaml Current Projects
Conclusion

Outline

1 About Incubaid

Why FP?
Current Projects

3 OCaml
OCaml vs. Haskell
Code Teaser

4 Conclusion


About Incubaid
Conclusion

Development Beneﬁts

Code quality
Correctness
Easy refactoring
Performance
Developer performance (Getting Things Done)
Developer satisfaction
Developer quality (hiring!)
Not the average “Enterprise Developer” ;-)
Mathematical background or interests
Good at logical, conceptual reasoning
Smaller prototype-to-production cycle


About Incubaid
Conclusion

Ideal ﬁt for our Application Domains

Handling large opaque data blobs (DSS)
System & application automation (SSO)
Highly scalable distributed systems
Large scale implies constant random failure
Transactions? Where, why, how?
Concurrency
Immutability of data, purity of operations, idempotent
actions,. . . allow to reason about systems and their
behaviour


About Incubaid
Conclusion

FP: more than languages

Note
Functional Programming is not only about languages. Insight in
FP concepts and techniques can bring radical improvements to
the architecture and design of software, especially in
large-scale, highly-unreliable environments.


About Incubaid
Conclusion

Amplidata DSS

Next-generation large object storage
Redundant storage of petabytes of big ﬁles
Dispersed storage using rateless codes
Initial version: C++
Current version: OCaml (90%), C (9%), Asm (1%)
More info @ amplidata.com


About Incubaid
Conclusion

Amplidata DSS (Cont.)
Math

2 a ... a0,n 3 b0
2 3
0,0
6 a1,0 ... a1,n 7 2 3 6 b1 7
7 x0
6. . . . . . . . . . . . . . . . . . .7 6 6 ... 7
6 6 7
x 7
6 a
6 n,0 ... an,n 7 6 1 7 = 6 bn 7
7 . . .5
74 6 7
6 an+1,0 ... an+1,n 7 6 bn+1 7
6 6 7
5 xn 4 ... 5
...................
4
an+m,0 ... an+m,n bn+m

Dataﬂow
Storage0

Client codec ........

StorageN


About Incubaid
Conclusion

Arakoon

Distributed, consistent, persistent key-value store
Multi-Paxos implementation
TokyoCabinet backend
OCaml
Open Source, see http://www.arakoon.org/

A nursery of two 3-node clusters
Cluster A A0 client0 client2 Cluster B B0 client1

A1 B1

A2 B2


About Incubaid
Conclusion

Baardskeerder

Local, in-process key-value database
Append-only B-tree-ish, persistent
Replace TokyoCabinet which can’t cope with large values
Beneﬁts of and doesn’t kill SSD drives
OCaml (future Arakoon backend)
Not publicly available yet
Follow @incubaid or blog.incubaid.com for updates


About Incubaid
Conclusion

Baardskeerder (Cont.)

Tree Construction
0: Value "A"
1: Leaf [ " a " , Outer 0 ]
2: Commit ( Outer 1 )
a b c 3: Value "B"
4: Leaf [ " a " , Outer 0 ; " b " , Outer 3 ]
6: Value "C"
A B C 7: Leaf [ " a " , Outer 0 ; " b " , Outer 3 ; " c " , Outer 6 ]


About Incubaid
Functional Programming @ Incubaid OCaml vs. Haskell
OCaml Code Teaser
Conclusion

Outline

1 About Incubaid

Why FP?
Current Projects

3 OCaml
OCaml vs. Haskell
Code Teaser

4 Conclusion


About Incubaid
OCaml Code Teaser
Conclusion

Introduction

Objective Caml
Objective Categorical Abstract Machine/Meta Language
Dialect of ML
Descendents include Microsoft F#, JoCaml (, Scala)
Static typing using type inference
Eager evaluation (by default)
Emphasis on performance, both in language features as
well as standard library implementation


About Incubaid
OCaml Code Teaser
Conclusion

Timeline

1970 1980 1990 2000 2010
Caml
Ocaml
F#
ML
Miranda
Haskell
Standard ML

Prolog
Erlang
C
C++
Scala
Java
C#

1970 1980 1990 2000 2010


About Incubaid
OCaml Code Teaser
Conclusion

Compiler and Runtime

Main implementation from INRIA (.fr)
Bytecode and native targets
Very fast compilation
Straight-forward compiler backend & code generation
Easy to reason about efﬁciency, predict compiler output
If necessary (unlike GHC) native assembly can be
read/interpreted by humans
Thanks to this, we were able to debug an obscure
non-reproducible bug in Arakoon once, which turned out to
be a ﬂaw in one of the libraries we use. Details on request!


About Incubaid
OCaml Code Teaser
Conclusion

Concurrency & Parallellism

Using native/system threads (1-to-1 mapping)
No parallel runtime (only one OCaml execution thread at
all times), a reentrant runtime is WIP
Lightweight “Monadic Threads” for concurrency using Lwt,
Async,. . . See W.L.Harrison, “Cheap (But Functional)
Threads”


About Incubaid
OCaml Code Teaser
Conclusion

Applications

Ocsigen: Web application framework, parent project of
Lwt: lightweight threading
Js_of_ocaml: OCaml to JavaScript compiler
Mirage: Xen-based exokernel
Coq: Proof assistant
MLDonkey: EDonkey P2P client
Unison: File synchronization
haXe compiler: High-level multiplatform language,
compiled to JS/SWF/AS3/PHP5/C++/. . .
Coccinelle: Semantic patches


About Incubaid
OCaml Code Teaser
Conclusion

Key Differences (1/3)

OCaml evaluation is eager by default
I once defined (>>) as (>>) a b = a >>= fun _ −> b. Lost quite
some time debugging the unexpected runtime behaviour.
No overloading (cfr. Haskell type-classes)
+ : int −> int −> int
+. : float −> float −> float (notice the dot)
+ : num −> num −> num
Both structural and referential equality operators: = and ==
Optional & named arguments
Higher-order modules & functors (not the Haskell ones)


About Incubaid
OCaml Code Teaser
Conclusion


OO support
Clean semantics
Quirky syntax
Type operators / coercions: :#, :>
Polymorphic variants (union types)
# [ ‘ Number 1 ; ‘ F l o a t 3 . 1 4 ; ‘ U n i t ; ] ; ;
− : [ > ‘ F l o a t o f f l o a t | ‘ Number o f i n t | ‘ Unit ] l i s t =
[ ‘ Number 1 ; ‘ F l o a t 3 . 1 4 ; ‘ U n i t ]

No purity restrictions (cfr. IO monad)
Side-effects (incl. IO) possible everywhere
Record ﬁelds can be mutable
In-place mutable variables (refs) implemented on top of this
Strings are in-place mutable


About Incubaid
OCaml Code Teaser
Conclusion


Sugared syntax: "abc ".[1] , [| 1; 2; 3; |].(1) vs. String .get,
Array.get
Exceptions are common, also in pure code (e.g. Not_found)
Smaller “standard library” compared to Haskell Platform
Syntax extensions
Macros: (IFDEF ... THEN ... ELSE ... END)
Comprehensions: [x | x <− [1;2;3]; x<2]
Succinctness for libraries
Normal lwt.syntax
lwt x = calc_x() in
c a l c _ x ( ) >>= fun x −> lwt y = calc_y() in
c a l c _ y ( ) >>= fun y −> ...
... (* concurrent evaluation! *)
lwt a = f () and b = g () in ...


About Incubaid
OCaml Code Teaser
Conclusion

Gotcha: Value Restriction (1/4)

Deﬁnition
The “value restriction” is a way to maintain correct typing in the
presence of side-effects: the type of an expression can only be
generalized if the expression is a “syntactic value” (or
“non-expansive expression”):
a literal or identiﬁer: 3, ’ a’ , . . .
an abstraction: fun x −> 2 ∗ x, . . .
a constructor applied to a syntactic value: Some 1, . . .


About Incubaid
OCaml Code Teaser
Conclusion


Example

# l e t i = fun x −> x ; ;
v a l i : ’ a −> ’ a = <fun >
# let r = ref i ; ;

The type of i is ∀α.(α → α). Without value restriction, the type of
r would be ∀α.(α → α) ref.
Since it’d be polymorphic, it could be used as type
(bool −> bool) ref or ( int −> int) ref , so
r : = fun ( t r u e | f a l s e ) −> f a l s e ;
(! r ) 0

would both type-check but the last expression would result in a
runtime error.


About Incubaid
OCaml Code Teaser
Conclusion


Demo
Here’s what happens for real:
# let r = ref i ; ;
v a l r : ( ’ _a −> ’ _a ) r e f = { c o n t e n t s = <fun >}
# r : = fun ( t r u e | f a l s e ) −> t r u e ; ;
− : unit = ()
# r ;;
− : ( b o o l −> b o o l ) r e f = { c o n t e n t s = <fun >}
# let s = ref i ; ;
v a l s : ( ’ _a −> ’ _a ) r e f = { c o n t e n t s = <fun >}
# (! s) 0;;
− : int = 0
# s;;
− : ( i n t −> i n t ) r e f = { c o n t e n t s = <fun >}
# l e t t = l e t l = r e f [ ] i n fun e −> l : = ( e : : (! l )); (! l );;
v a l t : ’ _a −> ’ _a l i s t = <fun >
# t 1;;
− : int l i s t = [1]
# t ;;
− : i n t −> i n t l i s t = <fun >
# t 2;;
− : i n t l i s t = [ 2 ; 1]


About Incubaid
OCaml Code Teaser
Conclusion


Work-around using eta-expansion
# let id x = x ; ;
v a l i d : ’ a −> ’ a = <fun >
# ( i d 1 , i d "abc" ) ; ;
− : i n t ∗ s t r i n g = ( 1 , "abc" )
# l e t id ’ = i d i d ; ;
v a l i d ’ : ’ _a −> ’ _a = <fun >
# id ’ 1 ; ;
− : int = 1
# id ’ ; ;
− : i n t −> i n t = <fun >
# i d ’ "abc" ; ;
E r r o r : T h i s e x p r e s s i o n has type s t r i n g b u t an e x p r e s s i o n was expected of type i n t
# l e t i d ’ ’ = fun x −> ( i d i d ) x ; ;
v a l i d ’ ’ : ’ a −> ’ a = <fun >
# ( i d ’ ’ 1 , i d ’ ’ "abc" ) ; ;
− : i n t ∗ s t r i n g = ( 1 , "abc" )
# l e t f 1 = L i s t . map i d ; ;
v a l f 1 : ’ _a l i s t −> ’ _a l i s t = <fun >
# l e t f 2 = fun l −> L i s t . map i d l ; ;
v a l f 2 : ’ a l i s t −> ’ a l i s t = <fun >


About Incubaid
OCaml Code Teaser
Conclusion

Rsync algo

Implementation of the classic “rsync” algorithm
Full source @
https://github.com/Incubaid/rsync-demo
Programming “in the large”
Expose use of
OO
Modules, functors
Imperative programming, mutability
...
but not necessarily the best, most advisable approach


About Incubaid
OCaml Code Teaser
Conclusion

module type WEAK = sig
type t
v a l make : u n i t −> t
v a l from : s t r i n g −> i n t −> i n t −> t
v a l r e s e t : t −> u n i t
v a l d i g e s t : t −> i n t
v a l r o t a t e : t −> char −> char −> u n i t
v a l update : t −> s t r i n g −> i n t −> i n t −> u n i t
end

module type STRONG = sig
type t
v a l to_hex : t −> s t r i n g
v a l f i l e : s t r i n g −> t
v a l s u b s t r i n g : s t r i n g −> i n t −> i n t −> t
v a l w r i t e : out_channel −> t −> u n i t
v a l read : i n _ c h a n n e l −> t
end

module SDigest = ( s t r u c t
include D i g e s t
l e t read i c = I o . r e a d _ s t r i n g i c
l e t w r i t e oc t = I o . w r i t e _ s t r i n g oc t

end : STRONG)


About Incubaid
OCaml Code Teaser
Conclusion

open Hash
open S i g n a t u r e
module Rsync = functor (W:WEAK) −> functor (S :STRONG) −> s t r u c t
module MySig = S i g n a t u r e (W) ( S)

class d e l t a _ e m i t t e r s i g n a t u r e new_fn h a n d l e r =
l e t bs = MySig . b l o c k _ s i z e s i g n a t u r e i n
l e t b u f f e r _ s i z e = 8 ∗ bs i n
l e t b u f f e r = S t r i n g . create b u f f e r _ s i z e in
object ( s e l f )
v a l mutable _read = 0
v a l mutable _ f i r s t _ f r e e = 0
v a l mutable _ n _ f r e e = b u f f e r _ s i z e
v a l mutable _ f i r s t _ t o d o = 0
v a l mutable _ p r e v i o u s _ a c t i o n = S t a r t bs
v a l mutable _ f i n i s h e d = f a l s e
v a l mutable _weak_ok = f a l s e
v a l _weak = W. make ( )

method _examine_block b u f f e r o f f s e t l e n g t h =
l e t wd = W. d i g e s t _weak i n
match MySig . lookup_weak s i g n a t u r e wd with
| None −> None
| Some bs −>
l e t strong = S. substring b u f f e r o f f s e t length in
i f s t r o n g = MySig . b s _s t r o n g bs
then Some ( MySig . bs_index bs )
else None


About Incubaid
OCaml
Conclusion

In Retrospect

No regrets
Productivity increased, less bugs
Bugs ﬁxed quickly, features added easily
Runtime issues mainly due to improper usage or
conﬁguration (the universe is still winning!), except Lwt
bugs
Limited tooling support compared to e.g. JVM
(JMX/JConsole, remote debugging, GC tuning,. . . )
No deal-breaker, because less needed
Would be some nice projects!


About Incubaid
OCaml
Conclusion

Future

Current focus on OCaml: in general, best known by the
team
Haskell for “compilable pseudocode”, once in a while
Haskell for future projects, why not?
Introduce devs, architects,. . . of non-FP projects to the FP
concepts and strengths to increase the quality of their
work: knowledge of FP concepts changes the way you
think about programming, even in non-FP settings


About Incubaid
OCaml
Conclusion

Internships & Jobs

Interested in working on our software and building future-proof
scalable computing platforms & looking for an internship or job?
Come join us!
recruiting@incubaid.com
wehire.be


About Incubaid
OCaml
Conclusion

Questions?


Real-World Functional Programming @ Incubaid

More Related Content

Viewers also liked (20)

Similar to Real-World Functional Programming @ Incubaid (20)

Recently uploaded (20)

Real-World Functional Programming @ Incubaid