CS589 paper presentation - What is in unison? A formal specification and reference implementation of a file synchronizer

WHAT’S IN UNISON?
A FORMAL SPECIFICATION AND
REFERENCE IMPLEMENTATION
OF A FILE SYNCHRONIZER
Presentation type: paper presentation
Class: CS 589 – Domain Specific Languages
Presenter: Sergii Shmarkatiuk
Date: 5/20/2014

TARGET DOMAIN: FILE REPLICATION AND
SYNCHRONIZATION
2
File
synchronization
Replication
MirroringDeployment
Cloud storage

COMMON USE CASE: CLOUD STORAGE AND
SYNCHRONIZATION BETWEEN DEVICES
3

COMMON USE CASE: DEPLOYMENT OF
WEB-APPLICATION
4
development instance production instance
http://localhost/MyApp http://myapp.com
3. upload
2. test
ftp://@myapp.com/var/www/html/
1. edit files
4. test
/var/www/html/MyApp

COMMON USE CASE: DEPLOYMENT OF
WEB-APPLICATION
5
development instance production instance
http://localhost/MyApp http://myapp.com
4. test
/var/www/html/MyApp ftp://@myapp.com/var/www/html/
1. edit files
5. test
2. edit files
3. sync
+

FILE SYNCHRONIZATION SOFTWARE
6
Rsync
Beyond Compare
Synchronize It!
Dropbox
Google Drive
Box.net
OneDrive
Manual synchronization
Cloud storage

SOFTWARE DEPLOYMENT TOOLS
7
Puppet
Chef
Capistrano
CFEngine
Ansible

UNISON
File synchronization tool
Command line interface
Implemented in OCaml (DSL) and C
(tool)
Available for all major platforms (UNIX,
Win, Mac)
Unlike rsync, Unison is not included into
basic UNIX distributions
8

PAPER CONTRIBUTIONS
Paper presents mathematical model,
DSL, mechanics and challenges of file
synchronization
Authors proved some properties of file
synchronization operations using Coq
Authors described the gap between
idealistic representation and actual tool
implementation
9

QUESTIONS
Keying Xu, Chao Peng:
What is the semantic domain of Unison? Is it a deep or
shallow embedded DSL?
10
Unison is a deep embedded DSL

UNISON: SYNTAX AND SEMANTIC DOMAIN
11
 Current states of the
replicas
 2 file trees
 Archive (last
synchronized state)
 Synchronized
replicas
 2 file trees
 Archive (last
synchronized state)
Syntax Semantic domain
sem
A B
~
A B
=

QUESTIONS
Chao Peng:
What is the basic object of Unison?
12

UNISON: BASIC DATA STRUCTURES
FILESYSTEM
13
type name = string
type contents = string
type properties = string
type fs = Dir of properties * dContents
| File of properties * contents
| Symlink of contents
| Bot
and dContents = (name * fs) list
OCaml

UPDATE DETECTION
14
type prevState = DIR
| FILE
| SYMLINK
| ABSENT
type ’a leafUpdate = LeafSame
| LeafUpdated of ’a * ’a option
OCaml

UPDATE DETECTION
15
type updateItem = Same
| Updated of updateContent * prevState
| Error
and updateContent =
UCDir of properties leafUpdate * updateChildren
| UCFile of properties leafUpdate * contents leafUpdate
| UCSymlink of contents leafUpdate
| UCAbsent
and updateChildren = (name * updateItem) list
OCaml

RECONCILIATION
16
type direction = Conflict
| LeftToRight
| RightToLeft
| Equal
type transportInstr =
Instr of updateItem * updateItem * direction
| NoInstr
| Problem
type transportInstrTree = Node of transportInstr * transp
ortInstrList
and transportInstrList = (name * transportInstrTree) list
OCaml

UNISON: BASIC OPERATIONS
17
• Comparison of two file trees
• Description of detected difference
Update detection (buildUpdates)
• Building set of transport instructions
Reconciliation (reconcile)
• Performing transport instructions
• Giving user the opportunity to verify changes
Propagation (propagate)

UNISON: SEMANTIC FUNCTIONS
FILESYSTEM OPERATIONS
18
• assoc
• lookupPath
• reverse
• mem
• fsKind
• remove
• deletions
• buildUpdates
• buildUpdateChildren

UPDATE DETECTION
19
• hasErrors
• hasErrorsChildren
• propagateErrors
• noConflictInstr
• reconcileNoConflict
• reconcileNoConflictChildren
• leafDirection
• combineDirections
• uassoc
• reconcile
• reconcileChildren
• reconcileLeft

RECONCILIATION
20
• leafApply
• updateArchive
• updateArchiveChildren
• emptySource
• copyRec
• copyChildren
• copy
• checkNoUpdates
• replaceRec
• replace

PROPAGATION
21
• performInstrLeaf
• newProps
• unchangedProps
• performInstrDir
• performInstr
• propagateLocally
• propagate
• propagateInChildren

QUESTIONS
Keying Xu:
How do authors deal with modeling gap between the
reference implementation and the specification?
22
Authors describe „modeling gap‟ limitations in
their paper

UNISON: THE “MODELING GAP”
23
 Functional program (Ocaml)
 Returns new replicas without
changing content
 Written as if it “owns”
filesystems
 Regards filesystems as simple,
mathematical tree structures
 Assumes that all operations
can be implemented
111111111111
 Treats archive as full-blown
filesystem
 Imperative program (C)
 Modifies real filesystems in-
place
 Runs with live filesystems
123123
 Operates on real
implementations of filesystems
(POSIX, NTFS, …)
 Deals with operations that might
be impossible to implement
 Stores just a fingerprint of each
file‟s contents
Reference implementation
(DSL)
Real implementation (software
tool)

QUESTIONS
Brent Carmer:
What is the connection between Unison and your DSL?
Panini Patapanchala:
Relation with the version control and what aspects you
can take from this paper.
24
• SCMF-DSL also uses concept of replication
• SCMF-DSL also operates with file trees
• SCMF-DSL also detects file changes to perform such
automatic actions as version numbering

UNISON VS SCMF-DSL
25
Unison
SCMF-
DSL
Deployment
Replication Version control
Version numberingConflicts

UNISON VS SCMF-DSL
26
 Takes into account only latest
synchronized state
 Allows synchronization only
between latest states
 Treats both file trees as equal
sources of changes (everything
is writable)11111111111
 There is no defined direction for
replication 1111111111111111111
 Operates with file trees
 Replicates contents of
filesystem
 Might generate incompatible
changes (conflicts)
 Saves information about all
synchronized states
 Allows to roll back to previous
states
 Treats file trees as primary and
secondary (can be writable or
read-only)
 Operates with certain direction
for replication: from primary
replica to secondary replica
 Operates with version trees
 Replicates contents of version
control system
 Does not generate incompatible
changes
Unison SCMF-DSL

UNISON VS SCMF-DSL
27
A B
~
A B
=
sem
Unison: file trees
SCMF-DSL: version trees
sem
1
x
2
3
57
8
9
11
x
4
6
x
10
1
x
2
3
57
8
9
11
x
4
6
x
10
12

UNISON VS SCMF-DSL
28
A B
~
A B
=
sem
Unison: file trees, 2 platforms (A, B)
SCMF-DSL: version trees, N platforms (P1, P2, … PN)
sem
1
x
2
3
57
8
9
11
x
4
6
x
10
1
x
2
3
57
8
9
11
x
4
6
x
10
12

QUESTIONS
Brent Carmer:
Does the user ever construct things using the types
listed in the reference implementation?
29
NO
User uses real implementation (tool) instead of
reference implementation (DSL)

QUESTIONS
Amin Alipour:
How can they make sure that function synch is run
atomically?
Brent Carmer:
How do they use Coq to verify their reference
implementation?
Maximal runs are unique can you justify the theorem with an
example.
30

QUESTIONS
31
Authors use Coq to prove following properties of their DSL:
• Laziness is safe (replication with itself is safe )
• Mirroring is a special case (replication with previously
synchronized state o and replica a gives replica a)
• Maximal runs are unique (it is impossible to generate
two different synchronizations on the same two replicas
a and b)
• Success in the absence of conflicts (if replication does
not generate conflicts first time, it won‟t generate
conflicts next time as well)

QUESTIONS
Amin Alipour:
The paper assumes that there are only two replica of filesystem's.
Is that right? If so, how it can scale synchronization to more than
two replicas?
Rui Qin:
How about more than two replicas, does it also work?
I feel the paper explained the base cases for reconciliation and
presently the more important problems are the one presented in
future scope like multi-replica synchronization for more number of
replicas.
32
NO
Unison works only with pairs of replicas

QUESTIONS
Amin Alipour:
What is the relation of conflict as described in the paper and
merge in git?
Chao Peng:
Is Unison easily extensible? can you conclude some low-
level or high-level aspects of Unison?
33

QUESTIONS
The buildupdate of the implementation is a bit like
imperative implementation than functional.
34
This is partly because of mixed nature of OCaml language -
it incorporates functional, imperative and
object-oriented paradigms

CS589 paper presentation - What is in unison? A formal specification and reference implementation of a file synchronizer

More Related Content

Viewers also liked (11)

Similar to CS589 paper presentation - What is in unison? A formal specification and reference implementation of a file synchronizer (20)

More from Sergii Shmarkatiuk (9)

Recently uploaded (20)

CS589 paper presentation - What is in unison? A formal specification and reference implementation of a file synchronizer

Editor's Notes