Aggregate Programming in Scala
1 like708 views
The document summarizes Roberto Casadei's presentation on scafi, a Scala-based framework for aggregate programming and distributed computational fields. Scafi includes (1) an internal DSL for the field calculus, integrated with Scala's type system, (2) an actor-based distributed platform using Akka for fault tolerance, communication, and synchronization, and (3) support for code mobility through message deserialization. The presentation evaluates scafi's achievement of key goals like field calculus support and distributed platform flexibility, while identifying opportunities for further improvement, testing, and applications.
![Spatial Computing
Space-oriented computation — Thematic areas:
A. Intensive computing
B. Computation embedded in space
C. Space computation
Computing “somewhere” [Duckham13]
• Location-related information
• Spatial constraints to communication
Space-time programming [PTRSL-15]
• Computation expressed in terms of properties
of the physical time and space in which it occurs
• Spatial abstractions
Discrete system vs. continuous space-time](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-5-320.jpg)
![The Computational Field CalculusComputing with fields
Local vs Global viewpoint
Globally, input fields are manipulated and produce output
Locally, nodes compute a “program” to globally achieve t
source destination
gradient distancegradient
<=
+
dilate
width
37
10
Mirko Viroli (Universit`a di Bologna) ISAC10 – SpatialComputing a.a.
Computational fields
• Mapping space-time to computational objects
Field calculus [ESOCC-13]
• Basis set of operators
• Minimal expressive model
• Space-time universal
Higher-order field calculus [SIP-15]
• First-class distributed functions
• Code mobility](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-6-320.jpg)
![Aggregate Programming [IEEEComp-15]
Programming the collective behaviour of aggregates
Viewpoints
• Local viewpoint (device-centric view)
• Global viewpoint (aggregate view)
Global-to-local mapping
• The code running on each device is generated from the aggregate-level
specification
Prominent approach founded on field calculus and self-org patterns
Composable self-organisation
• Self-stabilisation
• Building blocks for resilient coordination](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-7-320.jpg)
![Aggregate Programming Stack
Picture
from [IEEEComp-15]](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-8-320.jpg)
: A
def rep[A](init: A)
(fun: (A) => A): A
def branch[A](cond: => Boolean)
(th: => A)
(el: => A): A
def foldhood[A](init: => A)
(aggr: (A,A)=>A)
(expr: => A): A
def aggregate[A](f: => A): A
def sense[A](name: LSNS): A
def nbrvar[A](name: NSNS): A
def mid(): ID
def neighbour(): Option[ID]
}
}
trait MyLib { self: Constructs with Builtins =>
def nbrRange: Double = nbrvar[Double](NBR_RANGE_NAME)
def G[V: OrderingFoldable](src: Boolean, field: V,
acc: V => V, metric: => Double): V =
rep( (Double.MaxValue, field) ){
dv => mux(src) { (0.0, field) } {
minHoodPlus {
val (d, v) = nbr { (dv._1, dv._2) }
(d + metric, acc(v)) }
}
}._2
def distTo(source:Boolean): Double =
G[Double](source,0, _ + nbrRange, nbrRange)
def broadcast[V: OrderingFoldable](src:Boolean, field: V): V =
G[V](src,field, x=>x , nbrRange)
def distBetween(src:Boolean, target:Boolean): Double =
broadcast(src, distanceTo(target))
def channel(src:Boolean, target:Boolean, width:Double): Boolean =
distTo(src) + distTo(target) <= distBetween(src,target) + width
}](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-17-320.jpg)
]): SPACE[E] =
new Basic3DSpace(elems.toMap) { override val proximityThreshold = 1.1 }
}
// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA
class Demo3_AggregateProgram extends Demo3_Platform.AggregateProgram {
def hopGradient(source: Boolean): Double = {
rep(Double.PositiveInfinity){
hops => { mux(source) { 0.0 } { 1+minHood(nbr { hops }) } }
}
}
def main() = hopGradient(sense("source"))
}](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-22-320.jpg)
![References
[IEEEComp-15] Jacob Beal, Danilo Pianini, and Mirko Viroli. Aggregate Programming
for the Internet of Things. IEEE Computer, 48(9):22–30, 2015
[Duckham13] Matt Duckham. Decentralized Spatial Computing - Foundations of
Geosensor Networks. Springer, 2013.
[PTRSL-15] Jacob Beal and Mirko Viroli. Space–Time Programming. Philosophical
Transactions of the Royal Society of London A: Mathematical, Physical and Engineering
Sciences, 373(2046), 2015.
[ESOCC-13] Mirko Viroli, Ferruccio Damiani, and Jacob Beal. A calculus of
computational fields. In Carlos Canal and Massimo Villari, editors, ESOCC
Workshops, volume 393 of Communications in Computer and Information Science,
pages 114–128. Springer, 2013.
[SIP-15] Ferruccio Damiani, Mirko Viroli, Danilo Pianini, and Jacob Beal. Code mobil-
ity meets self-organisation: A higher-order calculus of computational fields. In
Formal Techniques for Distributed Objects, Components, and Systems, volume 9039 of
Lecture Notes in Computer Science, pages 113–128. Springer International Publishing,
2015.](https://image.slidesharecdn.com/scafipresentation-160318090010/85/Aggregate-Programming-in-Scala-27-320.jpg)
Aggregate Programming in Scala
- 1. Roberto Casadei Aggregate Programming in Scala: a Core Library and Actor-based Platform for Distributed Computational Fields
- 2. Presentation Summary Part I — Introduction Part II — Background: Aggregate Programming • Spatial computing and space-time programming • Computational fields and field calculus • Aggregate programming Part III — Background: Scala for Library Development • Advanced features • Advanced techniques Part IV — Development of scafi • Project • Design • Implementation Part V — Evaluation • Evaluation • Demo • Conclusion and contributions
- 3. Context
- 4. Context: development of complex artificial systems • Scenarios: pervasive computing, IoT, smart-* • Characteristics: large-scale, complex dynamics, situatedness • Challenges: unpredictability, scalability, robustness, … Inadequacy of traditional approaches • Centralisations, open-loop, fully bottom-up, device-centric … Requirements • Decentralised design • Self-adaptativeness • Balance of top-down and bottom-up problem solving Introduction
- 5. Spatial Computing Space-oriented computation — Thematic areas: A. Intensive computing B. Computation embedded in space C. Space computation Computing “somewhere” [Duckham13] • Location-related information • Spatial constraints to communication Space-time programming [PTRSL-15] • Computation expressed in terms of properties of the physical time and space in which it occurs • Spatial abstractions Discrete system vs. continuous space-time
- 6. The Computational Field CalculusComputing with fields Local vs Global viewpoint Globally, input fields are manipulated and produce output Locally, nodes compute a “program” to globally achieve t source destination gradient distancegradient <= + dilate width 37 10 Mirko Viroli (Universit`a di Bologna) ISAC10 – SpatialComputing a.a. Computational fields • Mapping space-time to computational objects Field calculus [ESOCC-13] • Basis set of operators • Minimal expressive model • Space-time universal Higher-order field calculus [SIP-15] • First-class distributed functions • Code mobility
- 7. Aggregate Programming [IEEEComp-15] Programming the collective behaviour of aggregates Viewpoints • Local viewpoint (device-centric view) • Global viewpoint (aggregate view) Global-to-local mapping • The code running on each device is generated from the aggregate-level specification Prominent approach founded on field calculus and self-org patterns Composable self-organisation • Self-stabilisation • Building blocks for resilient coordination
- 8. Aggregate Programming Stack Picture from [IEEEComp-15]
- 9. Aggregate Programming Toolchain State-of-art: Protelis + Alchemist • External DSL • Focus on simulation Technological gap: there is no aggregate programming framework aimed at the construction of real-world applications integrated in a mainstream language ➡ scafi (scala with computational fields) • Initial design and logic of the field calculus interpreter by prof. Viroli Towards a tool-chain for aggregate computing Field Calculus Protelis Xtext parsing Static analysis Interpreter Alchemist Platforms Formal foundation Programming Language Language tools Simulation Execution Properties Proto Lang Proto Sim Libraries Mirko Viroli (UNIBO) Aggregate Programming for VLS Ubicomp VLSUbicomp 2015
- 10. The Scala programming language Scala as general-purpose, multi-paradigm JVM language • Java integration • Smooth integration of OOP and FP • Expressive type system (with type inference) Scala in the real world • Increasing adoption in the industry • Used by: Twitter, Amazon, LinkedIn, Netflix, Coursera, … • Killer apps: Akka, Apache Spark, Play…
- 11. Scala for Library Development — Features • Typical FP features • First-class functions, lambdas, closures, HOFs, laziness, … • Traits and mix-in composition • Advanced typing • Self-types, abstract types, … • Generic programming support • Implicit system • Syntactic sugar
- 12. Scala for Library Development — Techniques • “Pimp-my-library” pattern • Type classes • Components and dependency injection • Cake pattern • Family polymorphism • Fluent API / Internal DSL development context("Collections") { describe("properties on lists") { List(1, 2, 2, 3, 3, 3) should contain inOrderOnly (1, 2, 3) List(1, 2, 3) should contain theSameElementsInOrderAs TreeSet(3, 2, 1) List("a","b","c") should (contain oneOf("b","d") and not contain value (7)) List(1, 2, 3) shouldBe sorted List(1, 2, 3) should have size 3 } }
- 13. scafi: Requirements and Goals Scala-based framework for aggregate programming • Project consolidation • Version control, project automation • Unit and functional tests • Internal DSL and related VM for field calculus • Correct, complete, reasonably performant • Reuse of Scala typing • Aggregate functions • Aggregate programming platform • Distributed • Code mobility • Non-functional requirements • Ease of use • Flexibility
- 14. scafi: Problem Analysis and Abstraction gap DSL — Issues • Implementation of field calculus constructs as method calls • Integration with the Scala type system Distributed platform — Issues • Fault-tolerance • Naming • Concurrency • Communication • Synchronisation
- 15. scafi: Design
- 17. Language trait Language { self: Core => trait Constructs { def nbr[A](expr: => A): A def rep[A](init: A) (fun: (A) => A): A def branch[A](cond: => Boolean) (th: => A) (el: => A): A def foldhood[A](init: => A) (aggr: (A,A)=>A) (expr: => A): A def aggregate[A](f: => A): A def sense[A](name: LSNS): A def nbrvar[A](name: NSNS): A def mid(): ID def neighbour(): Option[ID] } } trait MyLib { self: Constructs with Builtins => def nbrRange: Double = nbrvar[Double](NBR_RANGE_NAME) def G[V: OrderingFoldable](src: Boolean, field: V, acc: V => V, metric: => Double): V = rep( (Double.MaxValue, field) ){ dv => mux(src) { (0.0, field) } { minHoodPlus { val (d, v) = nbr { (dv._1, dv._2) } (d + metric, acc(v)) } } }._2 def distTo(source:Boolean): Double = G[Double](source,0, _ + nbrRange, nbrRange) def broadcast[V: OrderingFoldable](src:Boolean, field: V): V = G[V](src,field, x=>x , nbrRange) def distBetween(src:Boolean, target:Boolean): Double = broadcast(src, distanceTo(target)) def channel(src:Boolean, target:Boolean, width:Double): Boolean = distTo(src) + distTo(target) <= distBetween(src,target) + width }
- 20. scafi: three points on implementation Component-based design • Implemented via traits and family polymorphism Actor-based distributed platform • Akka (core + remoting) • Composition of reactive behaviour + trait stacking Code mobility (proof of concept) • Missing classes detected on message deserialisation trait ComputationDeviceActor extends BaseDeviceActor with BasicActorBehavior with SensingBehavior with SensorManagementBehavior with ActuatorManagementBehavior with BaseNbrManagementBehavior { … }
- 21. Example: p2p, ad-hoc net // STEP 1: CHOOSE INCARNATION import it.unibo.scafi.incarnations.{ BasicActorP2P => Platform } // STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA class Demo_AggregateProgram extends Platform.AggregateProgram { override def main(): Any = foldhood(0){_ + _}(1) } // STEP 3: DEFINE MAIN PROGRAM object Demo_MainProgram extends Platform.CmdLineMain 1) Demo_MainProgram -h 127.0.0.1 -p 9000 -e 1:2,4,5;2;3 --subsystems 127.0.0.1:9500:4:5 --program “demos.Demo_AggregateProgram” 2) Demo_MainProgram -h 127.0.0.1 -p 9500 -e 4;5:4 --program “demos.Demo_AggregateProgram”
- 22. Example: server-based, mobile spatial net (i) import it.unibo.scafi.distrib.actor.server.{SpatialPlatform => SpatialServerBasedActorPlatform} import it.unibo.scafi.incarnations.BasicAbstractActorIncarnation import it.unibo.scafi.space.{Point2D, BasicSpatialAbstraction} object Demo3_Platform extends BasicAbstractActorIncarnation with SpatialServerBasedActorPlatform with BasicSpatialAbstraction with Serializable { override val LocationSensorName: String = "LOCATION_SENSOR" override type P = Point2D override def buildNewSpace[E](elems: Iterable[(E,P)]): SPACE[E] = new Basic3DSpace(elems.toMap) { override val proximityThreshold = 1.1 } } // STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA class Demo3_AggregateProgram extends Demo3_Platform.AggregateProgram { def hopGradient(source: Boolean): Double = { rep(Double.PositiveInfinity){ hops => { mux(source) { 0.0 } { 1+minHood(nbr { hops }) } } } } def main() = hopGradient(sense("source")) }
- 23. Example: server-based, mobile spatial net (ii) // STEP 3: DEFINE MAIN PROGRAMS object Demo3_ServerMain extends Demo3_Platform.ServerCmdLineMain { override def refineSettings(s: Demo3_Platform.Settings) = { s.copy(profile = s.profile.copy( serverGuiActorProps = tm => Some(ServerGUIActor.props(Demo3_Platform, tm)) )) } } object Demo4_MainProgram extends Demo3_Platform.CmdLineMain { override def refineSettings(s: Demo3_Platform.Settings) = { s.copy(profile = s.profile.copy( devGuiActorProps = ref => Some(DevGUIActor.props(Demo3_Platform, ref)) )) } override def onDeviceStarted(dm: Demo3_Platform.DeviceManager, sys: Demo3_Platform.SystemFacade) = { dm.addSensorValue(Demo3_Platform.LocationSensorName, Point2D(dm.selfId%5,(dm.selfId/5.0).floor)) dm.addSensorValue("source", dm.selfId==4) dm.start } }
- 24. Example: server-based, mobile spatial net (iii)
- 25. Evaluation Project consolidation Field calculus support Complete Correct Higher-order FC with first-class distributed functions Distributed platform Flexibility, extension points Predefined profiles API Usability, ease of use Basic spatial computing support Code mobility
- 26. Conclusion Key goals have been achieved Of course, there is still room for improvement • General refactoring • Long-run implications of the component-based design • Code mobility — from PoC to sound implementation • Serialisation issues related to inner classes; closure cleaning; … • Actor-based platform testing • Documentation Related activities • scafi on Alchemist • scafi on Android • scafi on the cloud
- 27. References [IEEEComp-15] Jacob Beal, Danilo Pianini, and Mirko Viroli. Aggregate Programming for the Internet of Things. IEEE Computer, 48(9):22–30, 2015 [Duckham13] Matt Duckham. Decentralized Spatial Computing - Foundations of Geosensor Networks. Springer, 2013. [PTRSL-15] Jacob Beal and Mirko Viroli. Space–Time Programming. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 373(2046), 2015. [ESOCC-13] Mirko Viroli, Ferruccio Damiani, and Jacob Beal. A calculus of computational fields. In Carlos Canal and Massimo Villari, editors, ESOCC Workshops, volume 393 of Communications in Computer and Information Science, pages 114–128. Springer, 2013. [SIP-15] Ferruccio Damiani, Mirko Viroli, Danilo Pianini, and Jacob Beal. Code mobil- ity meets self-organisation: A higher-order calculus of computational fields. In Formal Techniques for Distributed Objects, Components, and Systems, volume 9039 of Lecture Notes in Computer Science, pages 113–128. Springer International Publishing, 2015.