Spec: a lisp-flavoured type system
3 likes702 views
The document discusses Clojure's type system, clojure.spec, which enhances automatic validation, error reporting, and generative testing to aid computation and facilitate data manipulation. It also explores a functional approach to parsing context-free grammars using Brzozowski's derivative method, emphasizing its challenges and performance issues, while advocating for laziness and memoization to improve efficiency. Lastly, it addresses the philosophies of building systems in Lisp, highlighting the importance of live programming and interactive development.
![Parsing with Derivatives
A Functional Pearl
Matthew Might David Darais
University of Utah
might@cs.utah.edu, david.darais@gmail.com
Daniel Spiewak
University of Wisconsin, Milwaukee
dspiewak@uwm.edu
Abstract
We present a functional approach to parsing unrestricted context-
free grammars based on Brzozowski’s derivative of regular expres-
sions. If we consider context-free grammars as recursive regular ex-
pressions, Brzozowski’s equational theory extends without modifi-
cation to context-free grammars (and it generalizes to parser combi-
nators). The supporting actors in this story are three concepts famil-
iar to functional programmers—laziness, memoization and fixed
points; these allow Brzozowski’s original equations to be translit-
erated into purely functional code in about 30 lines spread over
three functions.
Yet, this almost impossibly brief implementation has a draw-
back: its performance is sour—in both theory and practice. The
culprit? Each derivative can double the size of a grammar, and with
it, the cost of the next derivative.
Fortunately, much of the new structure inflicted by the derivative
is either dead on arrival, or it dies after the very next derivative.
To eliminate it, we once again exploit laziness and memoization
The derivative of regular expressions [1], if gently tempered
with laziness, memoization and fixed points, acts immediately
as a pure, functional technique for generating parse forests from
arbitrary context-free grammars. Despite—even because of—its
simplicity, the derivative transparently handles ambiguity, left-
recursion, right-recursion, ill-founded recursion or any combina-
tion thereof.
1.1 Outline
• After a review of formal languages, we introduce Brzozowski’s
derivative for regular languages. A brief implementation high-
lights its rugged elegance.
• As our implementation of the derivative engages context-free
languages, non-termination emerges as a problem.
• Three small, surgical modifications to the implementation (but
not the theory)—laziness, memoization and fixed points—
guarantee termination. Termination means the derivative can](https://image.slidesharecdn.com/spec-a-lisp-flavoured-type-system-170503161704/85/Spec-a-lisp-flavoured-type-system-7-320.jpg)
![System paradigmLanguage paradigm
infoq.com/presentations/Mixin-based-Inheritance
realworldclojure.com/the-system-paradigm
[Language:] a formal system of signs governed by
grammatical rules of combination to communicate
meaning
[System:] a set of interacting or interdependent
components forming an integrated whole](https://image.slidesharecdn.com/spec-a-lisp-flavoured-type-system-170503161704/85/Spec-a-lisp-flavoured-type-system-12-320.jpg)
Spec: a lisp-flavoured type system
- 2. Clojure at a glance • (lisp (running-on :JVM)) • Functional, dynamic, immutable • Excellent concurrency and state management support • Unparalleled data manipulation
- 4. Motivation • Communication (docs are not enough) • A lot of computation in Clojure is encoded with (naked) data • Generative (property based) testing • Manual parsing and error reporting is tedious and error prone
- 6. Writing a spec should enable automatic: • Validation • Error reporting • Destructuring • Instrumentation • Test-data generation • Generative test generation *http://clojure.org/about/spec
- 7. Parsing with Derivatives A Functional Pearl Matthew Might David Darais University of Utah [email protected], [email protected] Daniel Spiewak University of Wisconsin, Milwaukee [email protected] Abstract We present a functional approach to parsing unrestricted context- free grammars based on Brzozowski’s derivative of regular expres- sions. If we consider context-free grammars as recursive regular ex- pressions, Brzozowski’s equational theory extends without modifi- cation to context-free grammars (and it generalizes to parser combi- nators). The supporting actors in this story are three concepts famil- iar to functional programmers—laziness, memoization and fixed points; these allow Brzozowski’s original equations to be translit- erated into purely functional code in about 30 lines spread over three functions. Yet, this almost impossibly brief implementation has a draw- back: its performance is sour—in both theory and practice. The culprit? Each derivative can double the size of a grammar, and with it, the cost of the next derivative. Fortunately, much of the new structure inflicted by the derivative is either dead on arrival, or it dies after the very next derivative. To eliminate it, we once again exploit laziness and memoization The derivative of regular expressions [1], if gently tempered with laziness, memoization and fixed points, acts immediately as a pure, functional technique for generating parse forests from arbitrary context-free grammars. Despite—even because of—its simplicity, the derivative transparently handles ambiguity, left- recursion, right-recursion, ill-founded recursion or any combina- tion thereof. 1.1 Outline • After a review of formal languages, we introduce Brzozowski’s derivative for regular languages. A brief implementation high- lights its rugged elegance. • As our implementation of the derivative engages context-free languages, non-termination emerges as a problem. • Three small, surgical modifications to the implementation (but not the theory)—laziness, memoization and fixed points— guarantee termination. Termination means the derivative can
- 8. Parsing with derivates • Brzozowski derivative u−1S = { v ∈ Σ*: uv ∈ S } the set of all rest-strings obtainable from a string in S by cutting off its prefix u • code along: blog.klipse.tech/clojure/2016/10/02/parsing-with-derivatives-regular.html
- 9. Regular expressions (for data) + arbitrary predicates + data transformations (conformers) clojure.spec
- 12. System paradigmLanguage paradigm infoq.com/presentations/Mixin-based-Inheritance realworldclojure.com/the-system-paradigm [Language:] a formal system of signs governed by grammatical rules of combination to communicate meaning [System:] a set of interacting or interdependent components forming an integrated whole
- 13. The system paradigm 1. Nibble at the problem from different directions 2. Compose partial solutions into the final solution In lisp you build systems by altering the living, running one right in front of you — R. Gabriel Pascal is for building pyramids—imposing, breathtaking, static structures built by armies pushing heavy blocks into place. Lisp is for building organisms. — A. Perlis
- 14. The Artificial Intelligence people were endeavouring to write programs that no one knew how to write. The idea that you could sit down and say: 'Well, here is my problem, here are the requirements, let me come up with a specification and now code that up' (was) completely crazy as far as the AI people were concerned' The only way to write AI programs, then (as it still is now) was by taking an exploratory approach to development. The only way to do it was to experiment. 'Let me try this, let me add that. Let me try to add this fuzzy concept, let me try to add a scheduler, let me add agendas, let me add resources, let me have resource-limitations' ... you're not constructing it like making a ton of source code and compiling it periodically, you're constructing it the way you construct a city: build some of it, it's running all the time, so it's kind of like a live programming language. — R. Gabriel
- 15. Live programming
- 16. No clear delineation between environment and work code
- 17. spec: a fully interactive à la carte type system?
- 18. “Composition is about decomposing.” — E. Normand “Good design is not about making grand plans, but about taking things apart.” — R. Hickey
- 19. Destructuring • Pull apart and name • Factor out data shape
- 20. Data macros • Recursive transformations into canonical form • Do more without code macros * juxt.pro/blog/posts/data-macros.html
- 24. Generative testing • Limitations • sequences with internal structure (time series etc.) • generic higher-order functions (e.g. map) • Uncovers numerical instabilities • Better mocking in the REPL
- 25. TDD vs. REPL-driven development vs. spec-driven development?
- 26. Everything is data (aka. the lisp way) • generation • introspection • parsable errors
- 27. Building on top of spec
- 28. Queryable data descriptions Turn spec into a graph order promo code user account age country always always alwaysmaybe
- 29. Case study: autogenerating materialised views Kafka Materialised views Events External data Automatic view generation • Event & attribute ontology • Manual (via spec) • Inferred • Statistical analysis (seasonality detection, outlier removal, …) Onyx Onyx Onyx
- 30. Automatic view generation 1. Walk spec registry 2. Apply rules 1. Define new view (spec) 2. Trigger Onyx job that creates the view ⤾
- 31. Missing pieces github.com/arohner/spectrum — static analysis of (spec annotated) Clojure code github.com/stathissideris/spec-provider — infer spec from examples github.com/typedclojure/auto-annotation — infer spec from tests
- 32. Takeouts
- 34. Everything should be live and interactive
- 35. Blurring the line between environment and work is a powerful idea
- 36. Queryable data descriptions supercharge interactive development and are a great building block for automation