SlideShare a Scribd company logo
Mill builds in Scala 3,
a migration story
Jamie Thompson
@bishabosha (GitHub, Twitter/X)
● ex. Scala Center 2019-2024
● Engineer @ Mibex Software
Scala OSS Contributions
● Scala 2 TASTy Reader
● Scala 3 Pipelined builds
● TASTy Query
● Inline incremental compilation
● Mirror Framework
● Scala 3 enum improvements
bishabosha.github.io
About Me 📖
Learning Goals 🎯
● Process for migrating a large project
● What are expected blockers that are caused by
fundamental Scala 3 differences?
● Unexpected quirks and bugs found during migration
What is Mill? ⚙
● Build tool for Scala, Java, Kotlin
● Fast, CLI, hierarchical build
● Aggressive caching
● Introspectable tasks and modules
● Write build config in Scala
https://mill-build.org
package build
import mill._, scalalib._
object `package` extends RootModule with ScalaModule {
def scalaVersion = "2.13.15"
def ivyDeps = Agg(
ivy"com.lihaoyi::scalatags:0.12.0",
ivy"com.lihaoyi::mainargs:0.6.2"
)
object test extends ScalaTests {
def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
def testFramework = "utest.runner.Framework"
}
}
build.mill
Why Migrate to Scala 3? 🏝
● Users must use Scala 2.13 syntax in
build files (July 2024)
● Scala 2.13 is frozen except bug
fixes
● Scala 3 is >3 years old
🥱
2.13.14
🥳🏁
3.5.0
Success Criteria? 🤔
package build
import mill.*
object `package` extends RootModule:
enum DayValue derives CrossEnum:
case Monday, Tuesday, Wednesday, Thursday, Friday,
Saturday, Sunday
object day extends Cross[DayModule](DayValue.values.toSeq)
trait DayModule extends Cross.Module[DayValue]:
def myDay: DayValue = crossValue
def today(): Command[Unit] = Task.Command:
println(s"Today is $myDay")
end DayModule
end `package`
> mill 'day[Monday].today'
Today is Monday
terminal
Build.mill (Scala 3)
Demo
Project timeline 📆
● ☕ Begin coding: August 5th 2024
● ➡ Open PR with first passing tests: August 16th
● ✅ Fully passing CI: September 27th
Aug 5 2024
☕
Aug 16 ➡
Sep 27 ✅ Nov 7 2024
Oct 12-16 🔝
proj start
first test pass
full CI pass
rebase + improvements
First Steps Taken 1⃣2⃣3⃣…
1. Checkout fresh copy of main branch of github.com/com-lihaoyi/mill
2. Pick a test to run, verify it works
a. ./mill -i 'example.scalalib.basic[1-simple].local.test'
3. Change scalaVersion in build.mill
4. Try to run test again, fix compilation errors (stub if necessary) so test executes.
5. Fix any runtime errors due to stubbed implementations.
6. Iterate on another test
- val scalaVersion = "2.13.14"
+ val scalaVersion = "3.5.0"
Typical Issues
● Syntax errors
● Type inference changes/bugs
● Macro definitions + compiler plugins
● Some classes don’t get a compiler-synthetic Mirror
● Updating dependencies
● Transitive dependency problems
● Scalafix rules (no ExplicitResultTypes)
● Scalafmt quirks
Key Migration Problems 🛠
Syntax Errors ✏
- def bindDependency: Task[Dep => BoundDep] = Task.Anon { dep: Dep =>
+ def bindDependency: Task[Dep => BoundDep] = Task.Anon { (dep: Dep) =>
BoundDep((resolveCoursierDependency(): @nowarn).apply(dep), dep.force)
}
Parens on lambda args
Type Inference changes ✏
if (languageLevel.isDefined)
<language-level>{languageLevel.get}</language-level>
+ else
+ NodeSeq.Empty
inserted else branch
No else => () inserted on both branches
Implicit conversion from Unit => 💥
if (releaseTag == label) {
requests.post(
s"https://api.github.com/...",
data =
- ujson.Obj(
- "tag_name" -> releaseTag,
- "name" -> releaseTag),
headers = Seq("Authorization" -> ...)
)
}
Weird bugs 🐛
if (releaseTag == label) {
+ val jsonBlob = ujson.Obj(
+ "tag_name" -> releaseTag,
+ "name" -> releaseTag)
requests.post(
s"https://api.github.com/...",
+ data = (jsonBlob: requests.RequestBlob),
headers = Seq("Authorization" -> ...)
)
}
after
before
Implicit conversion + named args => 💥
Stubbing Macros ⏭
- import scala.reflect.macros.blackbox.Context
- def apply[T]: Discover = macro Router.applyImpl[T]
+ def apply[T]: Discover = ???
- private class Router(val ctx: Context)
- extends mainargs.Macros(ctx) {
- import c.universe._
- def applyImpl[T: WeakTypeTag]: Expr[Discover] = {
Stub macro implementation
Allow code to compile, wait for runtime crash
Stubbing Macros (cont.) ⏭
object Ctx {
- @compileTimeOnly("Target.ctx() must be used in a Task{...} block")
+ // @compileTimeOnly("Target.ctx() must be used in a Task{...} block")
@ImplicitStub
implicit def taskCtx: Ctx = ???
Stub macro implementation
Allow code to compile, wait for runtime crash
Third party macro errors ⏭
case class MillCliConfig(
@deprecated("No longer used", "Mill 0.12.0")
@arg(
hidden = true,
short = 'h',
doc = """(internal) The home directory where..."""
)
...
object MillCliConfig {
private[this] lazy val parser: ParserForClass[MillCliConfig] =
- mainargs.ParserForClass[MillCliConfig]
+ ???
com-lihaoyi/mainargs error
Crash in com-lihaoyi/mainargs
Remove compiler plugins ⏭
object MillBuildRootModule extends RootModule() with ScalaModule {
...
def lineNumberPluginClasspath: T[Agg[PathRef]] = Task {
- millProjectModule("mill-runner-linenumbers", repositoriesTask())
+ // millProjectModule("mill-runner-linenumbers", repositoriesTask())
+ Agg.empty
}
Remove linenumbers compiler plugin
Not critical, can implement later
Classes without Mirrors 🪞
package mill.scalalib
import upickle.default.{ReadWriter => RW}
trait JsonFormatters {
implicit lazy val publicationFormat: RW[coursier.core.Publication] =
upickle.default.macroRW
...
External type is not a “Generic Product”
No given instance of type
Mirror.Of[coursier.core.Publication]
Classes without Mirrors 🪞
@data(apply = false, settersCallApply = true) class Publication(
name: String,
`type`: Type,
ext: Extension,
classifier: Classifier
) {
def attributes: Attributes = Attributes(`type`, classifier)
def isEmpty: Boolean =
name.isEmpty && `type`.isEmpty && ext.isEmpty && classifier.isEmpty
final override lazy val hashCode = tuple.hashCode
}
External type is not a “Generic Product”
class Publication is not a generic
product because it is not a case class
External class in coursier
Classes without Mirrors 🪞
Serializing external classes to JSON is
necessary for caching of tasks
Choice 🧐:
a) Change uPickle to not need Mirror 🥱
b) Generate mirrors manually ✅
Best for controlled environment
Classes without Mirrors 🪞
Version 1 (manual codegen)
//> using scala 3.6.1
val cls = (
name = "",
sum = false,
hasApply = true,
modifier = "private",
ctor = Left(List("ES2015","ES2016","ES2017",...))
)
val padding = 4
// derived values
...
// GENERATED CODE BY manual_mirror_gen.sc - DO NOT EDIT
private type SingletonMirrorProxy[T <: AnyRef & Singleton] =
Mirror.SingletonProxy { val value: T }
private def genSingletonMirror[T <: AnyRef & Singleton](
ref: T
): SingletonMirrorProxy[T] = new Mirror.SingletonProxy(ref)
.asInstanceOf[SingletonMirrorProxy[T]]
private given Mirror_ES2015: SingletonMirrorProxy[ES2015.type] =
genSingletonMirror(ES2015)
private given Mirror_ES2016: SingletonMirrorProxy[ES2016.type] =
genSingletonMirror(ES2016)
private given Mirror_ES2017: SingletonMirrorProxy[ES2017.type] =
genSingletonMirror(ES2017)
private given Mirror_ES2018: SingletonMirrorProxy[ES2018.type] =
genSingletonMirror(ES2018)
...
manual_mirror_gen.sc
Singleton values (xN)
Classes without Mirrors 🪞
//> using scala 3.6.1
val cls = (
name = "Report",
sum = false,
hasApply = true,
modifier = "private",
ctor = Right(List(
"publicModules" -> "Iterable[Report.Module]",
"dest" -> "mill.PathRef"))
)
val padding = 4
...
// GENERATED CODE BY manual_mirror_gen.sc - DO NOT EDIT
private given Mirror_Report: Mirror.Product with {
final type MirroredMonoType = Report
final type MirroredType = Report
final type MirroredElemTypes =
(Iterable[Report.Module], mill.PathRef)
final type MirroredElemLabels =
("publicModules", "dest")
final def fromProduct(p: scala.Product): Report = {
val _1: Iterable[Report.Module] =
p.productElement(0).asInstanceOf[Iterable[Report.Module]]
val _2: mill.PathRef =
p.productElement(1).asInstanceOf[mill.PathRef]
Report.apply(_1,_2)
}
}
manual_mirror_gen.sc
Version 1 (manual codegen) Product type (x1)
Classes without Mirrors 🪞
//> using scala 3.6.1
val cls = (
name = "ModuleKind",
sum = true,
hasApply = true,
modifier = "private",
ctor = Right(List(
"NoModule" -> "NoModule.type",
"CommonJSModule" -> "CommonJSModule.type",
"ESModule" -> "ESModule.type"))
)
...
// GENERATED CODE BY manual_mirror_gen.sc - DO NOT EDIT
private given Mirror_ModuleKind: Mirror.Sum with {
final type MirroredMonoType = ModuleKind
final type MirroredType = ModuleKind
final type MirroredElemTypes =
(NoModule.type, CommonJSModule.type, ESModule.type)
final type MirroredElemLabels =
("NoModule", "CommonJSModule", "ESModule")
final def ordinal(p: ModuleKind): Int = {
p match {
case _: NoModule.type => 0
case _: CommonJSModule.type => 1
case _: ESModule.type => 2
}
}
}
manual_mirror_gen.sc
Version 1 (manual codegen) Sum type (x1)
Classes without Mirrors 🪞
Version 1 (manual codegen)
● Too verbose
● Fragile to change
● Worked as short-term solution
Classes without Mirrors 🪞
package mill.scalalib
import upickle.default.{ReadWriter => RW}
+import mill.api.Mirrors, Mirrors.autoMirror
trait JsonFormatters {
implicit lazy val publicationFormat: RW[coursier.core.Publication] =
upickle.default.macroRW
+ private given Root_Publication: Mirrors.Root[coursier.core.Publication] =
Mirrors.autoRoot[coursier.core.Publication]
...
External type is not a “Generic Product”
Version 2 (macros)
Much less boilerplate (1 Root per class hierarchy)
Provides Mirror.Of[T],
given a Root[R], &
T <: R
Classes without Mirrors 🪞
Version 2 (macros)
object Mirrors:
definitions
Classes without Mirrors 🪞
Version 2 (macros)
● Root[R] only stores mirrors that
the compiler won’t generate
object Mirrors:
sealed trait Root[R]:
def mirror[T <: R](key: ...): Mirror.Of[T]
inline def autoRoot[R]: Root[R] = …
Caches hierarchy of mirrors
Runtime Mirror lookup via key
Manually declare + generate Root[R]
Classes without Mirrors 🪞
Version 2 (macros)
object Mirrors:
sealed trait Root[R]:
def mirror[T <: R](key: Path[R, T]): Mirror.Of[T]
inline def autoRoot[R]: Root[R] = …
opaque type Path[R, T <: R] <: String = String
transparent inline given autoPath[R, T <: R]: Path[R, T] = ...
Proof that Root[R] stores Mirror.Of[T]
summon generated proof
Classes without Mirrors 🪞
Version 2 (macros)
object Mirrors:
sealed trait Root[R]:
def mirror[T <: R](key: Path[R, T]): Mirror.Of[T]
inline def autoRoot[R]: Root[R] = …
opaque type Path[R, T <: R] <: String = String
transparent inline given autoPath[R, T <: R]: Path[R, T] = ...
transparent inline given autoMirror[R, T <: R](using
inline r: Root[R],
inline p: Path[R, T]
): Mirror.Of[T] = ...
summon refined Mirror.Of[T]
using proof T has a mirror in Root[R]
lookup in Root[R]
Quick review 1⃣2⃣3⃣…
1. At this point, I had done enough for the test to execute
a. i.e. fixed necessary compile errors and stubbed macros/plugins.
2. Test does not pass, due to ???.
a. ./mill -i 'example.scalalib.basic[1-simple].local.test'
3. What’s next?
Mill Specific Issues
● com-lihaoyi/mainargs issues
● Discover macro
● Enclosing caller/class macros
● Applicative + Task DSL
● Mill Module plugin
● Cross Modules macro
● Script line number rewriting
● Wrapper codegen for build.mill
● Bytecode analyzer
● Custom Scala parser
Key Migration Problems 🛠
Next Steps - com-lihaoyi/mainargs
1. First ??? was stubbed com-lihaoyi/mainargs call
2. Clone com-lihaoyi/mainargs, fix problems, publishLocal
a. Didn’t handle multiple overloads of apply method.
3. Use local mainargs in build.mill, unstub call, validate
scalalib.basic[1-simple] progresses to next ???
4. Open PR to com-lihaoyi/mainargs
Next Steps - Discover macro 🔍
1. Straightforward conversion of Scala 2 macro code.
2. Problems in using com-lihaoyi/mainargs macro (for Command parameter
parsing):
a. No support for path-dependent types
b. No support for varargs
3. publishLocal fixes and open PR to com-lihaoyi/mainargs
Discover finds all the Tasks, Commands and Modules from a given Root module,
powering the CLI resolution mechanism
Next Steps - Discover macro 🔍 (cont)
c.Expr[Discover](
q"""import mill.main.TokenReaders._;
_root_.mill.define.Discover.apply2(
_root_.scala.collection.immutable.Map(..$mapping)
)"""
)
Old macro code
Duck Typing!
main.define
Discover
main
TokenReaders
dependsOn
Next Steps - EnclosingClass macro 🎁
object EnclosingClass {
implicit def generate: EnclosingClass = macro impl
def impl(c: Context): c.Tree = {
import c.universe._
q"new _root_.mill.define.EnclosingClass(this.getClass)"
}
EnclosingClass.scala (before)
Duck Typing!
Next Steps - EnclosingClass macro 🎁
object EnclosingClass {
inline given generate: EnclosingClass = ${ impl }
def impl(using Quotes): Expr[EnclosingClass] = {
import quotes.reflect.*
val cls = enclosingClass(Symbol.spliceOwner)
val this0 = This(cls)
val ref = this0.asExprOf[Any]
'{ new EnclosingClass($ref.getClass) }
}
EnclosingClass.scala (after)
Untyped
reflection
Next Steps - EnclosingClass macro 🎁
object EnclosingClass {
inline given generate: EnclosingClass = ${ impl }
def impl(using Quotes): Expr[EnclosingClass] = {
import quotes.reflect.*
val cls = enclosingClass(Symbol.spliceOwner)
val this0 = This(cls)
val ref = this0.asExprOf[Any]
'{ new EnclosingClass($ref.getClass) }
}
EnclosingClass.scala (after)
Checked cast to
typed Expr[Any]
Next Steps - EnclosingClass macro 🎁
object EnclosingClass {
inline given generate: EnclosingClass = ${ impl }
def impl(using Quotes): Expr[EnclosingClass] = {
import quotes.reflect.*
val cls = enclosingClass(Symbol.spliceOwner)
val this0 = This(cls)
val ref = this0.asExprOf[Any]
'{ new EnclosingClass($ref.getClass) }
}
EnclosingClass.scala (after)
Type safe
selection
Next Steps - Caller macro 📢
object Caller {
implicit def generate: Caller = macro impl
def impl(c: Context): c.Tree = {
import c.universe._
q"new _root_.mill.define.Caller(this)"
}
Caller.scala (before)
Duck Typing!
Again!!
Next Steps - Caller macro 📢 (cont)
Tried the same “trick” as for EnclosingClass macro, but there’s a bug
object `package` extends RootModule() {
object foo extends Module with BaseClass(using summon[Caller])
}
Scala 3 macro bug
Error: Expands to
Caller(foo.this)
Compiler Bug:
Enclosing should
be `package`.this
object Caller {
inline given generate: Caller = ${ impl }
}
Attempt 1 💥
Next Steps - Caller macro 📢 (cont)
Provide given explicitly in the enclosing module
trait Module extends Module.BaseClass {
final given millModuleCaller: Caller = Caller(this)
...
}
Attempt 2 ✅
No macros 😌
Next Steps - Applicative + Task macros
Transforms direct style Task { foo() } to traverseCtx
def qux = Task { "hello" * foo() ++ bar() }
Example (before)
def qux = MyClass.this.cachedTarget {
traverseCtx(Seq(foo,bar)) { (args, ctx) =>
Result.create {
"hello" * args(0).asInstanceOf[Int] ++ args(1).asInstanceOf[String]
}
}
}(using Enclosing("MyClass.qux"))
Example (after)
happens-before
results
Next Steps - Applicative + Task macros (cont)
Transforms direct style Task { foo() } to traverseCtx
def impl[...](c: blackbox.Context)(t: c.Expr[T]): c.Expr[M[T]] = {
q"""${c.prefix}.traverseCtx[_root_.scala.Any, ${weakTypeOf[T]}](
${exprs.toList}
){ $callback }"""
Applicative.scala (before)
Duck Typing!
Next Steps - Applicative + Task macros (cont)
def traverseCtxExpr[R: Type](caller: Expr[TraverseCtxHolder])(
args: Expr[Seq[Task[Any]]],
fn: Expr[(IndexedSeq[Any], mill.api.Ctx) => Result[R]]
)(using Quotes): Expr[Task[R]] =
'{ $caller.traverseCtx[Any, R]($args)($fn) }
Task.scala (after)
def impl[...](using Quotes)(
traverseCtx: (Expr[Seq[W[Any]]], Expr[(IndexedSeq[Any], Ctx) => Z[T]]) => Expr[M[T]],
t: Expr[Z[T]]
): Expr[M[T]]
Applicative.scala (after)
Next Steps - Applicative + Task macros (cont)
def targetResultImpl[T: c.WeakTypeTag](c: Context)(t: c.Expr[Result[T]])(
rw: c.Expr[RW[T]],
ctx: c.Expr[mill.define.Ctx]
): c.Expr[Target[T]] = {
import c.universe._
val taskIsPrivate = isPrivateTargetOption(c)
mill.moduledefs.Cacher.impl0[Target[T]](c)(
reify(
new TargetImpl[T](
Applicative.impl0[Task, T, mill.api.Ctx](c)(t.tree).splice,
ctx.splice,
rw.splice,
taskIsPrivate.splice
)
)
)
}
Task.scala (before)
Next Steps - Applicative + Task macros (cont)
def targetResultImpl[T: Type](using Quotes)(t: Expr[Result[T]])(
rw: Expr[RW[T]],
ctx: Expr[mill.define.Ctx],
caller: Expr[TraverseCtxHolder]
): Expr[Target[T]] = {
val taskIsPrivate = isPrivateTargetOption()
mill.moduledefs.Cacher.impl0[Target[T]](
'{
new TargetImpl[T](
${Applicative.impl[Task, Task, Result, T, mill.api.Ctx](traverseCtxExpr(caller), t)},
$ctx,
$rw,
$taskIsPrivate
)
}
)
}
Task.scala (after)
As close as possible
Quick review 1⃣2⃣3⃣…
1. At this point, I could pass several integration tests
a. e.g. ./mill -i 'example.fundamentals.tasks[2-primary-tasks].local'
2. These didn’t rely upon features I had stubbed or otherwise removed.
3. I opened a PR to com-lihaoyi/mill, where this test and more passed in CI.
Next Steps - mill-moduledefs plugin
● EnableScaladocAnnotation copies the scaladoc comment text of each definition to a
runtime annotation.
class EnableScaladocAnnotation extends PluginPhase {
private def cookComment(sym: Symbol, span: Span)(using Context): Unit = {
for { docCtx <- ctx.docCtx; comment <- docCtx.docstring(sym) } do {
val text = NamedArg(valueName, Literal(Constant(comment.raw))).withSpan(span)
val annot = Annotation(ScalaDocAnnot, text, span)
sym.addAnnotation(annot)
Option.when(sym.isTerm)(sym.moduleClass)
.filter(sym => sym.exists && !sym.hasAnnotation(ScalaDocAnnot))
.foreach(_.addAnnotation(annot))
}
}
override def prepareForTemplate(tree: Template)(using Context): Context = {
cookComment(tree.symbol, tree.span)
Phase 1
Next Steps - mill-moduledefs plugin
class AutoOverride extends PluginPhase {
private def isCacher(owner: Symbol)(using Context): Boolean =
owner.isClass && owner.asClass.baseClasses.exists(_ == Cacher)
override def prepareForDefDef(d: DefDef)(using Context): Context = {
val sym = d.symbol
if sym.allOverriddenSymbols.count(!_.is(Flags.Abstract)) >= 1
&& !sym.is(Flags.Override)
&& isCacher(sym.owner)
then
sym.flags = sym.flags | Flags.Override
ctx
}
Phase 2
● AutoOverride ensures that for each method in a Module, if it is an override, set the
Override flag.
Next Steps - Cross.Factory macro
object foo extends Cross[FooModule](Seq("x","y"))
Example (before)
object foo extends Cross[FooModule](new Cross.Factory[FooModule](
makeList = Seq("x","y").map { v2 =>
class FooModule_impl(using Ctx) extends FooModule {
def crossValue: String = v2
}
(classOf[FooModule_impl], ctx => new FooModule_impl(using ctx))
},
crossSegmentsList = ...
Example (after)
Next Steps - Cross.Factory macro (cont)
class FooModule_impl(using Ctx) extends FooModule {
def crossValue: String = v2
}
(classOf[FooModule_impl], ctx => new FooModule_impl(using ctx))
Code Gen
Needs params
trait SymbolModule { this: Symbol.type =>
@experimental def newClass(
parent: Symbol, name: String, parents: List[TypeRepr],
decls: Symbol => List[Symbol], selfType: Option[TypeRepr]
): Symbol
Quotes API
No param customisation
💥
Next Steps - Cross.Factory macro (cont)
trait ShimService[Q <: Quotes] {
val innerQuotes: Q
import innerQuotes.reflect.*
val Symbol: SymbolModule
trait SymbolModule { self: Symbol.type =>
def newClass(
parent: Symbol,
name: String,
parents: List[TypeRepr],
ctor: Symbol => (List[String], List[TypeRepr]),
decls: Symbol => List[Symbol],
selfType: Option[TypeRepr]
): Symbol
}
...
Shims
Call compiler internals
with casting
Next Steps - linenumbers
Mill wraps build scripts to add custom code, which offsets line numbers of code, these
need correcting in error reports and outputs such as bytecode.
● Syntax errors are caught before generated code is added
● Scala 2 linenumbers plugin runs on generated code after parsing, updating
source-files and offsets of positions.
● Scala 3 only allows plugins to run after type checking
○ Custom Zinc reporter intercepts messages from build-files, and corrects the positions.
● After type checking we can add the plugin to correct line numbers in byte code.
● Also need to rewrite expressions for sourcecode.{Line, FileName}
Next Steps - CodeSig analyzer
Mill invalidates tasks if after recompilation the bytecode is different
● Many errors in tests,
● After investigation, a lot were due to differences in type inference/scala
specialization, so could be fixed by updating the test sources with explicit types.
● One legitimate change in Scala 3 is the encoding of lambda methods, (instance vs
static method) which had to be updated in the analyzer.
Next Steps - “The Grind”
● A lot of extra errors in tests - mostly due to dependency conflicts, or changes to
error messages
● much willpower needed 🎧🤓.
Quick review 1⃣2⃣3⃣…
1. At this point, every existing test was passing in the CI!
2. But…
3. Still no scala 3 syntax allowed in build files! 😱
Next Steps - Custom Scala Parser
Status quo: customised Fastparse “Scalaparse” parser
Choice:
● Extend Scalaparse to support all of Scala 3 syntax? (stateful parsing!)
● Use Scalameta parser? (externalise maintenance)
● Reuse Scala 3’s own parser ✅
Next Steps - Custom Scala Parser (cont)
Problems using Scala 3:
● Need to load via reflection and isolated classpath
● Maintain compat with the Scalaparse implementation
Next Steps - Custom Scala Parser (cont)
trait MillScalaParser {
def splitScript(rawCode: String, fileName: String)
: Either[String, (Seq[String], Seq[String])]
def parseImportHooksWithIndices(stmts: Seq[String])
: Seq[(String, Seq[ImportTree])]
def parseObjectData(rawCode: String)
: Seq[ObjectData]
}
Lifted interface
trait ObjectData {
def obj: Snip
def name: Snip
def parent: Snip
def endMarker: Option[Snip]
def finalStat: Option[(String, Snip)]
}
ObjectData
Package decls +
expressions
Extract
imports
from
stats
Top level object
snippets
= text snippet (possibly
with position)
Next Steps - Custom Scala Parser (cont)
package build.runner
object `package` extends RootModule with BuildInfo {
object worker extends build.MillPublishScalaModule {
private[runner] def bootstrapDeps = T.task {
val moduleDep = {
val m = artifactMetadata()
s"${m.group}:${m.id}:${m.version}"
}
val nameFilter = "scala(.*)-compiler(.*)".r
Agg(moduleDep) ++ transitiveIvyDeps().collect {
case dep if nameFilter.matches(dep.name) =>
s"${dep.organization}:${dep.name}:${dep.version}"
}
}
}
def buildInfoMembers = Seq(
BuildInfo.Value(
"bootstrapDeps",
worker.bootstrapDeps().mkString(";"),
"Depedendencies used to bootstrap the scala compiler worker."
)
)
runner/package.mill
CodeGen the
artifact names
for runtime
resolution
Next Steps - Custom Scala Parser (cont)
def generateScriptSources: T[Seq[PathRef]] = Task {
val parsed = parseBuildFiles()
if (parsed.errors.nonEmpty) Result.Failure(parsed.errors.mkString("n"))
else {
...
CodeGen.generateWrappedSources(
...
T.dest,
rootModuleInfo.enclosingClasspath,
rootModuleInfo.compilerWorkerClasspath,
...
compilerWorker()
)
Result.Success(Seq(PathRef(T.dest)))
}
}
MillBuildRootModule.scala
Write classpath
of worker to
generated file
Compiler loaded
via reflection
Next Steps - Custom Scala Parser (cont)
def millCompilationUnit(source: SourceFile)(using Context): untpd.Tree = {
val parser = new OutlineParser(source) with MillParserCommon {
override def selfType(): untpd.ValDef = untpd.EmptyValDef
override def topStatSeq(outermost: Boolean): List[untpd.Tree] = {
val (_, stats) = templateStatSeq()
stats
}
}
parser.parse()
}
ScalaCompilerWorkerImpl.scala
Script body is
spliced into
object
Compiler worker parses outline of mill-dialect compilation unit,
then traverses trees to extract text snippets
Package decls
then expressions
Final review 💯
1. Finally, are we done? ✅
2. Custom scala 3 syntax is supported, with new tests
3. PR in process of being merged (piece by piece)
4. Occasional rebasing needed.
Final review 💯
bishabosha.github.io/projects
Full progress diary
Questions?

More Related Content

Similar to Scala.IO 2024: Mill builds in Scala 3, a migration story (20)

PDF
WebNet Conference 2012 - Designing complex applications using html5 and knock...
Fabio Franzini
 
PPTX
Scala Italy 2015 - Hands On ScalaJS
Alberto Paro
 
PPTX
Alberto Paro - Hands on Scala.js
Scala Italy
 
PPTX
Intro to Rails 4
Kartik Sahoo
 
PDF
SproutCore and the Future of Web Apps
Mike Subelsky
 
PDF
Dependency injection in scala
Michal Bigos
 
PPTX
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
Codemotion
 
PDF
Divide and Conquer – Microservices with Node.js
Sebastian Springer
 
PDF
In the Brain of Hans Dockter: Gradle
Skills Matter
 
ODP
Jenkins Job Builder: our experience
Timofey Turenko
 
PDF
Desiging for Modularity with Java 9
Sander Mak (@Sander_Mak)
 
ODP
Scala Reflection & Runtime MetaProgramming
Meir Maor
 
PDF
Spring Day | Spring and Scala | Eberhard Wolff
JAX London
 
PDF
Enter the gradle
Parameswari Ettiappan
 
KEY
modern module development - Ken Barber 2012 Edinburgh Puppet Camp
Puppet
 
PDF
The Naked Bundle - Tryout
Matthias Noback
 
PPTX
Rails Engine | Modular application
mirrec
 
PDF
Scala and Spring
Eberhard Wolff
 
PDF
[2015/2016] JavaScript
Ivano Malavolta
 
WebNet Conference 2012 - Designing complex applications using html5 and knock...
Fabio Franzini
 
Scala Italy 2015 - Hands On ScalaJS
Alberto Paro
 
Alberto Paro - Hands on Scala.js
Scala Italy
 
Intro to Rails 4
Kartik Sahoo
 
SproutCore and the Future of Web Apps
Mike Subelsky
 
Dependency injection in scala
Michal Bigos
 
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
Codemotion
 
Divide and Conquer – Microservices with Node.js
Sebastian Springer
 
In the Brain of Hans Dockter: Gradle
Skills Matter
 
Jenkins Job Builder: our experience
Timofey Turenko
 
Desiging for Modularity with Java 9
Sander Mak (@Sander_Mak)
 
Scala Reflection & Runtime MetaProgramming
Meir Maor
 
Spring Day | Spring and Scala | Eberhard Wolff
JAX London
 
Enter the gradle
Parameswari Ettiappan
 
modern module development - Ken Barber 2012 Edinburgh Puppet Camp
Puppet
 
The Naked Bundle - Tryout
Matthias Noback
 
Rails Engine | Modular application
mirrec
 
Scala and Spring
Eberhard Wolff
 
[2015/2016] JavaScript
Ivano Malavolta
 

Recently uploaded (20)

PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PDF
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
PDF
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Ad

Scala.IO 2024: Mill builds in Scala 3, a migration story

  • 1. Mill builds in Scala 3, a migration story Jamie Thompson @bishabosha (GitHub, Twitter/X)
  • 2. ● ex. Scala Center 2019-2024 ● Engineer @ Mibex Software Scala OSS Contributions ● Scala 2 TASTy Reader ● Scala 3 Pipelined builds ● TASTy Query ● Inline incremental compilation ● Mirror Framework ● Scala 3 enum improvements bishabosha.github.io About Me 📖
  • 3. Learning Goals 🎯 ● Process for migrating a large project ● What are expected blockers that are caused by fundamental Scala 3 differences? ● Unexpected quirks and bugs found during migration
  • 4. What is Mill? ⚙ ● Build tool for Scala, Java, Kotlin ● Fast, CLI, hierarchical build ● Aggressive caching ● Introspectable tasks and modules ● Write build config in Scala https://mill-build.org package build import mill._, scalalib._ object `package` extends RootModule with ScalaModule { def scalaVersion = "2.13.15" def ivyDeps = Agg( ivy"com.lihaoyi::scalatags:0.12.0", ivy"com.lihaoyi::mainargs:0.6.2" ) object test extends ScalaTests { def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4") def testFramework = "utest.runner.Framework" } } build.mill
  • 5. Why Migrate to Scala 3? 🏝 ● Users must use Scala 2.13 syntax in build files (July 2024) ● Scala 2.13 is frozen except bug fixes ● Scala 3 is >3 years old 🥱 2.13.14 🥳🏁 3.5.0
  • 6. Success Criteria? 🤔 package build import mill.* object `package` extends RootModule: enum DayValue derives CrossEnum: case Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday object day extends Cross[DayModule](DayValue.values.toSeq) trait DayModule extends Cross.Module[DayValue]: def myDay: DayValue = crossValue def today(): Command[Unit] = Task.Command: println(s"Today is $myDay") end DayModule end `package` > mill 'day[Monday].today' Today is Monday terminal Build.mill (Scala 3)
  • 8. Project timeline 📆 ● ☕ Begin coding: August 5th 2024 ● ➡ Open PR with first passing tests: August 16th ● ✅ Fully passing CI: September 27th Aug 5 2024 ☕ Aug 16 ➡ Sep 27 ✅ Nov 7 2024 Oct 12-16 🔝 proj start first test pass full CI pass rebase + improvements
  • 9. First Steps Taken 1⃣2⃣3⃣… 1. Checkout fresh copy of main branch of github.com/com-lihaoyi/mill 2. Pick a test to run, verify it works a. ./mill -i 'example.scalalib.basic[1-simple].local.test' 3. Change scalaVersion in build.mill 4. Try to run test again, fix compilation errors (stub if necessary) so test executes. 5. Fix any runtime errors due to stubbed implementations. 6. Iterate on another test - val scalaVersion = "2.13.14" + val scalaVersion = "3.5.0"
  • 10. Typical Issues ● Syntax errors ● Type inference changes/bugs ● Macro definitions + compiler plugins ● Some classes don’t get a compiler-synthetic Mirror ● Updating dependencies ● Transitive dependency problems ● Scalafix rules (no ExplicitResultTypes) ● Scalafmt quirks Key Migration Problems 🛠
  • 11. Syntax Errors ✏ - def bindDependency: Task[Dep => BoundDep] = Task.Anon { dep: Dep => + def bindDependency: Task[Dep => BoundDep] = Task.Anon { (dep: Dep) => BoundDep((resolveCoursierDependency(): @nowarn).apply(dep), dep.force) } Parens on lambda args
  • 12. Type Inference changes ✏ if (languageLevel.isDefined) <language-level>{languageLevel.get}</language-level> + else + NodeSeq.Empty inserted else branch No else => () inserted on both branches Implicit conversion from Unit => 💥
  • 13. if (releaseTag == label) { requests.post( s"https://api.github.com/...", data = - ujson.Obj( - "tag_name" -> releaseTag, - "name" -> releaseTag), headers = Seq("Authorization" -> ...) ) } Weird bugs 🐛 if (releaseTag == label) { + val jsonBlob = ujson.Obj( + "tag_name" -> releaseTag, + "name" -> releaseTag) requests.post( s"https://api.github.com/...", + data = (jsonBlob: requests.RequestBlob), headers = Seq("Authorization" -> ...) ) } after before Implicit conversion + named args => 💥
  • 14. Stubbing Macros ⏭ - import scala.reflect.macros.blackbox.Context - def apply[T]: Discover = macro Router.applyImpl[T] + def apply[T]: Discover = ??? - private class Router(val ctx: Context) - extends mainargs.Macros(ctx) { - import c.universe._ - def applyImpl[T: WeakTypeTag]: Expr[Discover] = { Stub macro implementation Allow code to compile, wait for runtime crash
  • 15. Stubbing Macros (cont.) ⏭ object Ctx { - @compileTimeOnly("Target.ctx() must be used in a Task{...} block") + // @compileTimeOnly("Target.ctx() must be used in a Task{...} block") @ImplicitStub implicit def taskCtx: Ctx = ??? Stub macro implementation Allow code to compile, wait for runtime crash
  • 16. Third party macro errors ⏭ case class MillCliConfig( @deprecated("No longer used", "Mill 0.12.0") @arg( hidden = true, short = 'h', doc = """(internal) The home directory where...""" ) ... object MillCliConfig { private[this] lazy val parser: ParserForClass[MillCliConfig] = - mainargs.ParserForClass[MillCliConfig] + ??? com-lihaoyi/mainargs error Crash in com-lihaoyi/mainargs
  • 17. Remove compiler plugins ⏭ object MillBuildRootModule extends RootModule() with ScalaModule { ... def lineNumberPluginClasspath: T[Agg[PathRef]] = Task { - millProjectModule("mill-runner-linenumbers", repositoriesTask()) + // millProjectModule("mill-runner-linenumbers", repositoriesTask()) + Agg.empty } Remove linenumbers compiler plugin Not critical, can implement later
  • 18. Classes without Mirrors 🪞 package mill.scalalib import upickle.default.{ReadWriter => RW} trait JsonFormatters { implicit lazy val publicationFormat: RW[coursier.core.Publication] = upickle.default.macroRW ... External type is not a “Generic Product” No given instance of type Mirror.Of[coursier.core.Publication]
  • 19. Classes without Mirrors 🪞 @data(apply = false, settersCallApply = true) class Publication( name: String, `type`: Type, ext: Extension, classifier: Classifier ) { def attributes: Attributes = Attributes(`type`, classifier) def isEmpty: Boolean = name.isEmpty && `type`.isEmpty && ext.isEmpty && classifier.isEmpty final override lazy val hashCode = tuple.hashCode } External type is not a “Generic Product” class Publication is not a generic product because it is not a case class External class in coursier
  • 20. Classes without Mirrors 🪞 Serializing external classes to JSON is necessary for caching of tasks Choice 🧐: a) Change uPickle to not need Mirror 🥱 b) Generate mirrors manually ✅ Best for controlled environment
  • 21. Classes without Mirrors 🪞 Version 1 (manual codegen) //> using scala 3.6.1 val cls = ( name = "", sum = false, hasApply = true, modifier = "private", ctor = Left(List("ES2015","ES2016","ES2017",...)) ) val padding = 4 // derived values ... // GENERATED CODE BY manual_mirror_gen.sc - DO NOT EDIT private type SingletonMirrorProxy[T <: AnyRef & Singleton] = Mirror.SingletonProxy { val value: T } private def genSingletonMirror[T <: AnyRef & Singleton]( ref: T ): SingletonMirrorProxy[T] = new Mirror.SingletonProxy(ref) .asInstanceOf[SingletonMirrorProxy[T]] private given Mirror_ES2015: SingletonMirrorProxy[ES2015.type] = genSingletonMirror(ES2015) private given Mirror_ES2016: SingletonMirrorProxy[ES2016.type] = genSingletonMirror(ES2016) private given Mirror_ES2017: SingletonMirrorProxy[ES2017.type] = genSingletonMirror(ES2017) private given Mirror_ES2018: SingletonMirrorProxy[ES2018.type] = genSingletonMirror(ES2018) ... manual_mirror_gen.sc Singleton values (xN)
  • 22. Classes without Mirrors 🪞 //> using scala 3.6.1 val cls = ( name = "Report", sum = false, hasApply = true, modifier = "private", ctor = Right(List( "publicModules" -> "Iterable[Report.Module]", "dest" -> "mill.PathRef")) ) val padding = 4 ... // GENERATED CODE BY manual_mirror_gen.sc - DO NOT EDIT private given Mirror_Report: Mirror.Product with { final type MirroredMonoType = Report final type MirroredType = Report final type MirroredElemTypes = (Iterable[Report.Module], mill.PathRef) final type MirroredElemLabels = ("publicModules", "dest") final def fromProduct(p: scala.Product): Report = { val _1: Iterable[Report.Module] = p.productElement(0).asInstanceOf[Iterable[Report.Module]] val _2: mill.PathRef = p.productElement(1).asInstanceOf[mill.PathRef] Report.apply(_1,_2) } } manual_mirror_gen.sc Version 1 (manual codegen) Product type (x1)
  • 23. Classes without Mirrors 🪞 //> using scala 3.6.1 val cls = ( name = "ModuleKind", sum = true, hasApply = true, modifier = "private", ctor = Right(List( "NoModule" -> "NoModule.type", "CommonJSModule" -> "CommonJSModule.type", "ESModule" -> "ESModule.type")) ) ... // GENERATED CODE BY manual_mirror_gen.sc - DO NOT EDIT private given Mirror_ModuleKind: Mirror.Sum with { final type MirroredMonoType = ModuleKind final type MirroredType = ModuleKind final type MirroredElemTypes = (NoModule.type, CommonJSModule.type, ESModule.type) final type MirroredElemLabels = ("NoModule", "CommonJSModule", "ESModule") final def ordinal(p: ModuleKind): Int = { p match { case _: NoModule.type => 0 case _: CommonJSModule.type => 1 case _: ESModule.type => 2 } } } manual_mirror_gen.sc Version 1 (manual codegen) Sum type (x1)
  • 24. Classes without Mirrors 🪞 Version 1 (manual codegen) ● Too verbose ● Fragile to change ● Worked as short-term solution
  • 25. Classes without Mirrors 🪞 package mill.scalalib import upickle.default.{ReadWriter => RW} +import mill.api.Mirrors, Mirrors.autoMirror trait JsonFormatters { implicit lazy val publicationFormat: RW[coursier.core.Publication] = upickle.default.macroRW + private given Root_Publication: Mirrors.Root[coursier.core.Publication] = Mirrors.autoRoot[coursier.core.Publication] ... External type is not a “Generic Product” Version 2 (macros) Much less boilerplate (1 Root per class hierarchy) Provides Mirror.Of[T], given a Root[R], & T <: R
  • 26. Classes without Mirrors 🪞 Version 2 (macros) object Mirrors: definitions
  • 27. Classes without Mirrors 🪞 Version 2 (macros) ● Root[R] only stores mirrors that the compiler won’t generate object Mirrors: sealed trait Root[R]: def mirror[T <: R](key: ...): Mirror.Of[T] inline def autoRoot[R]: Root[R] = … Caches hierarchy of mirrors Runtime Mirror lookup via key Manually declare + generate Root[R]
  • 28. Classes without Mirrors 🪞 Version 2 (macros) object Mirrors: sealed trait Root[R]: def mirror[T <: R](key: Path[R, T]): Mirror.Of[T] inline def autoRoot[R]: Root[R] = … opaque type Path[R, T <: R] <: String = String transparent inline given autoPath[R, T <: R]: Path[R, T] = ... Proof that Root[R] stores Mirror.Of[T] summon generated proof
  • 29. Classes without Mirrors 🪞 Version 2 (macros) object Mirrors: sealed trait Root[R]: def mirror[T <: R](key: Path[R, T]): Mirror.Of[T] inline def autoRoot[R]: Root[R] = … opaque type Path[R, T <: R] <: String = String transparent inline given autoPath[R, T <: R]: Path[R, T] = ... transparent inline given autoMirror[R, T <: R](using inline r: Root[R], inline p: Path[R, T] ): Mirror.Of[T] = ... summon refined Mirror.Of[T] using proof T has a mirror in Root[R] lookup in Root[R]
  • 30. Quick review 1⃣2⃣3⃣… 1. At this point, I had done enough for the test to execute a. i.e. fixed necessary compile errors and stubbed macros/plugins. 2. Test does not pass, due to ???. a. ./mill -i 'example.scalalib.basic[1-simple].local.test' 3. What’s next?
  • 31. Mill Specific Issues ● com-lihaoyi/mainargs issues ● Discover macro ● Enclosing caller/class macros ● Applicative + Task DSL ● Mill Module plugin ● Cross Modules macro ● Script line number rewriting ● Wrapper codegen for build.mill ● Bytecode analyzer ● Custom Scala parser Key Migration Problems 🛠
  • 32. Next Steps - com-lihaoyi/mainargs 1. First ??? was stubbed com-lihaoyi/mainargs call 2. Clone com-lihaoyi/mainargs, fix problems, publishLocal a. Didn’t handle multiple overloads of apply method. 3. Use local mainargs in build.mill, unstub call, validate scalalib.basic[1-simple] progresses to next ??? 4. Open PR to com-lihaoyi/mainargs
  • 33. Next Steps - Discover macro 🔍 1. Straightforward conversion of Scala 2 macro code. 2. Problems in using com-lihaoyi/mainargs macro (for Command parameter parsing): a. No support for path-dependent types b. No support for varargs 3. publishLocal fixes and open PR to com-lihaoyi/mainargs Discover finds all the Tasks, Commands and Modules from a given Root module, powering the CLI resolution mechanism
  • 34. Next Steps - Discover macro 🔍 (cont) c.Expr[Discover]( q"""import mill.main.TokenReaders._; _root_.mill.define.Discover.apply2( _root_.scala.collection.immutable.Map(..$mapping) )""" ) Old macro code Duck Typing! main.define Discover main TokenReaders dependsOn
  • 35. Next Steps - EnclosingClass macro 🎁 object EnclosingClass { implicit def generate: EnclosingClass = macro impl def impl(c: Context): c.Tree = { import c.universe._ q"new _root_.mill.define.EnclosingClass(this.getClass)" } EnclosingClass.scala (before) Duck Typing!
  • 36. Next Steps - EnclosingClass macro 🎁 object EnclosingClass { inline given generate: EnclosingClass = ${ impl } def impl(using Quotes): Expr[EnclosingClass] = { import quotes.reflect.* val cls = enclosingClass(Symbol.spliceOwner) val this0 = This(cls) val ref = this0.asExprOf[Any] '{ new EnclosingClass($ref.getClass) } } EnclosingClass.scala (after) Untyped reflection
  • 37. Next Steps - EnclosingClass macro 🎁 object EnclosingClass { inline given generate: EnclosingClass = ${ impl } def impl(using Quotes): Expr[EnclosingClass] = { import quotes.reflect.* val cls = enclosingClass(Symbol.spliceOwner) val this0 = This(cls) val ref = this0.asExprOf[Any] '{ new EnclosingClass($ref.getClass) } } EnclosingClass.scala (after) Checked cast to typed Expr[Any]
  • 38. Next Steps - EnclosingClass macro 🎁 object EnclosingClass { inline given generate: EnclosingClass = ${ impl } def impl(using Quotes): Expr[EnclosingClass] = { import quotes.reflect.* val cls = enclosingClass(Symbol.spliceOwner) val this0 = This(cls) val ref = this0.asExprOf[Any] '{ new EnclosingClass($ref.getClass) } } EnclosingClass.scala (after) Type safe selection
  • 39. Next Steps - Caller macro 📢 object Caller { implicit def generate: Caller = macro impl def impl(c: Context): c.Tree = { import c.universe._ q"new _root_.mill.define.Caller(this)" } Caller.scala (before) Duck Typing! Again!!
  • 40. Next Steps - Caller macro 📢 (cont) Tried the same “trick” as for EnclosingClass macro, but there’s a bug object `package` extends RootModule() { object foo extends Module with BaseClass(using summon[Caller]) } Scala 3 macro bug Error: Expands to Caller(foo.this) Compiler Bug: Enclosing should be `package`.this object Caller { inline given generate: Caller = ${ impl } } Attempt 1 💥
  • 41. Next Steps - Caller macro 📢 (cont) Provide given explicitly in the enclosing module trait Module extends Module.BaseClass { final given millModuleCaller: Caller = Caller(this) ... } Attempt 2 ✅ No macros 😌
  • 42. Next Steps - Applicative + Task macros Transforms direct style Task { foo() } to traverseCtx def qux = Task { "hello" * foo() ++ bar() } Example (before) def qux = MyClass.this.cachedTarget { traverseCtx(Seq(foo,bar)) { (args, ctx) => Result.create { "hello" * args(0).asInstanceOf[Int] ++ args(1).asInstanceOf[String] } } }(using Enclosing("MyClass.qux")) Example (after) happens-before results
  • 43. Next Steps - Applicative + Task macros (cont) Transforms direct style Task { foo() } to traverseCtx def impl[...](c: blackbox.Context)(t: c.Expr[T]): c.Expr[M[T]] = { q"""${c.prefix}.traverseCtx[_root_.scala.Any, ${weakTypeOf[T]}]( ${exprs.toList} ){ $callback }""" Applicative.scala (before) Duck Typing!
  • 44. Next Steps - Applicative + Task macros (cont) def traverseCtxExpr[R: Type](caller: Expr[TraverseCtxHolder])( args: Expr[Seq[Task[Any]]], fn: Expr[(IndexedSeq[Any], mill.api.Ctx) => Result[R]] )(using Quotes): Expr[Task[R]] = '{ $caller.traverseCtx[Any, R]($args)($fn) } Task.scala (after) def impl[...](using Quotes)( traverseCtx: (Expr[Seq[W[Any]]], Expr[(IndexedSeq[Any], Ctx) => Z[T]]) => Expr[M[T]], t: Expr[Z[T]] ): Expr[M[T]] Applicative.scala (after)
  • 45. Next Steps - Applicative + Task macros (cont) def targetResultImpl[T: c.WeakTypeTag](c: Context)(t: c.Expr[Result[T]])( rw: c.Expr[RW[T]], ctx: c.Expr[mill.define.Ctx] ): c.Expr[Target[T]] = { import c.universe._ val taskIsPrivate = isPrivateTargetOption(c) mill.moduledefs.Cacher.impl0[Target[T]](c)( reify( new TargetImpl[T]( Applicative.impl0[Task, T, mill.api.Ctx](c)(t.tree).splice, ctx.splice, rw.splice, taskIsPrivate.splice ) ) ) } Task.scala (before)
  • 46. Next Steps - Applicative + Task macros (cont) def targetResultImpl[T: Type](using Quotes)(t: Expr[Result[T]])( rw: Expr[RW[T]], ctx: Expr[mill.define.Ctx], caller: Expr[TraverseCtxHolder] ): Expr[Target[T]] = { val taskIsPrivate = isPrivateTargetOption() mill.moduledefs.Cacher.impl0[Target[T]]( '{ new TargetImpl[T]( ${Applicative.impl[Task, Task, Result, T, mill.api.Ctx](traverseCtxExpr(caller), t)}, $ctx, $rw, $taskIsPrivate ) } ) } Task.scala (after) As close as possible
  • 47. Quick review 1⃣2⃣3⃣… 1. At this point, I could pass several integration tests a. e.g. ./mill -i 'example.fundamentals.tasks[2-primary-tasks].local' 2. These didn’t rely upon features I had stubbed or otherwise removed. 3. I opened a PR to com-lihaoyi/mill, where this test and more passed in CI.
  • 48. Next Steps - mill-moduledefs plugin ● EnableScaladocAnnotation copies the scaladoc comment text of each definition to a runtime annotation. class EnableScaladocAnnotation extends PluginPhase { private def cookComment(sym: Symbol, span: Span)(using Context): Unit = { for { docCtx <- ctx.docCtx; comment <- docCtx.docstring(sym) } do { val text = NamedArg(valueName, Literal(Constant(comment.raw))).withSpan(span) val annot = Annotation(ScalaDocAnnot, text, span) sym.addAnnotation(annot) Option.when(sym.isTerm)(sym.moduleClass) .filter(sym => sym.exists && !sym.hasAnnotation(ScalaDocAnnot)) .foreach(_.addAnnotation(annot)) } } override def prepareForTemplate(tree: Template)(using Context): Context = { cookComment(tree.symbol, tree.span) Phase 1
  • 49. Next Steps - mill-moduledefs plugin class AutoOverride extends PluginPhase { private def isCacher(owner: Symbol)(using Context): Boolean = owner.isClass && owner.asClass.baseClasses.exists(_ == Cacher) override def prepareForDefDef(d: DefDef)(using Context): Context = { val sym = d.symbol if sym.allOverriddenSymbols.count(!_.is(Flags.Abstract)) >= 1 && !sym.is(Flags.Override) && isCacher(sym.owner) then sym.flags = sym.flags | Flags.Override ctx } Phase 2 ● AutoOverride ensures that for each method in a Module, if it is an override, set the Override flag.
  • 50. Next Steps - Cross.Factory macro object foo extends Cross[FooModule](Seq("x","y")) Example (before) object foo extends Cross[FooModule](new Cross.Factory[FooModule]( makeList = Seq("x","y").map { v2 => class FooModule_impl(using Ctx) extends FooModule { def crossValue: String = v2 } (classOf[FooModule_impl], ctx => new FooModule_impl(using ctx)) }, crossSegmentsList = ... Example (after)
  • 51. Next Steps - Cross.Factory macro (cont) class FooModule_impl(using Ctx) extends FooModule { def crossValue: String = v2 } (classOf[FooModule_impl], ctx => new FooModule_impl(using ctx)) Code Gen Needs params trait SymbolModule { this: Symbol.type => @experimental def newClass( parent: Symbol, name: String, parents: List[TypeRepr], decls: Symbol => List[Symbol], selfType: Option[TypeRepr] ): Symbol Quotes API No param customisation 💥
  • 52. Next Steps - Cross.Factory macro (cont) trait ShimService[Q <: Quotes] { val innerQuotes: Q import innerQuotes.reflect.* val Symbol: SymbolModule trait SymbolModule { self: Symbol.type => def newClass( parent: Symbol, name: String, parents: List[TypeRepr], ctor: Symbol => (List[String], List[TypeRepr]), decls: Symbol => List[Symbol], selfType: Option[TypeRepr] ): Symbol } ... Shims Call compiler internals with casting
  • 53. Next Steps - linenumbers Mill wraps build scripts to add custom code, which offsets line numbers of code, these need correcting in error reports and outputs such as bytecode. ● Syntax errors are caught before generated code is added ● Scala 2 linenumbers plugin runs on generated code after parsing, updating source-files and offsets of positions. ● Scala 3 only allows plugins to run after type checking ○ Custom Zinc reporter intercepts messages from build-files, and corrects the positions. ● After type checking we can add the plugin to correct line numbers in byte code. ● Also need to rewrite expressions for sourcecode.{Line, FileName}
  • 54. Next Steps - CodeSig analyzer Mill invalidates tasks if after recompilation the bytecode is different ● Many errors in tests, ● After investigation, a lot were due to differences in type inference/scala specialization, so could be fixed by updating the test sources with explicit types. ● One legitimate change in Scala 3 is the encoding of lambda methods, (instance vs static method) which had to be updated in the analyzer.
  • 55. Next Steps - “The Grind” ● A lot of extra errors in tests - mostly due to dependency conflicts, or changes to error messages ● much willpower needed 🎧🤓.
  • 56. Quick review 1⃣2⃣3⃣… 1. At this point, every existing test was passing in the CI! 2. But… 3. Still no scala 3 syntax allowed in build files! 😱
  • 57. Next Steps - Custom Scala Parser Status quo: customised Fastparse “Scalaparse” parser Choice: ● Extend Scalaparse to support all of Scala 3 syntax? (stateful parsing!) ● Use Scalameta parser? (externalise maintenance) ● Reuse Scala 3’s own parser ✅
  • 58. Next Steps - Custom Scala Parser (cont) Problems using Scala 3: ● Need to load via reflection and isolated classpath ● Maintain compat with the Scalaparse implementation
  • 59. Next Steps - Custom Scala Parser (cont) trait MillScalaParser { def splitScript(rawCode: String, fileName: String) : Either[String, (Seq[String], Seq[String])] def parseImportHooksWithIndices(stmts: Seq[String]) : Seq[(String, Seq[ImportTree])] def parseObjectData(rawCode: String) : Seq[ObjectData] } Lifted interface trait ObjectData { def obj: Snip def name: Snip def parent: Snip def endMarker: Option[Snip] def finalStat: Option[(String, Snip)] } ObjectData Package decls + expressions Extract imports from stats Top level object snippets = text snippet (possibly with position)
  • 60. Next Steps - Custom Scala Parser (cont) package build.runner object `package` extends RootModule with BuildInfo { object worker extends build.MillPublishScalaModule { private[runner] def bootstrapDeps = T.task { val moduleDep = { val m = artifactMetadata() s"${m.group}:${m.id}:${m.version}" } val nameFilter = "scala(.*)-compiler(.*)".r Agg(moduleDep) ++ transitiveIvyDeps().collect { case dep if nameFilter.matches(dep.name) => s"${dep.organization}:${dep.name}:${dep.version}" } } } def buildInfoMembers = Seq( BuildInfo.Value( "bootstrapDeps", worker.bootstrapDeps().mkString(";"), "Depedendencies used to bootstrap the scala compiler worker." ) ) runner/package.mill CodeGen the artifact names for runtime resolution
  • 61. Next Steps - Custom Scala Parser (cont) def generateScriptSources: T[Seq[PathRef]] = Task { val parsed = parseBuildFiles() if (parsed.errors.nonEmpty) Result.Failure(parsed.errors.mkString("n")) else { ... CodeGen.generateWrappedSources( ... T.dest, rootModuleInfo.enclosingClasspath, rootModuleInfo.compilerWorkerClasspath, ... compilerWorker() ) Result.Success(Seq(PathRef(T.dest))) } } MillBuildRootModule.scala Write classpath of worker to generated file Compiler loaded via reflection
  • 62. Next Steps - Custom Scala Parser (cont) def millCompilationUnit(source: SourceFile)(using Context): untpd.Tree = { val parser = new OutlineParser(source) with MillParserCommon { override def selfType(): untpd.ValDef = untpd.EmptyValDef override def topStatSeq(outermost: Boolean): List[untpd.Tree] = { val (_, stats) = templateStatSeq() stats } } parser.parse() } ScalaCompilerWorkerImpl.scala Script body is spliced into object Compiler worker parses outline of mill-dialect compilation unit, then traverses trees to extract text snippets Package decls then expressions
  • 63. Final review 💯 1. Finally, are we done? ✅ 2. Custom scala 3 syntax is supported, with new tests 3. PR in process of being merged (piece by piece) 4. Occasional rebasing needed.