How to write a TableGen backend

Min-Yih “Min” Hsu @ LLVM Dev Meeting 2021
How to write a TableGen backend

“Min” Hsu
• Computer Science PhD Candidate
in University of California, Irvine

• Code owner of M68k LLVM backend

• Author of book “LLVM Techniques,
Tips and Best Practices” (2021)
$ whoami
2

TableGen in a nutshell
• A Domain-Speci
fi
c Language (DSL) originated from the LLVM project.
• It’s Turing complete!
4

• A Domain-Speci
fi
• Originally invented to describe the instruction table of an LLVM target.
4

• A Domain-Speci
fi
• Ex. The operands, assembly syntax, and ISel rules for each instruction.
4

• A Domain-Speci
fi
• Now: Used in a wide variety of (completely) di
ff
erent areas inside LLVM.
4

• A Domain-Speci
fi
ff
• Instruction scheduling info.
4

• A Domain-Speci
fi
ff
• Declaring IR attributes.
4

• A Domain-Speci
fi
ff
• Declaring IR attributes.
• LLVM Option subsystem (e.g. Clang’s compiler
fl
ags).
4

class Stuff {

}

5
Layout of a template

class Stuff {

string Name;

int Quantity;

string Description;

}

6

class Stuff {

string Name;

int Quantity;

string Description;

}

6
Fields

class Stuff {

string Name;

int Quantity;

string Description;

}

def water_bottle : Stuff {

}

7
Fields

class Stuff {

string Name;

int Quantity;

string Description;

}


}

7
A record created with template Stuff
Fields

class Stuff {

string Name;

int Quantity;

string Description;

}


let Name = "Water bottle";

let Quantity = 1;

let Description = "Stuff that helps you hydrate.";

}

8
Fields

class Stuff {

string Name;

int Quantity;

string Description;

}


let Name = "Water bottle";

let Quantity = 1;

let Description = "Stuff that helps you hydrate.";

}

def smart_phone : Stuff {

let Name "Smart phone";

let Quantity = 2;

let Description = "Stuff that keeps you from hydrating.";

}
9
Fields
Another record created with template Stuff

class Stuff <string name, int quantity, string description> {

string Name = name;

int Quantity = quantity;

string Description = description;

}

10

class Stuff <string name, int quantity, string description> {

string Name = name;

int Quantity = quantity;

string Description = description;

}

def water_bottle : Stuff<"Water bottle", 1,

"Stuff that helps you hydrate.">;

def smart_phone : Stuff<"Smart phone", 2,

"Stuff that prevents you from hydrating.">;
11

12
class Person {

std::string Name;

int Age;

JobKind Job;

};

Person Me{"Min", 12, WEEBUS};

OOP v.s. TableGen

12
class Person {

std::string Name;

int Age;

JobKind Job;

};


Encapsulates data
OOP v.s. TableGen

13
class Person {

std::string Name;

int Age;

JobKind Job;

};


void foo(int N) {

Me.Name = "Max";

}

Encapsulates data
Records are immutable
OOP v.s. TableGen

14
class Person {

std::string Name;

int Age;

JobKind Job;

};


void foo(int N) {

Me.Name = "Max";

Person Rick{“Rick", N, SINGER};

}

Encapsulates data
Constant values in
fi
elds
OOP v.s. TableGen

15
class Person {

std::string Name;

int Age;

JobKind Job;

};


void foo(int N) {

Me.Name = "Max";

Person Rick{“Rick", N, SINGER};

for (i = 0; i < N; ++i)

SomeList.emplace_back(Person{...});

}

Encapsulates data
Constant values in
fi
elds
Constant number of records
OOP v.s. TableGen

16
Name Quantity Description
Water Bottle 1 …
Smart Phone 2 …
Table “Stu
f
”
Relational DB v.s. TableGen

16
Fields are similar to columns
Water Bottle 1 …
Smart Phone 2 …
Table “Stu
f
”

16
Water Bottle 1 …
Smart Phone 2 …
Table “Stu
f
”
Records do not belong to any table

16
Water Bottle 1 …
Smart Phone 2 …
Table “Stu
f
”
Strong structure is not required

16
Water Bottle 1 …
Smart Phone 2 …
Table “Stu
f
”
def foo {

string A = “foo”;

int B = 0;

}

def bar {

int Z = 1;

}

def zoo;
Strong structure is not required

TableGen data types
• Primitive types (common): int, string, bool, bit
17

TableGen data types
• Bit vector: bits<N>
17

TableGen data types
• List: list<T>
17

TableGen data types
• List: list<T>
• Direct Acyclic Graph (DAG): dag
17

TableGen data types
• List: list<T>
• Represent DAG data symbolically
17

TableGen data types
• List: list<T>
• Represent DAG data symbolically
• dag foo = (operator arg0, arg1, …)
17

How to use TableGen records?
18

How to use TableGen records?
19

"A stuff that…”>;



TableGen backends
20



TableGen
Backend 1
TableGen
Backend 2
TableGen
Backend 3

TableGen backends
21
TableGen
Backend 1
TableGen
Backend 2
TableGen
Backend 3
TableGen
Parser
TableGen
Code
Output 1
Output 2
Output 3

TableGen usage in LLVM
An example in LLVM backend
22
llvm-tblgen

22
llvm-tblgen
InstrInfo TG Backend
DAGISel TG Backend

22
llvm-tblgen
DAGISel TG Backend
X86.td

22
llvm-tblgen
DAGISel TG Backend
X86GenInstrInfo.inc
X86GenDAGISel.inc
X86.td

22
llvm-tblgen
DAGISel TG Backend
X86GenInstrInfo.inc
X86GenDAGISel.inc
X86.td
Files contain C/C++ code

Why should I learn to write a TG backend?
24

Despite being a DSL, TableGen is actually pretty versatile
24

Always require a specific TableGen backend
24

LLVM has provided nice infrastructures to work with TableGen code
24

Project overview
25
Recap: Comparison with Relational DB

Project overview
25
Recap: Comparison with Relational DB
A more
fl
exible way to represent static structural data

Project overview
SQLGen — Generate SQL from TableGen code
26
TableGen
Code

Project overview
26
TableGen
Code
TG Backend
of SQLGen
TableGen
Parser
SQLGen
A standalone tool

Project overview
26
TableGen
Code
TG Backend
of SQLGen
TableGen
Parser
SQLGen
SQL Code
A standalone tool

TableGen syntax in SQLGen
SQL table creation
27
CREATE TABLE Customer (

ID int,

Name varchar(255),

Affiliation varchar(255),

PRIMARY KEY (ID)

);
Generated SQL Code
Input TableGen Code

SQL table creation
27

ID int,

Name varchar(255),


PRIMARY KEY (ID)

);
Generated SQL Code
Input TableGen Code
class Table {

int PrimaryKey = 0;

}

SQL table creation
27

ID int,

Name varchar(255),


PRIMARY KEY (ID)

);
Generated SQL Code
Input TableGen Code
class Customer <string name,

string affiliation> : Table {

int ID = PrimaryKey;

string Name = name;

string Affiliation = affiliation;

}
class Table {

int PrimaryKey = 0;

}

Inserting rows into a SQL table
28
INSERT INTO Customer (

ID,

Name,

Affiliation

)

VALUES (0, "John Smith", "UC Irvine");
Generated SQL Code
Input TableGen Code

28

ID,

Name,

Affiliation

)

Generated SQL Code
Input TableGen Code
def john : Customer<"John Smith", "UC Irvine">;

28

ID,

Name,

Affiliation

)

Generated SQL Code
Input TableGen Code



string Name = name;


}

SQLGen entry point
29
int main(int argc, char **argv) {

cl::ParseCommandLineOptions(argc, argv);

return llvm::TableGenMain(argv[0], &CallbackFunc);

}

SQLGen entry point
29



}
bool CallbackFunc(raw_ostream &OS, RecordKeeper &Records)

SQLGen entry point
29



}
• OS: Stream to the output
fi
le (i.e. Output stream to print the SQL code)

SQLGen entry point
29



}
• OS: Stream to the output
fi
le (i.e. Output stream to print the SQL code)
• Records: In-memory representation of the parsed TableGen code

Recap: SQL table creation
31

ID int,

Name varchar(255),


PRIMARY KEY (ID)

);
Generated SQL Code
Input TableGen Code



string Name = name;


}
class Table {

int PrimaryKey = 0;

}

Enumerating TableGen class-es
32
bool CallbackFunc(raw_ostream &OS, RecordKeeper &Records) {

const auto &Classes = Records.getClasses();

...

}

Enumerating TableGen class-es
32


...

}
std::map<std::string, std::unique_ptr<llvm::Record>>

33
llvm::Record
TableGen record TableGen class

33
llvm::Record
int ID = 0;

string Name = "John Smith";

string Affiliation = "UC Irvine";

33
llvm::Record
int ID = 0;

string Name = ?;

string Affiliation = ?;
int ID = 0;



33
llvm::Record
llvm::Record::isClass() == false llvm::Record::isClass() == true
int ID = 0;

string Name = ?;

string Affiliation = ?;
int ID = 0;



34
class Customer <string name, string affiliation> : Table {


string Name = name;


}

35
def john {

int PrimaryKey = 0;

int ID = 0;



}


string Name = name;


}

36
def john {

int PrimaryKey = 0;

int ID = 0;



}


string Name = name;


}
llvm::Record

36
def john {

int PrimaryKey = 0;

int ID = 0;



}


string Name = name;


}
llvm::Record
llvm::RecordVal

37

for (const auto &P : Classes) {

auto ClassName = P.first;

Record &ClassRecord = *P.second;

}

38




if (!ClassRecord.isSubClassOf("Table"))

continue;

}

39





continue;

OS << "CREATE TABLE " << ClassName << " (";

OS << ");n";

}

);

40





continue;


for (const RecordVal &RV : ClassRecord.getValues()) {

}

OS << ");n";

}

);

41





continue;



OS << "t" << RV.getName() << " ";

}

OS << ");n";

}

ID

Name

Affiliation

);

42





continue;



OS << "t" << RV.getName() << " ";

if (isa<IntRecTy>(RV.getType()))

OS << "int,";

}

OS << ");n";

}

ID int,

Name

Affiliation

);

43





continue;



OS << "t" << RV.getName() << " ";

if (isa<IntRecTy>(RV.getType()))

OS << "int,";

if (isa<StringRecTy>(RV.getType()))

OS << "varchar(255),";

}

OS << ");n";

}

ID int,

Name varchar(255),


);

44
def john {

int PrimaryKey = 0;

int ID = 0;



}


string Name = name;


}
llvm::Record
llvm::RecordVal

44
def john {

int PrimaryKey = 0;

int ID = 0;



}


string Name = name;


}
llvm::Record
llvm::RecordVal
llvm::Init

Common derived classes of llvm::Init
45
llvm::Init llvm::TypedInit

45
llvm::IntInit
llvm::StringInit
llvm::DagInit

45
llvm::IntInit
llvm::StringInit
llvm::DagInit
llvm::VarInit

45
llvm::IntInit
llvm::StringInit
llvm::DagInit
int PrimaryKey = 0;

llvm::VarInit

46

...



OS << "t" << RV.getName() << " ";

...

Init *Val = RV.getValue();

if (auto *VI = dyn_cast<VarInit>(Val)) {

if (VI->getName() == "PrimaryKey")

OS << "PRIMARY KEY (" << RV.getName() << ")";

}

}

OS << ");n";

}

ID int,

Name varchar(255),


PRIMARY KEY (ID)

);

Recap: Inserting rows into a SQL table
48

ID,

Name,

Affiliation

)

Generated SQL Code
Input TableGen Code



string Name = name;


}

Enumerating TableGen Records
49

auto SQLRows = Records.getAllDerivedDefinitions("Table");

for (const Record *RowRecord : SQLRows) {

...

}

}

50
def john {

int PrimaryKey = 0;

int ID = 0;



}


string Name = name;


}
llvm::Record
llvm::RecordVal
llvm::Init
Recap: In-memory representations for TableGen records / classes

51

OS << "INSERT INTO " << ClassName << " (n";

for (const RecordVal &RV : RowRecord->getValues()) {

}

OS << ")n";

}

)

52



auto Name = RV.getName();

OS << "t" << Name << ",n";

}

OS << ")n";

}

ID,

Name,

Affiliation

)

53



auto Name = RV.getName();

OS << "t" << Name << ",n";

}

OS << ")n";

OS << "VALUES (";


const Init *Val = RV.getValue();

OS << Val->getAsString() << ", ";

}

OS << ");n";

}

ID,

Name,

Affiliation

)


Making queries
55
SELECT Affiliation FROM Customer

WHERE Name = "John Smith";
Generated SQL Code
Input TableGen Code
class Query <string table,

dag query_fields, dag condition> {

string TableName = table;

dag Fields = query_fields;

dag WhereClause = condition;

}

Making queries
56

Generated SQL Code
Input TableGen Code
def : Query<"Customer", (fields "Affiliation"),

(eq "Name", "John Smith":$str)>;





}

Making queries
56

Generated SQL Code
Input TableGen Code






}
Anonymous record

Example of the dag type
An expression tree
57
def plus;

def minus;

def var_x : Var {...}

dag expr = (plus 9, ("mul" 4, (minus var_x, 3)));
Modeling expression: 9 + 4 * (x - 3)

An expression tree
57
def plus;

def minus;


plus
9

An expression tree
57
def plus;

def minus;


plus
9 “mul”
4

An expression tree
57
def plus;

def minus;


plus
9 “mul”
4 minus
3
var_x

Making SQL queries
More examples
58
SELECT Person, Amount FROM Orders

WHERE Amount > 8;
def : Query<"Orders", (fields "Person", "Amount"),

(gt "Amount", 8)>
TableGen
SQL

Making SQL queries
More examples
58
SELECT Person, Amount FROM Orders

WHERE Amount > 8;
def : Query<"Orders", (fields "Person", "Amount"),

(gt "Amount", 8)>
TableGen
SQL
SELECT ProductName, Person FROM Orders

WHERE Amount > 8 AND Person <> 1;
def : Query<"Orders", (fields "ProductName", "Person"),

(and (gt "Amount", 8), (ne "Person", 1))>;
TableGen
SQL

Making queries
59

Generated SQL Code
Input TableGen Code






}

Making queries
59

Generated SQL Code
Input TableGen Code






}
An argument with tag

60
auto SQLQueries = Records.getAllDerivedDefinitions("Query");

for (const Record *Query : SQLQueries) {

OS << "SELECT ";

}
SELECT

61


auto TableName = Query->getValueAsString("TableName");

OS << "SELECT ";

OS << " FROM " << TableName << "n";

}

62


auto TableName = Query->getValueAsString("TableName");

const DagInit *Fields = Query->getValueAsDag("Fields");

OS << "SELECT ";

for (const Init *Arg : Fields->getArgs())

OS << Arg->getAsUnquotedString() << ",";

OS << " FROM " << TableName << "n";

}

63
void visitWhereClause(const DagInit *Term, raw_ostream &OS) {

}

WHERE
(eq "Name", "John Smith":$str)
TableGen
SQL

64

for (int i = 0; i < Term->arg_size(); ++i) {

const Init *Arg = Term->getArg(i);

}

}

WHERE
TableGen
SQL

65



if (const auto *ArgDag = dyn_cast<DagInit>(Arg))

visitWhereClause(ArgDag, OS);

}

}

WHERE
TableGen
SQL

66





else {

if (Term->getArgName(i) == "str")

OS << Arg->getAsString();

else

OS << Arg->getAsUnquotedString();

}

}

}

WHERE Name “John Smith";
TableGen
SQL

67

const Init *Operator = Term->getOperator();





else {

if (Term->getArgName(i) == "str")

OS << Arg->getAsString();

else

OS << Arg->getAsUnquotedString();

}

if (i < Term->arg_size() - 1)

printOperator(Operator);

}

}

TableGen
SQL

Recap: Why should I learn to write a TG backend?


LLVM has provided nice infrastructures to work with TableGen code
69

Other useful TableGen syntax
The multiclass - creating multiple records at once

TableGen operators (a.k.a bang operators) - e.g. !add, !mul, !or

Bits slice

String concatenation via ‘#’

Casting from string to a record

70

Sample code
71
https://github.com/mshockwave/SQLGen
• Hierarchical records via FOREIGN KEY

• Ordering
fi
elds via ORDER BY

• Using LLVM LIT for testing
Some additional features / highlights…

Thank you!
72
GitHub: mshockwave

Email: minyihh@uci.edu

25% book discount code: 25MINLLVM
Redeem:
Nov 15 ~ Nov 20, 2021

How to write a TableGen backend

More Related Content

What's hot (20)

Similar to How to write a TableGen backend (20)

More from Min-Yih Hsu (14)

Recently uploaded (20)

How to write a TableGen backend