GitBucket: Open source self-hosting Git server built by Scala
- 1. GitBucket Open source self-hosting Git server built by Scala Naoki Takezoe @takezoen
- 2. Who am I? Naoki Takezoe ● Software Engineer at Treasure Data ○ Presto (Trino) and Spark ● 10 years experience with Scala ○ GitBucket, Scalatra, Apache PredictionIO, etc ○ Japanese edition of Scala Puzzlers
- 4. GitBucket is... ● Open source self-hosting Git server ● Initial commit was April 2013 ● Built by Scala and Java technologies
- 5. Current status (Feb 6, 2021) ● 8266 stars ● 158 contributors ● 719 gitter users ● 98 releases
- 6. Start GitBucket right now! ● Download gitbucket.war from ○ https://github.com/gitbucket/gitbucket ● Run ● Official Docker image is available $ java -jar gitbucket.war $ docker run -d -p 8080:8080 gitbucket/gitbucket
- 7. Motivation
- 8. Motivation ● In-house Git repository ○ Due to company's security policy? ○ Due to contract with customers? ● Solutions ○ GitHub Enterprise was expensive ○ GitLab installation was awkward I must make it myself! by
- 9. Why ? From First Principles: Why Scala? by Li Haoyi https://www.lihaoyi.com/post/FromFirstPrinciplesWhyScala.html ● "A Compiled Language that feels Dynamic" ○ Compiled language + Scripting launguage ○ Easy to write with safety ● "A Broad and Deep Ecosystem" ○ Existing Java resources can be leveraged ○ JGit: Pure Java Git implementation ○ Zero dependencies (except JVM) inspired by Jenkins Personal reasons: ● Wanted to write a real-world applications in Scala ● Just for fun :-)
- 12. Issues
- 13. Pull requests
- 14. Major Features ● Public and private repository hosting ● Dashboard includes activity timeline ● Repository viewer which supports online file editing ● Markdown available Wiki ● Issues and pull requests ● Comment on source code ● Notification via E-mail ● User and group management ● LDAP integration ● Gravator integration ● GitHub compatible Web API ● Plugin system ● External database support (MySQL and PostgreSQL)
- 15. Kanban and Gantt chart
- 17. Major plugins Name Description Gist plugin Add code snippet Gist-like functionality to GitBucket Asciidoc plugin Add AsciiDoc support to GitBucket Bugspots plugin Apply Google Bugspots to code in GitBucket repositories Pages plugin Publish repository contents as web sites Network plugin Add the commit graph view to GitBucket Emoji plugin Emoji support in Wiki or Issues RST plugin Add ReStructuredText support to GitBucket Explorer plugin Add the tree view for repositories on GitBucket PlantUML plugin Render PlantUML files on GitBucket Jupyter plugin Render Jupyter or IPython files on GitBucket Fess Plugin Add full text search capability to GitBucket Maven repository plugin Host in-house maven repositories on GitBucket Visit https://gitbucket-plugins.github.io/ to find other plugins!!
- 18. Technology
- 19. Technology stack Git Repository RDBMS (H2 / MySQL / PostgreSQL) JGit Slick Apache MINA SSHD Jetty GitServlet (JGit) Scalatra + Twirl Git Client Web Browser SSH HTTP
- 20. Core technologies are Java components ● Jetty ● H2 ● JGit ● Apache MINA SSHD
- 21. Technology stack (Java parts) Git Repository RDBMS (H2 / MySQL / PostgreSQL) JGit Slick Apache MINA SSHD Jetty GitServlet (JGit) Scalatra + Twirl Git Client Web Browser SSH HTTP
- 22. Key to minimize development cost Minimizing dev cost is very important for sustainability of personal OSS projects ● Java interoperability ○ Scala has good Java interoperability ○ Benefit from existing Java software resources ● Plugin architecture ○ Keep core features minimum for maintainability ○ Leverage community resources Because GitBucket users are not necessarily mature Scala users, we avoid too much FP flavor in GitBucket in order to open the door to contribution and plugin development to them.
- 23. Challenges in long-life application with Scala
- 24. Upgrading Scala and libraries Scala's source code level backward compatibility is great, but… ● Need to rebuild libraries for new Scala major version ● Abandoned or not-well maintained libraries can be blockers ● Some libraries changed its public interface significantly (e.g. Slick2 -> Slick3)
- 25. Experienced Scala major upgrade 3 times ● 2.10 -> 2.11 ● 2.11 -> 2.12 ● 2.12 -> 2.13 The most painful upgrade!
- 26. Why upgrading to Scala 2.12 was so painful? Git Repository RDBMS (H2 / MySQL / PostgreSQL) JGit Slick Apache MINA SSHD Jetty GitServlet (JGit) Scalatra + Twirl Git Client Web Browser SSH HTTP Scalatra development was going down Destructive change in Slick3
- 27. Scalatra ● Simple we framework for Scala inspired by Ruby's Sinatra ○ Traditional Java servlet based framework ○ Declarative input validation and mapping framework like Play2 ○ json4s based JSON support
- 28. Scalatra development is going down ● Rise of reactive and functional programming in Scala ○ Emerge of new frameworks such as finagle, http4s and akka-http ○ Main Scalatra developers shifted to http4s ● Became a Scalatra committer ○ Boosted migration to Scala 2.12, and eventually Scala 2.13 ● Reduced maintenance cost to make it sustainable ○ Dropped minor features and library dependencies ○ Forked abandoned libraries, took some into Scalatra source tree if small enough
- 29. Slick ● Advanced Type-safe SQL builder (former Scala-Query) ○ Very powerful and flexible type-safe API ○ Sometimes generated SQL can cause performance issues, especially on MySQL, though... ● Super painful upgrade in Slick2 -> Slick3 ○ Monadic DBIO introduced in Slick3 affected all existing code
- 30. Slick2 -> Slick3 migration ● Amount of code that needs to be migrated ○ Affected all existing Slick2 based code including community developed plugins ● Difficulty of DBIO for GitBucket users ○ GitBucket users are not mature Scala programmers ○ We wanted to keep the bar low ● Scala 2.12 version of Slick2 was not available back then ○ Eventually Scala 2.12 version of Slick2 was released, though ○ Scala 2.13 version has not been released
- 31. How we migrated to Slick3? ● Created blocking-slick library ○ Slick2 compatible blocking API on the top of Slick3 https://github.com/takezoe/blocking-slick ● We could migrate to Slick3 with minimum effort https://github.com/gitbucket/gitbucket/pull/1381/files ○ Also, minimized the negative impact on plugin developers
- 32. Created Java libraries, not Scala libraries ● Markedj (GitHub flavored markdown parser, Java-port of marked.js) ● Solidbase (Multi-tenant and multi-database supported migration tool based on Liquibase) ● If the library interface is simple enough, no benefit to write in Scala for library users. ● We don't need to rebuild a library for each major Scala version.
- 33. Effective strategy for long-term maintenance ● Minimize library dependencies ● Use Java libraries if possible ● Fork or take over library maintenance if needed These strategies would be effective even for migration to Scala3!
- 34. Try GitBucket! ● GitHub: https://github.com/gitbucket/gitbucket ● Demo site: https://gitbucket.herokuapp.com/ ● Gitter: https://gitter.im/gitbucket/gitbucket ● Blog: https://gitbucket.github.io/gitbucket-news/ ● Community Plugins: https://gitbucket-plugins.github.io/
- 35. Appendix: How to create GitBucket plugin
- 36. Create project ● build.sbt ● project/plugins.sbt This sbt plugin adds necessary library dependencies to the project and provide configuration and sbt task useful for GitBucket plugin development.
- 37. Define plugin manifest ● Plugin.scala (plugin manifest) Register new controller via extension point
- 38. Implement plugin ● HelloWorldController.scala (Typical Scalatra controller) Bunch of extension points are available, such as: ● Add menus and tabs ● Inject JavaScript ● Register event hooks ● etc
- 39. Build and Test ● Create a package ● Install to local GitBucket ● Template project ○ https://github.com/gitbucket/gitbucket-plugin-template ● Tutorial ○ https://gitbucket.github.io/gitbucket-news/gitbucket/2015/06/29/ho w-to-create-plugin.html $ sbt assembly $ sbt install