2014 10-14: GitHub plus FOSS == 1 million SPDX
1 like•1,527 views
The document discusses the challenges and solutions related to managing software licenses in open source projects, highlighting the need for better tools and standards like SPDX. It outlines the large scale of open source projects and the burden on developers to appropriately apply licenses, presenting various tools and initiatives developed to facilitate this process. Additionally, it emphasizes the importance of open data and collaboration in achieving effective license transparency.
2014 10-14: GitHub plus FOSS == 1 million SPDX
- 1. + => 1 million SPDX Large-scale license transparency using open data, open standards and F/OSS http://triplecheck.net http://searchcode.com
- 2. Speaker Slide #2 Nuno Brito Free/open source contributor since 2005 Last 12 months wrote 100k F/OSS lines of code SPDX contributor, co-founder of TripleCheck Around the web http://nunobrito.eu
- 3. Transparency Slide #3 Take some source code as example Who developed the code? Which licenses are applicable? Was the code copied from somewhere else?
- 4. Size Slide #4 A problem of scale Open licenses? > 300 types to choose > 5 million F/OSS projects > 100 million source code files
- 5. Practice Slide #5 Applying licenses Burden on developer (do correctly, do enough) Expressed differently (difficult to understand) Scaling obstacles (scarce automation) Transparency?
- 6. What do? Slide #6 Ideally, we'd have tooling that is.. a) Reachable b) Cooperative c) Free Choose two. (sad reality)
- 7. Choose three Slide #7 Choose building blocks based on: a) Open standards b) Open data c) Reachable tools Learn, write, improve. Share.
- 8. Standards Slide #8 SPDX: Open standard for software licensing Standardizes license description Defines Id for license terms http://spdx.org Pro: Good docs, straightforward, getting better Cons: Slow adoption, scarce tooling
- 9. Open data Slide #9 GitHub: Targeting open data repositories API suited for intensive access Social coding Largest open source code collection Pro: Reachable, diverse Cons: Repositories processed one-by-one
- 10. Tooling Slide #10 Custom-built tools for software licenses Large-scale repository data-mining Find applicable licenses inside content Share millions of SPDX documents Pro: Learn by doing, modularized, single language Cons: Built from scratch, needs consolidation
- 11. Step 1 Slide #11 Desktop tool/engine to discover licenses SPDX format as storage medium Identify copyright and 18 license types Java, released in Feb 2014. EUPL http://spdx.org/tools/community/triplecheck-reporter
- 13. File detail Slide #13
- 14. SPDX file Slide #14
- 16. Details Slide #16 Underneath the hood 147 file extensions, 18 license types LOC, hashes (SHA1, MD5, SHA256, SSDEEP) Command line supported (Jenkins, cron) Fast, 40k files/minute (Pentium IV)
- 17. Step 2 Discovering repositories with gitFinder Create a list of projects online to use as components. Get basic licensing information from each project. Write text file with each github user (~7 million) For each user, find repositories not forked (~10M) Split each repository according to language (197) For each list of language/reps, download code Slide #17
- 18. Performance Slide #18 ~70k repositories/day Single machine (i7, 8Gb RAM, CentOS) 9 parallel threads Resume/recover supported Released in Jun. 2014 https://github.com/triplecheck/gitfinder
- 19. Output Slide #19
- 20. Storage? https://what-if.xkcd.com/29/ (CC BY-NC 2.5) Slide #20
- 21. Storage BigZip, +100 million files on a single download Slide #21 Flat-file, zip compression (per entry) Fast, simple, portable. Indexed search https://github.com/triplecheck/big
- 22. How it looks Slide #22
- 23. Step 3 Slide #23 SPDX search engine One-click SPDX creation from open data Visualize license and copyright data Visit at http://searchcode.com/spdx
- 24. Example Slide #24 Using the original URL.. https://github.com/iuly/europa_kernel/ => https://spdxhub.com/iuly/europa_kernel/
- 26. SPDX-1M “Do It Yourself” kit. Generate 1 million SPDX Slide #26 https://github.com/triplecheck/diy 1.2 million open source projects “Arduino” for s/w licenses detection 9Gb worth of SPDX? Grab: http://triplecheck.net/public/storage/spdx.big
- 28. Next step? Slide #28 F2F – pinpointing non-original code Decompose code into blocks Tokenize/anonymize data Find code matches across knowledge base ETA in Dec. 2014 https://github.com/triplecheck/f2f
- 30. Conclusion Slide #30 What is now available for everyone Desktop tooling / detection engine Extraction of open data in scale Search engine for SPDX
- 31. Questions? Slide #31 http://spdx.org http://searchcode.com/spdx http://github.com/triplecheck Interesting stuff? Let us know: @nn81 @boyte #linuxcon http://xkcd.com/1118/
- 32. Backup slides Slide #32
- 33. Engine Slide #33
- 34. License DB Slide #34