SFScon19 - Paolo Boldi - Software Ecosystems as Networks the FASTEN project
0 likes•34 views
The document discusses the importance of sharing programs and libraries among programmers, highlighting the evolution of software ecosystems into complex networks of dependencies. It emphasizes how these dependencies can lead to significant issues, such as security breaches, through historical incidents like the leftpad incident and others. The FASTEN project aims to provide thorough analysis tools for understanding software ecosystems, addressing challenges like change impact analysis and risk profiling.
![Updating a library
# Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Before # Check outdated dependencies
$ pip list --outdated
Package Version Latest Type
---------- ------- ------ -----
Pygments 2.2.0 2.3.1 wheel
Updating Pygments will affect:
foo.py: function colorize
bar.py: function parse
After
# Estimate update impact
$ pip install --dry-run Pygments
Function Pygments.Formatter.format[formatter.py]
changed ->
check <your_app> at colorize[foo.py]:32](https://image.slidesharecdn.com/sfscon19-paoloboldi-softwareecosystemsasnetworksthefastenproject-191205150645/85/SFScon19-Paolo-Boldi-Software-Ecosystems-as-Networks-the-FASTEN-project-62-320.jpg)
SFScon19 - Paolo Boldi - Software Ecosystems as Networks the FASTEN project
- 1. Software Ecosystems as Networks: the FASTEN project Paolo Boldi Università degli Studi di Milano Italy
- 3. Programmers and sharing ❖ The fundamental act of friendship among programmers is the sharing of programs The GNU manifesto (1985)
- 4. Sharing through software libraries
- 5. Sharing through software libraries ❖ One form of sharing is providing libraries
- 6. Sharing through software libraries ❖ One form of sharing is providing libraries ❖ Today, libraries are made available in the Internet
- 7. Sharing through software libraries ❖ One form of sharing is providing libraries ❖ Today, libraries are made available in the Internet ❖ on forges (SourceForge, GitHub, BitBucket, …)
- 8. Sharing through software libraries ❖ One form of sharing is providing libraries ❖ Today, libraries are made available in the Internet ❖ on forges (SourceForge, GitHub, BitBucket, …) ❖ or repositories (Maven, PyPi, CPAN, …)
- 9. Sharing through software libraries ❖ One form of sharing is providing libraries ❖ Today, libraries are made available in the Internet ❖ on forges (SourceForge, GitHub, BitBucket, …) ❖ or repositories (Maven, PyPi, CPAN, …) ❖ Internet made the dream of collaborative development a reality
- 10. Industrial revolution at the harbour of software development
- 11. Industrial revolution at the harbour of software development ❖ All trades, arts, and handiworks have gained by division of labour, namely, when, instead of one man doing everything, each confines himself to a certain kind of work distinct from others in the treatment it requires, so as to be able to perform it with greater facility and in the greatest perfection. Where the different kinds of work are not distinguished and divided, where everyone is a jack-of-all-trades, there manufactures remain still in the greatest barbarism. Immanuel Kant Groundwork for the Metaphysics of Morals (1785)
- 12. With libraries come dependencies…
- 13. With libraries come dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project>
- 14. With libraries come dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project> Library
- 15. With libraries come dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project> Library Dependencies
- 16. With libraries come dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project> Library Dependencies +version
- 17. With libraries come dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project> Library Dependencies +version +version constraints
- 19. …and transitive dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project>
- 20. …and transitive dependencies… <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>foo-bar</groupId> <artifactId>NeoImport</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> </dependencies> </project> <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>it.unimi.dsi</groupId> <artifactId>webgraph</artifactId> <version>3.6.1</version> </dependency> <dependency> <groupId>org.eclipse.jgit</groupId> <artifactId>org.eclipse.jgit</artifactId> <version>5.2.1.201812262042-r</version> </dependency> </dependencies> </project> Depends
- 22. Dependency graphs ❖ Library+versions and their dependencies form (complex, huge) dependency networks
- 23. Dependency graphs ❖ Library+versions and their dependencies form (complex, huge) dependency networks ❖ Version constraints make these networks more complicated than simple graphs
- 24. Dependency graphs ❖ Library+versions and their dependencies form (complex, huge) dependency networks ❖ Version constraints make these networks more complicated than simple graphs ❖ Package manager will finally determine which version is chosen for each library
- 26. The dependency heaven ❖ Relying on an ecosystem of easy-to- use well written libraries made the dream of code reuse a reality
- 28. The dependency hell ❖ A bug or security breach or legal issue concerning one single piece… ❖ …can make the whole tower fall!
- 30. Recent dependency nightmares ❖ The leftpad incident (2016): millions of websites affected
- 31. Recent dependency nightmares ❖ The leftpad incident (2016): millions of websites affected ❖ The Equifax breach (2017): costed 4B$
- 32. Epidemics in dependency graphs Lib A, vers 1.0 Lib B, vers 2.5 Lib C, vers 1.5Lib D, vers 3.0
- 33. Epidemics in dependency graphs Lib A, vers 1.0 Lib B, vers 2.5 Lib C, vers 1.5Lib D, vers 3.0 A vulnerability alert is issued about Lib D, vers 3.0
- 34. Epidemics in dependency graphs Lib A, vers 1.0 Lib B, vers 2.5 Lib C, vers 1.5Lib D, vers 3.0 A vulnerability alert is issued about Lib D, vers 3.0 All libraries in this graph are infected!
- 35. GitHub security alerts But is this enough?
- 36. The FASTEN Project ❖ Fine-Grained Analysis of SofTware Ecosystems as Networks ❖ Part of the EU H2020-ICT-2018-2020 Program ❖ Consortium
- 37. Epidemics in dependency graphs Lib A, vers 1.0 Lib B, vers 2.5 Lib C, vers 1.5Lib D, vers 3.0
- 38. Epidemics in dependency graphs A.f0 A.f2 A.f3 B.f1 B.f2 B.f3 C.f1 C.f2 D.f1 D.f2 D.f3
- 39. Epidemics in dependency graphs A.f0 A.f2 A.f3 B.f1 B.f2 B.f3 C.f1 C.f2 D.f1 D.f2 D.f3 A vulnerability alert is issued about Lib D, vers 3.0, function f3
- 40. Epidemics in dependency graphs A.f0 A.f2 A.f3 B.f1 B.f2 B.f3 C.f1 C.f2 D.f1 D.f2 D.f3 A vulnerability alert is issued about Lib D, vers 3.0, function f3
- 41. Epidemics in dependency graphs A.f0 A.f2 A.f3 B.f1 B.f2 B.f3 C.f1 C.f2 D.f1 D.f2 D.f3 A vulnerability alert is issued about Lib D, vers 3.0, function f3 Much more informative!
- 42. Examples
- 43. Examples ❖ Fully precise change impact analysis: “How many libraries are affected if I remove/modify a certain method/interface?”
- 44. Examples ❖ Fully precise change impact analysis: “How many libraries are affected if I remove/modify a certain method/interface?” ❖ Fully precise license compliance: “Is my library compliant with the licenses of the libraries that I depend from (directly or indirectly)? (e.g., am I linking any GPL code?)”
- 45. Examples ❖ Fully precise change impact analysis: “How many libraries are affected if I remove/modify a certain method/interface?” ❖ Fully precise license compliance: “Is my library compliant with the licenses of the libraries that I depend from (directly or indirectly)? (e.g., am I linking any GPL code?)” ❖ Fully precise risk profiling: “Does this vulnerability affect my code?”
- 46. Examples ❖ Fully precise change impact analysis: “How many libraries are affected if I remove/modify a certain method/interface?” ❖ Fully precise license compliance: “Is my library compliant with the licenses of the libraries that I depend from (directly or indirectly)? (e.g., am I linking any GPL code?)” ❖ Fully precise risk profiling: “Does this vulnerability affect my code?” ❖ Centrality analysis: “What methods/functions are more central within a given ecosystem? are there bottlenecks? critical points?”
- 48. The FASTEN toolchain Project information Security alerts Repositories
- 49. The FASTEN toolchain Project information Security alerts Repositories publish Data stream publish publish
- 50. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server publish publish
- 51. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction publish publish
- 52. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer publish publish
- 53. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer Analysis layer publish publish
- 54. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer Analysis layer RESTApi publish publish
- 55. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer Analysis layer RESTApiWebUI publish publish
- 56. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer Analysis layer RESTApiWebUI publish publish Continuous integration server
- 57. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer Analysis layer RESTApiWebUI publish publish Continuous integration server
- 58. The FASTEN toolchain Project information Security alerts Repositories publish Data stream FASTEN server Call-graph construction Storage layer Analysis layer RESTApiWebUI publish publish Continuous integration server Developer
- 60. Updating a library # Check outdated dependencies $ pip list --outdated Package Version Latest Type ---------- ------- ------ ----- Pygments 2.2.0 2.3.1 wheel Before
- 61. Updating a library # Check outdated dependencies $ pip list --outdated Package Version Latest Type ---------- ------- ------ ----- Pygments 2.2.0 2.3.1 wheel Before # Check outdated dependencies $ pip list --outdated Package Version Latest Type ---------- ------- ------ ----- Pygments 2.2.0 2.3.1 wheel Updating Pygments will affect: foo.py: function colorize bar.py: function parse After
- 62. Updating a library # Check outdated dependencies $ pip list --outdated Package Version Latest Type ---------- ------- ------ ----- Pygments 2.2.0 2.3.1 wheel Before # Check outdated dependencies $ pip list --outdated Package Version Latest Type ---------- ------- ------ ----- Pygments 2.2.0 2.3.1 wheel Updating Pygments will affect: foo.py: function colorize bar.py: function parse After # Estimate update impact $ pip install --dry-run Pygments Function Pygments.Formatter.format[formatter.py] changed -> check <your_app> at colorize[foo.py]:32
- 63. Information about a library
- 64. Information about a library # Checking info about the library $ pip show tornado Name: tornado Version: 5.0 Summary: Tornado is a Python web framework ... Home-page: http://www.tornadoweb.org/ Author: Facebook Author-email: ... License: http://www.apache.org/licenses/ LICENSE-2.0 Location: ... Requires: backports-abc, futures, singledispatch Required-by: Before
- 65. Information about a library # Checking info about the library $ pip show tornado Name: tornado Version: 5.0 Summary: Tornado is a Python web framework ... Home-page: http://www.tornadoweb.org/ Author: Facebook Author-email: ... License: http://www.apache.org/licenses/ LICENSE-2.0 Location: ... Requires: backports-abc, futures, singledispatch Required-by: Before # Checking info about the library $ pip show tornado Name: tornado Version: 5.0 License: http://www.apache.org/licenses/ LICENSE-2.0 ... Maintainers: 3 Community size: 15 Used by: 145 on PyPI, 34433 on GitHub Latest vulnerability: 13 months ago All known vulnerabilities: 25 (best 10%) License rating: Compatible After
- 66. Who uses my function?
- 67. Who uses my function? # Check uses of function pkg.list() in dependents $ pip query --uses pkg.list depA(v1.2).parse() depA(v1.2).test() depB(0.0.2).foo() depC(1.2.1).calculate()X
- 68. Who uses my function? # Check uses of function pkg.list() in dependents $ pip query --uses pkg.list depA(v1.2).parse() depA(v1.2).test() depB(0.0.2).foo() depC(1.2.1).calculate()X # Rank them by importance (aka centrality) $ pip query --uses --rank pkg.list depC(1.2.1).calculate() depB(0.0.2).foo() depA(v1.2).parse() depA(v1.2).test()
- 69. Who uses my function? # Check uses of function pkg.list() in dependents $ pip query --uses pkg.list depA(v1.2).parse() depA(v1.2).test() depB(0.0.2).foo() depC(1.2.1).calculate()X # Estimate "damage" if pkg.list will be updated $ pip query —total pkg.list 3 direct and 223 indirect dependencies will be affected) # Rank them by importance (aka centrality) $ pip query --uses --rank pkg.list depC(1.2.1).calculate() depB(0.0.2).foo() depA(v1.2).parse() depA(v1.2).test()
- 70. Who uses my function? # Check uses of function pkg.list() in dependents $ pip query --uses pkg.list depA(v1.2).parse() depA(v1.2).test() depB(0.0.2).foo() depC(1.2.1).calculate()X # Estimate "damage" if pkg.list will be updated $ pip query —total pkg.list 3 direct and 223 indirect dependencies will be affected) # Notify direct dependencies of upcoming breakage $ pip query --uses pkg.list | cut -f 1 -d '(' | xargs -I {} pip show {} | grep Author-email: | cut -f 2 -d ':' | xargs mail -s 'MyProject update will break yours!' # Rank them by importance (aka centrality) $ pip query --uses --rank pkg.list depC(1.2.1).calculate() depB(0.0.2).foo() depA(v1.2).parse() depA(v1.2).test()
- 72. FASTEN challenges ❖ Sound call-graph generation
- 73. FASTEN challenges ❖ Sound call-graph generation ❖ Scalability (huge graphs!)
- 74. FASTEN challenges ❖ Sound call-graph generation ❖ Scalability (huge graphs!) ❖ Real-time responsiveness
- 75. FASTEN challenges ❖ Sound call-graph generation ❖ Scalability (huge graphs!) ❖ Real-time responsiveness ❖ Data integration
- 76. FASTEN challenges ❖ Sound call-graph generation ❖ Scalability (huge graphs!) ❖ Real-time responsiveness ❖ Data integration ❖ Tool development for end users
- 77. Network analysis will be the next step for the future of software development
- 78. Network analysis will be the next step for the future of software development
- 79. Questions? Paolo Boldi Università degli Studi di Milano Italy [email protected]