![# requirements.lock
boto3==1.10.2
botocore==1.13.2 # via boto3, s3transfer
certifi==2019.9.11 # via requests
chardet==3.0.4 # via requests
docutils==0.15.2 # via botocore
futures==3.3.0 # via s3transfer
idna==2.8 # via requests
jmespath==0.9.4 # via boto3, botocore
python-dateutil==2.8.0 # via botocore
requests==2.22.0
s3transfer==0.2.1 # via boto3
six==1.12.0 # via python-dateutil
urllib3==1.25.6 # via botocore,
requests
py_binary(
name = "my_app",
deps = ["lib_a"],
requires = ["requests>=2.21"],
)
py_library(
name = "lib_a",
deps = ["lib_b"],
requires = ["requests>=2.18"],
)
py_library(
name = "lib_b",
deps = ["lib_c"],
requires = ["boto3~=1.9"],
)
py_venv(
name = "my_app_env",
targets = [":my_app"],
)
# requirements.in
boto3~=1.9
requests>=2.18
requests>=2.21](https://image.slidesharecdn.com/bazelcon2019-beeswaxpublish-200208213918/85/Python-Dependencies-at-Beeswax-BazelCon-2019-Lightning-Talk-Ron-Rothman-4-320.jpg)
Python Dependencies at Beeswax - BazelCon 2019 Lightning Talk - Ron Rothman
- 1. Python Dependencies The Right* Way Bazel + Python @ Beeswax Ron Rothman * For us. YMMV.
- 2. ● 30 engineers ● monorepo ● 100 applications ● 300 kLoPyC 1.Global requirements.txt 2.rules_python 3.pipenv We have: We tried: ● bazel test ● bazel run ● deployments We need deps for:
- 4. # requirements.lock boto3==1.10.2 botocore==1.13.2 # via boto3, s3transfer certifi==2019.9.11 # via requests chardet==3.0.4 # via requests docutils==0.15.2 # via botocore futures==3.3.0 # via s3transfer idna==2.8 # via requests jmespath==0.9.4 # via boto3, botocore python-dateutil==2.8.0 # via botocore requests==2.22.0 s3transfer==0.2.1 # via boto3 six==1.12.0 # via python-dateutil urllib3==1.25.6 # via botocore, requests py_binary( name = "my_app", deps = ["lib_a"], requires = ["requests>=2.21"], ) py_library( name = "lib_a", deps = ["lib_b"], requires = ["requests>=2.18"], ) py_library( name = "lib_b", deps = ["lib_c"], requires = ["boto3~=1.9"], ) py_venv( name = "my_app_env", targets = [":my_app"], ) # requirements.in boto3~=1.9 requests>=2.18 requests>=2.21
- 5. Needs improvement: ● Dev/CI workflow more complicated than before ● Bazel cache misses ● Dependency changes are not propagated automatically Working well: ● 100% correct (and minimal) dependencies! ● Monorepo-friendly ● History of dependency changes (lock files) ● Wheels are easy
- 6. Thank You 🙏🏻 ron {at} beeswax.com
Editor's Notes
- #2: Hi everyone...
- #3: We're 30 / monorepo / 100 / 300k The problem... how to manage our Py 3p deps By third-party... [C]When we bāzel-test or bāzel-run some target... also deps of every other target in its xtive clo same in prod [C]When had one or two apps, we solved... reqs.txt ...this didn't scale well once critical mass by multiple engs rules_python - can't guarantee import correct version pipenv Which led us to roll...
- #4: Our solution, which we call Beeve... set of tools pip-tools is... composed of starlark macros/rules/aspects + command line tools [C]Beeve's output is... A lockfile is a set of 3p deps, each pinned an application's reqs.txt, but autogenerated as needed We leveraged: Bāzel's... walk the graph, collect 3pd; pip-tools's... resolve deps [C]We take each app's lockfile and use it...
- #5: Here's an example. toy BUILD Note we've annotated each py rule... new attr requires... dep spec a dep spec is just a constraint on py_venv - tells Beeve we want lockfile for the... When run Bāzel on py_venv rule, our Starlark code... [C] requirements.in... union [C]then calls pip-compile to generate... the lockfile contains pinned versions of... as well as their transitive deps... ie it's the complete set of Py pkgs that our py_binary... not hermetic; nor reproducible; current state PyPI if you run... different lockfiles guaranteed that any resulting lockfile conforms to constraints... !!!
- #6: one downside is... devs must now be aware of py envs, switch between them Bāzel-test error prone Bāzel caching less effective 3p dep changes are not propagated automatically to downstream... [C]on plus side... most important goal... reliably know precise set of 3p deps... contains everything/nothing monorepo-friendly... go without saying nice side benefit... track dep changes over time, since... can easily produce wheels for our Py apps, has made deployments...
- #7: I'm happy to talk more about Beeve or Python dependencies ...email me with any questions or just find me at the conference. Thank you!