Add production image support #7832

potiuk · 2020-03-23T12:37:59Z

Issue link: WILL BE INSERTED BY boring-cyborg

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

Dockerfile

mik-laj · 2020-03-23T15:45:48Z

There is some error in the scripts.
When I ran following command:

bash breeze restart -a 1.10.6 && bash breeze -a 1.10.6

I got following error:

Installs version of Airflow: 1.10.6

Restarts the environment. Includes emptying the databases.

Fixing group permissions
Fixed group permissions

Docker image build is forced for CI image

Checking if the remote image needs to be pulled

/Users/kamilbregula/devel/google-airflow/airflow/scripts/ci/_utils.sh: line 628: AIRFLOW_REMOTE_MANIFEST_IMAGE: unbound variable

potiuk · 2020-03-23T15:48:18Z

/Users/kamilbregula/devel/google-airflow/airflow/scripts/ci/_utils.sh: line 628: AIRFLOW_REMOTE_MANIFEST_IMAGE: unbound variable

will take a look thanks!

BREEZE.rst

Dockerfile

entrypoint.sh

BREEZE.rst

potiuk · 2020-04-02T02:16:09Z

All corrected - please take another look

potiuk · 2020-04-02T09:46:33Z

Hey @ashb @kaxil - I want to merge it to 1.10.10 before we release, Is there anything else you want to comment on?

ashb · 2020-04-02T13:38:16Z

Taking another look now

Dockerfile

IMAGES.rst

kaxil

Let's do other minor changes in follow up PRs

potiuk · 2020-04-02T16:02:48Z

🎉 🎉 !

Dockerfile

dimberman · 2020-04-02T17:09:07Z

@potiuk I can't tell if I need to reload something but I'm not seeing build-image in auto-complete

(airflow) dimberman airflow ((HEAD detached at polidea/add-production-docker-image)) (⎈ |N/A|N/A)
$ ./breeze build-
build-docs  build-only
(airflow) dimberman airflow ((HEAD detached at polidea/add-production-docker-image)) (⎈ |N/A|N/A)
$ ./breeze build-image --production-image

feluelle · 2020-04-02T17:14:48Z

@dimberman run breeze-complete script again. (. ./breeze-complete did it for me)

dimberman · 2020-04-02T17:16:33Z

@feluelle works! had to run ./breeze setup-autocomplete and then restart shell. Might be worth mentioning in the docs or UPDATING.md?

feluelle · 2020-04-02T17:42:41Z

IMAGES.rst

+
+.. code-block::
+
+  docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \


Suggested change

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.6-slim-buster" \

feluelle · 2020-04-02T17:49:10Z

IMAGES.rst

+  docker build . --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+    --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 --build-arg COPY_SOURCE=. \
+    --build-arg COPY_TARGET=/opt/airflow --build-arg AIRFLOW_SOURCES=/opt/airflow \
+    --build-arg CONSTRAINT_REQUIREMENTS=requirements/requirements-python3.7.txt" \
+    --build-arg ENTRYPOINT_FILE=entrypoint.sh \
+    --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+    --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
+    --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"
+    --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \
+    --build-arg SOURCES_FROM="Dockerfile" \
+    --build-arg SOURCES_TO="/Dockerfile" \
+    --build-arg WWW_FOLDER="www_rbac"


There is sth. wrong.

Suggested change

docker build . --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \

--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 --build-arg COPY_SOURCE=. \

--build-arg COPY_TARGET=/opt/airflow --build-arg AIRFLOW_SOURCES=/opt/airflow \

--build-arg CONSTRAINT_REQUIREMENTS=requirements/requirements-python3.7.txt" \

--build-arg ENTRYPOINT_FILE=entrypoint.sh \

--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \

--build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \

--build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"

--build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \

--build-arg SOURCES_FROM="Dockerfile" \

--build-arg SOURCES_TO="/Dockerfile" \

--build-arg WWW_FOLDER="www_rbac"

docker build .

--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \

--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \

--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \

--build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \

--build-arg CONSTRAINT_REQUIREMENTS=requirements/requirements-python3.7.txt" \

--build-arg ENTRYPOINT_FILE=entrypoint.sh \

--build-arg AIRFLOW_SOURCES_FROM="Dockerfile" \

--build-arg AIRFLOW_SOURCES_TO="/Dockerfile" \

--build-arg WWW_FOLDER="www_rbac"

Like this @potiuk ?

IMAGES.rst

(cherry-picked from 07fd0d7)

potiuk · 2020-04-02T18:13:30Z

@feluelle works! had to run ./breeze setup-autocomplete and then restart shell. Might be worth mentioning in the docs or UPDATING.md?

I think it really depend on your setup. simply '. ./breeze-complete' should do. Did you have another directory with airflow somewhere? Tejh setup-autocomplete sets up a symbolic link to breeze-complete and only one can be active at the same time ... It's a bit tricky to get it right if you have more than one breeze command... The command itself will work though (it will use correct definition). I keep the master v1_10_test in sync so most of the auto-complete is good for both (some values like python version are not) and eventually you will get it right next time you get into the shell...

Not sure if UPDATING.md is for those dev tools though.

potiuk requested review from BasPH, ashb, dimberman, feluelle, kaxil, mik-laj and turbaszek March 23, 2020 12:37

boring-cyborg bot added the area:dev-tools label Mar 23, 2020

mik-laj reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

mik-laj reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

mik-laj reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

mik-laj reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

potiuk force-pushed the add-production-docker-image branch from ff1d713 to ba1927a Compare March 23, 2020 13:26

mik-laj reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

mik-laj reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

potiuk mentioned this pull request Mar 23, 2020

[AIRFLOW-5831] Prod image support. Depends on [AIRFLOW-5704] [AIRFLOW-5842] [AIRFLOW-5828] #6501

Closed

5 tasks

dimberman reviewed Mar 23, 2020

View reviewed changes

BREEZE.rst Outdated Show resolved Hide resolved

dimberman reviewed Mar 23, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

potiuk force-pushed the add-production-docker-image branch from 550e8ef to 91a424e Compare March 23, 2020 21:56

xinbinhuang reviewed Mar 24, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

turbaszek reviewed Mar 24, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

xinbinhuang reviewed Mar 24, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

potiuk force-pushed the add-production-docker-image branch 6 times, most recently from f0c94f8 to 7dce17b Compare March 25, 2020 05:22

ashb reviewed Mar 31, 2020

View reviewed changes

entrypoint.sh Outdated Show resolved Hide resolved

feluelle reviewed Apr 1, 2020

View reviewed changes

BREEZE.rst Outdated Show resolved Hide resolved

potiuk force-pushed the add-production-docker-image branch from 182a141 to dc4c5a5 Compare April 2, 2020 02:15

potiuk force-pushed the add-production-docker-image branch from dc4c5a5 to f0ee1c7 Compare April 2, 2020 07:45

ashb reviewed Apr 2, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

ashb reviewed Apr 2, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

turbaszek reviewed Apr 2, 2020

View reviewed changes

IMAGES.rst Outdated Show resolved Hide resolved

turbaszek reviewed Apr 2, 2020

View reviewed changes

IMAGES.rst Outdated Show resolved Hide resolved

potiuk force-pushed the add-production-docker-image branch from f0ee1c7 to 37014a4 Compare April 2, 2020 15:25

turbaszek reviewed Apr 2, 2020

View reviewed changes

IMAGES.rst Outdated Show resolved Hide resolved

Add production image support

314f919

potiuk force-pushed the add-production-docker-image branch from 37014a4 to 314f919 Compare April 2, 2020 15:48

ashb approved these changes Apr 2, 2020

View reviewed changes

kaxil approved these changes Apr 2, 2020

View reviewed changes

turbaszek approved these changes Apr 2, 2020

View reviewed changes

kaxil added this to the Airflow 1.10.10 milestone Apr 2, 2020

zhongjiajie reviewed Apr 2, 2020

View reviewed changes

Dockerfile Show resolved Hide resolved

feluelle reviewed Apr 2, 2020

View reviewed changes

kaxil merged commit 07fd0d7 into apache:master Apr 2, 2020

kaxil pushed a commit that referenced this pull request Apr 2, 2020

Add Production Docker image support (#7832)

f49afcc

(cherry-picked from 07fd0d7)


		.. code-block::

		docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \

Add production image support #7832

Add production image support #7832

Uh oh!

Conversation

potiuk commented Mar 23, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mik-laj commented Mar 23, 2020

Uh oh!

potiuk commented Mar 23, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

potiuk commented Apr 2, 2020

Uh oh!

potiuk commented Apr 2, 2020

Uh oh!

ashb commented Apr 2, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaxil left a comment

Choose a reason for hiding this comment

Uh oh!

potiuk commented Apr 2, 2020

Uh oh!

Uh oh!

dimberman commented Apr 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

feluelle commented Apr 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimberman commented Apr 2, 2020

Uh oh!

feluelle Apr 2, 2020

Choose a reason for hiding this comment

Uh oh!

potiuk Apr 2, 2020

Choose a reason for hiding this comment

Uh oh!

feluelle Apr 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

potiuk commented Apr 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

dimberman commented Apr 2, 2020 •

edited

Loading

feluelle commented Apr 2, 2020 •

edited

Loading

feluelle Apr 2, 2020 •

edited

Loading