Continuous Integration & Continuous Delivery with GCP
12 likes3,974 views
The document discusses the Google Cloud Platform User Group in Taiwan, focusing on sharing knowledge and experiences related to Google Cloud services. It covers topics like App Engine development, continuous integration, and delivery with tools such as Docker and GitLab, aiming to enhance collaboration between software developers and IT professionals. Additionally, it provides links to relevant resources and study information for users of varying expertise in Google Cloud Platform.
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-10-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-11-320.jpg)
![Devops
DevOps (a clipped compound of
development and operations) is
a culture, movement or practice
that emphasizes the
collaboration and
communication of both software
developers and other
information-technology (IT)
professionals while automating
the process of software delivery
and infrastructure changes.[1][2]
It aims at establishing a culture
and environment where
building, testing, and releasing
software, can happen rapidly,
frequently, and more reliably -
DevOps - Wikipedia
[image](https://goo.gl/pMC6UZ)](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-13-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-14-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-15-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-22-320.jpg)
// list and register gitlab-ci-runner
$ gitlab-ci-multi-runner list
Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml
gae-flask-gitlab-ci Executor=docker Token=95d56b26104856b9f3852f2d82b1e5 URL=https://gitlab.com/ci
// prepare a docker-machine for gitlab-ci-multi-runner
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0
// run a gitlab-ci-multi-runner test via docker
$ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY
GCLOUD_ACCOUNT=$GCLOUD_ACCOUNT GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION"
// push git
(gae-flask-gitlab-ci) $ git push master](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-25-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-30-320.jpg)
![Google Cloud Source Repositories repo
$ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default
// generates a file named source-context.json, which contains information about the version of the
source code used to build the application
$ gcloud preview app gen-repo-info-file
// push code to google cloud source repositories
$ git push google master
[/ - GAE Labs](https://goo.gl/c99Dz8)](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-31-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-32-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-33-320.jpg)
](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-36-320.jpg)
[Vendorization for deployment - Google Groups](https://groups.google.com/forum/#!topic/google-appengine-go/umcPV_4tAm8)](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-38-320.jpg)
○ [Configuration of your builds with .gitlab-ci.yml](http://doc.gitlab.
com/ee/ci/yaml/README.html)
○ [GitLab.org / gitlab-ci-multi-runner · GitLab](https://gitlab.
com/gitlab-org/gitlab-ci-multi-runner)
◎ Deploying to App Engine or Managed VMs from Travis CI
○ [Deploying to App Engine or Managed VMs from Travis CI - YouTube]
(https://www.youtube.com/watch?v=7U4jjRw_AJk&feature=youtu.
be)
○ [GoogleCloudPlatform/continuous-deployment-demo](https:
//github.com/GoogleCloudPlatform/continuous-deployment-
demo)](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-39-320.jpg)
○ [Stackdriver Debugger - Production Debugging - Google Cloud
Platform — Google Cloud Platform](https://cloud.google.
com/debugger/)
○ [Managed VMs lab · kaichu.io](http://kaichu.io/2015/09/managed-
vms-lab/)](https://image.slidesharecdn.com/continuousintegrationwithgcp-160513023721/85/Continuous-Integration-Continuous-Delivery-with-GCP-40-320.jpg)
Continuous Integration & Continuous Delivery with GCP
- 2. https://www.facebook.com/groups/GCPUG.TW/ https://plus.google.com/u/0/communities/116100913832589966421 Google Cloud Platform User Group Taiwan 我們是Google Cloud Platform Taiwan User Group。在Google雲端服務在台灣地區展露頭角之後， 有許多新的服務、新的知識、新的創意，歡迎大家一起分享，一起了解 Google雲端服務... GCPUG透過網際網路串聯喜好 Google Cloud的使用者，分享與交流使用 GCP的點滴鑑驗。如果您 是Google Cloud Platform的初學者，您應該來聽聽前輩們的使用經驗；如果您是 Google Cloud Platform的Expert，您應該來分享一下寶貴的經驗，並與更多高手互相交流；如果您還沒開始用 Google Cloud Platform，那麼您應該馬上來聽聽我們是怎麼使用 Google Cloud的！
- 3. Hello! I am Cage Chung I am here because I like to share my experiences. You can find me at: http://kaichu.io
- 4. Outline ◎ App Engine Development ◎ Devops: CI & CD (Docker, gitlab, GCP) ◎ Tips & Study information
- 5. 1. App Engine Development Let’s start with the first set of slides
- 6. Who is using GAE? Angry Birds Khan Academy Royal Wedding Snapchat EA Games Forbes Bestbuy https://cloud.google.com/customers/ Appengine
- 7. Google Cloud Platform | Compute PaasIaaS Application Data Runtime Middleware OS Virtualization Servers Storage Networking Application Data Runtime Middleware OS Virtualization Servers Storage Networking Saas Application Data Runtime Middleware OS Virtualization Servers Storage Networking Infrastructure as a Service Platform as a Service Service as a Service You manage Google manage
- 9. App Engine APIs ◎ Storage ◎ Cloud Storage ◎ Cloud SQL ◎ Cloud Datastore ◎ Memcache ◎ Task Queue ◎ Cloud Endpoints Find more https://cloud.google.com/appengine/
- 10. App Engine runtime Python2.7 Java7 Php Go RubyNode.js standard Flexible Beta Python2.7/3.4 Java8 Go custom [Google App Engine Documentation - App Engine — Google Cloud Platform](https://cloud.google.com/appengine/docs)
- 11. App Engine | waldo queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints standard Flexible Beta [Waldo-gcp](http://www.slideshare.net/cagechung/waldogcp)
- 12. 2. Devops: CI & CD (Docker, gitlab, GCP) Let’s start with the second set of slides
- 13. Devops DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.[1][2] It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably - DevOps - Wikipedia [image](https://goo.gl/pMC6UZ)
- 14. Devops: CI & CD (Docker, gitlab, GCP) [image](https://goo.gl/pMC6UZ)
- 15. Origin imgae from and modified [DevOps：持續整合＆持續交付（Docker、CircleCI、AWS）](http://blog.amowu.com/2015/04/devops- continuous-integration-delivery-docker-circleci-aws-beanstalk.html)
- 16. Simple flow ◎ Python (gae-flask) ○ Local Development ○ Local Test ◎ Gitlab ◎ Code Review with GitHub(Gitlab) Flow ◎ gitlab-ci-multi-runner ○ Local runner exec test docker ○ Gitlab runner test docker ◎ Google App Engine ○ Google Cloud Source Repositories ○ Stackdriver Debugger
- 17. Local Development / Test | gae-flask // clone repo $ git clone [email protected]:cage1016/gae-flask.git // make a new virtualenv $ mkvirtualenv gae-flask // pip install requirements.test.txt or link env as lib folder (gae-flask-gitlab-ci) $ pip install -r requirements.testing.txt -t lib // or (gae-flask-gitlab-ci) $ pip install git+https://github.com/ze-phyr-us/linkenv.git (gae-flask-gitlab-ci) $ linkenv env/lib/python2.7/site-packages lib // local run app engine project (gae-flask-gitlab-ci) $ dev_appserver.py app.yaml // local run test (gae-flask-gitlab-ci) $ python ./runtests.py
- 18. Local Development / mian.py import flask import requests app = flask.Flask(__name__) @app.route("/") def hello(): return "hello world" if __name__ == "__main__": app.run()
- 19. Local Development / test_main.py import pytest import main @pytest.fixture def app(): return main.app @pytest.fixture def test_client(app): return app.test_client() def test_hello(test_client): response = test_client.get("/") assert response.data.decode("utf-8") == "hello world"
- 22. gitlab-ci.yml image: cage1016/gitlab-ci-gcp:v1.5 before_script: - export CLOUDSDK_CORE_DISABLE_PROMPTS=1 - export CLOUDSDK_PYTHON_SITEPACKAGES=1 - export GCP_PROJECT=gae-lab-001 - export GCP_PROJECT_VERSION=uat types: - test - deploy test: stage: test script: - sh ./scripts/tests.sh deploy: stage: deploy script: - sh ./scripts/deploy.sh only: master [GitLab Documentation](http://doc.gitlab.com/ee/ci/yaml/README.html#only-and-except)
- 23. gitlab-ci.yml (tests.sh) #!/usr/bin/env bash virtualenv env source env/bin/activate # install test env packages from requirements.testing.txt pip install -r requirements.testing.txt -t lib # link env as lib folder if you just execute `pip install -r requirements.testing.txt` # pip install git+https://github.com/ze-phyr-us/linkenv.git # linkenv env/lib/python2.7/site-packages lib # run tests python runtests.py
- 24. gitlab-ci.yml (deploy.sh) #!/usr/bin/env bash pip install -r requirements.txt -t lib/ echo $GCLOUD_KEY > key.json gcloud auth activate-service-account $GCLOUD_ACCOUNT --key-file key.json gcloud --quiet config set project $GCP_PROJECT gcloud --quiet preview app deploy app.yaml --no-promote --version $GCP_PROJECT_VERSION
- 25. Local runner exec test docker // [install gitlab-ci-multi-runner](https://goo.gl/a99Aa4) // list and register gitlab-ci-runner $ gitlab-ci-multi-runner list Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml gae-flask-gitlab-ci Executor=docker Token=95d56b26104856b9f3852f2d82b1e5 URL=https://gitlab.com/ci // prepare a docker-machine for gitlab-ci-multi-runner $ docker-machine ls NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0 // run a gitlab-ci-multi-runner test via docker $ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY GCLOUD_ACCOUNT=$GCLOUD_ACCOUNT GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION" // push git (gae-flask-gitlab-ci) $ git push master
- 26. Docker cage1016/gitlab-ci-gcp:v1.5 FROM buildpack-deps:jessie # remove several traces of debian python RUN apt-get purge -y python.* RUN apt-get update && apt-get install -y -qq --no-install-recommends unzip openssh-client python-openssl # install "virtualenv", since the vast majority of users of this image will want it RUN pip install --no-cache-dir virtualenv # Install the Google Cloud SDK. # archive gcloud sdk https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-105.0.0- linux-x86_64.tar.gz ENV HOME / ENV CLOUDSDK_PYTHON_SITEPACKAGES 1 RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-102.0.0-linux-x86_64. tar.gz && tar -zxvf google-cloud-sdk-102.0.0-linux-x86_64.tar.gz && rm google-cloud-sdk-102.0.0-linux- x86_64.tar.gz RUN google-cloud-sdk/install.sh --usage-reporting=true --path-update=true --bash-completion=true --rc- path=/.bashrc --additional-components app-engine-java app-engine-python app kubectl alpha beta ... https://github.com/cage1016/gitlab-ci-gcp/blob/master/Dockerfile
- 29. Push a new branch to Gitlab https://gitlab.com/cage1016/gae-flask/builds
- 30. Google Cloud Source Repositories [/ - GAE Labs](https://goo.gl/c99Dz8)
- 31. Google Cloud Source Repositories repo $ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default // generates a file named source-context.json, which contains information about the version of the source code used to build the application $ gcloud preview app gen-repo-info-file // push code to google cloud source repositories $ git push google master [/ - GAE Labs](https://goo.gl/c99Dz8)
- 32. Stackdriver Debugger | Source Code [Debug - GAE Labs](https://goo.gl/I3K6J2)
- 33. Stackdriver Debugger | Debug [Debug - GAE Labs](https://goo.gl/I3K6J2)
- 34. 3. Tips & Study information Let’s start with the third set of slides
- 35. “ 5G Google cloud storage for App engine for FREE <project-id>.appspot.com staging.<project-id>.appspot.com
- 36. “ App Engine Managed VMs is renamed to App Engine flexible environment. March 24, 2016 [Python Release Notes - Python — Google Cloud Platform](https://cloud.google.com/appengine/docs/python/release-notes#1.9.35)
- 37. “ App Engine Admin API Cloud Source Repositories API Google Cloud Storage
- 38. “ At the moment, the Go app engine SDK is still based on Go 1.4, even if you use the go1.6beta version for your app. This means that the SDK doesn't understand 1.6-style vendoring, and so doesn't correctly upload the vendored dependencies. This will be fixed when 1.6 becomes the stable (go1) version and is used as the basis of the SDK. Stay tuned! [gcpug.tw.go/.travis.yml at master · gcpug-tw/gcpug.tw.go](https://github.com/gcpug-tw/gcpug.tw.go/blob/master/.travis.yml) [Vendorization for deployment - Google Groups](https://groups.google.com/forum/#!topic/google-appengine-go/umcPV_4tAm8)
- 39. Study information ◎ Gitlab ○ [Quick Start](http://doc.gitlab.com/ce/ci/quick_start/README. html) ○ [Configuration of your builds with .gitlab-ci.yml](http://doc.gitlab. com/ee/ci/yaml/README.html) ○ [GitLab.org / gitlab-ci-multi-runner · GitLab](https://gitlab. com/gitlab-org/gitlab-ci-multi-runner) ◎ Deploying to App Engine or Managed VMs from Travis CI ○ [Deploying to App Engine or Managed VMs from Travis CI - YouTube] (https://www.youtube.com/watch?v=7U4jjRw_AJk&feature=youtu. be) ○ [GoogleCloudPlatform/continuous-deployment-demo](https: //github.com/GoogleCloudPlatform/continuous-deployment- demo)
- 40. Study information cont. ◎ GCP ○ [Cloud Source Repositories - Private Git Repositories — Google Cloud Platform](https://cloud.google.com/source-repositories/) ○ [Stackdriver Debugger - Production Debugging - Google Cloud Platform — Google Cloud Platform](https://cloud.google. com/debugger/) ○ [Managed VMs lab · kaichu.io](http://kaichu.io/2015/09/managed- vms-lab/)
- 41. Thanks! Any questions? You can find me at: http://kaichu.io [email protected]