Getting Started on Google Cloud Platform
4 likes660 views
The document outlines Meta's strategies for using Google Cloud Platform (GCP) to address various operational challenges, such as user onboarding and processing heterogeneous data sources. It highlights the advantages of GCP's App Engine, task queues, and autoscaling features, enabling a focus on product development while maintaining a lean team. Furthermore, it discusses the current development processes, recent advancements, and provides examples for getting started with GCP using the Go programming language.
![Datastoretype Greeting struct {
Author string
Content string
Date time.Time
}
// guestbookKey returns the key used for all guestbook entries.
func guestbookKey(c appengine.Context) *datastore.Key {
// The string "default_guestbook" here could be varied to have multiple guestbooks.
return datastore.NewKey(c, "Guestbook", "default_guestbook", 0, nil)
}
func root(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
// Ancestor queries, as shown here, are strongly consistent with the High
// Replication Datastore. Queries that span entity groups are eventually
// consistent. If we omitted the .Ancestor from this query there would be
// a slight chance that Greeting that had just been written would not
// show up in a query.
q := datastore.NewQuery("Greeting").Ancestor(guestbookKey(c)).Order("-Date").Limit(10)
greetings := make([]Greeting, 0, 10)
if _, err := q.GetAll(c, &greetings); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if err := guestbookTemplate.Execute(w, greetings); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}](https://image.slidesharecdn.com/usinggooglev3-160614162026/85/Getting-Started-on-Google-Cloud-Platform-31-320.jpg)
Getting Started on Google Cloud Platform
- 1. Getting Started on Google Cloud Platform Aaron Taylor @ataylor0123
- 2. access any file in seconds, wherever it is. www.meta.sc
- 7. Talk Roadmap • What problems we face at Meta • How we are solving them using GCP • How you can get started on GCP
- 8. Building a product • No baggage, free to choose whatever stack we want • Take advantage of latest technologies • but not quite bleeding edge
- 9. Engineering Goals • This will be a complex product, it needs to be comprehensible to everyone on our team • Keep the team as lean as possible • Focus on product, not sysadmin and dev ops
- 10. Language Choices • Go chosen as our primary language • Python for NLP and data analysis • enables easy experimentation, comfortable for data scientists and developers • Java/Scala interacting with Dataflow, Apache Tika, etc.
- 11. Our Hard Problems • User onboarding load • Heterogeneous (changing) data sources • Unpredictable traffic from web hooks • Compute loads for file content analysis • Processing streaming data
- 12. User Onboarding • Crawl multiple cloud accounts at once • Parallel computation • In-process using Go • Distributed using tasks • App Engine Taskqueues
- 13. Heterogeneous Data • Remove complexity of third-party services • Detect changes/ breakages in APIs • Distributed by nature • Continuous Deployment • Datastore • BigQuery
- 14. Unpredictable Traffic • Changes are pushed to us through web hooks • Dropping changes generally unacceptable • One user should not negatively impact others • App Engine autoscaling • Asynchronous task queues
- 15. Compute loads • Rich file content analysis • Parallel computation • App Engine Flexible Runtimes • CPU-based autoscaling
- 16. Stream Processing • Efficient handling of high-volume changes • Collate events in succession, from multiple users • Google Cloud Pub/Sub • Google Cloud Dataflow
- 17. How we started off • App Engine is our entry point • Service Oriented Architecture • Currently ~37 different services • Cloud Datastore is our persistence layer • BigQuery as a data warehouse
- 18. Documentation • Lots of information for getting started • Quality resources for our growing team • Onboarding new developers without GCP experience has been a breeze • Google is devoting lots of resources to this area
- 19. App Engine • Don’t worry about servers • Cache, task queues, cron, database, logging, monitoring, and more all built in • Powerful, configurable autoscaling • Heavy compute on App Engine Flexible Runtimes
- 20. Development Process • Build, run, and test services locally • Continuous deployment to a development project • Incremental releases go to production project • Logging and monitoring easy to setup
- 21. Problems we faced • Mantra of “don’t worry about scalability”didn’t take us very far • Users have lots and lots of files • Datastore use optimizations • Cost issues with App Engine • Trimming auto-scaling parameters • Migrated heavy compute to Flexible Runtimes
- 22. Outside GCP • Algolia • Hosts infrastructure for our search indices • Pusher • realtime socket connections • Postmark/Mailchimp • transactional and campaign-based email
- 23. Growth of the platform • Rapid changes and improvements taking place • Flexible Runtimes • Container Engine • Dataflow • Investing in a documentation overhaul soon • Support is generally quite responsive
- 24. Recent Developments • Introduction of Pub/Sub to our system for all event processing • Experimenting with Kubernetes/Container Engine • Dataflow stream processing jobs • Splitting functionality into multiple projects
- 25. Quickstart Documentation for Go How you can start off
- 26. Hello World in Go https://cloud.google.com/appengine/docs/go/quickstart
- 27. Server package hello import ( "fmt" "net/http" ) func init() { http.HandleFunc("/", handler) } func handler(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "Hello, world!") } hello.go
- 28. Configuration runtime: go api_version: go1 handlers: - url: /.* script: _go_app app.yaml
- 31. Datastoretype Greeting struct { Author string Content string Date time.Time } // guestbookKey returns the key used for all guestbook entries. func guestbookKey(c appengine.Context) *datastore.Key { // The string "default_guestbook" here could be varied to have multiple guestbooks. return datastore.NewKey(c, "Guestbook", "default_guestbook", 0, nil) } func root(w http.ResponseWriter, r *http.Request) { c := appengine.NewContext(r) // Ancestor queries, as shown here, are strongly consistent with the High // Replication Datastore. Queries that span entity groups are eventually // consistent. If we omitted the .Ancestor from this query there would be // a slight chance that Greeting that had just been written would not // show up in a query. q := datastore.NewQuery("Greeting").Ancestor(guestbookKey(c)).Order("-Date").Limit(10) greetings := make([]Greeting, 0, 10) if _, err := q.GetAll(c, &greetings); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := guestbookTemplate.Execute(w, greetings); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }
- 32. Templates var guestbookTemplate = template.Must(template.New("book").Parse(` <html> <head> <title>Go Guestbook</title> </head> <body> {{range .}} {{with .Author}} <p><b>{{.}}</b> wrote:</p> {{else}} <p>An anonymous person wrote:</p> {{end}} <pre>{{.Content}}</pre> {{end}} <form action="/sign" method="post"> <div><textarea name="content" rows="3" cols="60"></textarea></div> <div><input type="submit" value="Sign Guestbook"></div> </form> </body> </html> `))
- 33. Forms func sign(w http.ResponseWriter, r *http.Request) { c := appengine.NewContext(r) g := Greeting{ Content: r.FormValue("content"), Date: time.Now(), } if u := user.Current(c); u != nil { g.Author = u.String() } // We set the same parent key on every Greeting entity to ensure each Greeting // is in the same entity group. Queries across the single entity group // will be consistent. However, the write rate to a single entity group // should be limited to ~1/second. key := datastore.NewIncompleteKey(c, "Greeting", guestbookKey(c)) _, err := datastore.Put(c, key, &g) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } http.Redirect(w, r, "/", http.StatusFound) }
- 34. Conclusions • Google Cloud Platform has allowed us to build out Meta in ways that wouldn’t otherwise be feasible • Simplicity of App Engine allows us to focus on product • Scalability/Availability are built in to the platform
- 35. access any file in seconds, wherever it is. www.meta.sc/careers [email protected]