SlideShare a Scribd company logo
GAE Developer - Day3
Simon @ MiCloud
2014Q1
Last Class
● Datastore basic
● Datastore operation
● Restrictions
● Transaction
Create an Entity with Key
Entity job = new Entity("User", "simonsu@mitac.com.tw");
job.setProperty("name", "Simon Su");
job.setProperty("start", "20140103");
job.setProperty("create", new Date());
try {
Key k = ds.put(job);
out.println("Done..." + k.getId());
} catch (Exception e) {
e.printStackTrace();
}
Create an Entity with Ancestor
Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
Entity job = new Entity("Jobs", userKey);
job.setProperty("name", "engineer");
job.setProperty("start", new Date());
try {
Key k = ds.put(job);
out.println("Done..." + k.getId());
} catch (Exception e) {
e.printStackTrace();
}
Get data with Key
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Key key = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
try {
Entity ent = ds.get(key);
out.print(ent);
Map m = ent.getProperties();
out.println(m.toString());
} catch (EntityNotFoundException e) {
e.printStackTrace();
}
Get data with Ancestor
Query q = new Query("Jobs");
Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
q.setAncestor(ancestor);
PreparedQuery results = ds.prepare(q);
Iterator iter = results.asIterable().iterator();
while(iter.hasNext()) {
Entity ent = (Entity) iter.next();
out.println(ent);
}
Today
● Memcache basic
● Task Queue concept
● Push Queue
● Pull Queue
Memcache
Why memcache
● Improve Application Performance
● Reduce Application Cost
● Caching for Read heavy operations
● Caching In Front of Datastore
● Semi-durable Shared state Across App
Instances
Memcache - operation
MemcacheService cache = MemcacheServiceFactory.getMemcacheService();
cache.setErrorHandler(
ErrorHandlers.getConsistentLogAndContinue(Level.INFO));
// read from cache
value = (byte[]) cache.get(key);
// save to cache
if (value == null) {
// ........
cache.put(key, value);
}
Async memcache - get
// Using the asynchronous cache
AsyncMemcacheService asyncCache =
MemcacheServiceFactory.getAsyncMemcacheService();
asyncCache.setErrorHandler(
ErrorHandlers.getConsistentLogAndContinue(Level.INFO));
// read from cache
Future<Object> futureValue = asyncCache.get(key);
// ... do other work in parallel to cache retrieval
value = (byte[]) futureValue.get();
Async memcache - put
if (value == null) {
// get value from other source
// ........
// asynchronously populate the cache
// Returns a Future<Void> which can be used to block until completion
asyncCache.put(key, value);
}
Using jcache - init
import java.util.Collections;
import net.sf.jsr107cache.*;
Cache cache;
try {
CacheFactory cacheFactory =
CacheManager.getInstance().getCacheFactory();
cache = cacheFactory.createCache(Collections.emptyMap());
} catch (CacheException e) {
// ...
}
Using jcache - get & put
String key; // ...
byte[] value; // ...
// Put the value into the cache.
cache.put(key, value);
// Get the value from the cache.
value = (byte[]) cache.get(key);
Testing with ab benchmark
ab -c 50 -n 50 http://[app-address]/[servlet_path]
or
ab -c 50 -t 50 http://[app-address]/[servlet_path]
Other Operation
● getAll(), putAll(), deleteAll()
A single read or write operation for multiple memcache
entries
● increment(key, delta), incrementAll(...)
Provide atomic increment of numeric value(s)
● getIdentifiable(), putIfUntouched()
A mechanism to update a value consistently by
concurrent requests
Memcache Caveat
● Memcache Is Volatile
● Performance suggest: Batch size < 32 MB
● Dedicated memcache (pay): 10K OPS/s/GB,
range 1-20 GB
Task Queue
Why Task Queue
● Frontend restriction of 60 second limit
● Need longer-running tasks
● Asynchronously outside front end request
● Best practice:
○ Email
○ Write datastore
○ Web crawler
○ Backend jobs
Task Queue Types
● By name
○ Default Queue (Push Queue)
○ Named Queue
● By type
○ Push queues
■ tasks execute automatically
■ only for App Engine app
○ Pull queues
■ tasks wait to be leased
■ available to workers outside the app
■ tasks can be batched
queue.xml
<queue-entries>
<queue>
<name>Queue-Name</name>
<rate>1/s</rate>
...
</queue>
<queue>
<name>Queue-Name2</name>
<mode>pull</mode>
...
</queue>
</queue-entries>
Push Queue
Pull Queue
Task Queue Manage Console
Task Queue - Default Push Queue
Queue queue = QueueFactory.getDefaultQueue();
//named queue:
//QueueFactory.getQueue("Qname");
TaskOptions task =
TaskOptions.Builder
.withUrl("/path-to-my-worker")
.param(key, value);
queue.add(task);
Reachable Servlet
Detail Configure
● Task queues use token bucket algorithm [ref]
● rate - usual rate
● bucket-size - cap on peak demand
● max concurrent requests
● Retry options
○ task-retry-limit - min retries
○ task-age-limit - min elapsed time to keep retrying
(Both task-retry-limit and task-age-limit are reached, task is deleted)
○ min-backoff-seconds -- min delay between retries
○ max-backoff-seconds -- max delay between retries
○ max-doublings -- max times to double delay
● more parameters see DOC
queue.xml parameters
Task Queue - With Transaction
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Queue queue = QueueFactory.getDefaultQueue();
try {
Transaction txn = ds.beginTransaction();
// … other operations
queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker"));
// … other operations
txn.commit();
} catch (DatastoreFailureException e) {
// … exception handle
}
Pull Queue - Create Queue
Queue queue =
QueueFactory.getQueue("queue_name");
TaskOptions task =
TaskOptions.Builder
.withMethod(TaskOptions.Method.PULL)
.payload(payload);
queue.add(task);
Pull Queue - Lease
Queue queue =
QueueFactory.getQueue("queue_name");
tasks = q.leaseTasks(
duration,
TimeUnit.SECONDS,
how-many);
Pull Queue - Delete Task
//delete named task
queue.deleteTask("task_ name");
//purge all tasks from queue
QueueFactory.getQueue("queue_name").purge();
Task Queue ACL - queue.xml
<queue>
<name>pull-queue1</name>
<mode>pull</mode>
<acl>
<user-email>me@gmail.com</user-email>
<writer-email>you@gmail.com</writer-email>
<writer-email>she@gmail.com</writer-email>
</acl>
</queue>
Cron
Cron
<cronentries>
<cron>
<url>/recache</url>
<description>
Put your description here
</description>
<schedule>every 5 minutes</schedule>
</cron>
</cronentries>
● every 12 hours
● every 5 minutes from 10:00 to 14:00
● 2nd,third mon,wed,thu of march 17:00
● every monday 09:00
● 1st monday of sep,oct,nov 17:00
● every day 00:00
● every N (hours|mins|minutes) ["from" (time) "to" (time)]
● every 2 hours synchronized
Cron Schedule Format
Cron Manage Console
Restrictions
● 20 cron jobs for free apps
● 100 for paid apps
Task Queue / Cron Reference
● Task Queues Developer Guide (Java) (Python)
● Task Queue Configuration (Java) (Python)
● Task Queue API Ref (JavaDoc) (Python)
● Task Queue REST API Ref
● Scheduled Tasks with Cron (Java) (Python)
● Task queue quota and limits
● Get started with Task API article (Python)
● Google I/O 2011: Putting Task Queues to Work (video)

More Related Content

What's hot (20)

PDF
Jggug 2010 330 Grails 1.3 観察
Tsuyoshi Yamamoto
 
PDF
Wprowadzenie do technologi Big Data i Apache Hadoop
Sages
 
PDF
The Ring programming language version 1.10 book - Part 79 of 212
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.2 book - Part 48 of 84
Mahmoud Samir Fayed
 
PPTX
Mysql handle socket
Philip Zhong
 
PDF
Zone.js 2017
Jia Li
 
PDF
The Ring programming language version 1.2 book - Part 42 of 84
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.9 book - Part 78 of 210
Mahmoud Samir Fayed
 
PPTX
Compare mysql5.1.50 mysql5.5.8
Philip Zhong
 
PDF
The Ring programming language version 1.2 book - Part 47 of 84
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.6 book - Part 69 of 189
Mahmoud Samir Fayed
 
PPTX
Mysql5.1 character set testing
Philip Zhong
 
PPT
Moar tools for asynchrony!
Joachim Bengtsson
 
PDF
The Ring programming language version 1.6 book - Part 64 of 189
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.6 book - Part 70 of 189
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.7 book - Part 16 of 196
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.2 book - Part 46 of 84
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.5.1 book - Part 63 of 180
Mahmoud Samir Fayed
 
PDF
Ns2: Introduction - Part I
Ajit Nayak
 
PDF
BDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
Christian Baranowski
 
Jggug 2010 330 Grails 1.3 観察
Tsuyoshi Yamamoto
 
Wprowadzenie do technologi Big Data i Apache Hadoop
Sages
 
The Ring programming language version 1.10 book - Part 79 of 212
Mahmoud Samir Fayed
 
The Ring programming language version 1.2 book - Part 48 of 84
Mahmoud Samir Fayed
 
Mysql handle socket
Philip Zhong
 
Zone.js 2017
Jia Li
 
The Ring programming language version 1.2 book - Part 42 of 84
Mahmoud Samir Fayed
 
The Ring programming language version 1.9 book - Part 78 of 210
Mahmoud Samir Fayed
 
Compare mysql5.1.50 mysql5.5.8
Philip Zhong
 
The Ring programming language version 1.2 book - Part 47 of 84
Mahmoud Samir Fayed
 
The Ring programming language version 1.6 book - Part 69 of 189
Mahmoud Samir Fayed
 
Mysql5.1 character set testing
Philip Zhong
 
Moar tools for asynchrony!
Joachim Bengtsson
 
The Ring programming language version 1.6 book - Part 64 of 189
Mahmoud Samir Fayed
 
The Ring programming language version 1.6 book - Part 70 of 189
Mahmoud Samir Fayed
 
The Ring programming language version 1.7 book - Part 16 of 196
Mahmoud Samir Fayed
 
The Ring programming language version 1.2 book - Part 46 of 84
Mahmoud Samir Fayed
 
The Ring programming language version 1.5.1 book - Part 63 of 180
Mahmoud Samir Fayed
 
Ns2: Introduction - Part I
Ajit Nayak
 
BDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
Christian Baranowski
 

Similar to Google App Engine Developer - Day3 (20)

PDF
JCConf 2015 - 輕鬆學google的雲端開發 - Google App Engine入門(上)
Simon Su
 
PDF
Google App Engine Developer - Day2
Simon Su
 
PPT
The 90-Day Startup with Google AppEngine for Java
David Chandler
 
PDF
Google app engine - Soft Uni 19.06.2014
Dimitar Danailov
 
PDF
Java Web Programming on Google Cloud Platform [2/3] : Datastore
IMC Institute
 
PDF
Appengine Java Night #2a
Shinichi Ogawa
 
KEY
Appengine Nljug
Paul Bakker
 
PDF
Java Support On Google App Engine
Xebia IT Architects
 
PDF
Developing, deploying and monitoring Java applications using Google App Engine
IndicThreads
 
PDF
Introduction to Datastore
Software Park Thailand
 
PPT
Talk 1: Google App Engine Development: Java, Data Models, and other things yo...
Singapore Google Technology User Group
 
PDF
Google Developer Days Brazil 2009 - Java Appengine
Patrick Chanezon
 
PDF
Memcache basics on google app engine
Ido Green
 
PDF
Google Cloud Platform 2014Q1 - Starter Guide
Simon Su
 
PPTX
CiklumJavaSat_15112011:Alex Kruk VMForce
Ciklum Ukraine
 
PDF
Hands on App Engine
Simon Su
 
KEY
2011 aug-gdd-mexico-city-high-replication-datastore
ikailan
 
PPT
LatJUG. Google App Engine
denis Udod
 
PDF
Java on Google App engine
Michael Parker
 
PDF
Easy ORM-ness with Objectify-Appengine - Indicthreads cloud computing confere...
IndicThreads
 
JCConf 2015 - 輕鬆學google的雲端開發 - Google App Engine入門(上)
Simon Su
 
Google App Engine Developer - Day2
Simon Su
 
The 90-Day Startup with Google AppEngine for Java
David Chandler
 
Google app engine - Soft Uni 19.06.2014
Dimitar Danailov
 
Java Web Programming on Google Cloud Platform [2/3] : Datastore
IMC Institute
 
Appengine Java Night #2a
Shinichi Ogawa
 
Appengine Nljug
Paul Bakker
 
Java Support On Google App Engine
Xebia IT Architects
 
Developing, deploying and monitoring Java applications using Google App Engine
IndicThreads
 
Introduction to Datastore
Software Park Thailand
 
Talk 1: Google App Engine Development: Java, Data Models, and other things yo...
Singapore Google Technology User Group
 
Google Developer Days Brazil 2009 - Java Appengine
Patrick Chanezon
 
Memcache basics on google app engine
Ido Green
 
Google Cloud Platform 2014Q1 - Starter Guide
Simon Su
 
CiklumJavaSat_15112011:Alex Kruk VMForce
Ciklum Ukraine
 
Hands on App Engine
Simon Su
 
2011 aug-gdd-mexico-city-high-replication-datastore
ikailan
 
LatJUG. Google App Engine
denis Udod
 
Java on Google App engine
Michael Parker
 
Easy ORM-ness with Objectify-Appengine - Indicthreads cloud computing confere...
IndicThreads
 
Ad

More from Simon Su (20)

PDF
Kubernetes Basic Operation
Simon Su
 
PDF
Google IoT Core 初體驗
Simon Su
 
PDF
JSDC 2017 - 使用google cloud 從雲到端,動手刻個IoT
Simon Su
 
PDF
GCPUG.TW meetup #28 - GKE上運作您的k8s服務
Simon Su
 
PDF
Google Cloud Platform Special Training
Simon Su
 
PDF
GCE Windows Serial Console Usage Guide
Simon Su
 
PDF
GCPNext17' Extend 開始GCP了嗎?
Simon Su
 
PDF
Try Cloud Spanner
Simon Su
 
PDF
Google Cloud Monitoring
Simon Su
 
PDF
Google Cloud Computing compares GCE, GAE and GKE
Simon Su
 
PDF
JCConf 2016 - Google Dataflow 小試
Simon Su
 
PDF
JCConf 2016 - Dataflow Workshop Labs
Simon Su
 
PDF
JCConf2016 - Dataflow Workshop Setup
Simon Su
 
PDF
GCPUG meetup 201610 - Dataflow Introduction
Simon Su
 
PDF
Brocade - Stingray Application Firewall
Simon Su
 
PDF
使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析
Simon Su
 
PDF
Docker in Action
Simon Su
 
PDF
Google I/O 2016 Recap - Google Cloud Platform News Update
Simon Su
 
PDF
IThome DevOps Summit - IoT、docker與DevOps
Simon Su
 
PDF
Google Cloud Platform Introduction - 2016Q3
Simon Su
 
Kubernetes Basic Operation
Simon Su
 
Google IoT Core 初體驗
Simon Su
 
JSDC 2017 - 使用google cloud 從雲到端,動手刻個IoT
Simon Su
 
GCPUG.TW meetup #28 - GKE上運作您的k8s服務
Simon Su
 
Google Cloud Platform Special Training
Simon Su
 
GCE Windows Serial Console Usage Guide
Simon Su
 
GCPNext17' Extend 開始GCP了嗎?
Simon Su
 
Try Cloud Spanner
Simon Su
 
Google Cloud Monitoring
Simon Su
 
Google Cloud Computing compares GCE, GAE and GKE
Simon Su
 
JCConf 2016 - Google Dataflow 小試
Simon Su
 
JCConf 2016 - Dataflow Workshop Labs
Simon Su
 
JCConf2016 - Dataflow Workshop Setup
Simon Su
 
GCPUG meetup 201610 - Dataflow Introduction
Simon Su
 
Brocade - Stingray Application Firewall
Simon Su
 
使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析
Simon Su
 
Docker in Action
Simon Su
 
Google I/O 2016 Recap - Google Cloud Platform News Update
Simon Su
 
IThome DevOps Summit - IoT、docker與DevOps
Simon Su
 
Google Cloud Platform Introduction - 2016Q3
Simon Su
 
Ad

Recently uploaded (20)

PDF
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
PPT
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
“Voice Interfaces on a Budget: Building Real-time Speech Recognition on Low-c...
Edge AI and Vision Alliance
 
PDF
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PPTX
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
PDF
NLJUG Speaker academy 2025 - first session
Bert Jan Schrijver
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PPTX
Digital Circuits, important subject in CS
contactparinay1
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
“Voice Interfaces on a Budget: Building Real-time Speech Recognition on Low-c...
Edge AI and Vision Alliance
 
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
NLJUG Speaker academy 2025 - first session
Bert Jan Schrijver
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Digital Circuits, important subject in CS
contactparinay1
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 

Google App Engine Developer - Day3

  • 1. GAE Developer - Day3 Simon @ MiCloud 2014Q1
  • 2. Last Class ● Datastore basic ● Datastore operation ● Restrictions ● Transaction
  • 3. Create an Entity with Key Entity job = new Entity("User", "[email protected]"); job.setProperty("name", "Simon Su"); job.setProperty("start", "20140103"); job.setProperty("create", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  • 4. Create an Entity with Ancestor Key userKey = KeyFactory.createKey("User", "[email protected]"); Entity job = new Entity("Jobs", userKey); job.setProperty("name", "engineer"); job.setProperty("start", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  • 5. Get data with Key DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Key key = KeyFactory.createKey("User", "[email protected]"); try { Entity ent = ds.get(key); out.print(ent); Map m = ent.getProperties(); out.println(m.toString()); } catch (EntityNotFoundException e) { e.printStackTrace(); }
  • 6. Get data with Ancestor Query q = new Query("Jobs"); Key ancestor = KeyFactory.createKey("User", "[email protected]"); q.setAncestor(ancestor); PreparedQuery results = ds.prepare(q); Iterator iter = results.asIterable().iterator(); while(iter.hasNext()) { Entity ent = (Entity) iter.next(); out.println(ent); }
  • 7. Today ● Memcache basic ● Task Queue concept ● Push Queue ● Pull Queue
  • 9. Why memcache ● Improve Application Performance ● Reduce Application Cost ● Caching for Read heavy operations ● Caching In Front of Datastore ● Semi-durable Shared state Across App Instances
  • 10. Memcache - operation MemcacheService cache = MemcacheServiceFactory.getMemcacheService(); cache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache value = (byte[]) cache.get(key); // save to cache if (value == null) { // ........ cache.put(key, value); }
  • 11. Async memcache - get // Using the asynchronous cache AsyncMemcacheService asyncCache = MemcacheServiceFactory.getAsyncMemcacheService(); asyncCache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache Future<Object> futureValue = asyncCache.get(key); // ... do other work in parallel to cache retrieval value = (byte[]) futureValue.get();
  • 12. Async memcache - put if (value == null) { // get value from other source // ........ // asynchronously populate the cache // Returns a Future<Void> which can be used to block until completion asyncCache.put(key, value); }
  • 13. Using jcache - init import java.util.Collections; import net.sf.jsr107cache.*; Cache cache; try { CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory(); cache = cacheFactory.createCache(Collections.emptyMap()); } catch (CacheException e) { // ... }
  • 14. Using jcache - get & put String key; // ... byte[] value; // ... // Put the value into the cache. cache.put(key, value); // Get the value from the cache. value = (byte[]) cache.get(key);
  • 15. Testing with ab benchmark ab -c 50 -n 50 http://[app-address]/[servlet_path] or ab -c 50 -t 50 http://[app-address]/[servlet_path]
  • 16. Other Operation ● getAll(), putAll(), deleteAll() A single read or write operation for multiple memcache entries ● increment(key, delta), incrementAll(...) Provide atomic increment of numeric value(s) ● getIdentifiable(), putIfUntouched() A mechanism to update a value consistently by concurrent requests
  • 17. Memcache Caveat ● Memcache Is Volatile ● Performance suggest: Batch size < 32 MB ● Dedicated memcache (pay): 10K OPS/s/GB, range 1-20 GB
  • 19. Why Task Queue ● Frontend restriction of 60 second limit ● Need longer-running tasks ● Asynchronously outside front end request ● Best practice: ○ Email ○ Write datastore ○ Web crawler ○ Backend jobs
  • 20. Task Queue Types ● By name ○ Default Queue (Push Queue) ○ Named Queue ● By type ○ Push queues ■ tasks execute automatically ■ only for App Engine app ○ Pull queues ■ tasks wait to be leased ■ available to workers outside the app ■ tasks can be batched
  • 22. Task Queue Manage Console
  • 23. Task Queue - Default Push Queue Queue queue = QueueFactory.getDefaultQueue(); //named queue: //QueueFactory.getQueue("Qname"); TaskOptions task = TaskOptions.Builder .withUrl("/path-to-my-worker") .param(key, value); queue.add(task); Reachable Servlet
  • 24. Detail Configure ● Task queues use token bucket algorithm [ref]
  • 25. ● rate - usual rate ● bucket-size - cap on peak demand ● max concurrent requests ● Retry options ○ task-retry-limit - min retries ○ task-age-limit - min elapsed time to keep retrying (Both task-retry-limit and task-age-limit are reached, task is deleted) ○ min-backoff-seconds -- min delay between retries ○ max-backoff-seconds -- max delay between retries ○ max-doublings -- max times to double delay ● more parameters see DOC queue.xml parameters
  • 26. Task Queue - With Transaction DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Queue queue = QueueFactory.getDefaultQueue(); try { Transaction txn = ds.beginTransaction(); // … other operations queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker")); // … other operations txn.commit(); } catch (DatastoreFailureException e) { // … exception handle }
  • 27. Pull Queue - Create Queue Queue queue = QueueFactory.getQueue("queue_name"); TaskOptions task = TaskOptions.Builder .withMethod(TaskOptions.Method.PULL) .payload(payload); queue.add(task);
  • 28. Pull Queue - Lease Queue queue = QueueFactory.getQueue("queue_name"); tasks = q.leaseTasks( duration, TimeUnit.SECONDS, how-many);
  • 29. Pull Queue - Delete Task //delete named task queue.deleteTask("task_ name"); //purge all tasks from queue QueueFactory.getQueue("queue_name").purge();
  • 30. Task Queue ACL - queue.xml <queue> <name>pull-queue1</name> <mode>pull</mode> <acl> <user-email>[email protected]</user-email> <writer-email>[email protected]</writer-email> <writer-email>[email protected]</writer-email> </acl> </queue>
  • 31. Cron
  • 32. Cron <cronentries> <cron> <url>/recache</url> <description> Put your description here </description> <schedule>every 5 minutes</schedule> </cron> </cronentries>
  • 33. ● every 12 hours ● every 5 minutes from 10:00 to 14:00 ● 2nd,third mon,wed,thu of march 17:00 ● every monday 09:00 ● 1st monday of sep,oct,nov 17:00 ● every day 00:00 ● every N (hours|mins|minutes) ["from" (time) "to" (time)] ● every 2 hours synchronized Cron Schedule Format
  • 35. Restrictions ● 20 cron jobs for free apps ● 100 for paid apps
  • 36. Task Queue / Cron Reference ● Task Queues Developer Guide (Java) (Python) ● Task Queue Configuration (Java) (Python) ● Task Queue API Ref (JavaDoc) (Python) ● Task Queue REST API Ref ● Scheduled Tasks with Cron (Java) (Python) ● Task queue quota and limits ● Get started with Task API article (Python) ● Google I/O 2011: Putting Task Queues to Work (video)