Nuxeo World Session: Scaling Nuxeo Applications
1 like•609 views
This document discusses scaling Nuxeo applications by ensuring good performance. It emphasizes defining usage hypotheses and constraints, conducting performance testing, and ongoing tuning. Key points include spreading services across VMs, using a VCS cluster for redundancy and scaling out, and partitioning data across multiple repositories. Benchmark results show document retrieval and insertion operations can achieve throughput of hundreds per second depending on the scenario. Regular performance monitoring and tuning of software configurations and databases is important for scaling Nuxeo applications.
![Setup the monitoring
●
Use your production monitoring
(nagios, hyperic, sysstat …)
●
Setup GC log in the nuxeo.conf fle:
JAVA_OPTS=$JAVA_OPTS -Xloggc:$DIRNAME/../log/gc.log -verbose:gc -XX:+PrintGC
●
Setup the monitor JBoss templates in the
nuxeo.conf fle :
nuxeo.templates=postgresql,monitor
●
Use monitorctl.sh
./jboss/bin/monitorctl.sh
Usage: monitorctl.sh (start|stop|status|heapdump [TAG]|info|vacuumdb|help)
35](https://image.slidesharecdn.com/nuxeoscalingv3-101124222159-phpapp01/85/Nuxeo-World-Session-Scaling-Nuxeo-Applications-35-320.jpg)
Nuxeo World Session: Scaling Nuxeo Applications
- 1. Scaling Nuxeo Applications Building scalable content apps with Nuxeo Benoit Delbosc / Thierry Delprat 1
- 2. Performance questions ... How can I ensure 0.1s response time ? What CPU should I use to host 1TB data ? Will 2 servers be enough for 1000 users ? Can I run Nuxeo DM with 512 MB of heap ? Is DELL PExyz ok to host Nuxeo ? 2
- 3. Why is this not that simple ? ● Nuxeo EP is an ECM platform ● there are several ways to use it – hundreds of possible user actions – lot of screens ● there are several distributions ● there are several possible confgurations (security, fling plan, doc types ...) ● you can not size without knowing ● the hypothesis ● the constraints 3
- 4. Agenda ● Approach for managing performance ● Nuxeo Architecture and performance ● Performance testing ● Performance tuning ● Benchmark fgures 4
- 5. Ensuring good performance ● Carefully defne your hypothesis ➔ what processing will be needed ? ➔ batch processing vs interactive processing ➔ what data you will be processed ? ➔ big image transformation vs simple page rendering ➔ what are the expectations ? ➔ how many concurrent users ? ➔ are users hyper-active ? ➔ be able to defne a usage scenario 5
- 6. Ensuring good performance ● Defne your architecture according to ● constraints ● initial benchmark ➔ mono server, multi-servers, cluster … ● Organize periodic benchmarks ➔ ideally integrate in CI chain ● Plan some tuning ➔ DB, memory, connections … ➔ and code too ➔ this is not a one shot task 6
- 7. Impacting factors ● Security policies ● ACLs inheritance and custom security policies ● Web Layer ● Stateless vs Stateful ● JSF traps and screen design ● Document types ● complex document types impact DB schema ● Number of active documents ● number of rows ● size of indexes vs Memory 7
- 8. Impacting factors ● Simultaneous access ● how many concurrent requests/s ● include batch processing ● Application server ● Tomcat 6 is signifcantly faster than JBoss 5 ● DB choice ● OS ● a 32 bit OS is too limiting for JVM Memory ● JVM seems to run faster under Linux 64 8
- 9. No (or low) impact factors ● Total volume of binary fles ● only network and low level storage are impacted ● Average number of document per folder ● VCS has not the same limitations as JackRabbit ● Number of parallel sessions ● can only be impacting in JSF ● Documents that are almost never accessed ● DB caches should do their job 9
- 10. Scaling Nuxeo Applications Architecture considerations 10
- 11. Architecture solutions ● 3 possibles axises ● Simple Clustering ● Spreading services on multiple JVMs ● Multiple repositories 11
- 12. VCS Cluster ● VCS Cluster is simple ● only one confg parameter to turn on ● do not rely on App Server level cluster ● “Nuxeo Boxes” are swappable ● VCS Cluster scales well ● as long as the backend DB scales ! ➔ VCS Cluster is a good solution for both ➔ scaling out ➔ providing redundancy 12
- 13. 1 node deployment Nuxeo DB Instance FS 13
- 14. 2 nodes deployment Nuxeo Instance DB NLB NAS Nuxeo Instance 14
- 15. 3 nodes deployment Nuxeo Instance Nuxeo DB NLB Instance NAS Nuxeo Instance 15
- 16. Multi VM deployment ● Nuxeo Services can be spread across JVMs ● externalize batch processing (mass i/o) ● externalize heavy transformations ● externalize slow interaction with external app ● … ➔ build dedicated processing servers ● Nuxeo services can be coupled with a GRID ● integration with GRID Gain 16
- 17. Multi VM : mono node batch Nuxeo DB Instance FS 17
- 18. Multi VM : 2 nodes batch Nuxeo Instance Nuxeo DB Instance FS 18
- 19. Multi VM : GRID batch batch GRID batch Nuxeo Nuxeo Nuxeo Instance Instance Instance Nuxeo DB Instance FS 19
- 20. Multi VM : perspectives ● Technically each service could be accessed remotely via RMI (on JEE server) ● rendering layer ● wf engine ● … ● But it's a little bit more complex ● Tx management ● binding confguration ➔ In most case VCS Cluster is much more simple 20
- 21. Multi-Repositories ● A single Nuxeo Application can be bound to several repositories ● Each repository ● has its own database and cache ➔ scale out solution if DB is the bottleneck ➔ Useful to do data partitioning ➔ manage documents with different constraints (ex: Live documents and archives) ➔ manage documents for different user groups (ex: multi-tenant) 21
- 22. Multi-Repositories Database 2 Archived Repo documents 2 FS 2 Nuxeo Server Repo Database Live 1 1 documents FS 1 22
- 23. Scaling Nuxeo Applications Performance testing 23
- 24. Performance testing overview 24
- 25. Setup a realistic environment ● Replicate as much as possible the production environment ● Use historical data when possible ● Populate the database ● Custom mass importer tools (nuxeo-platform-importer) ● Load generating tools ● Feed the database at the SQL level 25
- 26. Monitoring ● Performance testing without monitor don't bother ● Collecting general information: ● Hardware metrics: type and number of cpu, memory, disk usage ● Software versions: os, jvm, middlewar, application ● Application confguration: nuxeo.conf ● Database confguration and database statistics 26
- 27. Monitoring CPU ● Is the CPU a bottleneck ? ● Is the CPU waiting for IO ? ● Does the system use all the available CPU ? 27
- 28. Monitoring Disk ● Is there a device saturation ? ● Writing or reading operations ? 28
- 29. Monitoring GC ● Is the JVM spend too much time in the garbage collector ? 29
- 30. Monitoring web request processor ● How many requests ? ● How much input/output ? 30
- 31. Monitoring web thread pool ● Is the thread pool a bottle neck ? 31
- 32. Monitoring datasources ● Is there enough connection in the pool ? 32
- 33. Monitoring JVM ● How much JVM threads ? ● After a longevity testing, is there a memory leak ? 33
- 34. Monitoring database ● How much time is spent processing SQL ? ● Which query took up the most time ? 34
- 35. Setup the monitoring ● Use your production monitoring (nagios, hyperic, sysstat …) ● Setup GC log in the nuxeo.conf fle: JAVA_OPTS=$JAVA_OPTS -Xloggc:$DIRNAME/../log/gc.log -verbose:gc -XX:+PrintGC ● Setup the monitor JBoss templates in the nuxeo.conf fle : nuxeo.templates=postgresql,monitor ● Use monitorctl.sh ./jboss/bin/monitorctl.sh Usage: monitorctl.sh (start|stop|status|heapdump [TAG]|info|vacuumdb|help) 35
- 36. Load generating tools ● The application has to cooperate a bit to ease test writing ● Tools ● In-house tools ● Vendor tools ● Open source tools – JMeter (GUI script, java) – FunkLoad (Python) 36
- 37. Writing test scripts Try to simulate the expected user actions 37
- 38. Performance report ● Speed (response time) ● Throughput (requests/s) ● User satisfaction (Apdex) http://www.apdex.org/ 38
- 39. Monitoring report ● All in one logchart.py http://svn.nuxeo.org/nuxeo/tools/qa/logchart/trunk ● Other tools: kSar, pgfouine, gcviewer ... 39
- 41. Tuning process 41
- 42. Jboss/Tomcat tuning ● JVM, heap size (nuxeo.conf) JAVA_OPTS=$JAVA_OPTS -Xms2g -Xmx2g ● In special case you may remove SoftRef JAVA_OPTS=$JAVA_OPTS -XX:SoftRefLRUPolicyMSPerMB=0 ● Datasource connection pool (nuxeo.conf) nuxeo.db.max-pool-size=40 nuxeo.vcs.max-pool-size=40 ● HTTP or AJP Thread pooling <Connector port=”8080” … maxThreads=”32” ... acceptCount=”256” .. /> 42
- 43. Database tuning ● Read the Nuxeo KB https://doc.nuxeo.com/display/KB/Confguring+PostgreSQL ● EXPLAIN ANALYZE helper: http://explain.despez.com ● Check for missing index on custom schema ● PostgreSQL Performance ml mailto:[email protected] 43
- 44. Scaling Nuxeo Applications Benchmarking results 44
- 45. Mass injection ● Using load generating tools: from 3 to 10 doc/s works fne up to 100k docs (otherwise it takes to much time) ● Using nuxeo-platform-importer: from 30 to 100 doc/s works fne up to 1m docs (otherwise it takes to much time) ● Using SQL injection : from 1000 to 3000 doc/s 45
- 46. Document retrieval and insertion operations ● Nuxeo DM 5.4 / tomcat / Sun JVM 6 (Heap 3G) ● 10M of documents, 1TB of data ● Dell PE 2900 2xQuad-Core 20g RAM / Linux ● http://public.dev.nuxeo.com/~ben/bench-10m-tomcat/ Operation Speed (s) Throughput (req/s) Extrapolation VU JSF View of a 0.6 25 250 random document JSF View of a 0.2 30 300 cached document Web Engine 0.1 100 1000 navigation JSF Creating 0.8 16 160 new file 46
- 47. Thank you! 47