Alfresco tuning part2

Editor's Notes

• #5: Tune the memory and garbage collection parameters for the JVM to be appropriate for your situation. Tuning and deeper inspection of the Java Virtual Machine is a very important activity for java-based applications. In an enterprise-class installation of Alfresco where often one or multiple clustered instances are running, huge amounts of data are stored with a lot of user access that cause often a http overload. Alfresco can be optimized by a careful and meticulous tuning of the hardware resources, specifically cpu and ram used by the JVM.
• #6: There are some available tools to analyze garbage collection logs, but the one I think generates a more accurate report is Censum from Jclarity. It’s possible to download a trial version of this tool as use it to analyze the GC logs during 7 days. Censum is a nice tool that takes log files from the complex Java™ (JVM) garbage collection sub-system and gives you meaningful answers by providing clear infographics and making recommendations based on the analysis results. You can also use a completely free tool like GCViewer, it does not provide so the same recommendations but it’s still a very useful tool. https://github.com/chewiebug/GCViewer
• #7: 1 ) Periodic calls to trigger a Full GC either via System.gc() or Runtime.gc() preempt the natural flow of GC and corrupt the numerous metrics that are used to help the collectors run as optimally as is possible. There are many possible sources for these calls, which may include: An RMI call from a remote JVM. A Timer, TimerTask or ScheduleExecutor thread. A cron or Quartz job. Some external trigger causing the Memory MXBean to trigger a Full GC. add the -XX:+DisableExplicitGC flag 3) There are a number of possible causes for this problem: The Eden and/or Survivor spaces may be too small. The -XX:MaxTenuringThreshold flag may have been set too low.   There are number of possible solutions for this problem:   Alter the size of the young space via the -XX:NewRatio property. Alter the size of Survivor Spaces (relative to Eden) via the -XX:SurvivorRatio=<N> property using the information provided by the Tenuring graphs. This flag works to divide Young into N + 2 chunks. N chunks will be assigned to Eden and 1 chunk each will be assigned to the To and From spaces respectively. Alter the -XX:MaxTenuringThreshold property.   Note: enlarging Survivor spaces takes will result in less space being assigned to Eden. The size of Eden times your allocation rate yields the frequency of collections in young generation. Be sure to increase the size of young so that Eden stays the same size in order to avoid increasing the number of young generational collections.
• #8: Tracing Hazelcast usage Transactional Caches (Level 1) Alfresco version 5.0 introduced a way to trace the transactional cache usage (much similar to the previous ehcache tracing mechanism). Unfortunately that tracing is not available in version 4.2.X, one way to get this feature would be to open a support ticket requesting a back-port of this feature. HazelCast caches (Level 2) Using hazelcast mancenter you can trace the L2 cache usage, for more information check http://docs.alfresco.com/4.2/tasks/hazelcast-setup.html
• #9: Cache-Factory - Allows for the creation of caches Messenger-Factory - Abstraction over the hazelcast topic (published subscribe messaging system) LockStore Factory - Where in-memory locks are kept You can define your own caches as per the example below   <bean name=“contentDataSharedCache" factory-bean=“cacheFactory" factory-method="createCache"> <constructor-arg value="cache.customContDataCache"/> </bean>   cache.customContDataCache.maxItems=130000 cache.customContDataCache.timeToLiveSeconds=0 cache.customContDataCache.maxIdleSeconds=0 cache.customContDataCache.cluster.type=fully-distributed cache.customContDataCache.backup-count=1 cache.customContDataCache.eviction-policy=LRU cache.customContDataCache.eviction-percentage=25 cache.customContDataCache.merge-policy=hz.ADD_NEW_ENTRY   Note that there is no corresponding hazelcast-tcp.xml entry for the custom cache, the factory does all the configuration programmatically using the name of the cache customContDataCache as a root to discover the remaining configuration properties.
• #10: Invalidating Can be useful to store something that its not serializable, because all the values stored in a hazelcast cache must be serializable. That is the way hazlecast send the information to the member its going to be stored on. If you got a cache where there is an enormous number of reads and very rare writes, using a invalidating cache can be the best approach, very likely to the way ehcache used to work. This was introduced also because there were some non serializable values in alfresco that could not reside on a fully-distributed cache. The way we define the caches (on our cache.properties file is as follows) cache.aclSharedCache.tx.maxItems=40000 cache.aclSharedCache.maxItems=100000 cache.aclSharedCache.timeToLiveSeconds=0 cache.aclSharedCache.maxIdleSeconds=0 cache.aclSharedCache.cluster.type=fully-distributed cache.aclSharedCache.backup-count=1 cache.aclSharedCache.eviction-policy=LRU cache.aclSharedCache.eviction-percentage=25 cache.aclSharedCache.merge-policy=hz.ADD_NEW_ENTRY   Note the notion of cache backups (backup-count=??), that guarantees that a distributed cache has a specific number of backups, in case of a node holding that bit of the cache dies, those caches are still accessible. The more backups you have, the more memory will be consumed.
• #11: We also need to consider, that in distributed caches, when there are a lot of remote gets the network traffic will increase. On the other hand, if we choose and invalidation cache mechanism and the caches are changing often, the Invalidation messages can also be a single point of network stress. So overall it’s all about analyzing the trade-offs of each mechanism and to choose the more appropriate for each use case.
• #12: LRU = Least Recently Used LFU = Least Frequently Used For debugging purposes, you can disable the L2 cache. The database will keep working, but at a slower rate.The Level 2 (L2) cache provides out-of-transaction caching of Java objects inside the Alfresco system. The L2 cache objects are stored in memory attached to the application scope of the server. Sticky sessions must be used to keep a user that has already established a session on one server for the entire session. By default, the cache replication makes use of RMI to replicate changes to all nodes in the cluster using the Peer Cache Replicator. Each replicated cache member notifies all other cache instances when its content has changed, in case of invalidating cache types. Fully distributes is a different approach introduced with hazelcast. If you have issues with the replication of information in clustered systems, that is, the cache cluster test fails, you may want to confirm this by setting the following properties to true in the alfresco-global.properties file as follows : system.cache.disableMutableSharedCaches=true system.cache.disableImmutableSharedCaches=true An important indicator that you need to increase your caches is when you see messages like the ones below on your alfresco.log file indicating that some specific caches are full. 13:25:12,901 WARN [cache.node.nodesTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.nodesTransactionalCache’ is full (125000). 13:25:14,182 WARN [cache.node.aspectsTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.aspectsTransactionalCache’ is full (65000). 13:25:14,214 WARN [cache.node.propertiesTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.propertiesTransactionalCache’ is full (65000).
• #13: Let’s follow the money  , In this case we “Follow the Request”
• #14: jstack prints Java stack traces of Java threads for a given Java process or core file or a remote debug server. For each Java frame, the full class name, method name, 'bci' (byte code index) and line number, if available, are printed. Your repository can get easier maxed out if you are setting the application server to process more connections/threads than the machine can handle it. Consider first “bottlenecking”/reducing the number of application server connector’s threads in order each call gets CPU and memory resources needed to perform properly, so that your system gets a more stable response time over peak usages (scale the database connection pool maximum accordingly as mentioned before). You may need to scale up (or scale out your cluster) for handling a bigger number of calls during peak usages in place of just increasing the number of threads of one node application server.
• #15: Alfresco (like all Java applications) uses JDBC to access the database, inspecting JDBC can yield useful information for developers, the tools below can be used to inspect JDBC.   jdbcspy Log4jdbc – more active development, easier to configure (IMHO) Regular maintenance and tuning of the Alfresco database is necessary. Specifically, all of the database servers that Alfresco supports require at the very least that some form of index statistics maintenance be performed at frequent, regular intervals to maintain optimal Alfresco performance.   Index maintenance can have a severe impact on Alfresco performance while in progress, hence it needs to be discussed with your project team and scheduled appropriately. Make sure your database is tuned for your usage patterns: high throughput, long running queries, decision support, mixed usage. Finally check that the specific supported database being used is configured properly and according to Alfresco documentation. In regards to latency in communication, the golden rule is that the response time from the DB in general, should be around 4ms or lower. If the amount of nodes and the expected concurrency increases, for example when the database grows to more than 20 million nodes and the number of concurrent users increase considerably we suggest adopting an active-active database cluster approach.  
• #18: Understand JMX configuration and priority Alfresco allows for setting most of its configuration through JMX (or the Share admin console) in hot without need to restart and in a cluster aware mode. This is very handy for solving issues in hot and testing in general. But beware that those settings are stored in the database and have priority over the static settings (the file configurations on your server in general in alfresco-global.properties) will be active again when you restart the system.   This has a drawback, which is the fact that your static configuration may not reflect the real configuration state of your system what can make life complicated for system administrators in certain circumstances (especially newcomers). Keep your static configuration up to date You should consider as a best practice keeping the static files up to date with your configuration and also try to make sure they are as much as possible the ones being actively configuring your system. This would include in general reverting any configuration setup through JMX as soon as possible in order the system uses again the static configuration with whatever values you find appropriate.