SlideShare a Scribd company logo
4
Most read
5
Most read
8
Most read
Best Practices in Handling Performance Issues
Best Practices in Handling
Performance Issues
Components (POP)
Odoo
Python
PostgreSQL
➔ On what platform (Operating System, version) Odoo is deployed?
➔ What version of Odoo is installed? Is it up to date with the latest
official version?
➔ What is the deployment architecture: hardware specs, number
of machines, any load-balancer, VMs?
◆ $ inxi -Fxz | $ hwinfo --short
➔ Any Web Service integration via Odoo API? If yes, what kind of
volume and what kinds of transactions? WS calls sent in parallel?
➔ Any monitoring tool system performance
Deployment Information
➔ Linux Desktop Computer
◆ most commonly used Ubuntu LTS releases
➔ Setup SSH Access
➔ Command Line (CLI) Tools
The Setup Access Tools
CPU and Memory timeouts limit
concurrency / serialization
Request (RPC) processing time
Important Details in Logs
grep -n -B15 -A5 'CPU time' ./odoo/log | grep -n -B15 -A5 'memory limit' ./odoo/log
grep -n -B15 -A5 'SERIALIZATION' ./odoo.log
Request (RPC) Processing Time
2020-09-11 00:26:07,967 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:07] "POST /web/webclient/bootstrap_translations HTTP/1.0" 200 - 1 0.001 0.002
2020-09-11 00:26:08,881 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:08] "POST /web/dataset/call_button HTTP/1.0" 200 - 1169 2.159 0.619
2020-09-11 00:26:09,344 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/stock.picking/read HTTP/1.0" 200 - 109 0.292 0.070
2020-09-11 00:26:09,533 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/stock.move.line/read HTTP/1.0" 200 - 24 0.033 0.032
2020-09-11 00:26:09,578 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/stock.move/read HTTP/1.0" 200 - 49 0.059 0.046
2020-09-11 00:26:09,797 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/mail.message/message_format HTTP/1.0" 200 - 33 0.023 0.020
2020-09-11 00:26:09,942 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "GET /web/image/res.partner/33747/image_small HTTP/1.0" 304 - 16 0.009 0.012
2020-09-11 00:26:15,432 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:15] "POST /web/dataset/search_read HTTP/1.0" 200 - 14 5.017 0.018
2020-09-11 00:26:20,762 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:20] "POST /web/dataset/search_read HTTP/1.0" 200 - 14 5.208 0.018
2020-09-11 00:26:23,980 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:23] "POST /web/dataset/call_kw/stock.picking/read HTTP/1.0" 200 - 74 0.242 0.065
2020-09-11 00:26:24,182 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /web/dataset/call_kw/stock.move.line/read HTTP/1.0" 200 - 50 0.055 0.045
2020-09-11 00:26:24,209 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /web/dataset/call_kw/stock.move/read HTTP/1.0" 200 - 64 0.068 0.057
2020-09-11 00:26:24,425 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /mail/read_followers HTTP/1.0" 200 - 6 0.004 0.009
2020-09-11 00:26:24,448 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /web/dataset/call_kw/mail.message/message_format HTTP/1.0" 200 - 25 0.017 0.018
https://github.com/odoo/odoo/blob/master/odoo/netsvc.py#L76
Normal
>=
Warning
>=
Alarming
>=
1. Number of queries (query cont) → 0 100 1000
2. Query Execution Time (query_time) → 0 0.1 3
3.
Total Execution Time minus query time
(remaining_time) → 0 1 5
Parsing & Sorting Request Odoo Processing Time
Parsing & Sorting Request Query Execution Time
Request Processing Time
Parsing Request (RPC) Time from Logs
cat odoo.log | grep -v longpolling | awk '{print $NF-1 " " $0}' | sort -nr | less
cat odoo.log | grep -v longpolling | awk '{split($2, a, ":"); arr[a[1]]+=$N2;arr2[a[1]]+=1;} END {for (i in arr) {print i,arr[i],arr2[i]}}' | sort -n
grep -e '[^?] werkzeug:.*(GET|POST)' ./odoo.log | awk '{ d[$13]+=$18} END{for (k in d) print k, d[k] }' | sort -rnk 2 | head
egrep -o "([0-9]{1,3}[.]){3}[0-9]{1,3}" ./odoo.log | uniq -c | sort -nr | head -n 20
cat odoo.log | grep -v longpolling | awk '{print $NF-1 " " $0}' | sort -nr | less
Reproducing is a Key
Reproducing and automating it founding stone
Profile the code
Profile the code using Odoo Profiler
Graph a method
This embedded profiler output can be used to generate
a graph of calls triggered by the method, number of
queries, percentage of time taken in the method itself
as well as the time that the method took and its sub-
called methods.
from odoo.tools.misc import profile
[...]
@profile('/temp/prof.profile')
def mymethod(...)
This produces a file called /temp/prof.profile
A tool called gprof2dot will produce a graph with this
result:
gprof2dot -f pstats -o /temp/prof.xdot
/temp/prof.profile
A tool called xdot will display the resulting graph:
xdot /temp/prof.xdot
This profiler can be used to log statistics on a method:
from odoo.tools.profiler import profile
[...]
@profile
@api.model
def mymethod(...):
The statistics will be displayed into the logs once the method
to be analysed is completely reviewed.
2018-03-28 06:18:23,196 22878 INFO openerp odoo.tools.profiler:
calls queries ms
project.task ------------------------ /home/odoo/src/odoo/addons/project/models/project.py, 638
1 0 0.02 @profile
@api.model
def create(self, vals):
1 0 0.01 context = dict(self.env.context, mail_create_nolog=True)
1 108 631.8 task = super(Task, self.with_context(context)).create(vals)
1 0 0.01 return task
Total:
1 108 631.84
Log a method
py-spy: Sampling Profiler
https://github.com/benfred/py-spy
Py-spy `record` allows generating a flame graph or speedscope of python process :
py-spy record -o profile.svg --pid 12345
sudo py-spy record --pid 12345 --format speedscpe -o profile.speedscope.json
Py-spy `top` shows a live view of what functions are taking the most time in your python program:
py-spy top --pid 12345
py-spy can also display the current call stack for each python thread with the dump command:
py-spy dump --pid 12345
PostgreSQL
Settings in postgresql.conf
➔ There are few high-level setting:
● shared_buffers
● wal_buffers
● effective_cache_size
● work_mem
● maintenance_work_mem
➔ Using a tool such as https://pgtune.leopard.in.ua/#/ to get an initial
configuration is worthwhile, but the key to getting the absolute best
performance is by monitoring and changing the configuration as required
and repeat.
pg_activity
➔ Command line tool for PostgreSQL server activity monitoring
➔ pg_activity is a curses-based realtime utility to display information from a running
PostgreSQL database server in top like fashion
➔ It also provides interactive commands to sort, filter, scroll and extend the queries
list displayed
➔ Running pg_activity with superuser or root like access will give access to more
information, will also not fallback to a degraded mode, and will not stop
displaying system informations
➔ Give window is to live execution, do not require reboot and allows interactive
operations
pg_stat_statements
-- creates pg_stat_statements extension
CREATE EXTENSION pg_stat_statements;
-- discards statistics gathered so far in pg_stat_statements
select pg_stat_statements_reset();
-- gives 20 most time expensive queries
SELECT
substring(query, 1, 50) AS short_query,
round(total_time::numeric, 2) AS total_time,
calls as nbr_calls,
round(mean_time::numeric, 2) AS mean,
round((100 * total_time / sum(total_time::numeric) OVER
())::numeric, 2) AS percentage_cpu
FROM pg_stat_statements
ORDER BY
total_time DESC
LIMIT 20;
-- drops pg_stat_statements EXTENSION
DROP EXTENSION pg_stat_statements;
https://www.postgresql.org/docs/current/pgstatstatements.html
➔ The pg_stat_statements module
provides a means for tracking
execution statistics of all SQL
statements executed by a server.
➔ Pg_stat_statements has overhead
when enabled but it is fairly low
overhead, and should be safe to use in
production. Use it when needed.
➔ User COPY command to export result
to csv for analyzing and reporting.
pgbadger
https://github.com/darold/pgbadger
➔ pgBadger - PostgreSQL log analyzer
with fully detailed reports and graphs.
➔ pgBadger is a PostgreSQL log analyzer
built for speed with fully reports from
your PostgreSQL log file. It's a single
and small Perl script that outperforms
any other PostgreSQL log analyzer.
➔ It is written in pure Perl and uses a
JavaScript library (flotr2) to draw
graphs
Reproduce
Reproducing right way is important
Doing it manually is not efficient
Fix It
Apply necessary based on analysis;
and repeat by reproducing
Analyze
Analyze the results, find root cause & fix it
Check if solution is acceptable
Observe
Profile, Monitor, Collect results
Conquering
Performance
Issue
Setup
Release and Deploy
Thank You

More Related Content

What's hot (20)

PDF
Tools for Solving Performance Issues
Odoo
 
PDF
Odoo Performance Limits
Odoo
 
PPTX
Common Performance Pitfalls in Odoo apps
Odoo
 
PDF
New Framework - ORM
Odoo
 
PDF
Odoo Online platform: architecture and challenges
Odoo
 
PDF
Impact of the New ORM on Your Modules
Odoo
 
PPTX
Odoo's Test Framework - Learn Best Practices
Odoo
 
PDF
Load Testing - How to Stress Your Odoo with Locust
Odoo
 
PDF
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
PDF
Odoo External API
Odoo
 
DOCX
Simple Odoo ERP auto scaling on AWS
Julien Lecadou,MSc.
 
PDF
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
 
PDF
PostgreSQL WAL for DBAs
PGConf APAC
 
PDF
Monitoring with Prometheus
Shiao-An Yuan
 
PDF
What is new in PostgreSQL 14?
Mydbops
 
PDF
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Altinity Ltd
 
PPTX
Query Optimizer – MySQL vs. PostgreSQL
Christian Antognini
 
PDF
PromQL Deep Dive - The Prometheus Query Language
Weaveworks
 
PPTX
Introduction to GCP BigQuery and DataPrep
Paweł Mitruś
 
PDF
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
PostgreSQL-Consulting
 
Tools for Solving Performance Issues
Odoo
 
Odoo Performance Limits
Odoo
 
Common Performance Pitfalls in Odoo apps
Odoo
 
New Framework - ORM
Odoo
 
Odoo Online platform: architecture and challenges
Odoo
 
Impact of the New ORM on Your Modules
Odoo
 
Odoo's Test Framework - Learn Best Practices
Odoo
 
Load Testing - How to Stress Your Odoo with Locust
Odoo
 
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Odoo External API
Odoo
 
Simple Odoo ERP auto scaling on AWS
Julien Lecadou,MSc.
 
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
 
PostgreSQL WAL for DBAs
PGConf APAC
 
Monitoring with Prometheus
Shiao-An Yuan
 
What is new in PostgreSQL 14?
Mydbops
 
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Altinity Ltd
 
Query Optimizer – MySQL vs. PostgreSQL
Christian Antognini
 
PromQL Deep Dive - The Prometheus Query Language
Weaveworks
 
Introduction to GCP BigQuery and DataPrep
Paweł Mitruś
 
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
PostgreSQL-Consulting
 

Similar to Best Practices in Handling Performance Issues (20)

PDF
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
PDF
Presto anatomy
Dongmin Yu
 
PPTX
ql.io: Consuming HTTP at Scale
Subbu Allamaraju
 
PDF
Osol Pgsql
Emanuel Calvo
 
PPTX
Oracle Basics and Architecture
Sidney Chen
 
PDF
Logstash for SEO: come monitorare i Log del Web Server in realtime
Andrea Cardinale
 
PDF
[245] presto 내부구조 파헤치기
NAVER D2
 
PDF
How to measure everything - a million metrics per second with minimal develop...
Jos Boumans
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
PPT
TopicMapReduceComet log analysis by using splunk
akashkale0756
 
PDF
Why you should be using structured logs
Stefan Krawczyk
 
PPTX
Being HAPI! Reverse Proxying on Purpose
Aman Kohli
 
PDF
Divolte Collector - meetup presentation
fvanvollenhoven
 
PPTX
Debugging & profiling node.js
tomasperezv
 
KEY
fog or: How I Learned to Stop Worrying and Love the Cloud (OpenStack Edition)
Wesley Beary
 
KEY
fog or: How I Learned to Stop Worrying and Love the Cloud
Wesley Beary
 
PDF
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
Andrey Kudryavtsev
 
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
Presto anatomy
Dongmin Yu
 
ql.io: Consuming HTTP at Scale
Subbu Allamaraju
 
Osol Pgsql
Emanuel Calvo
 
Oracle Basics and Architecture
Sidney Chen
 
Logstash for SEO: come monitorare i Log del Web Server in realtime
Andrea Cardinale
 
[245] presto 내부구조 파헤치기
NAVER D2
 
How to measure everything - a million metrics per second with minimal develop...
Jos Boumans
 
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
TopicMapReduceComet log analysis by using splunk
akashkale0756
 
Why you should be using structured logs
Stefan Krawczyk
 
Being HAPI! Reverse Proxying on Purpose
Aman Kohli
 
Divolte Collector - meetup presentation
fvanvollenhoven
 
Debugging & profiling node.js
tomasperezv
 
fog or: How I Learned to Stop Worrying and Love the Cloud (OpenStack Edition)
Wesley Beary
 
fog or: How I Learned to Stop Worrying and Love the Cloud
Wesley Beary
 
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
Andrey Kudryavtsev
 
Ad

More from Odoo (20)

PPTX
Timesheet Workshop: The Timesheet App People Love!
Odoo
 
PPTX
Odoo 3D Product View with Google Model-Viewer
Odoo
 
PPTX
Keynote - Vision & Strategy
Odoo
 
PPTX
Opening Keynote - Unveilling Odoo 14
Odoo
 
PDF
Extending Odoo with a Comprehensive Budgeting and Forecasting Capability
Odoo
 
PDF
Managing Multi-channel Selling with Odoo
Odoo
 
PPTX
Product Configurator: Advanced Use Case
Odoo
 
PDF
Accounting Automation: How Much Money We Saved and How?
Odoo
 
PPTX
Rock Your Logistics with Advanced Operations
Odoo
 
PPTX
Transition from a cost to a flow-centric organization
Odoo
 
PDF
Synchronization: The Supply Chain Response to Overcome the Crisis
Odoo
 
PPTX
Running a University with Odoo
Odoo
 
PPTX
Down Payments on Purchase Orders in Odoo
Odoo
 
PPTX
Odoo Implementation in Phases - Success Story of a Retail Chain 3Sach food
Odoo
 
PPTX
Migration from Salesforce to Odoo
Odoo
 
PPTX
Preventing User Mistakes by Using Machine Learning
Odoo
 
PPTX
Becoming an Odoo Expert: How to Prepare for the Certification
Odoo
 
PPTX
Instant Printing of any Odoo Report or Shipping Label
Odoo
 
PPTX
How Odoo helped an Organization Grow 3 Fold
Odoo
 
PPTX
From Shopify to Odoo
Odoo
 
Timesheet Workshop: The Timesheet App People Love!
Odoo
 
Odoo 3D Product View with Google Model-Viewer
Odoo
 
Keynote - Vision & Strategy
Odoo
 
Opening Keynote - Unveilling Odoo 14
Odoo
 
Extending Odoo with a Comprehensive Budgeting and Forecasting Capability
Odoo
 
Managing Multi-channel Selling with Odoo
Odoo
 
Product Configurator: Advanced Use Case
Odoo
 
Accounting Automation: How Much Money We Saved and How?
Odoo
 
Rock Your Logistics with Advanced Operations
Odoo
 
Transition from a cost to a flow-centric organization
Odoo
 
Synchronization: The Supply Chain Response to Overcome the Crisis
Odoo
 
Running a University with Odoo
Odoo
 
Down Payments on Purchase Orders in Odoo
Odoo
 
Odoo Implementation in Phases - Success Story of a Retail Chain 3Sach food
Odoo
 
Migration from Salesforce to Odoo
Odoo
 
Preventing User Mistakes by Using Machine Learning
Odoo
 
Becoming an Odoo Expert: How to Prepare for the Certification
Odoo
 
Instant Printing of any Odoo Report or Shipping Label
Odoo
 
How Odoo helped an Organization Grow 3 Fold
Odoo
 
From Shopify to Odoo
Odoo
 
Ad

Recently uploaded (20)

PPTX
Washington University of Health and Science A Choice You Can Trust
Washington University of Health and Science
 
PDF
Jordan Minnesota City Codes and Ordinances
Forklift Trucks in Minnesota
 
PDF
Maksym Vyshnivetskyi: Управління закупівлями (UA)
Lviv Startup Club
 
PDF
Raman Bhaumik - A Passion For Service
Raman Bhaumik
 
PDF
20250703_A. Stotz All Weather Strategy - Performance review July
FINNOMENAMarketing
 
DOCX
TCP Communication Flag Txzczczxcxzzxypes.docx
esso24
 
PDF
NewBase 07 July 2025 Energy News issue - 1800 by Khaled Al Awadi_compressed.pdf
Khaled Al Awadi
 
PDF
Pyrolysis Oil Manufacturing Plant Project Report.pdf
procurement resource
 
PPTX
World First Cardiovascular & Thoracic CT Scanner
arineta37
 
PDF
"Complete Guide to the Partner Visa 2025
Zealand Immigration
 
PPTX
DECODING AI AGENTS AND WORKFLOW AUTOMATION FOR MODERN RECRUITMENT
José Kadlec
 
PPTX
Top Oil and Gas Companies in India Fuelling the Nation’s Growth.pptx
Essar Group
 
PDF
HOW TO RECOVER LOST CRYPTOCURRENCY - VISIT iBOLT CYBER HACKER COMPANY
diegovalentin771
 
PPTX
GE%205.pptx BUSINESS EMAIL this report aims
KenSantos27
 
PDF
Azumah Resources reaffirms commitment to Ghana amid dispute with Engineers & ...
Kweku Zurek
 
PPTX
25 Future Mega Trends Reshaping the World in 2025 and Beyond
presentifyai
 
PPTX
Build Wealth & Protect Your Legacy with Indexed Universal Life Insurance
iulfinancial6
 
DOCX
RECLAIM STOLEN CRYPTO REVIEW WITH RECUVA HACKER SOLUTIONS
camilamichaelj7
 
PDF
NewBase 03 July 2025 Energy News issue - 1799 by Khaled Al Awadi_compressed.pdf
Khaled Al Awadi
 
PPTX
2025 July - ABM for B2B in Hubspot - Demand Gen HUG.pptx
mjenkins13
 
Washington University of Health and Science A Choice You Can Trust
Washington University of Health and Science
 
Jordan Minnesota City Codes and Ordinances
Forklift Trucks in Minnesota
 
Maksym Vyshnivetskyi: Управління закупівлями (UA)
Lviv Startup Club
 
Raman Bhaumik - A Passion For Service
Raman Bhaumik
 
20250703_A. Stotz All Weather Strategy - Performance review July
FINNOMENAMarketing
 
TCP Communication Flag Txzczczxcxzzxypes.docx
esso24
 
NewBase 07 July 2025 Energy News issue - 1800 by Khaled Al Awadi_compressed.pdf
Khaled Al Awadi
 
Pyrolysis Oil Manufacturing Plant Project Report.pdf
procurement resource
 
World First Cardiovascular & Thoracic CT Scanner
arineta37
 
"Complete Guide to the Partner Visa 2025
Zealand Immigration
 
DECODING AI AGENTS AND WORKFLOW AUTOMATION FOR MODERN RECRUITMENT
José Kadlec
 
Top Oil and Gas Companies in India Fuelling the Nation’s Growth.pptx
Essar Group
 
HOW TO RECOVER LOST CRYPTOCURRENCY - VISIT iBOLT CYBER HACKER COMPANY
diegovalentin771
 
GE%205.pptx BUSINESS EMAIL this report aims
KenSantos27
 
Azumah Resources reaffirms commitment to Ghana amid dispute with Engineers & ...
Kweku Zurek
 
25 Future Mega Trends Reshaping the World in 2025 and Beyond
presentifyai
 
Build Wealth & Protect Your Legacy with Indexed Universal Life Insurance
iulfinancial6
 
RECLAIM STOLEN CRYPTO REVIEW WITH RECUVA HACKER SOLUTIONS
camilamichaelj7
 
NewBase 03 July 2025 Energy News issue - 1799 by Khaled Al Awadi_compressed.pdf
Khaled Al Awadi
 
2025 July - ABM for B2B in Hubspot - Demand Gen HUG.pptx
mjenkins13
 

Best Practices in Handling Performance Issues

  • 2. Best Practices in Handling Performance Issues
  • 4. ➔ On what platform (Operating System, version) Odoo is deployed? ➔ What version of Odoo is installed? Is it up to date with the latest official version? ➔ What is the deployment architecture: hardware specs, number of machines, any load-balancer, VMs? ◆ $ inxi -Fxz | $ hwinfo --short ➔ Any Web Service integration via Odoo API? If yes, what kind of volume and what kinds of transactions? WS calls sent in parallel? ➔ Any monitoring tool system performance Deployment Information
  • 5. ➔ Linux Desktop Computer ◆ most commonly used Ubuntu LTS releases ➔ Setup SSH Access ➔ Command Line (CLI) Tools The Setup Access Tools
  • 6. CPU and Memory timeouts limit concurrency / serialization Request (RPC) processing time Important Details in Logs grep -n -B15 -A5 'CPU time' ./odoo/log | grep -n -B15 -A5 'memory limit' ./odoo/log grep -n -B15 -A5 'SERIALIZATION' ./odoo.log
  • 7. Request (RPC) Processing Time 2020-09-11 00:26:07,967 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:07] "POST /web/webclient/bootstrap_translations HTTP/1.0" 200 - 1 0.001 0.002 2020-09-11 00:26:08,881 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:08] "POST /web/dataset/call_button HTTP/1.0" 200 - 1169 2.159 0.619 2020-09-11 00:26:09,344 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/stock.picking/read HTTP/1.0" 200 - 109 0.292 0.070 2020-09-11 00:26:09,533 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/stock.move.line/read HTTP/1.0" 200 - 24 0.033 0.032 2020-09-11 00:26:09,578 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/stock.move/read HTTP/1.0" 200 - 49 0.059 0.046 2020-09-11 00:26:09,797 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "POST /web/dataset/call_kw/mail.message/message_format HTTP/1.0" 200 - 33 0.023 0.020 2020-09-11 00:26:09,942 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:09] "GET /web/image/res.partner/33747/image_small HTTP/1.0" 304 - 16 0.009 0.012 2020-09-11 00:26:15,432 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:15] "POST /web/dataset/search_read HTTP/1.0" 200 - 14 5.017 0.018 2020-09-11 00:26:20,762 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:20] "POST /web/dataset/search_read HTTP/1.0" 200 - 14 5.208 0.018 2020-09-11 00:26:23,980 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:23] "POST /web/dataset/call_kw/stock.picking/read HTTP/1.0" 200 - 74 0.242 0.065 2020-09-11 00:26:24,182 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /web/dataset/call_kw/stock.move.line/read HTTP/1.0" 200 - 50 0.055 0.045 2020-09-11 00:26:24,209 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /web/dataset/call_kw/stock.move/read HTTP/1.0" 200 - 64 0.068 0.057 2020-09-11 00:26:24,425 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /mail/read_followers HTTP/1.0" 200 - 6 0.004 0.009 2020-09-11 00:26:24,448 3398 INFO dummy werkzeug: 127.0.0.1 - - [22/Sep/2020 00:26:24] "POST /web/dataset/call_kw/mail.message/message_format HTTP/1.0" 200 - 25 0.017 0.018 https://github.com/odoo/odoo/blob/master/odoo/netsvc.py#L76 Normal >= Warning >= Alarming >= 1. Number of queries (query cont) → 0 100 1000 2. Query Execution Time (query_time) → 0 0.1 3 3. Total Execution Time minus query time (remaining_time) → 0 1 5
  • 8. Parsing & Sorting Request Odoo Processing Time Parsing & Sorting Request Query Execution Time Request Processing Time Parsing Request (RPC) Time from Logs cat odoo.log | grep -v longpolling | awk '{print $NF-1 " " $0}' | sort -nr | less cat odoo.log | grep -v longpolling | awk '{split($2, a, ":"); arr[a[1]]+=$N2;arr2[a[1]]+=1;} END {for (i in arr) {print i,arr[i],arr2[i]}}' | sort -n grep -e '[^?] werkzeug:.*(GET|POST)' ./odoo.log | awk '{ d[$13]+=$18} END{for (k in d) print k, d[k] }' | sort -rnk 2 | head egrep -o "([0-9]{1,3}[.]){3}[0-9]{1,3}" ./odoo.log | uniq -c | sort -nr | head -n 20 cat odoo.log | grep -v longpolling | awk '{print $NF-1 " " $0}' | sort -nr | less
  • 9. Reproducing is a Key Reproducing and automating it founding stone
  • 10. Profile the code Profile the code using Odoo Profiler
  • 11. Graph a method This embedded profiler output can be used to generate a graph of calls triggered by the method, number of queries, percentage of time taken in the method itself as well as the time that the method took and its sub- called methods. from odoo.tools.misc import profile [...] @profile('/temp/prof.profile') def mymethod(...) This produces a file called /temp/prof.profile A tool called gprof2dot will produce a graph with this result: gprof2dot -f pstats -o /temp/prof.xdot /temp/prof.profile A tool called xdot will display the resulting graph: xdot /temp/prof.xdot This profiler can be used to log statistics on a method: from odoo.tools.profiler import profile [...] @profile @api.model def mymethod(...): The statistics will be displayed into the logs once the method to be analysed is completely reviewed. 2018-03-28 06:18:23,196 22878 INFO openerp odoo.tools.profiler: calls queries ms project.task ------------------------ /home/odoo/src/odoo/addons/project/models/project.py, 638 1 0 0.02 @profile @api.model def create(self, vals): 1 0 0.01 context = dict(self.env.context, mail_create_nolog=True) 1 108 631.8 task = super(Task, self.with_context(context)).create(vals) 1 0 0.01 return task Total: 1 108 631.84 Log a method
  • 12. py-spy: Sampling Profiler https://github.com/benfred/py-spy Py-spy `record` allows generating a flame graph or speedscope of python process : py-spy record -o profile.svg --pid 12345 sudo py-spy record --pid 12345 --format speedscpe -o profile.speedscope.json Py-spy `top` shows a live view of what functions are taking the most time in your python program: py-spy top --pid 12345 py-spy can also display the current call stack for each python thread with the dump command: py-spy dump --pid 12345
  • 14. Settings in postgresql.conf ➔ There are few high-level setting: ● shared_buffers ● wal_buffers ● effective_cache_size ● work_mem ● maintenance_work_mem ➔ Using a tool such as https://pgtune.leopard.in.ua/#/ to get an initial configuration is worthwhile, but the key to getting the absolute best performance is by monitoring and changing the configuration as required and repeat.
  • 15. pg_activity ➔ Command line tool for PostgreSQL server activity monitoring ➔ pg_activity is a curses-based realtime utility to display information from a running PostgreSQL database server in top like fashion ➔ It also provides interactive commands to sort, filter, scroll and extend the queries list displayed ➔ Running pg_activity with superuser or root like access will give access to more information, will also not fallback to a degraded mode, and will not stop displaying system informations ➔ Give window is to live execution, do not require reboot and allows interactive operations
  • 16. pg_stat_statements -- creates pg_stat_statements extension CREATE EXTENSION pg_stat_statements; -- discards statistics gathered so far in pg_stat_statements select pg_stat_statements_reset(); -- gives 20 most time expensive queries SELECT substring(query, 1, 50) AS short_query, round(total_time::numeric, 2) AS total_time, calls as nbr_calls, round(mean_time::numeric, 2) AS mean, round((100 * total_time / sum(total_time::numeric) OVER ())::numeric, 2) AS percentage_cpu FROM pg_stat_statements ORDER BY total_time DESC LIMIT 20; -- drops pg_stat_statements EXTENSION DROP EXTENSION pg_stat_statements; https://www.postgresql.org/docs/current/pgstatstatements.html ➔ The pg_stat_statements module provides a means for tracking execution statistics of all SQL statements executed by a server. ➔ Pg_stat_statements has overhead when enabled but it is fairly low overhead, and should be safe to use in production. Use it when needed. ➔ User COPY command to export result to csv for analyzing and reporting.
  • 17. pgbadger https://github.com/darold/pgbadger ➔ pgBadger - PostgreSQL log analyzer with fully detailed reports and graphs. ➔ pgBadger is a PostgreSQL log analyzer built for speed with fully reports from your PostgreSQL log file. It's a single and small Perl script that outperforms any other PostgreSQL log analyzer. ➔ It is written in pure Perl and uses a JavaScript library (flotr2) to draw graphs
  • 18. Reproduce Reproducing right way is important Doing it manually is not efficient Fix It Apply necessary based on analysis; and repeat by reproducing Analyze Analyze the results, find root cause & fix it Check if solution is acceptable Observe Profile, Monitor, Collect results Conquering Performance Issue Setup Release and Deploy

Editor's Notes

  • #6: I am using linux (ubuntu lts) computer to perform all my work, we have most instances are running on cloud staging/copy server so you will need tools like SSH to access them. I am using most tools that are CLI based as they are most easiest to access
  • #7: I am using linux (ubuntu lts) computer to perform all my work, we have most instances are running on cloud staging/copy server so you will need tools like SSH to access them. I am using most tools that are CLI based as they are most easiest to access
  • #9: Hourly processing time to find the peak hours Most time expensive route to see high transection IP with highest request
  • #13: py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way. py-spy is extremely low overhead: it is written in Rust for speed and doesn't run in the same process as the profiled Python program. This means py-spy is safe to use against production Python code.
  • #15: shared_buffers PostgreSQL uses 'double buffering', meaning that PostgreSQL uses its own internal buffer as well as kernel buffered IO. In short, this means that data is stored in memory twice. The PostgreSQL buffer is named shared_buffers and it defines how much dedicated system memory PostgreSQL will use for cache. Because of PostgreSQL's design choice to ensure compatibility on all supported machines and operating systems, this value is set conservatively low by default. As such, updating the shared_buffers is one of the settings which will be the most effective in improving overall performance on most modern operating systems. There is not one specific recommended value for shared_buffers, but the calculation to determine the value for a particular system is not especially difficult. Generally speaking, the value for shared_buffers should be roughly 25% of the total system RAM for a dedicated DB server. The value for shared_buffers should never be set to reserve all of the system RAM for PostgreSQL. A value over 25% of the system RAM can be useful if, for example, it is set such that the entire database working set of data can fit in cache, as this would greatly reduce the amount of time reading from disk. Alternately, while a larger shared_buffers value can increase performance in 'read heavy' use cases, having a large shared_buffer value can be detrimental for 'write heavy' use cases, as the entire contents of shared_buffers must be processed during writes. wal_buffers Write-Ahead Logging (WAL) is a standard method for ensuring integrity of data. Much like in the shared_buffers setting, PostgreSQL writes WAL records into buffers and then these buffers are flushed to disk. The default size of the buffer is set by the wal_buffers setting- initially at 16MB. If the system being tuned has a large number of concurrent connections, then a higher value for wal_buffers can provide better performance. effective_cache_size This value is used only by the PostgreSQL query planner to figure out whether plans it's considering would be expected to fit in RAM or not. As such, it's a bit of a fuzzy number to define for general use cases. A conservative value for effective_cache_size would be 1/2 of the total memory available on the system. Most commonly, the value is set to 75% of the total system memory on a dedicated DB server, but can vary depending on the specific discrete needs on a particular server workload. It's also important to remember that even the most well tuned database cannot salvage poorly formed queries. Developers creating applications which interface with the database need to be mindful of how queries are written.
  • #16: pg_activity works locally or remotely. In local execution context, to obtain sufficient rights to display system informations, the system user running pg_activity must be the same user running postgresql server (postgres by default), or have more rights like root. Otherwise, pg_activity can fallback to a degraded mode without displaying system informations. On the same way, PostgreSQL user used to connect to the database must be super-user Install it from pypi to get latest updates
  • #17: The module must be loaded by adding pg_stat_statements to shared_preload_libraries in postgresql.conf, because it requires additional shared memory. This means that a server restart is needed to add or remove the module. For security reasons, only superusers and members of the pg_read_all_stats role are allowed to see the SQL text and queryid of queries executed by other users. Other users can see the statistics, however, if the view has been installed in their database.
  • #18: The module must be loaded by adding pg_stat_statements to shared_preload_libraries in postgresql.conf, because it requires additional shared memory. This means that a server restart is needed to add or remove the module. For security reasons, only superusers and members of the pg_read_all_stats role are allowed to see the SQL text and queryid of queries executed by other users. Other users can see the statistics, however, if the view has been installed in their database.
  • #20: And, now, we have 580 employees.