Application performance monitoring with Elastic APM and the ELK stack

Senacor Technologies AG 04.05.2019 1
Application performances
monitoring with
Elastic APM
And the ELK stack
Alain Lompo
Senacor Technologies AG
#DevOps PRO Europe alain.lompo@senacor.com @alainlompo

Plan
 Motivation
 Introducing the ELK Stack
 Setting up the ELK Stack
 Introducing Elastic APM
 Setting up Elastic APM
 Monitoring Spring-boot apps with Elastic APM

Motivation

The increased complexity of software development
Programming
Language(s)
Methodology
Team
IDE
Application server
Frameworks
And Dev technologies
Integration with
External services
Orchestration
Virtualisation
Containairization
Build tools
Versioning system
Qualimetry tools
Dev Environment
Commit Environment
Acceptance Environment
PRE-PROD Environment
PROD Environment
Solution
architecture
Application
architecture

Monitoring complex apps is…complex
Internet
Web server Email server Database server Search server
Application server Application server
Application server Application server
SAN
?
?
?

Elastic APM to the rescue

Elastic makes APM simple and beautiful

Introducing the ELK stack

Introducing Elasticsearch
 Elasticsearch is an open source search server based
on the Apache Lucene server
 It is cross-platform (build in Java)
 It is designed from origin to be scalable and distributed
 It is data source agnostic (can take any data source and
search through it)

Elasticsearch’s role in the ELK stack
Kibana Elasticsearch Logstash / Beats
UI for Datas and
Dashboard visualisation
Powerful and flexible
Search server
Collection, transformation
And shipment of Log datas
http (S) http (S)
/ REST
Elastic
Transport

Other interesting Elasticsearch features
 Real time analytics
 Full text search
 Document oriented
 Restful API

Introducing Logstash
Analysis
Archiving
Monitoring
Alerting

Introducing Kibana

Senacor Technologies AG
Application Tier
(container)
04.05.2019 14
ELK Big picture
Database Tier
(container)
Logging/Monitoring Tier (ELK)
nodejs
WebServer
JBOSS EAP
Application server
Traders Admin
Portal
WebApplication
logstash
ELK component
Container
log_endpoint
logstash_server
Compute
Container
elasticsearch
ELK component
Container
search_endpoint
elasticsearch
_server
Compute
kibana
ELK component
s
Container
kibana_server
Compute
search_endpoint
ConnectsTo
HostedOn HostedOn HostedOn
ConnectsTo
search_endpoint
ContainerContainer
ConnectsTo

Setting up the ELK stack

Starting simple
GET _search
{
"query": {
"match_all": {}
}
}

Starting simple
POST /talks/devops
{
"title": "Application monitoring with ELK stack
and Elastic APM",
"speaker": "Alain Lompo",
"category": "APM",
"tags": ["Elastic", "Search", "Monitoring", "APM"],
"duration_in_minutes": 45
}

Sample configuration for logstash (in a .conf file)
input {
stdin {}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout { codec => rubydebug }
}

Connecting Kibana to Elasticsearch server (in kibana.yml)
elasticsearch.hosts: ["http://localhost:9200"]

Connecting apm-server to elasticsearch server (in apm-server.yml)
output.elasticsearch:
hosts: ["localhost:9200"]

Introducing Elastic APM

On top of the stack…
Elastic APM is built on top of the ELK stack:
 Uses Elasticsearch as its datastore
 Uses logstash to collect, parse and transform logs
 Uses Kibana to render and visualise all kinds
of metrics related dashboards

Other interesting features of Elastic APM
Elastic APM…
 Can monitor in real time the performances
of thousands of applications.
 Automatically gathers performance relevant
informations.
 Automatically instrumentalise our apps and
measure response times for requests

How does Elastic APM fits into the Elastic ecosystem?
Logs
Metrics
Packets
Beats
Logstash
Datastore JMX
Elasticsearch Kibana
APM agent APM server

Setting up Elastic APM

What’s an agent?
APM agents are…
 Open source libraries written in the same
language as our application.
 Agents are hooks in your application that
collect performance metrics and errors.
 All collected datas are sent to the APM server

Deep diving into the APM server…
The APM server is…
 An application written in GO that runs on your
server just like the elasticsearch server.
 Listens by default on port 8200.
 Expose a simple json based Restfull http Api

Deep diving into the APM server…
The APM server …
 Receives datas from agents periodically.
 Builds Elasticsearch documents from the
datas collected by agents.
 The documents are stored in the
Elasticsearch cluster

Extended visualisation capabilities with Kibana and Xpack…
Using Kibana you can…
 Visualize the collected datas
 Create Dashboards
 Detect anomalies and create alerts

Drilling down into traces…
With Kibana’s APM UI you can…
 Drill-down into transaction traces
to see executed methods

Monitoring applications

Analysing results in Kibana
Elastic APM will provide us out of the box informations
about:
 The web request
 The background jobs
 The errors

Elastic APM will provide us out of the box informations
about:
 The average for any measure
 The 95th percentile
 The 99th percentile
 The number of requests per minute

It is also possible to:
 Create customized dashboard
 Choose your indexes
 Choose the datas to visualize

Handling identified issues
The analysis result:
 Will highlight bottlenecks, errors and issues that are
deeply rooted in the code
 Elastic APM allows us to zoom in and pinpoint precisely
the origin of problems

Setting up notifications and alerts
Notifications and alerts are very usefull and:
 Will periodically inform us when a metrics value exceeds
a specified threshold
 Elastic APM and the ELK stack allows us to configure
them with a great degree of flexibility

Wrapping up

Wrapping up
 Software development and lifecycle has grown complex
over time
 We can properly monitor our application with Elastic APM
 Elastic APM makes APM simple and beautful
 It is built on top of the ELK stack: which provides a
powerful and flexible search server, datasource agnostic
data collecting and transformation pipeline and a
fantastic report and dashboard generation capabilities

Resources
Images are from:
https://unsplash.com/:
 photo-1531956656798-56686eeef3d4.jpg
 candice-picard-800024-unsplash.jpg
https://pixabay.com/
 coast-guard-training-1138169_1280.jpg
 computer-159830_640.png
 computer-23261_640.png
 firewall-37925_640.png
 server-146615_640.png
 wifi-891896_640.png
 pebbles-796943_1920.jpg
https://github.com/kubernetes/kubernetes/tree/master/logo
 Logo.png
https://iconscout.com/
 Elasticsearch.png
https://www.iconfinder.com/icons/202809/redis_icon
 iconfinder_Untitled-2_202809.png
https://www.freeiconspng.com/img/3705
 computer-server-icon-png-19.png
https://icons8.com/:
 icons8-cloud-storage-64.png
 icons8-code-file-64.png
 icons8-conference-48.png
 icons8-connected-64.png
 icons8-docker-48.png
 icons8-edit-property-64.png
 icons8-email-64.png
 icons8-gears-48.png
 icons8-globe-earth-64.png
 icons8-jenkins-48.png
 icons8-new-file-64.png
 icons8-onedrive-48.png
 icons8-scatter-plot-48.png
 icons8-signal-40.png
 icons8-skyatlas-48.png
 icons8-steam-64.png
 icons8-web-shield-64.png
 icons8-website-64.png
https://www.elastic.co/brand

Licences
 https://unsplash.com/license
 https://pixabay.com/service/license/
 https://github.com/kubernetes/kubernetes/blob/master/logo/LICENSE
 https://support.iconscout.com/hc/en-us/articles/360007897374-Licensing
 https://www.freeiconspng.com/copyright-policy.html
 http://support.iconfinder.com/ownership-and-licenses/license-basic
 http://support.iconfinder.com/ownership-and-licenses/license-extended
 https://icons8.com/license
 https://www.elastic.co/brand

Application performance monitoring with Elastic APM and the ELK stack

More Related Content

What's hot (20)

Similar to Application performance monitoring with Elastic APM and the ELK stack (20)

Recently uploaded (20)

Application performance monitoring with Elastic APM and the ELK stack

Editor's Notes