Do you know what your Drupal is doing Observe it! (DrupalCon Prague 2022)

lussoluca
Do you know what your Drupal is doing?
Observe it!

Luca Lusso
Drupal / PHP / Go developer @ SparkFabrik
Drupal contributor (WebProﬁler, Monolog, …) and speaker
Drupal.org: https://www.drupal.org/u/lussoluca
Twitter: https://www.twitter.com/lussoluca
LinkedIn: www.linkedin.com/in/lussoluca
Slack (drupal.slack.com): lussoluca

WE ARE A TECH COMPANY OF ENGINEERS,
DEVELOPERS AND DESIGNERS WHO WILL
THINK, DESIGN AND BUILD YOUR CUSTOM APPLICATIONS,
MODERNIZE YOUR LEGACY AND TAKE YOU TO THE
CLOUD NATIVE ERA.
www.sparkfabrik.com
SPARKFABRIK

We help italian businesses to bridge
the gap with China thanks to our
Official Partnership with
Alibaba Cloud.
SparkFabrik is Cloud Native
Computing Foundation
Silver Member.
SparkFabrik is Google Cloud Platform
Technology Partner.
SparkFabrik is AWS
Official Partner.
PROUD OF OUR PARTNERSHIPS

Almost everyone is working with distributed systems.
There are microservices, containers, cloud, serverless, headless, message queues
and a lot of combinations of these technologies.

All of these, increase the number of failures that systems may encounter
because there are too many parts interacting.
And because of the distributed system’s diversity, it’s
complex to understand present problems and predict
future ones

Observability is a measure of how well
internal states of a system can be
inferred from knowledge of its external
outputs

We need more data to correlate classic metrics like CPU
and memory spikes with application behaviours

3 pillars of
observability
● Structured logs
● Metrics
● (Distributed) Traces

Tools
We need different tools and technologies to collect logs, traces
and metrics, and we need a way to query and visualize them.
Logs -> Monolog + Promtail + Loki
Metrics -> Prometheus
Traces -> OpenTelemetry + Tempo
Visualization -> Grafana

Tools
Grafana
Grafana allows to query, visualize, alert on
and understand metrics, traces and logs no
matter where they are stored.
grafana.com

1. Logs are about storing speciﬁc events

Tools
Monolog
Monolog is a standard PHP library and it can
be included in a Drupal website using a
contrib module (drupal.org/project/monolog)
Monolog sends logs to ﬁles, sockets,
inboxes, databases and various web
services.
Monolog implements the PSR-3 interface
that can be type-hint against in code to keep
a maximum of interoperability.
github.com/Seldaek/monolog

Download the Monolog module using
Composer, to have both the Drupal module and the PHP library
composer require drupal/monolog:^2.2

Monolog module doesn't have a UI, it's conﬁgured using yml ﬁles,
for example
sites/default/monolog.services.yml

services:
monolog.handler.rotating_file:
class: MonologHandlerRotatingFileHandler
arguments: ['private://logs/debug.log', 10, 'INFO']
First of all deﬁne a new service in the service container

parameters:
monolog.channel_handlers:
default:
handlers:
- name: 'rotating_file'
formatter: 'json'
monolog.processors: [
'message_placeholder', 'current_user',
'request_uri', 'ip', 'referer',
'filter_backtrace', 'introspection'
]
Then define handlers, formatter
and processors using service
container parameters. Here we're
configuring the default channel to
catch all log messages and to save
them using the
monolog.handler.rotating_file
service, in json format and after
being processed by a set of
processors

$settings['container_yamls'][] =
DRUPAL_ROOT . '/sites/default/monolog.services.yml';
Add monolog.services.yml to
the list of container’s yamls in settings.php ﬁle
sites/default/settings.php

Drupal::logger('drupalcon')->notice('Data from remote microservice.');

{
"message": "Data from remote microservice.",
"context": {},
"level": 250,
"level_name": "NOTICE",
"channel": "drupalcon",
"datetime": "2022-09-20T10:05:53.811568+02:00",
"extra": {
"referer": "",
"ip": "172.20.0.9",
"request_uri": "https://drupal10.ddev.site/microservice",
"uid": "1",
"user": "admin",
"file": "/.../drupalcon/src/Controller/MicroserviceController.php",
"line": 26,
"class": "DrupaldrupalconControllerMicroserviceController",
"function": "view"
}
}
private://logs/debug-2022-09-20.log

Structured logs makes it simple to query them for any
sort of useful information
We can write custom Monolog processors to add
application’s custom data to our logs
(for example: pod name, cluster id, order id, …)

In a Cloud Native environment, the application runs on multiple servers (or pods). We
need a way to export all those logs generated by every instance of the application.
In this case our logs are ﬁles stored in the local ﬁlesystem of every instance.
We have to discover, scrape and send them to a log collector.
Promtail is an agent which ships the contents of local logs to a private Grafana Loki
instance or Grafana Cloud. It is usually deployed to every machine that has applications
needed to be monitored.

Scraping logs with Grafana Promtail
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /mnt/ddev_config/o11y/loki/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: drupal
pipeline_stages:
- json:
expressions:
level: level_name
uid: extra.uid
- labels:
level:
uid:
- static_labels:
filename: 'drupal.log'
static_configs:
- targets:
- localhost
labels:
job: drupal
__path__: /var/www/html/web/sites/default/files/private/logs/*log

Scraping and sending logs to Grafana
Monolog -> logs (ﬁlesystem) -> Promtail -> Loki -> Grafana

2. Metrics are a measurement at a point in time for the system

Tools
Prometheus
Prometheus is an open-source systems
monitoring and alerting toolkit originally built
at SoundCloud. It is now a standalone open
source project and it’s maintained
independently of any company.
Prometheus collects and stores metrics as
time series data.
Prometheus was the second project to join
the Cloud Native Computing Foundation after
Kubernetes.
prometheus.io

Examples of metrics you might need:
● the number of times you receive an HTTP request
● how much time was spent handling requests
● how many users/nodes has been created (over time)
● the number of modules that need a security update
● The number of orders in an ecommerce site
● And many more …

There’s a module for that!
Observability suite
https://www.drupal.org/project/o11y
composer require drupal/o11y:1.x-dev
drush pm:enable o11y_metrics o11y_metrics_requests

/admin/people/permissions
/admin/conﬁg/system/o11y_metrics/plugins-settings

Prometheus scrapes data at the /metrics endpoint at a conﬁgured rate
PHP uses a shared-nothing architecture by default
o11y needs a way to store data between one scrape and the next
default implementation uses Database as a storage backend (but you can use Memcache
or Redis as well)

The main module (o11y_metrics) provides the following metrics:
PHP info
Node count: total and with bundle labels
Node revision count: total and with bundle labels
Extensions: list of modules/themes/profiles installed with name and version labels
Queue size: list of queues with number of items in them
User count: total, with status (active/blocked) and role labels
A set of submodules can be installed to provide additional metrics:
o11y_metrics_cache: cache total hits, miss with bin labels. Tag invalidations, with tag and request path labels
o11y_metrics_config: information whether the drupal config is out of sync or not
o11y_metrics_database: histograms for time spent on select queries, with database target name and route labels
o11y_metrics_requests: histograms for time spent on requests, with http method, route name and http code status labels
o11y_metrics_update: info about existing core/module/theme updates
o11y_metrics_comment: comments count, with status labels

The module exposes an URL with metrics in
Prometheus format (/metrics)
# HELP drupal_http_requests Timing metrics for requests.
# TYPE drupal_http_requests histogram
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.005"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="1"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="+Inf"} 6
drupal_http_requests_count{method="GET",route="user_admin_create",status="2xx"} 6

scrape_configs:
- job_name: 'drupal'
scrape_interval: 5s
static_configs:
- targets: ['web:80']
Scraping and sending metrics to Grafana Prometheus

o11y_metrics -> local database -> Prometheus -> Grafana

2. Metrics are a measurement at a point in time for the system
3. Distributed traces deals with information that is request-scoped

Tools
OpenTelemetry
OpenTelemetry is a collection of tools, APIs,
and SDKs. It can be used to instrument,
generate, collect, and export telemetry data
(metrics, logs, and traces) to help analyze
software’s performance and behavior.
OpenTelemetry is an incubating project from
the Cloud Native Computing Foundation,
created after the merger of OpenCensus
(from Google) and OpenTracing (from Uber).
The data collected with OpenTelemetry is
vendor-agnostic and can be exported in
many formats.
https://opentelemetry.io

Main cloud vendor support for OpenTelemetry
● AWS Distro for OpenTelemetry: aws-otel.github.io/
● Google Cloud OpenTelemetry: google-cloud-opentelemetry.readthedocs.io
● Azure Monitor OpenTelemetry:
docs.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview

Tools
OpenTelemetry
Language support
PHP:
Go:
Javascript:
Tracing Metrics Logging
pre-alpha pre-alpha not yet implemented
stable alpha not yet implemented
stable RC in development

The OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export
telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors. It allows your service to
offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even
sensitive data ﬁltering.

Tools
Tempo
Grafana Tempo is an open source,
easy-to-use, and high-scale distributed
tracing backend. Tempo is cost-efficient,
requiring only object storage to operate, and
is deeply integrated with Grafana,
Prometheus, and Loki. Tempo can ingest
common open source tracing protocols,
including Jaeger, Zipkin, and
OpenTelemetry.
grafana.com/oss/tempo

We will use the Observability suite module to instrument our
application
Internally the module uses OpenTelemetry to do the hard work
drush pm:enable o11y_traces

$class_loader->addPsr4('Drupaltracer', [ __DIR__ . '/../../modules/contrib/tracer/src']);
$settings['container_base_class'] = 'DrupaltracerDependencyInjectionTraceableContainer';
$settings['tracer_plugin'] = 'o11y_tracer';
sites/default/settings.php
There should be only one module that instruments the code at a time. We need to replace a lot
of services and subsystems with traceable versions.
To avoid code duplication we create a third project: Tracer
(https://www.drupal.org/project/tracer) which both WebProﬁler and Observability suite
depend on.
https://blog.sparkfabrik.com/en/webproﬁler-updates

Per-process logging and metric monitoring have their
place, but neither can reconstruct the elaborate
journeys that transactions take as they propagate
across a distributed system. Distributed traces are
these journeys

We take for example a Drupal 10 website
that renders a page with some data that comes from a
remote microservice

class MicroserviceController extends ControllerBase {
private Client $httpClient;
public static function create(ContainerInterface $container) {
return new static(
$container->get('http_client')
);
}
final public function __construct(Client $httpClient) {
$this->httpClient = $httpClient;
}
public function view() {
$response = $this->httpClient->get('http://ddev-drupal10-microservice:8080/endpoint1');
$json = json_decode($response->getBody()->getContents());
$this->loggerFactory->get(drupalcon)->notice($json->message);
return [
'#theme => 'microservice',
'#message' => $json->message,
];
}
}

receivers:
otlp:
protocols:
http:
processors:
batch:
exporters:
otlphttp:
endpoint: http://tempo:4318
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
OpenTelemetry collector to send traces to a
Grafana Tempo instance

Observability suite automatically instrument
● Events
● Twig templates
● HTTP calls
● Database queries
● Services (optional)
but you can trace your own code too!

class MicroserviceController extends ControllerBase {
public function view() {
$response =
$this->httpClient->get('http://ddev-drupal10-microservice:8080/endpoint2');
$json = json_decode($response->getBody()->getContents());
$this->someComplexMethod();
return [...];
}
private function someComplexMethod() {
$tracer = Drupal::service('tracer.tracer');
$span = $tracer->start('custom','someComplexMethod',
['someAttribute' => 'someValue']
);
sleep(1);
$tracer->stop($span);
}
}

o11y_traces -> OpenTelemetry collector -> Tempo -> Grafana

One last thing we need, is to correlate traces with logs,
so when we found a problem with a request we can go
from the trace to the logs (and viceversa)

The o11y module provides a new processor for
Monolog that adds a trace_id argument to every log

parameters:
monolog.channel_handlers:
default:
handlers:
- name: 'rotating_file'
formatter: 'json'
monolog.processors: [
'message_placeholder', 'current_user',
'request_uri', 'ip', 'referer',
'filter_backtrace', 'introspection', 'tracer'
]

❯ curl -I https://drupalcon-prague-2022.ddev.site/en
HTTP/2 200
server: nginx/1.20.1
date: Mon, 19 Sep 2022 10:39:26 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
cache-control: must-revalidate, no-cache, private
x-drupal-dynamic-cache: MISS
x-drupal-trace-id: 355f26d048b5f8bb5e4aa232ce35a81d
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
permissions-policy: interest-cohort=()
expires: Sun, 19 Nov 1978 05:00:00 GMT
x-generator: Drupal 10 (https://www.drupal.org)
x-debug-token: a79d3a
x-debug-token-link:
https://drupalcon-prague-2022.ddev.site/en/admin/reports/profiler/view/a79d3a
x-drupal-cache: HIT

Example stack and source code:
https://github.com/lussoluca/drupalcon_prague_2022

Join us for
contribution opportunities
20-23 September, 2022
Room C2 + C3
Mentored
Contribution
First Time
Contributor Workshop
General
Contribution
#DrupalContributions
20 - 22 September: 9:00 - 18:00
Room C3
23 September: 9:00 - 18:00
Room C2 + C3
20 September: 17:15 - 18:00
Room D9
21 September: 10:30 - 11:15
Room D9
23 September: 09:00 - 12:30
Room C2
23 September: 09:00 - 18:00
Room C2 + C3

What did you think?
Please ﬁll in this session survey directly from the Mobile App.

We appreciate your feedback!
Please take a moment to fill out:
the general
conference survey
Flash the QR code
OR
It will be sent by email
the Individual
session surveys
(located under each session description)
1 2

Do you know what your Drupal is doing Observe it! (DrupalCon Prague 2022)

More Related Content

Similar to Do you know what your Drupal is doing Observe it! (DrupalCon Prague 2022) (20)

More from sparkfabrik (20)

Recently uploaded (20)

Do you know what your Drupal is doing Observe it! (DrupalCon Prague 2022)