AdGuard Home v0.106.1 high CPU usage and slow DNS response

[N1ghtshad0w]

Issue Details

Version of AdGuard Home server:
    AdGuard Home, version v0.106.1
How did you install AdGuard Home:
    Docker Image
How did you setup DNS configuration:
    System
If it's a router or IoT, please write device model:
    Raspberry Pi 3 Model B+
CPU architecture:
    arm
Operating system and version:
    Raspberry Pi OS Kernel version: 5.10.17

Expected Behavior
AGH should run with minimal CPU usage.

Actual Behavior
AGH runs with high CPU usage dragging the whole system down resulting in a slow DNS response.

Additional Information
AGH was running fine using v0.105.2.
Upgraded to v0.106.1 over the previous version without changing any settings.
Slow DNS responses noticed immediately.
Running DNS benchmark revealed that AGH was running slower than external DNS.
Further investigation revealed that AGH is using high CPU %.
Clean installed v0.106.1 and reconfigured from scratch and still problem exists (high CPU usage and slow DNS response).
Rolling back to v0.105.2 fixes the problem.

[ameshkov]

We don't see anything like that on our side.

Probably, we'll need to see the pprof dumps from your RPi.

Here's how you can get them:

1. Stop AGH
2. Set debug_pprof to true in AdGuardHome.yaml
3. Start AGH
4. It'll start listening on localhost:6060

Now if you open http://localhost:6060/debug/pprof/profile?seconds=30, you'll download the CPU profile dump.
I'd like to see two of them - one for v0.105.2, the other one for v0.106.x.

Also, if you have go and graphwiz installed anywhere, you can take a look yourself: go tool pprof -http localhost:3434 path_to_dump will open a nice UI interface for exploring the dump.

[N1ghtshad0w]

We don't see anything like that on our side.

Probably, we'll need to see the pprof dumps from your RPi.

Here's how you can get them:

1. Stop AGH

2. Set `debug_pprof` to `true` in AdGuardHome.yaml

3. Start AGH

4. It'll start listening on `localhost:6060`

Now if you open http://localhost:6060/debug/pprof/profile?seconds=30, you'll download the CPU profile dump.
I'd like to see two of them - one for v0.105.2, the other one for v0.106.x.

Also, if you have go and graphwiz installed anywhere, you can take a look yourself: go tool pprof -http localhost:3434 path_to_dump will open a nice UI interface for exploring the dump.

Followed your steps to download the dump but it couldn't connect to http://adguardhome container ip:6060/debug/pprof/profile?seconds=30 most likely cause I am using macvlan schema.

I took 2 screenshots for htop one with v0.105.2 and one with v0.106.2
With v0.106.2 I see steady heavy CPU usage involving AdGuardHome.yaml
With v0.105.2 I don't see such usage.

v0.105.2

v0.106.2

[DeuceCode]

Can confirm was also experiencing this issue on v0.106.1 and v0.106.2 on a Raspberry Pi 4 4GB. Rolled back to v0.105.2 and this issue was no longer present.

I first noticed this issue when checking my Adguard Home Dashboard and noticed the average processing time for queries had jumped up from 20-30ms to several 100s of milliseconds. A quick look through the query logs showed a lot of Reverse DNS lookups taking on the order of 700-900ms to complete. My top queried domain had quickly become 1.0.28.172.in-addr.arpa the address of the adguard container with almost 5000 queries in a 24 hour period. Having reverted about 12 hours ago, queries to that address have already halved,

I've also followded your steps to download the dumps, but was also unable to connect. I'm not using macvlan so I'm sure its not that. I'll keep trying a few other configs and update if I get it working.

[ameshkov]

Followed your steps to download the dump but it couldn't connect to http://adguardhome container ip:6060/debug/pprof/profile?seconds=30 most likely cause I am using macvlan schema.

Hmm, I don't think this port is exposed from the container.

You need to run the container with --expose 6060/tcp and add port mapping.

[ameshkov]

@DeuceCode @N1ghtshad0w could you please record verbose-level log? Maybe we can figure something from it.

1. Configure AdGuard Home to write verbose-level log.
2. Reproduce the issue.
3. Post the log file here.

[N1ghtshad0w]

Followed your steps to download the dump but it couldn't connect to http://adguardhome container ip:6060/debug/pprof/profile?seconds=30 most likely cause I am using macvlan schema.

Hmm, I don't think this port is exposed from the container.

You need to run the container with --expose 6060/tcp and add port mapping.

You are right port 6060 is not exposed from the container.
Only these ports are

EXPOSE map[3000/tcp:{} 3001/tcp:{} 443/tcp:{} 443/udp:{} 53/tcp:{} 53/udp:{} 5443/tcp:{} 5443/udp:{} 67/udp:{} 68/udp:{} 784/udp:{} 80/tcp:{} 853/tcp:{}]

macvlan is an overlay network that assigns an IP address to your container, all declared expose ports are accessible on the assigned IP and nothing needs to be forwarded to the host.

Port 6060 need to be added to the dockerfile EXPOSE command, adding it to the docker run command doesn't do anything.

[ameshkov]

Well, the difference is rather huge, hopefully, we'll be able to figure it out from the verbose log.

[CaesarRulezzz]

I have this issue on v0.106.2 (and v0.106.1). As a temporary measure I have disabled almost all filtering lists. This has helped to reduce latency from 18k ms to 100 ms.

[ameshkov]

@CaesarRulezzz what's your config? Is it docker as well?

Could you please reproduce the issue and record the verbose-level logs #3064 (comment) ?

@ all - if you want to share logs privately, please send them to devteam@adguard.com and mention this issue.

[CaesarRulezzz]

@CaesarRulezzz what's your config? Is it docker as well?

Could you please reproduce the issue and record the verbose-level logs #3064 (comment) ?

@ all - if you want to share logs privately, please send them to devteam@adguard.com and mention this issue.

Yes, I use it in a docker container.

[CaesarRulezzz]

@CaesarRulezzz what's your config? Is it docker as well?

Could you please reproduce the issue and record the verbose-level logs #3064 (comment) ?

@ all - if you want to share logs privately, please send them to devteam@adguard.com and mention this issue.

I've sent logs to devteam@adguard.com

[ameshkov]

Received, thank you!