From: "felix.wolfsteller@... (Felix Wolfsteller) via ruby-core" <ruby-core@...>
Date: 2023-04-28T10:05:15+00:00
Subject: [ruby-core:113373] [Ruby master Bug#19621] Resolv::Hosts uses ineffective File.read, making using big hosts file 'impossible'

Issue #19621 has been updated by felix.wolfsteller@betterplace.org (Felix Wolfsteller).


The reading does not seem to be the problem, but the hash operations.


----------------------------------------
Bug #19621: Resolv::Hosts uses ineffective File.read, making using big hosts file 'impossible'
https://bugs.ruby-lang.org/issues/19621#change-102940

* Author: felix.wolfsteller@betterplace.org (Felix Wolfsteller)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
By default on unixoid systems, `Resolv` will read `/etc/hosts` once. Privacy- and security aware people might use the file to prevent unwanted traffic, developers use it to quickly manipulate address resolution.

`Resolv::Hosts` uses [`IO.read`](https://github.com/betterplace/ruby/blob/9b07d30df8c6bf65c2558c023fd6452405915610/lib/resolv.rb#LL195C4-L195C4), which seems to be inefficient when dealing with large amounts of data that should be consumed by line.

E.g. if you install the `/etc/hosts` additions by [hblock](https://hblock.molinero.dev/hosts) (https://github.com/hectorm/hblock), the first call to resolve an address will likely take **minutes**.

Unfortunately, replacing `.open ... .each` with `IO.foreach` does not help.

Benchmarking with partial examplary `/etc/hosts` from above (172751 line) with this

```ruby
require 'resolv'
require 'benchmark'

Benchmark.measure do
  Resolv::Hosts.new.lazy_initialize
end
```

yields to
```
25.622515   8.821095  34.443610 ( 34.495448)
```
..

Reading in all the lines into memory first and then consuming them  (`File.readlines`) might improve the situation, but is probably not desirable due to memory concerns.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/