Issue description
I’ve used the DNS filter on the logstash, but i can clearly see that the indexing speed has decreased by adding resolve.
Does it have to be so slow?
Logstash config from documentation:
filter {
dns {
reverse => [ "source_host", "field_with_address" ]
resolve => [ "field_with_fqdn" ]
action => "replace"
}
}
Issue solution
In older versions of logstash (2018 *), after using the cache_size / failed_cache_size directive, there was a bug that prevented parallel cache polling.
A very nice analysis with performance graphs was carried out by the git user named robcowart:
https://github.com/logstash-plugins/logstash-filter-dns/pull/42
A ready config to use below – please note that full performance is obtained when the cache is full with data.
It’s also worth using fast dns, e.g. 1.1.1.1/1.0.0.1
filter{
# dns resolve
dns {
reverse => [ "hostname" ]
action => "replace"
nameserver => ["1.1.1.1", "1.0.0.1"]
hit_cache_size => 131072
hit_cache_ttl => 900
failed_cache_size => 131072
failed_cache_ttl => 900
}
# filter performance
metrics {
meter => "events"
add_tag => "metric"
}
}
output {
if "metric" in [tags] {
stdout {
codec => line {
format => "DNS filter rate: %{[events][rate_1m]}"
}
}
}
}