How to remove duplicated or not important messages from syslog?

Issue description

we all know this entry in the syslog:
... last message repeated ... times

can it somehow be easily ruled out?

Issue solution

Yes, they can. There are many ways to do so and below is only one such example:


filter {
  if [source] == "/ var / log / messages" {
    if [message] =~ / last message repeated [0-9] + times / {
      drop {}
    }
  }
}

DNS logstash filter is slow

Issue description

 

I've used the DNS filter on the logstash, but i can clearly see that the indexing speed has decreased by adding resolve.
Does it have to be so slow?

Logstash config from documentation:

filter {
  dns {
    reverse => [ "source_host", "field_with_address" ]
    resolve => [ "field_with_fqdn" ]
    action => "replace"
  }
}

Issue solution

 

In older versions of logstash (2018 *), after using the cache_size / failed_cache_size directive, there was a bug that prevented parallel cache polling.

A very nice analysis with performance graphs was carried out by the git user named robcowart:
https://github.com/logstash-plugins/logstash-filter-dns/pull/42

A ready config to use below - please note that full performance is obtained when the cache is full with data.
It's also worth using fast dns, e.g. 1.1.1.1/1.0.0.1

filter{
  # dns resolve
  dns {
    reverse => [ "hostname" ]
    action => "replace"
    nameserver => ["1.1.1.1", "1.0.0.1"]
    hit_cache_size => 131072
    hit_cache_ttl => 900
    failed_cache_size => 131072
    failed_cache_ttl => 900
  }

  # filter performance
  metrics {
    meter => "events"
    add_tag => "metric"
  }
}

output {
  if "metric" in [tags] {
    stdout {
      codec => line {
        format => "DNS filter rate: %{[events][rate_1m]}"
      }
    }
  }
}

Future Tech Event with our partner – CyberX

We are proud to announce the Future Tech Event conference in Oman, whose platinum sponsor is our partner from the MENA region - CyberX.

Future Tech Event is an event presenting the latest ICT products and services, the latest devices, consumer electronics and the most modern intelligent technology in all sectors - including cybersecurity.

At this event, we will have the opportunity to listen to presentations by the founder of CyberX, Mohannad Alkalash, and our engineer - Szymon Ćwieka.

 

To sign up for the event and listen to the lectures, please click here: https://www.futuretechevent.com