Logging and monitoring

Logging is the process of monitoring events, mostly to

  • 🔎 to detect suspicious activities
  • ⚖️ to comply with investigations or legal regulations
  • 🚀 to monitor and improve performances
  • 🐛 to troubleshoot issues

➡️ It's important to analyze what the organization needs to log.

Logs can be generated from multiple sources which must be identified

  • 💻 Workstations (Windows 10/11...) and Servers
  • 📚 Applications and databases
  • 📶 Routers, Switches, Firewalls, Antivirus, IDS...
  • ...

What to log?

There are 4 main categories of logs:

  • User actions 🧔: successful and failed login attempts, creation or suppression of users, change of permissions, attempts at privilege escalation (su/sudo/runas)...
  • System actions 💻: application/services started/stopped, startup and shutdown, change to system settings...
  • Application events 🖨️: user actions, warnings, errors...
  • Network events 📶: incoming/outgoing traffic...

Usually, the minimal information needed to provide useful logs are:

  • Date and time of the event
  • Source and destination IP addresses or hostnames
  • Event type or category (e.g. login, file access, network traffic)
  • User ID (if applicable)
  • Outcome of the event (success or failure)

🎯 Ensure that clocks are synchronized (see the NTP protocol).

How to collect logs

There are two methods to collect logs:

  • with an agent (an additional software)
  • without an agent (agentless)

👉 Try to use native "services" such as syslog (if applicable).

There are two methods to transfer logs:

  • Pull 🛻: the server requests the logs (sftp...)
  • Push 🚀: the component sends its logs without being requested (syslog, snmp, using a script, SFTP...)

We can use UDP to transfer logs if we can accept a potential loss, otherwise, TCP should be used instead.


Collectors

It's possible to set up a collector that will collect the logs of a "zone", and transfer them to the server. This has a cost but allows better use of the bandwidth.

A collector can be Agent-based, Syslog-based, or Forwarding-based.

Centralized logs

Centralized logging means centralizing every log on one server.

  • we must ensure that logs are not tampered
  • we must closely monitor them to detect unusual activity
  • we must ensure that the person accessing the logs (read/write) has the right to do it. For instance, if there are logs with confidential data, their access should be restricted.
  • it's recommended to keep a raw version of the logs

Security Information and Event Management (SIEM)

A SIEM is a centralized logging solution that should detect attacks by correlating logs (temporal, based on the source/destination...) and alert/notify people. It can also generate reports.

  • Splunk
  • Microsoft
  • IBM
  • ...

➡️ Search for "Gartner SIEM". You should examine the number of events per second and the cost of configuring it when picking one.

Prioritization and alerting

As we will have many logs, we need to prioritize them

  • Priority 1: critical events that require immediate intervention (ex: system/IDS/antivirus crash, access to admin accounts...)
  • Priority 2: important events that require attention (ex: failed login attempts, system/application warnings/errors...)
  • Priority 3: informal events (ex: successful login attempts...)

When events occur, there is a need to notify the stakeholders. Both persons to contact and methods of contact should have been defined, most likely in accordance with the event priority.

Some methods of contact 📨: SMS, email, push notification...

Common log files locations

linuxfundamentals password_attacks linuxfundamentalspart3 adventofcyber4

  • /var/log/syslog and /var/log/messages (general and system logs)
  • /var/log/auth.log and /var/log/secure (authentication attempts)
  • /var/log/kern.log (kernel messages)
  • /var/log/dmesg (hardware and driver messages)
  • /var/log/audit/audit.log (audit logs)
  • /var/log/nginx/ and /var/log/apache2/ (webserver logs)
  • /var/log/mysql/ and /var/log/postgresql/ (dbms logs)
  • /var/log/cron (cron logs)
  • /var/log/boot.log (boot messages)
  • /var/log/lastlog (last logins)
  • /var/log/faillog (failed logins)
  • /var/log/btmp (failed logins)
  • /var/log/utmp (current users)
  • /var/log/wtmp (record of every login/logout)
  • /var/log/maillog (mail events)
  • /var/log/dpkg.log (installed packages)

On Windows, check out Windows Event Viewer.

👻 To-do 👻

Stuff that I found, but never read/used yet.

  • SIEM (SIM+SEM)
  • SIM=Security Information Management=collect, monitor, analyze logs
  • SEM=Security Event Management=real-time analysis, alerts
  • Configuration come at a cost, logs may still have to be checked manually if not correctly configured
  • windowseventlogs
  • endpoint-security-monitoring
  • security-information-event-management
  • var/log/kern.log (outdated drivers, system crashes, resource limitations)
  • Application logs (/var/log/fail2ban.log, apache, etc.)
  • Auditing
  • Connected as unusual hours
  • Unusual traffic at unusual hours
  • Copy or a big amount of data
  • AD Audit policy best practices
  • Handle scenario in which you can't access the logs
  • How log do we store logs? Minimum of 3y?
  • "You're monitored" (users more responsible)
  • access to resources
  • protect logs
  • confidential data in logs
  • Syslog server/rsyslog
  • prometheus