Logging and monitoring

Logging is the process of monitoring events, mostly to

  • 🔎 to detect suspicious activities
  • ⚖️ to comply with investigations or legal regulations
  • 🚀 to monitor and improve performances
  • 🐛 to troubleshoot issues

➡️ It's important to analyze what the organization needs to log.

Logs can be generated from multiple sources which must be identified

  • 💻 Workstations (Windows 10/11...) and Servers
  • 📚 Applications and databases
  • 📶 Routers, Switches, Firewalls, Antivirus, IDS...
  • ...

What to log?

There are 4 main categories of logs:

  • User actions 🧔: successful and failed login attempts, creation or suppression of users, change of permissions, attempts at privilege escalation (su/sudo/runas)...
  • System actions 💻: application/services started/stopped, startup and shutdown, change to system settings...
  • Application events 🖨️: user actions, warnings, errors...
  • Network events 📶: incoming/outgoing traffic...

Usually, the minimal information needed to provide useful logs are:

  • Date and time of the event
  • Source and destination IP addresses or hostnames
  • Event type or category (e.g. login, file access, network traffic)
  • User ID (if applicable)
  • Outcome of the event (success or failure)

🎯 Ensure that clocks are synchronized (see the NTP protocol).

How to collect logs

There are two methods to collect logs:

  • with an agent (an additional software)
  • without an agent (agentless)

👉 Try to use native "services" such as syslog (if applicable).

There are two methods to transfer logs:

  • Pull 🛻: the server requests the logs (sftp...)
  • Push 🚀: the component sends its logs without being requested (syslog, snmp, using a script, SFTP...)

We can use UDP to transfer logs if we can accept a potential loss, otherwise, TCP should be used instead.

Collectors

It's possible to set up a collector that will collect the logs of a "zone", and transfer them to the server. This has a cost but allows better use of the bandwidth.

A collector can be Agent-based, Syslog-based, or Forwarding-based.


Centralized logs

Centralized logging means centralizing every log on one server.

  • we must ensure that logs are not tampered
  • we must closely monitor them to detect unusual activity
  • we must ensure that the person accessing the logs (read/write) has the right to do it. For instance, if there are logs with confidential data, their access should be restricted.
  • it's recommended to keep a raw version of the logs

Managing your Logs

Prioritization and alerting

As we will have many logs, we need to prioritize them:

  • Priority 1 🔥: critical events that require immediate intervention (ex: system/IDS/antivirus crash, access to admin accounts...)

  • Priority 2 😰: important events that require attention (ex: failed login attempts, system/application warnings/errors...)

  • Priority 3 💥: informal events (ex: successful login attempts...)

When events occur, there is a need to notify the stakeholders. Both persons to contact and methods of contact should have been identified, most likely in accordance with the event priority.

Some methods of contact 📨: SMS, email, push notification...

Security Information and Event Management (SIEM)

A SIEM is a centralized logging solution that should detect attacks by correlating logs (temporal, based on the source/destination...) and alert/notify people. It can also generate reports.

  • Splunk
  • Microsoft
  • IBM
  • ...

➡️ Search for "Gartner SIEM". You should examine the number of events per second and the cost of configuring it when picking one.

📚 You may use Sigma rules to filter logs.

Common log files locations

linuxfundamentals password_attacks linuxfundamentalspart3 linuxfilesystemanalysis adventofcyber4

On Windows, check out Windows Event Viewer.

On Linux, check the following files:

  • /var/log/syslog and /var/log/messages (general and system logs)
  • /var/log/auth.log and /var/log/secure (authentication attempts)
  • /var/log/kern.log (kernel messages)
  • /var/log/dmesg (hardware and driver messages)
  • /var/log/cron (cron logs)
  • /var/log/boot.log (boot messages)
  • /var/log/lastlog (last logins)
  • /var/log/faillog (failed logins)
  • /var/log/btmp (failed logins)
  • /var/log/utmp (current users)
  • /var/log/wtmp (record of every login/logout)
  • /var/log/audit/audit.log (audit logs)
  • /var/log/maillog (mail events)
  • /var/log/dpkg.log (installed packages)
  • /var/log/apache2/ for access.log and error.log (webserver logs)
  • /var/log/nginx/ (webserver logs)
  • /var/log/mysql/ and /var/log/postgresql/ (dbms logs)
  • /var/log/sshd.log (SSH logs)
  • /var/log/vsftpd.log (FTP logs)

👻 To-do 👻

Stuff that I found, but never read/used yet.

  • SIEM (SIM+SEM)
  • SIM=Security Information Management=collect, monitor, analyze logs
  • SEM=Security Event Management=real-time analysis, alerts
  • Configuration come at a cost, logs may still have to be checked manually if not correctly configured
  • windowseventlogs
  • endpoint-security-monitoring
  • security-information-event-management
  • var/log/kern.log (outdated drivers, system crashes, resource limitations)
  • Application logs (/var/log/fail2ban.log, apache, etc.)
  • Auditing
  • Connected as unusual hours
  • Unusual traffic at unusual hours
  • Copy or a big amount of data
  • AD Audit policy best practices
  • Handle scenario in which you can't access the logs
  • How log do we store logs? Minimum of 3y?
  • "You're monitored" (users more responsible)
  • access to resources
  • protect logs
  • confidential data in logs
  • Syslog server/rsyslog
  • prometheus