If an attacker manages to bypass your edge firewalls, dodge your systemd sandboxes, and exploit a local service, what happens next? In a standard environment, sophisticated attackers immediately look to do two things: establish persistence (by modifying binaries or adding cron jobs) and cover their tracks (by erasing standard log files).

To defend against this, your hardening strategy needs post-compromise visibility and system integrity verification. This is achieved by combining two powerful tools: Auditd (The Linux Auditing System) for real-time kernel-level event tracking, and AIDE (Advanced Intrusion Detection Environment) for host-based file integrity monitoring.

Here is how to deploy them to create an unalterable forensic trail.

1. Auditd: Tracking Kernel-Level Events in Real Time

Unlike standard application logs that can be easily manipulated or turned off by a privileged user, auditd hooks directly into the Linux kernel. It monitors system calls (syscalls) at the lowest level, tracking exactly who executed a command, what file they touched, and when.

Writing Bulletproof Audit Rules

Auditd relies on rules defined in /etc/audit/rules.d/audit.rules. To track an attacker’s movements effectively, we want to watch critical system configuration files and user space activity.

Here are essential production rules to add to your configuration:

# First, clear any existing rules
-D

# Increase the buffer size to handle high-volume event logging
-b 8192

# Watch for changes to user accounts and authentication
-w /etc/passwd -p wa -k user_changes
-w /etc/shadow -p wa -k user_changes
-w /etc/sudoers -p wa -k sudo_changes

# Watch for modifications to network and SSH configurations
-w /etc/ssh/sshd_config -p wa -k sshd_changes
-w /etc/network/interfaces -p wa -k network_changes

# Track execution of specific administrative tools
-a always,exit -F arch=b64 -S execve -F euid=0 -k root_commands

• -w: Specifies the file or directory path to watch.

• -p wa: Triggers an alert on writes and attribute alterations (like chmod or chown).

• -k: Assigns a unique search key (e.g., user_changes) so you can quickly filter your logs later.

Searching the Forensic Trail

When an anomaly occurs, you use ausearch to parse the audit logs. For example, to see every event tied to your SSH configuration rule, run:

sudo ausearch -k sshd_changes

To see a summary of all commands executed by user IDs over a specific timeframe, use aureport:

sudo aureport -x --summary

2. AIDE: Catching Unauthorized File Modifications

While auditd tells you who is doing something in real-time, AIDE tells you what changed on the filesystem after the fact. AIDE takes a snapshot of your system’s binaries, libraries, and configurations, hashes them, and stores the hashes in a secure database.

If an attacker drops a rootkit, alters /bin/bash, or adds a malicious backdoor line to a system library, AIDE will flag it on the next run.

Step 1: Initialize the Baseline Database

After installing AIDE, you must generate the initial trusted snapshot of your clean operating system.

sudo aideinit

This creates a baseline database file (typically aide.db.new.gz). To make it active, move it to the production path:

sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

Step 2: Running regular Integrity Checks

To compare the current state of the server against your trusted baseline, execute a check:

sudo aide --check

If a malicious actor added a backdoored file or altered an existing binary, AIDE will print an aggressive summary showing exact cryptographic hash mismatches, size differences, and modified timestamps.

Step 3: Automating the System Audit

Manually running integrity checks is inefficient. Automate AIDE via a daily cron job that emails you the output or pipes it to your centralized logging dashboard:

0 3 * * * /usr/bin/aide --check | mail -s "Daily AIDE Integrity Report for $(hostname)" [email protected]

Crucial Operational Note: Whenever you intentionally update packages or modify configuration files via apt, yum, or Ansible, your AIDE checks will fail because the files legitimately changed. You must update your baseline database immediately after any maintenance window:

sudo aide --update
sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

3. The Ultimate Rule: Securing the Logs

Deploying Auditd and AIDE on the local machine is only half the battle. If an attacker manages to achieve full root privileges, they can eventually delete the local AIDE database or clear the /var/log/audit/audit.log file.

To truly secure your forensic logging:

1. Ship Logs Instantly: Use tools like rsyslog or vector to immediately forward all system and audit logs to a dedicated, offsite log management server.
2. Read-Only Databases: Store a backup copy of your master aide.db.gz file off-server, or put it on a read-only filesystem mount. If you suspect a breach, copy the verified off-site database back to the server to run your integrity check.

By implementing this dual-layered forensic setup, you ensure that even if an intrusion occurs, the attacker cannot hide, and you possess the absolute clarity required to remediate the system completely.