How to Scan Linux for Malware and Rootkits

Linux systems are generally considered more secure than other operating systems, but they're not immune to malware, rootkits, and other security threats. Regular security scanning is essential for maintaining system integrity and detecting potential compromises. This guide walks you through using powerful open-source tools to scan your Linux system for malware, rootkits, and suspicious activities.

Introduction

While Linux may be less frequently targeted than Windows, the rising popularity of Linux servers and the increasing sophistication of attackers means Linux systems are increasingly at risk. Security scanning should be a regular part of your system maintenance routine, helping you to:

• Detect malware and viruses that may have infected your system
• Identify rootkits that hide deep in your system to maintain unauthorized access
• Discover suspicious files, processes, or system modifications
• Verify the integrity of critical system files
• Maintain compliance with security policies and regulations

In this guide, we'll explore several powerful security tools including ClamAV, Rootkit Hunter, chkrootkit, and Lynis that provide comprehensive scanning capabilities for Linux systems.

Prerequisites

• A Linux system (this guide uses Ubuntu/Debian commands, but can be adapted)
• Root or sudo access
• Basic knowledge of Linux command line
• Internet connection (for downloading tools and virus definitions)

Step 1: Update Your System

Before installing security tools, ensure your system is up-to-date with the latest security patches:

sudo apt update
sudo apt upgrade -y

Step 2: Install and Configure ClamAV

ClamAV is an open-source antivirus engine designed for detecting trojans, viruses, malware, and other malicious threats.

Installing ClamAV

sudo apt install clamav clamav-daemon -y

After installation, the ClamAV daemon (clamd) will automatically start, and the freshclam service will periodically update virus definitions.

Updating Virus Definitions

To manually update virus definitions:

sudo systemctl stop clamav-freshclam
sudo freshclam
sudo systemctl start clamav-freshclam

Basic ClamAV Scanning

To scan a specific directory:

sudo clamscan -r /path/to/directory

To scan your home directory and show only infected files:

sudo clamscan -r --infected /home/username

To scan the entire system (this will take a long time):

sudo clamscan -r /

Advanced ClamAV Scanning

For a more comprehensive scan with detailed output:

sudo clamscan -r --infected --detect-pua=yes --alert-broken=yes --alert-encrypted=yes --recursive=yes /home

This command:

• --infected: Only shows infected files
• --detect-pua=yes: Detects Potentially Unwanted Applications
• --alert-broken=yes: Alerts on broken executables
• --alert-encrypted=yes: Alerts on encrypted files
• --recursive=yes: Scans subdirectories recursively

To save scan results to a log file:

sudo clamscan -r --infected /home -l ~/clamscan.log

Step 3: Install and Use Rootkit Hunter (rkhunter)

Rootkit Hunter is a security monitoring and analyzing tool that scans for rootkits, backdoors, and possible local exploits.

Installing rkhunter

sudo apt install rkhunter -y

Updating rkhunter

Update the tool's database before scanning:

sudo rkhunter --update

Running a Rootkit Scan

Perform a comprehensive system check:

sudo rkhunter --check

This scan checks for:

• Hidden files and directories
• Wrong file permissions
• Suspicious strings in kernel modules
• Known rootkit signatures
• System command modifications

For a more detailed scan with all tests:

sudo rkhunter --check --sk

The --sk option skips the key press requirement between test sections.

Understanding rkhunter Warnings

After running rkhunter, review any warnings in the log file:

sudo cat /var/log/rkhunter.log | grep -i warning

Not all warnings indicate an actual threat. Some common false positives include:

• Hidden directories used by legitimate package managers
• Custom configurations that differ from defaults
• Recently updated system utilities

Updating rkhunter's File Properties Database

After system updates or legitimate system changes, update rkhunter's file properties database to prevent false positives:

sudo rkhunter --propupd

Step 4: Install and Use chkrootkit

chkrootkit is another rootkit scanner that uses a different detection approach than rkhunter, making it valuable as a secondary verification tool.

Installing chkrootkit

sudo apt install chkrootkit -y

Running chkrootkit

Run a basic scan:

sudo chkrootkit

For a more silent operation with only warning messages:

sudo chkrootkit -q

Understanding chkrootkit Output

chkrootkit scans for signs of:

• Known rootkits
• Trojanized system binaries
• Unusual network interfaces
• Signs of LKM trojan activity

Pay particular attention to "INFECTED" results, but be aware that false positives can occur with chkrootkit as well.

Step 5: Install and Use Lynis

Lynis is an auditing tool that performs in-depth security scans and provides hardening recommendations. It's more comprehensive than just a malware scanner.

Installing Lynis

sudo apt install lynis -y

Running a System Audit with Lynis

sudo lynis audit system

This comprehensive scan checks:

• File system security
• Available security updates
• Firewall configurations
• Malware scanners
• User accounts and authentication
• System hardening levels

Reviewing Lynis Results

After the scan completes, Lynis provides:

• Suggestions for improving security
• Warnings about potential security issues
• A hardening index score

Review the detailed report:

sudo cat /var/log/lynis.log

For security warnings only:

sudo cat /var/log/lynis.log | grep Warning

Step 6: Creating a Comprehensive Scanning Schedule

Regular scanning is crucial for maintaining system security. Let's set up automated scans using cron jobs.

Setting Up a Daily Malware Scan

Create a script for daily scanning:

sudo nano /usr/local/bin/daily-security-scan.sh

Add the following content:

#!/bin/bash
# Daily security scan script

SCAN_DATE=$(date +%Y-%m-%d)
LOG_DIR="/var/log/security-scans"
CLAM_LOG="$LOG_DIR/clamav-$SCAN_DATE.log"
RKHUNTER_LOG="$LOG_DIR/rkhunter-$SCAN_DATE.log"
EMAIL="root@localhost"

# Create log directory if it doesn't exist
mkdir -p $LOG_DIR

# Update virus definitions
freshclam --quiet

# Run ClamAV scan on important directories
echo "Starting ClamAV scan at $(date)" > $CLAM_LOG
clamscan -r --infected --detect-pua=yes /home /var/www /tmp /var/tmp /etc >> $CLAM_LOG 2>&1
echo "Completed ClamAV scan at $(date)" >> $CLAM_LOG

# Update and run rkhunter
echo "Starting rkhunter scan at $(date)" > $RKHUNTER_LOG
rkhunter --update --quiet
rkhunter --check --skip-keypress --quiet >> $RKHUNTER_LOG 2>&1
echo "Completed rkhunter scan at $(date)" >> $RKHUNTER_LOG

# Check for infected files or warnings
INFECTED=$(grep -c "Infected files: [1-9]" $CLAM_LOG)
WARNINGS=$(grep -c -i warning $RKHUNTER_LOG)

# Send email if issues found
if [ $INFECTED -gt 0 ] || [ $WARNINGS -gt 0 ]; then
    echo "Security issues detected on $HOSTNAME" | mail -s "SECURITY ALERT: Issues Found on $HOSTNAME" $EMAIL -A $CLAM_LOG -A $RKHUNTER_LOG
fi

# Rotate logs older than 30 days
find $LOG_DIR -type f -name "*.log" -mtime +30 -delete

Make the script executable:

sudo chmod +x /usr/local/bin/daily-security-scan.sh

Setting Up a Weekly Full System Scan

Create a script for weekly comprehensive scanning:

sudo nano /usr/local/bin/weekly-security-scan.sh

Add the following content:

#!/bin/bash
# Weekly security scan script

SCAN_DATE=$(date +%Y-%m-%d)
LOG_DIR="/var/log/security-scans"
FULL_LOG="$LOG_DIR/full-scan-$SCAN_DATE.log"
CHKROOTKIT_LOG="$LOG_DIR/chkrootkit-$SCAN_DATE.log"
LYNIS_LOG="$LOG_DIR/lynis-$SCAN_DATE.log"
EMAIL="root@localhost"

# Create log directory if it doesn't exist
mkdir -p $LOG_DIR

echo "Starting weekly security scan at $(date)" > $FULL_LOG

# Run chkrootkit
echo "Running chkrootkit..." >> $FULL_LOG
chkrootkit > $CHKROOTKIT_LOG 2>&1
grep -i "infected\|suspicious" $CHKROOTKIT_LOG >> $FULL_LOG

# Run Lynis audit
echo "Running Lynis system audit..." >> $FULL_LOG
lynis audit system --quiet > $LYNIS_LOG 2>&1
grep -i "warning\|suggestion" /var/log/lynis.log | tail -n 50 >> $FULL_LOG

# Run full ClamAV scan
echo "Running full ClamAV scan (this will take a long time)..." >> $FULL_LOG
clamscan -r --infected --detect-pua=yes / --exclude-dir=/sys --exclude-dir=/proc --exclude-dir=/dev >> $FULL_LOG 2>&1

echo "Completed weekly security scan at $(date)" >> $FULL_LOG

# Send email with results
cat $FULL_LOG | mail -s "Weekly Security Scan Results: $HOSTNAME" $EMAIL

# Update rkhunter file properties after scan to reduce false positives
rkhunter --propupd

Make the script executable:

sudo chmod +x /usr/local/bin/weekly-security-scan.sh

Adding to Cron

Add the scripts to cron for automatic execution:

sudo crontab -e

Add these lines:

# Run daily security scan at 3:30 AM
30 3 * * * /usr/local/bin/daily-security-scan.sh

# Run weekly security scan at 2:00 AM on Sundays
0 2 * * 0 /usr/local/bin/weekly-security-scan.sh

Step 7: Real-time File Integrity Monitoring

For continuous protection, consider setting up AIDE (Advanced Intrusion Detection Environment) for file integrity monitoring.

Installing AIDE

sudo apt install aide -y

Initializing the AIDE Database

Initialize AIDE's database of file checksums:

sudo aideinit

This process may take some time as AIDE creates checksums for all monitored files.

Daily Integrity Checks

Create a script for daily integrity checks:

sudo nano /usr/local/bin/aide-check.sh

Add the following content:

#!/bin/bash
# AIDE integrity check script

SCAN_DATE=$(date +%Y-%m-%d)
LOG_FILE="/var/log/aide/aide-check-$SCAN_DATE.log"
EMAIL="root@localhost"

# Create log directory if it doesn't exist
mkdir -p /var/log/aide

# Run AIDE check
echo "Starting AIDE integrity check at $(date)" > $LOG_FILE
aide --check >> $LOG_FILE 2>&1
echo "Completed AIDE check at $(date)" >> $LOG_FILE

# Check for changes
CHANGES=$(grep -c "changed" $LOG_FILE)

# Send email if changes detected
if [ $CHANGES -gt 0 ]; then
    echo "File integrity changes detected on $HOSTNAME" | mail -s "AIDE Alert: File Changes on $HOSTNAME" $EMAIL -A $LOG_FILE
fi

Make the script executable and add to cron:

sudo chmod +x /usr/local/bin/aide-check.sh
sudo crontab -e

Add this line:

# Run AIDE integrity check at 4:00 AM daily
0 4 * * * /usr/local/bin/aide-check.sh

Troubleshooting Section

Common Issues and Solutions

1. ClamAV Resource Usage

Problem: ClamAV scans consume too much CPU or memory.

Solution: Adjust scanning parameters and scheduling:

# Use the --max-filesize and --max-scansize options
sudo clamscan -r --max-filesize=100M --max-scansize=500M /path/to/scan

Also consider using ionice and nice to reduce impact:

nice -n 19 ionice -c 3 sudo clamscan -r /path/to/scan

2. False Positives in rkhunter/chkrootkit

Problem: Security scanners report warnings on legitimate files.

Solution: For rkhunter, update the file properties database after system updates:

sudo rkhunter --propupd

For persistent false positives, consider whitelisting in rkhunter.conf:

sudo nano /etc/rkhunter.conf

Add the appropriate ALLOWHIDDENDIR, ALLOWHIDDENFILE, or ALLOWDEVFILE entry.

3. AIDE Database Management

Problem: After system updates, AIDE reports numerous legitimate changes.

Solution: Update the AIDE database after planned system changes:

sudo aide --update
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

4. Email Notifications Not Working

Problem: Security scripts aren't sending email alerts.

Solution: Install and configure a mail transport agent:

sudo apt install postfix mailutils

During postfix installation, select "Local only" for a basic setup.

Best Practices & Optimization Tips

Security Enhancement

• Scan isolation: For critical servers, run intensive scans in maintenance windows
• Layered approach: Use multiple scanning tools to catch different types of threats
• Monitor logs: Centralize security logs in a SIEM system for analysis
• Secure scanner updates: Verify integrity of scanner updates and databases
• Physical security: Remember that malware scanners can't protect against physical access

Performance Optimization

• Exclude virtual filesystems: Always exclude /proc, /sys, /dev from scans
• Target high-risk directories: Focus regular scans on /tmp, /var/tmp, /home, and /var/www
• Schedule during low-usage periods: Run intensive scans when system activity is minimal
• Configure resource limits: Use cgroups or nice to limit scanner resource usage
• Incremental scanning: Implement progressive scanning strategies for large systems

# Example: Limit ClamAV memory usage
echo "MaxFileSize 100M" | sudo tee -a /etc/clamav/clamd.conf
echo "MaxScanSize 500M" | sudo tee -a /etc/clamav/clamd.conf
sudo systemctl restart clamav-daemon

Responding to Security Incidents

If your scans detect genuine security issues, follow these steps:

1. Isolate the System

If possible, disconnect the compromised system from the network to prevent further damage or data exfiltration.

2. Preserve Evidence

Before making changes, capture the current state for analysis:

sudo dd if=/dev/sda of=/path/to/external/disk/disk_image.dd bs=4M
ps aux > running_processes.txt
netstat -tulanp > network_connections.txt
lsof -i > open_files.txt

3. Analyze the Threat

Use the scanner logs to identify affected files and potential attack vectors. For suspicious files, consider uploading them to VirusTotal (for non-sensitive data only).

4. Clean or Rebuild

Depending on the severity:

• For isolated malware: Remove infected files and check for persistence mechanisms
• For rootkits or serious compromises: Complete system rebuild is often safest

5. Report the Incident

Document the incident and, if applicable, report it to relevant authorities or security teams.

Conclusion

Regularly scanning your Linux systems for malware and rootkits is an essential practice for maintaining security. By implementing the tools and procedures outlined in this guide—ClamAV, Rootkit Hunter, chkrootkit, Lynis, and AIDE—you create a robust, multi-layered approach to detecting and preventing security threats.

Remember that scanning is just one component of a comprehensive security strategy. Combine these tools with other security practices such as:

• Keeping systems and applications updated
• Implementing proper user access controls
• Configuring firewalls and intrusion detection
• Regular security audits and penetration testing
• Staff security awareness training

With vigilant monitoring and proper security protocols, you can significantly reduce the risk of successful attacks against your Linux systems.