From GoBlueMich Wiki
Jump to navigation Jump to search

Picard Facepalm

Why Was IP Blocked

Script from Mark's Wiki:

Email Applications Screen Shots

Email App Screen Shots:


Exim Commands

Exim Cheat Sheet:

List queue size:

 exim -bpcList queue messages:
 exim -bp

Show message with headers:

 exim -Mvh $message_idShow message with body:
 exim -Mvb $message_id

Show message logs:

 exim -Mvl $message_id

Clear Mail Directory

Clear 'cur'

 find /home/$USER/mail/cur -type f -exec rm -f {} \;

Clear 'new'

 find /home/$USER/mail/cur -type f -exec rm -f {} \;


Spamfu PDF:

Determine SMTP vs Script spam. SMTP spam will always have a login ID in the -Mvl entry:
Fixed_login, Courier_login, or Dovecot_login

If SMTP spam -
“Lock” the e-mail account right away.

    • Make sure the e-mail account that is spamming is not the customers primary e-mail contact in billing

Set a random password via cPanel. Don't bother e-mailing it to the customer, just tell them to set a new one themselves.
Set WHM - Password Strength
Clear the queue

If Script spam -
Exim logs (or -Mvl) will show U=(username) P=local (instead of P=esmtpa).
Username can be 'nobody' (DSO) or a local user (SuPHP).
Get the CWD for one message

 grep -B3 $message_id /var/log/exim_mainlog |grep -i cwd 


 grep -B3 $message_id /var/log/exim_mainlog |grep -i home 

Consult the domlogs (grep POST) if the cwd doesn't contain obvious malicious scripts.
Simply stat/chmod 000 the offending script.
Clean the mail queue
If the spam script was a malicious file, search for and disable other malicious files, noting their file `stat` prior to disabling them.
Investigate the origin of the files (FTP upload, cPanel upload, vulnerable web app)

If Darkmailer:

 lsof -i :25

PHP Spam

Only works with PHP 5.3 or higher

This has been used on live spamming servers and using the mail.log does pinpoint the correct file nicely.

Please note the variables below are not enabled by default in php.ini.

If you suspect there is a PHP script sending out email (and it is still doing so) try adding these two lines:

mail.add_x_header = On
mail.log = /var/log/php_maillog

to the [mail function] section of:


Also make sure to create the log file manually otherwise you may get permissions errors and it won't work:

touch /var/log/php_maillog
chmod 666 /var/log/php_maillog

Or if you prefer the easy oneliner method you can use:

if [[ "$(php -v | grep -oP 'PHP 5.[^12]')" != '' ]]; then if [[ -z $(egrep '(^mail.add_x_header|^mail.log)' /usr/local/lib/php.ini) ]]; then cp -a /usr/local/lib/php.ini{,.pre_php_mail_log_addition}; perl -n -i -e 'print; print "mail.add_x_header = On\nmail.log = /var/log/php_maillog\n" if /(\[mail function\])/' /usr/local/lib/php.ini; echo -e "\nVariables Added to [mail function]:\n"; touch /var/log/php_maillog; chmod 666 /var/log/php_maillog; /etc/init.d/httpd restart 2>/dev/null; egrep --color=never '(^mail.add_x_header = On|^mail.log = /var/log/php_maillog)' /usr/local/lib/php.ini; echo -e "\nLog File Created:";ls -l /var/log | grep php_maillog | awk '{print $1"  /var/log/"$9}'; else echo -e "\nNothing Done.\nCheck /usr/local/lib/php.ini:\n"; egrep -n --color=never '(mail.add_x_header|mail.log)' /usr/local/lib/php.ini; fi; else echo -e "\nNothing Done.\nThis only works with 5.3 or higher"; fi

This will check if PHP 5.3 or higher installed and if so it checks if the mail.add_x_header or mail.log variables exists. if they do it stops and outputs what it found and the line numbers it found them on. If it does not see them (or they are commented out) it will make a backup of /usr/local/lib/php.ini, add the variables, create the log file in /var/log, chmod it, restart Apache and output what it did.

Example of failure:

Nothing Done.                                                                                                                                                                                                                                
Check /usr/local/lib/php.ini:                                                                                                                                                                                                                
601:mail.add_x_header = On                                                                                                                                                                                                                   
602:mail.log = /var/log/php_maillog

Example of success:

Variables Added to [mail function]:                                                                                                                                                                                                          
mail.add_x_header = On                                                                                                                                                                                                                       
mail.log = /var/log/php_maillog
Log File Created:
-rw-rw-rw-  /var/log/php_maillog

The first variable adds:

This checks if PHP 5.3 or higher is installed and if it is it will then check if either of the above variables are present. If they are


to the exim email header (the header variable should only show up when PHP does the sending). So, for example if you had a a PHP script sending from bad_script.php the header would look something like:

X-PHP-Originating-Script: 500:bad_script.php

You can actually search the queue for this header (doesn't show up in the regular exim_mainlog) by using:

exiqgrep 'X-PHP-Originating-Script'

This should give you a list of emails that have been sent using PHP.

Or, if you know the script name, you could also use:

exiqgrep 'bad_script.php'

This should give you a list of emails that have that have that script name in it.

And if you're REALLY sure that only spam emails are listed you could use:

exiqgrep -i 'bad_script.php' | xargs exim -Mrm

which filters out all the emails with bad_script.php in it, then only displays the message ids and then delete them.

The above header stuff is really useful, but when combined with the mail.log variable can be useful. This causes PHP to record whenever:


is used. A simple output would be similar to:

mail() on [/home/user/public_html/bad_script.php:11]: To: -- Headers: From:  Reply-To:  Content-type: text/html; charset=iso-8859-1

If you were to compare the cwd output of Spamfu to this log you could get a pretty good idea of where the spam is coming from.

Spamfu script

wget -O /scripts/ 
chmod +x /scripts/

A script designed to help you find the source of spam quickly. It currently can parse spam from the email queue as well as the exim logs.

It works very well, however it is a work in progress so please email any bugs or feature requests to If you find a server that has a large amount of spam coming from it and the script does not pick up on it, be sure to include the name of the server in your email.


There is a spam deletion script at - just download it and run it. It may already be on the server in /root or /scripts.

cd /scripts
rm -f nukespam*
chmod 700 nukespam

Manually Clear Mail Queue

Manually stop exim:

 touch /etc/eximdisable
 service exim stop

When finished:

 rm /etc/eximdisable
 service exim restart

Clear the entire exim queue:

find /var/spool/exim/input/ -type f -name '*-H' | sed -e 's#/var/spool/exim/input/[A-Z0-9a-z]/\([A-Z0-9a-z\-]\{16\}\)\-[HDJ].*#\1#' |xargs exim -Mrm

or for a more CLI friendlier string (if the queue isn't over 50000 or so emails):

exiqgrep -i | xargs exim -Mrm

Remove specific email from an account, -r for recipient and -f for sender. Include the -i flag to use the message ID.

exiqgrep -ir | xargs exim -Mrm


exiqgrep -if | xargs exim -Mrm

Thaw frozen messages from the queue:

exiqgrep -i | xargs exim -Mt

Remove frozen messages from the queue:

exiqgrep -z -i | xargs exim -Mrm

Forcing exim to attempt to deliver all messages:

exim -q -v

Remove message in queue by auth'd sender:

 cd /var/spool/exim/input/ 
 find . -type f | xargs grep -m 1 -s -i "auth_id $USER@DOMAIN.TLD" | sed -e 's/\.\/[A-Z0-9a-z]\/\([A-Z0-9a-z\-]\{16\}\)\-[HDJ].*/\1/' | xargs exim -Mrm