AWStats

From GoBlueMich Wiki
Jump to navigation Jump to search

Things to check for Initially

Before you do anything with awstats configurations check the cpanel stats_log to make sure they didn't actually process or there aren't any specific errors.

/usr/local/cpanel/logs/stats_log

Next make sure the CustomLog entries exists in the Apache config. (this should exists on per vhosts basis)

CustomLog path/to/domlog

If not most likely you will need to run EasyApache so cPanel rebuilts the apache config and puts them back in place.

Another reason it may have problems is if the domlog path has been changed. SSublett has seen some randomness when the domlog path has been changed. See cPanel ticket refernence below.

https://tickets.cpanel.net/review/msg.cgi?ticketid=6003329

EasyApache 4

Our dedicated kick currently includes a file at /etc/apache2/conf.d/logformat.conf - requested by support - that modifies the LogFormat outside of cPanel. This modified LogFormat causes Apache to write the domlogs in the incorrect format - and then Awstats, Analog, and Webalizer may fail to parse the domlogs - excluding those domlog entries.

If the file /etc/apache2/conf.d/logformat.conf contains the following, simply remove it and restart Apache.

<IfModule log_config_module>
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent
    # NOTE: "combined" and "common" are required by WHM
    LogFormat "%h %l %u %t \"%r\" %T %>s %b" common
    LogFormat "%h %l %u %t \"%r\" %T %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

    # access_log format can be set in WHM under 'Basic cPanel & WHM Setup'
    CustomLog logs/access_log combined
</IfModule>

If the customer would like to modify the log format in a format similar to this, instead modify /var/cpanel/conf/apache/local to include the relevant section from /var/cpanel/conf/apache/main. After making these changes, make sure you monitor logs processing for a few days to watch for a drop in logs.

Allow Updating Through cPanel

Sometimes, it may be easier to just enable your cPanel user accounts to update manually.

1. Login to WHM, then select Server Configuration -> Tweak Settings. 
2. Scroll down to Stats and Logs.
3. Check the box "Allow users to update Awstats from cPanel"

If The Link Doesn't Appear

The link for "Update Now" in AWStats will be located at the top, next to "Last Update". However, it may not show immediately. You may need to manually run the stats using /scripts/runweblogs $user You can also verify that the setting is actually enabled, by checking the AWStats Configuration File for a particular user.

1. Login via SSH as root
2. cd /home/username/tmp/awstats
3. grep AllowToUpdateStatsFromBrowser awstats.example.com.conf
4. It should be set to AllowToUpdateStatsFromBrowser=1
5. If not, edit the file and save.
6. restart cpanel : service cpanel restart

Error: Log is currently being processed in the background. Cannot update.

If the customer sees the above error when trying to manually update awstats from cpanel, log in via SSH, and verify that cpanellogd isn't actually processing logs. If cpanellogd is sleeping for logs, then search /usr/local/apache/domlogs/ for a file called domain.com.bkup, domain.com being the domain in question that is not updating manually. If the file exists DO NOT DELETE THIS FILE. This file is a queued up domlog file left behind when cpanel stats failed to finish recording this log info. Deleting this will prevent awstats from updating for that segment of time. Instead run the following command, replace $user with the cpanel user in question:

/scripts/runweblogs $user

If that fails to run, try forcing a stats run for the entire server. Ask the customer about this first, since stats can be a rather server intensive process. You can force a stats run by running the following:

/scripts/runlogsnow

Awstats should now update properly.

AWstats reporting higher bandwidth usage than other bandwidth tools

Log processing programs like webalizer and AWstats measure bandwidth by showing the size of the data that was requested, not the amount the data that was actually transferred.

What is most likely happening is that there are a lot of large requests that are then cancelled. Apache will log the request but does not log\track what actually transfered before the cancellation.

AWStats don't exist

Check to see if the perl script has proper permissions. /usr/local/cpanel/3rdparty/bin/awstats.pl should be 755.

AWStats not updating

Sometimes AWStats decides to stop generating and a customer will call or open a HD ticket asking why. Here is a quick fix to get it up and running again:

Login to shell and change to the users awstats directory:

cd /home/username/tmp/awstats

Now locate a copy of awstats.pl copy that to the users awstats directory and set permissions:

locate awstats.pl
cp /usr/local/cpanel/base/awstats.pl .
chown usrname:usrname awstats.pl

Now run the following command to update awstats for that domain:

perl awstats.pl -config=domain.com -update

Check awstats in cpanel now.

You should be able to check cPanel now and show some sort of an update for the current day.


NOTE: If this does not work, the following is the less-than-quick fix to get it all running again.

Make certain that /home/user/tmp/awstats/awstats.domain.com.conf exists. If it does not:

1. cd /home/user/tmp/awstats #(if you're not there already)
2. touch /home/user/tmp/awstats/awstats.domain.com.conf
3. chown user.user awstats.domain.com.conf
4. From the list generated from the above 'locate awstats.pl', above, find an existing awstats.otherdomain.com.conf (or run 'locate awstats' )
5. cp /home/otheruser/tmp/awstats/awstats.otherdomain.com.conf > awstats.domain.com.conf
6. chown user.user awstats.domain.com.conf
7. Edit your newly-populated awstats.domain.com.conf and replace the previous username and domain name appropriately. There should be one instance of username:
DirData="/home/user/tmp/awstats"

And three instances of domain name:
SiteDomain="domain.com"
HostAliases="domain.com www.domain.com localhost 127.0.0.1"
LogFile="/usr/local/apache/domlogs/domain.com" #Make certain this is set to the correct path! May also be "/home/domlogs/domain.com"

8. perl awstats.pl - config=domain.com - update

If this still does not update awstats, make sure there are domlogs for the domain. They may be in either /usr/local/apache/domlogs/ OR in /home/domlogs/ . If the domlogs do not exist, they will need to be created:

1. touch /usr/local/apache/domlogs/domain.com
2. touch /usr/local/apache/domlogs/domain.com-bytes_log
3. chown root.user /usr/local/apache/domlogs/domain.com

OR

1. touch /home/domlogs/domain.com
2. touch /home/domlogs/domain.com-bytes_log
3. chown root.user /home/domlogs/domain.com

Make sure the httpd.conf knows to write to the domlogs. There should be these two lines in the httpd.conf for that vhost:

   CustomLog /usr/local/apache/domlogs/domain.com combined
   CustomLog /usr/local/apache/domlogs/domain.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
OR
CustomLog /home/domlogs/domain.com combined CustomLog /home/domlogs/domain.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."

If these lines are not present in the httpd.conf for the vhost, do the following:

1. mkdir /usr/local/apache/conf/userdata/std/2/username
2. mkdir /usr/local/apache/conf/userdata/std/2/username/domain.com
3. touch /usr/local/apache/conf/userdata/std/2/username/domain.com/domlogs.conf

edit /usr/local/apache/conf/userdata/std/2/username/domain.com/domlogs.conf and add either of the CustomLog pairs from above, as appropriate.

4. /usr/local/cpanel/bin/apache_conf_distiller --update
5. /usr/local/cpanel/bin/build_apache_conf
6. /etc/init.d/httpd restart

For testing and/or luls, go to domain.com (mainly so there's info in the domlogs to work with)
Then:
cd /home/username/tmp/awstats
perl awstats.pl - config=domain.com - update
Double-check that awstats is working in cpanel.

If awstats is still not working, please make certain that the domain has been registered AND has DNS pointing to the server.

AWStats is installed by default and enabled in tweak settings and/or statistic software management, this script is for viewing the stats outside of cPanel



Create a file anywhere in the public_html tree called awstats.php.
You will need to modify the following:

$user = 'username';//your cpanel username
$pass = 'password';//your cpanel password
$domain = 'mydomain.com';//do not include 'http://' or 'www.'

awstats.php

<?php
/*
dv at josheli.com

Proxy for viewing Awstats outside of cpanel. I assume no liability.


1 out of 3 people ask me if it's "safe" to have their username and password
in this file. Here's my answer:

When you signed up with your web hosting provider, they probably provided
you with an email with your login/password, right? Do you ever use FTP
with your site? Do you login to your mail server, to hotmail, to yahoo, to
anywhere else? When you log in to cpanel or WHM, do you do it through SSL
or not? Have you installed any other web software like osCommerce or phpBB
or any other script?

In all cases, your user/password is either sent through dozens of
computers in plain text and is sitting in someone else's harddrive or
database, or is stored in plain text on some file on your webserver. You
are never safe.

So, if someone wants to steal ANY user/password, it's pretty easy. In
fact, probably half a dozen people could look at any password of yours
right now. But to answer what i think you're specifically asking about about
this script, no, not just anyone can find out the user/pass.

And besides that, there are other precautions you could take. Ask around.
*/

$user = 'username';//your cpanel username
$pass = 'password';//your cpanel password
$domain = 'mydomain.com';//do not include 'http://' or 'www.'

/*
Domain of the stats you wish to view, e.g. a subdomain like "cvs.mydomain.com".
If left blank, defaults to the "domain" above
Another option is to set the "config" parameter in the url of your browser, e.g.:
http://www.domain.com/awstats.php?config=sub.domain.com
*/
$config_domain = ;

/*
If you don't know what you're doing, set $dynamic_images equal
to TRUE, and don't worry about the $image_directory variable.
Otherwise,
    - Normally, this script will load images by proxy, i.e. awstats.php
      is called for each  tag and will send the correct
      image to the browser. This is not the way the web is designed
      to work. So, if you wish to improve performance and lower
      bandwidth, you can:
      1. Set $dynamic_images to FALSE
      2. Create an image directory in your webroot
      3. Copy all of awstats image sub-directories to this new directory
      4. Point the $image_directory variable to your new directory
    You will get all the benefits of cached, static images.
    In order to get the Awstats images and their directories, you will
    probably need to download an awstats distribution from
    awstats.sourceforge.net. The final layout will probably look like this:

      awstats_imagedir/
                    browser/
                    clock/
                    cpu/
                    flags/
                    mime/********
                    os/
                    other/

    Under each of those sub-directories will be dozens of .png files.
*/

$dynamic_images = false;
$image_directory = './awstats_images/';

//lame attempt to combat referrer spam
$spam_words = array('mortgage', 'sex', 'porn', 'cock', 'slut', 'facial', 'loving', 'gay', '.ro');


/***********
NO NEED TO TOUCH ANYTHING BELOW HERE
************/

//retrieves the file, either .pl or .png
function get_file($fileQuery)
{
  global $user, $pass, $domain;
  return file_get_contents("http://$user:$pass@$domain:2082/".$fileQuery);
}

$requesting_image = (strpos($_SERVER['QUERY_STRING'],'.png')===false)?false:true;

if($requesting_image) //it's a .png file...
{
  if(!$dynamic_images && !is_dir($image_directory))
  {
    exit;
  }
  $fileQuery = $_SERVER['QUERY_STRING'];
}
elseif(empty($_SERVER['QUERY_STRING']))//probably first time to access page...
{
    if(empty($config_domain))
    {
        $config_domain = $domain;
    }
  $fileQuery = "awstats.pl?config=$config_domain";
}
else //otherwise, all other accesses
{
  $fileQuery = 'awstats.pl?'.$_SERVER['QUERY_STRING'];
}

$file = get_file($fileQuery);

//check again to see if it was a .png file
//if it's not, replace the links
if(!$requesting_image)
{
  $file = str_replace('awstats.pl', basename($_SERVER['PHP_SELF']), $file);

  if($dynamic_images)
  {
    $imgsrc_search = '="/images';
    $imgsrc_replace = '="'.basename($_SERVER['PHP_SELF']).'?images';
  }
  else
  {
    $imgsrc_search = 'src="/images/awstats/';
    $imgsrc_replace = 'src="'.$image_directory;
  }

  $file = str_replace($imgsrc_search, $imgsrc_replace, $file);
  $file = str_replace($spam_words, 'SPAM', $file);
}
else //if it is a png, output appropriate header
{
  header("Content-type: image/png");
}

//output the file
echo $file;
?>


IF AUTH IS Failing The above "usually" works.

I just had an instance where it took the following to get it working:

-changed port 2082 -> 2083
-changed http -> https
-the last step was cpanel was resolving to hostname, therefor had to change domain to host name and specify a $config_domain

This was all determined by viewing the awstats through cpanel and reproducing the URL


ERROR FIXES

AWStats will only read domlogs that exist. If there are only domlogs for the month of January, it will only display statistics for the month of January.

If you come across this error:

Error: Couldn't open config file "awstats.solrobots.com.conf" nor "awstats.conf" after searching in path ".,/home/t1000/tmp/awstats/,/etc/opt/awstats,/etc/awstats,/etc,/usr/local/etc/awstats": No such file or directory
run /scripts/runweblogs 

If that^ doesn't work, the following will as tested on cpanel 11.44 cent6.5:

mkdir /usr/local/etc/awstats
cp /home/$USER/tmp/awstats/awstats.$DOMAIN.TLD.conf /usr/local/etc/awstats/
/usr/local/cpanel/3rdparty/bin/awstats.pl -config=$DOMAIN.TLD

This does a Create/Update database for the particular domain.


If Awstats stops:

Check the size of the domlogs. When the logs reach 2GB it stops working because it can't write to them anymore.

If you get this error:

Error: Not same number of records of BrowsersSearchIDOrder (164 entries) and BrowsersHashIDLib (166 entries without 
msie,netscape,) in Browsers database. May be you updated AWStats without updating browsers.pm file or you made 
changed into browsers.pm not correctly. Check your file /web/joeri/www/cgi-bin/lib/browsers.pm is up to date.
Setup ('/etc/awstats/awstats.www.joeri.net.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory). 

Then it is likely that 'firefox' and 'svn' need to be added to the BrowsersSearchIDOrder section of the browsers.pm file. (These were the offending browsers missing in the file on two shared servers at this point.)

Converting awstats to PDF files

Customers on an infrequent basis will request that awstats output be dumped for them into PDF format. There's a little legwork involved, but is ultimately fairly simple to accomplish.

To begin with, we must enable the EPEL repo so we can install some dependent software:

https://wiki.int.liquidweb.com/articles/Epel

Now we can proceed with the installation and first run. Pay attention to the bold wording, as that involves input from you, the ever-vigilant hero:

yum update
yum install -y htmldoc
su - <cpanel_user>
cd tmp/
wget http://prdownloads.sourceforge.net/awstats/awstats-7.3.tar.gz
tar -xzf awstats-7.3.tar.gz
mv awstats-7.3/tools/* awstats/
/usr/bin/perl ~/tmp/awstats/awstats_buildstaticpages.pl -update -config=<domain_name>  -dir=/home/<user>/public_html -awstatsprog=/usr/local/cpanel/3rdparty/bin/awstats.pl -buildpdf

Should see a bunch of output, followed by:

Main HTML page is 'awstats.<domain_name>.html'.
PDF file is 'awstats.<domain_name>.pdf'.

The above will run this for the first time, generating a bunch of HTML pages in /home/<user>/public_html, in addition to a PDF. Of course, if the customer doesn't want their public_html cluttered, it might behoove you to create a "stats" directory to output these files to.

AWStats with GeoIP Country/City on cPanel

If a customer wants GeoIP for country and city on a cPanel box, this isn't too hard to do. Assuming that GeoIP hasn't been installed on the server yet, heres how you do it.

Starting with the C libraries that the Perl module will need:

cd /usr/local/src
wget http://www.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
tar zxf GeoIP.tar.gz
cd GeoIP*
./configure
make && make install

Now the Perl module for GeoIP. More than likely, a cpan install will fail, so:

wget http://search.cpan.org/CPAN/authors/id/B/BO/BORISZ/Geo-IP-1.40.tar.gz
tar zxf Geo-IP-1.40.tar.gz
cd Geo-IP-1.40
perl Makefile.PL LIBS="-L/usr/local/lib -lGeoIP" INC=-I/usr/local/include

Now we'll need the dats:

cd /usr/local/share/GeoIP
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

And make cPanel play load the modules and play nice:

cp /usr/local/cpanel/src/3rdparty/gpl/awstats-7.0/wwwroot/cgi-bin/plugins/geoip_city_maxmind.pm /usr/local/cpanel/3rdparty/bin/plugins/
sed -i.bak 's/LoadPlugin\=\"geoipfree\"/LoadPlugin=\"geoip GEOIP_STANDARD \/usr\/local\/share\/GeoIP\/GeoIP\.dat\"\nLoadPlugin=\"geoip_city_maxmind GEOIP_STANDARD \/usr\/local\/share\/GeoIP\/GeoLiteCity\.dat\"/' /usr/local/cpanel/etc/awstats.conf

You should be done after that. This will update each user's conf file when stats are next run to include this information. You can force this run by:

/scripts/runstatsonce

or check it out on a single user:

/scripts/runweblogs $user

This should be pretty copy/paste-able, but if you encounter any issues, feel free to let cbyerly@lw know.


Please see the Help Center articles for updated information: https://help.liquidweb.com/s/article/Viewing-Website-Statistics-in-cPanel and https://help.liquidweb.com/s/article/Troubleshooting-AWStats