Monday, April 23, 2012

Awstats Rebuilding files from old logs

SkyHi @ Monday, April 23, 2012

This is using AWStats 6.3 on Windows Server 2003 and IIS 6.0
My log files will not be processed for 2010. The stats are all fine up to 31/12/2009.
The last file in my DirData folder is 'awstats122009.[config].txt' – there is no 'awstats012010.[config].txt' or later.
If I change my LogFile entry in the conf file to LogFile="C:/Inetpub/vhosts/[domain]/statistics/logs/W3SVC[nnnnn]/ex100101.log" and run "perl awstats.pl -config=[domain] -update", the expected awstats012010.[config].txt is not generated and I get:
Update for config "./awstats.[domain].conf"
With data in log file "C:/Inetpub/vhosts/[domain]/statistics/logs/W3SVC[nnnnn]
/ex100101.log"…
Phase 1 : First bypass old records, searching new record…
Direct access to last remembered record has fallen on another record.
So searching new records from beginning of log file…
Jumped lines in file: 0
Parsed lines in file: 35616
 Found 6 dropped records,
 Found 18 corrupted records,
 Found 35592 old records,
 Found 0 new qualified records.
Attempts at subsequent log files result in the same problem; AWStats thinks these are all old records even though the file for that month does not exist. Any pointers to how to resolve this would be greatly appreciated.



Solution:

Temporarily move all files contained in DirData to another directory and try again. This will allow you to generate the AWStats data files for January 2010. When this is done, move the 2009 data files back to DirData.

OR
backup and comment the "highlighted part" /var/lib/awstats/domain.txt


# LastLine    = Date of last record processed - Last record line number in last log - Last record offset in la
st log - Last record signature value
# FirstTime   = Date of first visit for history file
# LastTime    = Date of last visit for history file
# LastUpdate  = Date of last update - Nb of parsed records - Nb of parsed old records - Nb of parsed new records - Nb of parsed corrupted - Nb of parsed dropped
# TotalVisits = Number of visits
# TotalUnique = Number of unique visitors
# MonthHostsKnown   = Number of hosts known
# MonthHostsUnKnown = Number of hosts unknown
BEGIN_GENERAL 8
LastLine 20120101000113 257865 25669630 36390796923
FirstTime 20111201000008
LastTime 20111231235838
LastUpdate 20120101040228 26376 0 26375 0 0
TotalVisits 16978
TotalUnique 7482
MonthHostsKnown 0
MonthHostsUnknown 11032
END_GENERAL




 So you have a bunch of old logs and you want to put it all together into AWStats to put the historical data in.  Firstly you have to remove any existing AWStats monthly data files from the data directory as AWStats won't enter log information from the past.

Then you need to use a script like the one below to feed the logfiles back into AWStats and have it regenerate the monthly data files:
for %i in (D:\sites\www.fridaynightgaming.com\logs\W3SVC2008\*.log) do perl awstats.pl -config=fridaynightgaming -logfile=%i -update
Obviously change out the example logfile path above for your own.  This is the only way I know of how to import historical log data into AWstats.




REFERENCES
http://www.internetofficer.com/forum/awstats-iis-installation-and-configuration/problem-generating-old-files/
http://www.nakedmcse.com/Home/tabid/39/forumid/1/postid/9/scope/posts/Default.aspx