Website downtime

Uh oh. Woke up on Sunday morning with an email from ServiceUptime.com to inform me my website was down.

Immediate worry was I forgot to renew the VPS at RamNode but it no, it was still pingable and I was able to log in via SSH, it was just that none of the sites were responding.

To the Apache error logs!

tail -n 200 /var/log/apache/error.log

Oh, hmm:

[Sun Apr 02 01:51:03 2017] [notice] Graceful restart requested, doing restart

Interesting:

(2)No such file or directory: apache2: could not open error log file /var/www//logs/error.log.
Unable to open logs

Ah of course. User error! I renamed a virtual host's document root but never updated it in the config.

But why did Apache decide to restart at 2AM on Sunday morning?

The answer was logrotate:

$ cat /etc/logrotate.d/apache2
/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                /etc/init.d/apache2 reload > /dev/null
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

Big thanks to DO for that one.

Problem solved! My googlebot visitor can be happy again.

Incidentally I've probably been using the free version of ServiceUptime.com for 8 years now and frankly since moving away from Heart over to a RamNode VPS I forgot I had it, there has been so little downtime. Highly recommend it for simple uptime checking.