如何解决Apache每半夜自行重启导致apscheduler任务未完成
嗨,我有一个使用 Apache 服务的 Pyramid wsgi webapp。 webapp 有一个每小时作业,必须在第 0 分钟运行以获取时间敏感数据并写入我的 MysqL 数据库。我注意到有时(并非所有时间)数据可能不会在任务的午夜 00:00:00 运行时写入数据库。查看日志,Apache 似乎在每个午夜后不久重新启动,这可能会导致问题。
通过 stackoverflow 搜索后,似乎 logrotate 可能是重新启动的罪魁祸首。但是,我也注意到 logrotate 是由 crontab 调用的,它默认为上午 6:25,所以我不知道为什么重新启动发生在午夜。 (我的 Ubuntu 服务器没有安装 anacron)
这是过去几天来自 Apache 的日志文件
[Tue May 11 00:00:35.534821 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Tue May 11 00:00:35.534867 2021] [core:notice] [pid 72273:tid 140034084613184] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Wed May 12 00:00:00.029412 2021] [wsgi:error] [pid 72660:tid 140033624434432] 2021-05-12 00:00:00,029 INFO [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'],next run at: 2021-05-12 01:00:00 HKT)" (scheduled at 2021-05-12 00:00:00+08:00)
[Wed May 12 00:00:00.621944 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00493: SIGUSR1 received. Doing graceful restart
[Wed May 12 00:00:03.614647 2021] [wsgi:error] [pid 72660:tid 140033624434432] 2021-05-12 00:00:03,614 INFO [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'],next run at: 2021-05-12 01:00:00 HKT)" executed successfully
从上面的日志中有趣地注意到,我的apscheduler似乎仍在运行(数据库写入成功)并在Doing graceful restart
之后和创建新日志文件之前打印到日志中(内容如下所示)
[Wed May 12 00:00:03.641095 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Wed May 12 00:00:03.641146 2021] [core:notice] [pid 72273:tid 140034084613184] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Thu May 13 00:00:00.032261 2021] [wsgi:error] [pid 95013:tid 140083656877824] 2021-05-13 00:00:00,032 INFO [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'],next run at: 2021-05-13 01:00:00 HKT)" (scheduled at 2021-05-13 00:00:00+08:00)
[Thu May 13 00:00:03.764471 2021] [wsgi:error] [pid 95013:tid 140083656877824] 2021-05-13 00:00:03,764 INFO [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'],next run at: 2021-05-13 01:00:00 HKT)" executed successfully
[Thu May 13 00:00:34.829438 2021] [mpm_event:notice] [pid 95012:tid 140084121332800] AH00493: SIGUSR1 received. Doing graceful restart
在上面的日志文件中,我的 apscheduler 作业在重新启动之前完成,因此我的数据库也被正确写入。
[Thu May 13 00:00:35.588354 2021] [mpm_event:notice] [pid 95012:tid 140084121332800] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Thu May 13 00:00:35.588433 2021] [core:notice] [pid 95012:tid 140084121332800] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Fri May 14 00:00:00.020559 2021] [wsgi:error] [pid 2120:tid 140241617286912] 2021-05-14 00:00:00,020 INFO [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'],next run at: 2021-05-14 01:00:00 HKT)" (scheduled at 2021-05-14 00:00:00+08:00)
[Fri May 14 00:00:00.558072 2021] [mpm_event:notice] [pid 2119:tid 140242151496768] AH00493: SIGUSR1 received. Doing graceful restart
刚刚过去的午夜,作业没有完成,数据库也没有写入。由于作业在完成之前突然终止,因此在午夜之前和之后的日志中也没有写入任何伴随的 INFO [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'],next run at: xxxxxxx)" executed successfully
行
[Fri May 14 00:00:03.588691 2021] [mpm_event:notice] [pid 2119:tid 140242151496768] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Fri May 14 00:00:03.588744 2021] [core:notice] [pid 2119:tid 140242151496768] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
day hasn't ended yet
这是我的 crontab 文件,我认为它是标准文件,并声明每日作业应该在早上 6:25 而非午夜运行。
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job deFinition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
我的 Ubuntu 服务器又没有安装 anacron
ubuntu@xxx:~$ anacron --help
Command 'anacron' not found,but can be installed with:
sudo apt install anacron
logrotate 和 apache 有 cron.daily 任务
ubuntu@xxx:~$ ls -ln /etc/cron.daily/
total 40
-rwxr-xr-x 1 0 0 539 Apr 14 2020 apache2
-rwxr-xr-x 1 0 0 376 Dec 5 2019 apport
-rwxr-xr-x 1 0 0 1478 Apr 9 2020 apt-compat
-rwxr-xr-x 1 0 0 355 Dec 29 2017 bsdmainutils
-rwxr-xr-x 1 0 0 1187 Sep 6 2019 dpkg
-rwxr-xr-x 1 0 0 377 Jan 21 2019 logrotate
-rwxr-xr-x 1 0 0 1123 Feb 26 2020 man-db
-rwxr-xr-x 1 0 0 4574 Jul 18 2019 popularity-contest
-rwxr-xr-x 1 0 0 214 Dec 7 23:35 update-notifier-common
vi /etc/cron.daily/logrotate
#!/bin/sh
# skip in favour of systemd timer
if [ -d /run/systemd/system ]; then
exit 0
fi
# this cronjob persists removals (but not purges)
if [ ! -x /usr/sbin/logrotate ]; then
exit 0
fi
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE
vi /etc/cron.daily/apache2
#!/bin/sh
# run htcacheclean if set to 'cron' mode
set -e
set -u
type htcacheclean > /dev/null 2>&1 || exit 0
[ -e /etc/default/apache-htcacheclean ] || exit 0
# edit /etc/default/apache-htcacheclean to change this
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
HTCACHECLEAN_OPTIONS=""
. /etc/default/apache-htcacheclean
[ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0
htcacheclean ${HTCACHECLEAN_OPTIONS} \
-p${HTCACHECLEAN_PATH} \
-l${HTCACHECLEAN_SIZE}
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the adm group by default,since this is the owning group
# of /var/log/syslog.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.
apache2 包含在 logrotate.d 下
ubuntu@xxx:~$ ls -ln /etc/logrotate.d
total 52
-rw-r--r-- 1 0 0 120 Sep 6 2019 alternatives
-rw-r--r-- 1 0 0 442 Apr 14 2020 apache2
-rw-r--r-- 1 0 0 126 Dec 5 2019 apport
-rw-r--r-- 1 0 0 173 Apr 9 2020 apt
-rw-r--r-- 1 0 0 91 Nov 2 2020 bootlog
-rw-r--r-- 1 0 0 130 Jan 21 2019 btmp
-rw-r--r-- 1 0 0 112 Sep 6 2019 dpkg
-rw-r--r-- 1 0 0 845 Nov 7 2019 MysqL-server
-rw-r--r-- 1 0 0 501 Mar 7 2019 rsyslog
-rw-r--r-- 1 0 0 119 Mar 31 2020 ubuntu-advantage-tools
-rw-r--r-- 1 0 0 178 Jan 22 2020 ufw
-rw-r--r-- 1 0 0 235 Jul 21 2020 unattended-upgrades
-rw-r--r-- 1 0 0 145 Feb 19 2018 wtmp
vi /etc/logrotate.d/apache2
/var/log/apache2/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if invoke-rc.d apache2 status > /dev/null 2>&1; then \
invoke-rc.d apache2 reload > /dev/null 2>&1; \
fi;
endscript
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi; \
endscript
}
我只想知道为什么我的 Apache 在午夜重新启动,而根据 crontab 应该在早上 6:25 并更改时间以避免与我的第 0 分钟每小时工作发生冲突。谢谢!
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。