当时同事说他在批量导入数据，然后又同事说网站怎么打开没有反应，我就去看了网站是否能打开，按照平时也不说打不开，也没有用户暴增的情况，怎么回事！
然后就是自己打开的那个页面确实是等了很久才响应过来，然后就只能去查询日志咯
第一次故障
- 显示查看了nginx日志，也没有说出现什么明显的错误
- 然后就查询PHP的日志，也就是php-fpm.log，发现确实是有问题
当时的日志大概和这个类似吧：
[07-Aug-2019 10:12:06] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 32 idle, and 67 total children
[07-Aug-2019 10:12:07] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 40 idle, and 75 total children
然后发现是php-fpm的配置确实是配置创建进程数有些少，就把php-fpm.conf的内容进行了调整：
pm = dynamic
pm.max_children = 80
pm.start_servers = 60
pm.min_spare_servers = 50
pm.max_spare_servers = 80
pm.max_requests = 2048
pm.process_idle_timeout = 10s
request_terminate_timeout = 120
request_slowlog_timeout = 0
然后 service php-fpm restart 重启php-fpm 就可以了。
第二次故障
我仍然是先看nging的access.log 和 error.log ,发现请求太多，但也看出了写问题，最后截图发个工作群，发现可能java接口处理数据有问题，接着找，就看了php-fpm.log的日志
出现了下面的一幕：
[07-Aug-2019 10:12:08] WARNING: [pool www] server reached pm.max_children setting (80), consider raising it
[07-Aug-2019 14:09:21] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 96 idle, and 106 total children
[07-Aug-2019 14:09:22] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 93 idle, and 110 total children
[07-Aug-2019 14:09:23] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 99 idle, and 117 total children
[07-Aug-2019 14:09:28] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 97 idle, and 122 total children
[07-Aug-2019 14:09:56] WARNING: [pool www] server reached pm.max_children setting (135), consider raising it
我清楚的记得我在上一次出现问题的时候就把这个配置在php-fpm.conf中改大了，我在想服务器的内存挺大，我就经过计算再改大一点，然后重启PHP-fpm。
这一次的修改让我失望了，修改完了之后一直提示我要改大这个配置，我觉得不对劲啊，根据现有的客户数，这个请求应该是不会这样的。后面java的同学发现确实他的代码中有个地方有问题，修改后提交，情况稍微好了点。
后面运维说数据挂掉过一次，我们很吃惊，怎么会这样，之后运维开启了mysql的慢查询，发现有个select的语句是慢查询，我看了下，就一张表，90多万的数据，然后没有创建索引，只有增加，每次有一个请求过来都要进程一次全表扫描的查询，当请求的量大的时候，请求就阻塞在这里。
最后给这个表加上索引，然后写定时器去删除里面的数据。
网友评论