2026-08-08 Hung threads in apache mod php zend
This website has been getting hit at >5qps intermittently for a while now likely due to all those AI driven bots scraping the internet. Today, I noticed that my website had stopped responding, and went on a debugging journey.
I first noticed that my access log contained a lot of lines like:
113.x.x.x - - [08/Aug/2026:17:37:01 +1000] "-" 408 - "-" "-"
408 is "Request Timeout" client error; the server failed to read a request from the connection before giving up. That's odd, especially given I was seeing thousands of these consecutively.
Since I'm expected to use AI heavily at $DAYJOB, I gave Gemini a shot. First, how about debug logging? Apart from actual debug logging, the recommendation was to check the status output from mod_status - which I didn't have configured. After adding that, I saw that I had exhausted concurrent requests, and was guided through tuning my event mode server. I then noticed that many of the httpd processes got stuck in "accepting no" state, without changing in any counters. After telling Gemini that I was running NetBSD, not Linux (I guess that's a common assumption), which means kqueue, not epoll, etc, suggested running gdb against one of these wedged processes, and check all thread backtraces.
Thread 1 (LWP 10837 of process 19497 ""):
#0 0x00007f7ecc80abba in ___lwp_park60 () from /usr/libexec/ld.elf_so
#1 0x00007f7ecc805d45 in _rtld_exclusive_enter () from /usr/libexec/ld.elf_so
#2 0x00007f7ecc809480 in _rtld_tls_get_addr () from /usr/libexec/ld.elf_so
#3 0x0000736b21cdd9dd in tsrm_is_shutdown () from /usr/pkg/lib/httpd/mod_php83.so
#4 0x0000736b21dd579b in zend_signal_handler_defer () from /usr/pkg/lib/httpd/mod_php83.so
#5 <signal handler called>
#6 0x00007f7ecc807ff3 in _rtld_symlook_obj () from /usr/libexec/ld.elf_so
#7 0x00007f7ecc8083ea in _rtld_symlook_list () from /usr/libexec/ld.elf_so
#8 0x00007f7ecc80889f in _rtld_symlook_default () from /usr/libexec/ld.elf_so
#9 0x00007f7ecc808d4a in _rtld_find_plt_symdef () from /usr/libexec/ld.elf_so
#10 0x00007f7ecc800bc0 in _rtld_bind () from /usr/libexec/ld.elf_so
#11 0x00007f7ecc80082d in _rtld_bind_start () from /usr/libexec/ld.elf_so
#12 0x000000007fffff21 in ?? ()
#13 0x0000000000000000 in ?? ()
This was the "ah-ha" moment, and while I knew what it meant, Gemini also gave solutions:
- Stop using mod_php, and migrate to PHP-FPM.
- Disable zend signal handling, via zend.signal_handling = Off.
- Force immediate symbol binding via export LD_BIND_NOW=1.
I opted for disabling zend signal handling, since I haven't had any other issues running mod_php multi-threaded to date, and this seems to have worked perfectly.
