2020-11-17 virtual memory limits

From Wikistix
Jump to navigation Jump to search

Just spent too much time trying to figure out why my mediawiki thumbnail generation using ImageMagick convert(1) was failing. After enabling mediawiki debugging via:

$wgDebugLogFile = "/tmp/debug-{$wgDBname}.log";
$wgShowExceptionDetails = true;

The error was singularly uninformative, with an apparently different, random library appearing on each attempt:

[thumbnail] thumbnail failed on www.stix.id.au: error 1 "/usr/lib/libgssapi.so.11: Shared object "libkrb5.so.27" not found" from "'/usr/pkg/bin/convert' '-quality' '95' …

The clue was uncovered by ktrace -dip <pid>, on the parent apache process. This uncovered the real error:

   581      1 convert  CALL  close(3)                                                                                                  
   581      1 convert  RET   close 0                                                                                                   
   581      1 convert  CALL  open(0x7f7fff9ed3f8,0,0x63)                                                                               
   581      1 convert  NAMI  "/lib/libcrypt.so.1"                                                                                      
   581      1 convert  RET   open 3                                                                                                    
   581      1 convert  CALL  __fstat50(3,0x7f7fff9ed2f8)                                                                               
   581      1 convert  RET   __fstat50 0                                                                                               
   581      1 convert  CALL  mmap(0,0x1000,PROT_READ,0x1<SHARED,FILE,ALIGN=NONE>,3,0,0)                                                
   581      1 convert  RET   mmap 135607257300992/0x7b5586668000                                                                       
   581      1 convert  CALL  munmap(0x7b5586668000,0x1000)                                                                             
   581      1 convert  RET   munmap 0                                                                                                  
   581      1 convert  CALL  mmap(0,0x20a000,PROT_READ|PROT_EXEC,0x15000002<PRIVATE,FILE,ALIGN=2MB>,3,0,0)                             
   581      1 convert  RET   mmap -1 errno 12 Cannot allocate memory                                                                   
   581      1 convert  CALL  close(3)                                                                                                  
   581      1 convert  RET   close 0

So it turns out that mediawiki limits the virtual memory of sub-processes by default, using a simple wrapper around ulimit (or cgroup magic where supported). Sure enough, on my NetBSD system, the default 300MiB is insufficient.

ksh$ ulimit -v $((300*1024))
ksh$ ulimit -a
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         1048576
stack(kbytes)        4096
lockedmem(kbytes)    5401636
memory(kbytes)       16204908
nofiles(descriptors) 1024
processes            512
threads              1024
vmemory(kbytes)      307200
sbsize(bytes)        unlimited
ksh$ /usr/pkg/bin/convert -?
/usr/lib/libgssapi.so.11: Shared object "libroken.so.20" not found

Bumping the default in LocalSettings.php fixed the issue for me.

# $wgMaxShellMemory = 307200;
$wgMaxShellMemory = 524288;

Misinformation found herein copyright Paul Ripke (aka “stix”) stixpjr@gmail.com.