Cleaning up SAP resources

After an abnormal shutdown, it may be necessary to clean up the leftover resources that won't go away with stopsap. These are processes, System V Shared Memory segments and System V Semaphores. Examples below are from a system running AIX. Other UNIX systems may be subtly different.

Processes

The fastest way to clean up all processes is to become the sidadm user and issue kill -1 -1, which will send SIGHUP to all processes owned by sidadm. SIGHUP can be trapped by processes, but does give them a chance to shutdown more gracefully. Be aware that if the system has paged any processes out into swap space, they may take some time to exit. Additionally, it is not unusual for the shell initiating the kill to also be killed. Any remaining processes can be individually killed with kill -9 <pid>.

# su - sidadm
sidadm$ ps ux | head -6
USER        PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
sidadm   516350  0.7  3.0 455268 457300      - A      May 17 569:58 dw.sapSID_D10 pf
sidadm   508154  0.4  1.0 102812 104568      - A      May 17 362:39 dw.sapSID_D10 pf
sidadm   483462  0.4  1.0 91172 93060      - A      May 17 296:33 dw.sapSID_D10 pf
sidadm   512252  0.3  3.0 390192 392192      - A      May 17 275:30 dw.sapSID_D10 pf
sidadm   471264  0.3  1.0 83328 85096      - A      May 17 258:40 dw.sapSID_D10 pf
sidadm$ kill -1 -1
Hangup
# su - sidadm
sidadm$ ps ux
USER        PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
sidadm   651328  0.0  0.0 27964 27360      - A      May 17  0:04 [disp+wor]
sidadm   917540  0.0  0.0  804  844  pts/1 A    18:57:50  0:00 ksh
sidadm   938072  0.0  0.0  456  472  pts/1 A    18:58:13  0:00 ps ux
sidadm$ kill -9 651328
sidadm$ ps ux
USER        PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
sidadm   917540  0.0  0.0  804  844  pts/1 A    18:57:50  0:00 ksh
sidadm   938078  0.0  0.0  456  472  pts/1 A    18:58:30  0:00 ps ux
sidadm$

System V Shared Memory

SAP is a heavy user of shared memory, and these must be cleaned up before SAP will successfully restart. A one line script can be used to delete the segments easily.

First check that NATTCH (number of attached processes) is zero for all the users segments, since segments will only be deleted when NATTCH is zero, otherwise they will be marked for deletion. Then delete the segments. If NATTCH is not zero, then there are still processes hanging around. Doing all this as the sidadm user is safer, it is less likely to impact anything else running on the system.

sidadm$ ipcs -ma | egrep '^T|sidadm' | head -6
T        ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NATTCH     SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
m         6 0x0382be8e --rw-rw-rw-   sidadm   sapsys   sidadm   sapsys      0      4096 434218 651328 18:36:47 18:59:14 23:18:27
m    524295 0xffffffff --rw-------   sidadm   sapsys   sidadm   sapsys      0 268435456 675916 860326 13:02:46 13:02:46 11:10:50
m    524296 0xffffffff --rw-------   sidadm   sapsys   sidadm   sapsys      0 268435456 802954 815248 16:29:28 16:29:28 11:10:50
m    524297 0xffffffff --rw-------   sidadm   sapsys   sidadm   sapsys      0 268435456 905336 815248 16:06:53 16:06:53 11:11:13
m    524298 0xffffffff --rw-------   sidadm   sapsys   sidadm   sapsys      0 268435456 401624 462986 16:17:04 16:17:04 11:11:13
sidadm$ ipcs -m | awk '/^m.*sidadm/{print $2}' | xargs -n 1 ipcrm -m
sidadm$ ipcs -ma | egrep '^T|sidadm'
T        ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NATTCH     SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
sidadm$

There is also a SAP command cleanipc which is designed to do this correctly. As a UNIX Sysadmin, I have no experience with this command. It's usage appears to be:

sidadm$ /usr/sap/SID/SYS/exe/run/cleanipc <instance number> remove

System V Semaphores

Depending on the type of UNIX, these may be less critical. Since AIX does not enforce any easily reachable limit on System V objects, SAP will simply allocate more semaphores when restarted. Other UNIX systems may find that system-wide limits (SEMMNI, SEMMNS, SEMMSL, etc) are reached if these are not deleted.

sidadm$ ipcs -s | egrep '^T|sidadm' | head -6
T        ID     KEY        MODE       OWNER    GROUP
s    131074 0x0000520a --ra-ra-ra-   sidadm   sapsys
s    131075 0x00005209 --ra-ra-ra-   sidadm   sapsys
s    131076 0x00005208 --ra-ra-ra-   sidadm   sapsys
s    131077 0x002f741b --ra-r-----   sidadm   sapsys
s    131078 0x002f741c --ra-r-----   sidadm   sapsys
sidadm$ ipcs -s | awk '/^s.*sidadm/{print $2}' | xargs -n 1 ipcrm -s
sidadm$ ipcs -s | egrep '^T|sidadm'
T        ID     KEY        MODE       OWNER    GROUP
sidadm$