|
|
Yes I did reboot. The attached file I sent has the slabcache up to when
the OOM Kill happened I will paste the last few results from that...
--PASTE--
Thu Feb 3 12:26:26 EST 2005
slabinfo - version: 1.1 (SMP)
kmem_cache 112 112 244 7 7 1 : 1008 252
nfs_write_data 0 0 384 0 0 1 : 496 124
nfs_read_data 0 0 384 0 0 1 : 496 124
nfs_page 0 0 128 0 0 1 : 1008 252
ocfs_fileentry 96 96 512 12 12 1 : 496 124
ocfs_lockres 180 180 256 12 12 1 : 1008 252
ocfs_ofile 430 430 384 43 43 1 : 496 124
ocfs_oin 180 180 256 12 12 1 : 1008 252
ip_fib_hash 16 448 32 4 4 1 : 1008 252
ext3_xattr 0 0 44 0 0 1 : 1008 252
journal_head 280 1078 48 14 14 1 : 1008 252
revoke_table 6 750 12 3 3 1 : 1008 252
revoke_record 0 0 32 0 0 1 : 1008 252
clip_arp_cache 0 0 256 0 0 1 : 1008 252
ip_mrt_cache 0 0 128 0 0 1 : 1008 252
tcp_tw_bucket 12 150 128 5 5 1 : 1008 252
tcp_bind_bucket 49 1120 32 10 10 1 : 1008 252
tcp_open_request 0 0 128 0 0 1 : 1008 252
inet_peer_cache 4 174 64 3 3 1 : 1008 252
secpath_cache 0 0 128 0 0 1 : 1008 252
xfrm_dst_cache 0 0 256 0 0 1 : 1008 252
ip_dst_cache 60 225 256 15 15 1 : 1008 252
arp_cache 7 45 256 3 3 1 : 1008 252
flow_cache 0 0 128 0 0 1 : 1008 252
blkdev_requests 15360 15450 128 515 515 1 : 1008 252
kioctx 0 0 128 0 0 1 : 1008 252
kiocb 0 0 128 0 0 1 : 1008 252
dnotify_cache 0 0 20 0 0 1 : 1008 252
file_lock_cache 66 400 96 10 10 1 : 1008 252
async_poll_table 0 0 140 0 0 1 : 1008 252
fasync_cache 0 0 16 0 0 1 : 1008 252
uid_cache 9 560 32 5 5 1 : 1008 252
skbuff_head_cache 949 7015 168 305 305 1 : 1008 252
sock 2172 2255 1408 451 451 2 : 240 60
sigqueue 232 232 132 8 8 1 : 1008 252
kiobuf 270 720 128 24 24 1 : 1008 252
cdev_cache 18 406 64 7 7 1 : 1008 252
bdev_cache 20 348 64 6 6 1 : 1008 252
mnt_cache 26 348 64 6 6 1 : 1008 252
inode_cache 71214 77595 512 11085 11085 1 : 496 124
dentry_cache 4214 5940 128 198 198 1 : 1008 252
dquot 0 0 128 0 0 1 : 1008 252
filp 5760 5970 128 199 199 1 : 1008 252
names_cache 23 23 4096 23 23 1 : 240 60
buffer_head 4935182 4936190 108 141009 141034 1 : 1008 252
mm_struct 285 390 384 39 39 1 : 496 124
vm_area_struct 10765 14000 68 250 250 1 : 1008 252
fs_cache 653 1102 64 19 19 1 : 1008 252
files_cache 296 378 512 54 54 1 : 496 124
signal_cache 452 1102 64 19 19 1 : 1008 252
sighand_cache 235 355 1408 71 71 2 : 240 60
pte_chain 60829 72300 128 2410 2410 1 : 1008 252
pae_pgd 647 1044 64 18 18 1 : 1008 252
size-131072(DMA) 0 0 131072 0 0 32 : 0 0
size-131072 0 0 131072 0 0 32 : 0 0
size-65536(DMA) 0 0 65536 0 0 16 : 0 0
size-65536 0 0 65536 0 0 16 : 0 0
size-32768(DMA) 0 0 32768 0 0 8 : 0 0
size-32768 13 13 32768 13 13 8 : 0 0
size-16384(DMA) 0 0 16384 0 0 4 : 0 0
size-16384 22 22 16384 22 22 4 : 0 0
size-8192(DMA) 0 0 8192 0 0 2 : 0 0
size-8192 44 44 8192 44 44 2 : 0 0
size-4096(DMA) 0 0 4096 0 0 1 : 240 60
size-4096 1079 1079 4096 1079 1079 1 : 240 60
size-2048(DMA) 0 0 2048 0 0 1 : 240 60
size-2048 262 262 2048 131 131 1 : 240 60
size-1024(DMA) 0 0 1024 0 0 1 : 496 124
size-1024 508 628 1024 157 157 1 : 496 124
size-512(DMA) 0 0 512 0 0 1 : 496 124
size-512 741 744 512 93 93 1 : 496 124
size-256(DMA) 0 0 256 0 0 1 : 1008 252
size-256 2220 2220 256 148 148 1 : 1008 252
size-128(DMA) 0 0 128 0 0 1 : 1008 252
size-128 3839 6510 128 217 217 1 : 1008 252
size-64(DMA) 0 0 128 0 0 1 : 1008 252
size-64 724 870 128 29 29 1 : 1008 252
size-32(DMA) 0 0 64 0 0 1 : 1008 252
size-32 3509 4640 64 80 80 1 : 1008 252
--END PASTE--
That was the last entry before the system stopped responding... If this
still does not have enough I can re-run the process again and capture...
--
-Ryan
Allvac Unix Services
Monroe, NC
(704) 282-1586
-----Original Message-----
From: taroon-list-bounces@xxxxxxxxxx
[mailto:taroon-list-bounces@xxxxxxxxxx] On Behalf Of Larry Woodman
Sent: Thursday, February 03, 2005 2:00 PM
To: Discussion of Red Hat Enterprise Linux 3 (Taroon)
Subject: Re: OCFS and Out Of Memory issue
Frank, Ryan wrote:
>Here is a paste of the current /proc/slabinfo
>
You must have rebooted your system, there is only about 7000 slabcache
pages now. Please get the /proc/slabinfo output right at the OOM kill.
Larry
>
>--BEGIN PASTE--
>slabinfo - version: 1.1 (SMP)
>kmem_cache 144 144 244 9 9 1 : 1008 252
>nfs_write_data 30 30 384 3 3 1 : 496 124
>nfs_read_data 10 10 384 1 1 1 : 496 124
>nfs_page 60 60 128 2 2 1 : 1008 252
>ocfs_fileentry 72 72 512 9 9 1 : 496 124
>ocfs_lockres 30 30 256 2 2 1 : 1008 252
>ocfs_ofile 80 80 384 8 8 1 : 496 124
>ocfs_oin 45 45 256 3 3 1 : 1008 252
>ip_fib_hash 560 560 32 5 5 1 : 1008 252
>ext3_xattr 0 0 44 0 0 1 : 1008 252
>journal_head 1309 1309 48 17 17 1 : 1008 252
>revoke_table 750 750 12 3 3 1 : 1008 252
>revoke_record 448 448 32 4 4 1 : 1008 252
>clip_arp_cache 0 0 256 0 0 1 : 1008 252
>ip_mrt_cache 0 0 128 0 0 1 : 1008 252
>tcp_tw_bucket 120 120 128 4 4 1 : 1008 252
>tcp_bind_bucket 896 896 32 8 8 1 : 1008 252
>tcp_open_request 60 60 128 2 2 1 : 1008 252
>inet_peer_cache 116 116 64 2 2 1 : 1008 252
>secpath_cache 0 0 128 0 0 1 : 1008 252
>xfrm_dst_cache 0 0 256 0 0 1 : 1008 252
>ip_dst_cache 360 360 256 24 24 1 : 1008 252
>arp_cache 45 45 256 3 3 1 : 1008 252
>flow_cache 0 0 128 0 0 1 : 1008 252
>blkdev_requests 15420 15420 128 514 514 1 : 1008 252
>kioctx 0 0 128 0 0 1 : 1008 252
>kiocb 0 0 128 0 0 1 : 1008 252
>dnotify_cache 0 0 20 0 0 1 : 1008 252
>file_lock_cache 320 320 96 8 8 1 : 1008 252
>async_poll_table 0 0 140 0 0 1 : 1008 252
>fasync_cache 0 0 16 0 0 1 : 1008 252
>uid_cache 784 784 32 7 7 1 : 1008 252
>skbuff_head_cache 6004 6256 168 272 272 1 : 1008 252
>sock 375 375 1408 75 75 2 : 240 60
>sigqueue 290 290 132 10 10 1 : 1008 252
>kiobuf 240 240 128 8 8 1 : 1008 252
>cdev_cache 2900 2900 64 50 50 1 : 1008 252
>bdev_cache 464 464 64 8 8 1 : 1008 252
>mnt_cache 290 290 64 5 5 1 : 1008 252
>inode_cache 23289 23289 512 3327 3327 1 : 496 124
>dentry_cache 25350 25350 128 845 845 1 : 1008 252
>dquot 0 0 128 0 0 1 : 1008 252
>filp 870 870 128 29 29 1 : 1008 252
>names_cache 28 28 4096 28 28 1 : 240 60
>buffer_head 11200 11200 108 320 320 1 : 1008 252
>mm_struct 290 290 384 29 29 1 : 496 124
>vm_area_struct 5264 5264 68 94 94 1 : 1008 252
>fs_cache 522 522 64 9 9 1 : 1008 252
>files_cache 315 315 512 45 45 1 : 496 124
>signal_cache 696 696 64 12 12 1 : 1008 252
>sighand_cache 455 455 1408 91 91 2 : 240 60
>pte_chain 6366 6870 128 229 229 1 : 1008 252
>pae_pgd 580 580 64 10 10 1 : 1008 252
>size-131072(DMA) 0 0 131072 0 0 32 : 0 0
>size-131072 0 0 131072 0 0 32 : 0 0
>size-65536(DMA) 0 0 65536 0 0 16 : 0 0
>size-65536 0 0 65536 0 0 16 : 0 0
>size-32768(DMA) 0 0 32768 0 0 8 : 0 0
>size-32768 11 12 32768 11 12 8 : 0 0
>size-16384(DMA) 0 0 16384 0 0 4 : 0 0
>size-16384 22 23 16384 22 23 4 : 0 0
>size-8192(DMA) 0 0 8192 0 0 2 : 0 0
>size-8192 21 23 8192 21 23 2 : 0 0
>size-4096(DMA) 0 0 4096 0 0 1 : 240 60
>size-4096 2297 2477 4096 2297 2477 1 : 240 60
>size-2048(DMA) 0 0 2048 0 0 1 : 240 60
>size-2048 720 780 2048 360 390 1 : 240 60
>size-1024(DMA) 0 0 1024 0 0 1 : 496 124
>size-1024 1056 1056 1024 264 264 1 : 496 124
>size-512(DMA) 0 0 512 0 0 1 : 496 124
>size-512 1288 1288 512 161 161 1 : 496 124
>size-256(DMA) 0 0 256 0 0 1 : 1008 252
>size-256 1785 3045 256 152 203 1 : 1008 252
>size-128(DMA) 0 0 128 0 0 1 : 1008 252
>size-128 4590 4590 128 153 153 1 : 1008 252
>size-64(DMA) 0 0 128 0 0 1 : 1008 252
>size-64 570 570 128 19 19 1 : 1008 252
>size-32(DMA) 0 0 64 0 0 1 : 1008 252
>size-32 2802 3306 64 57 57 1 : 1008 252
>--END PASTE--
>
>Also attached is the slabinfo results while running the `dd` (requested
>by RedHat)
>
>--
>-Ryan (RHCE, SCSA)
>Allvac Unix Services
>Monroe, NC
>(704) 282-1586
>
>-----Original Message-----
>From: taroon-list-bounces@xxxxxxxxxx
>[mailto:taroon-list-bounces@xxxxxxxxxx] On Behalf Of Larry Woodman
>Sent: Thursday, February 03, 2005 1:35 PM
>To: Discussion of Red Hat Enterprise Linux 3 (Taroon)
>Subject: Re: OCFS and Out Of Memory issue
>
>Frank, Ryan wrote:
>
>
>
>>Hello all!
>>
>>I am using AS 3.0 update 3 (Kernel 2.4.21-20.Elsmp) on a pair of
>>Compaq DL-580's with 12gb of RAM. We are running Oracle 10g release 3,
>>
>>
>
>
>
>>in a RAC configuration. I am using SAN storage (Hitachi 9585) via
>>Emulex LP9002 HBA's. Cluster file system is provided by OCFS 1.0.13.
>>
>>The problem is like this... When I try to create a large file on an
>>
>>
>OCFS
>
>
>>volume I get the following error from the system when the cached
>>memory approaches 4GB.
>>
>>
>>
>
>The problem here is the slab cache has consumed all of the Normal
>memory
>zone:
>
> >>>Feb 3 11:53:54 alvmnrlnx004 kernel: aa:0 ac:340 id:2 il:74 ic:0
>fr:617 >>>Feb 3 11:53:54 alvmnrlnx004 kernel: 161776 pages of
>slabcache
>
>Please send along a "cat /proc/slabinfo" output.
>
>Larry Woodman
>
>
>--
>Taroon-list mailing list
>Taroon-list@xxxxxxxxxx
>http://www.redhat.com/mailman/listinfo/taroon-list
>
>
>
>-----------------------------------------------------------------------
>-
>
>--
>Taroon-list mailing list
>Taroon-list@xxxxxxxxxx
>http://www.redhat.com/mailman/listinfo/taroon-list
>
--
Taroon-list mailing list
Taroon-list@xxxxxxxxxx
http://www.redhat.com/mailman/listinfo/taroon-list
--
Taroon-list mailing list
Taroon-list@xxxxxxxxxx
http://www.redhat.com/mailman/listinfo/taroon-list
|
|