|
|
0.4 seconds of CPU on average to do a dmu_object_alloc is a wee sluggish!
I suspect, however, it's contention on osi->os_obj_lock as we don't seem
to be looping in dmu_object_alloc. I'm still confused though, I believe
that locking an adaptive mutex will spin for a short period then context
switch and so they shouldn't be burning CPU - at least not .4s worth!
EricK suggested running "lockstat -kgIW -D 60 sleep 60" which I think
is the next step.
Looking back at your original email, I see you had observed the
same contention on os_obj_lock. I don't enough much about the dmu
to know if anything has been fixed in that area.
Neil.
Ben Rockwood wrote On 12/07/06 13:53,:
Neil Perrin wrote:
Ben,
The attached dscript might help determining the zfs_create issue.
It prints:
- a count of all functions called from zfs_create
- average wall count time of the 30 highest functions
- average cpu time of the 30 highest functions
Note, please ignore warnings of the following type:
dtrace: 1346 dynamic variable drops with non-empty dirty list
The results are below. I keep bumping into dmu_object_alloc, but I'm
uncertain what is or isn't normal in that reguard.
benr.
[private:/tmp] root# dtrace -s zfs_create.d
dtrace: script 'zfs_create.d' matched 2524 probes
dtrace: 508 dynamic variable drops with non-empty dirty list
^C
CPU ID FUNCTION:NAME
1 2 :END
counts
arc_read 1
arc_set_callback 1
buf_hash_find 1
dbuf_read_done 1
dmu_object_next 2
dmu_zfetch_dofetch 2
dmu_zfetch_fetch 2
dmu_zfetch_fetchsz 2
dnode_next_offset 2
dnode_next_offset_level 2
dbuf_hash_insert 3
dbuf_set_data 3
dmu_zfetch 3
dmu_zfetch_find 3
arc_buf_alloc 6
zfs_v4_to_unix 6
dbuf_prefetch 10
mzap_addent 11
mze_find 11
mze_find_unused_cd 11
mze_insert 11
dbuf_findbp 13
dmu_tx_check_ioerr 13
dnode_block_freed 13
spa_get_dsl 13
buf_hash_remove 36
arc_change_state 37
arc_release 40
dbuf_fix_old_data 40
fzap_add 41
fzap_add_cd 41
zap_entry_create 41
zap_increment_num_entries 41
zap_put_leaf_maybe_grow_ptrtbl 41
arc_tempreserve_clear 52
dmu_tx_commit 52
dsl_dir_tempreserve_clear 52
txg_rele_to_sync 52
zap_add 52
zfs_dirent_unlock 52
zfs_link_create 52
zfs_log_create 52
zil_commit 52
zfs_mknode 55
zfs_acl_chmod 56
zfs_acl_fixup_canonical_six 56
zfs_aclset_common 56
zfs_mode_compute 56
zfs_perm_init 56
zfs_znode_dmu_init 56
dbuf_create_bonus 57
dmu_bonus_hold 57
dmu_object_alloc 57
zfs_acl_append 57
zfs_znode_alloc 57
dmu_tx_add_new_object 58
dnode_allocate 58
dmu_buf_set_user 59
dmu_buf_update_user 59
dbuf_create 60
dbuf_read_impl 60
dmu_zfetch_init 61
dnode_create 61
dnode_new_blkid 61
dnode_setdblksz 61
dnode_clear_range 62
zio_buf_alloc 67
zap_table_load 68
dsl_dataset_dirty 69
dbuf_fix_old_bonus_data 71
fzap_lookup 74
zap_lookup 74
zap_leaf_array_create 82
dmu_tx_hold_zap 85
arc_tempreserve_space 86
dmu_tx_assign 86
dmu_tx_count_write 86
dmu_tx_get_txg 86
dmu_tx_try_assign 86
dsl_dir_tempreserve_impl 86
dsl_dir_tempreserve_space 86
dsl_pool_adjustedsize 86
spa_get_dspace 86
txg_hold_open 86
txg_rele_to_quiesce 86
dmu_tx_create 95
dmu_tx_create_dd 95
zfs_acl_node_read 95
zfs_acl_node_read_internal 95
zfs_dirent_lock 95
zfs_zaccess 95
zfs_zaccess_common 95
zfs_time_stamper_locked 109
dbuf_hold_level 112
bp_get_dasize 115
dbuf_new_block 115
dmu_tx_is_syncing 115
fzap_checksize 115
zap_deref_leaf 115
zap_get_leaf_byblk 115
zap_idx_to_blk 115
zap_leaf_lookup 115
zap_put_leaf 115
dnode_cons 122
zap_hash 126
zap_lockdir 126
zap_unlockdir 126
dnode_add_ref 130
zio_next_stage_async 133
zio_pop_transform 133
zio_ready 133
zio_wait 133
zio_wait_children_ready 133
spa_config_enter 134
spa_config_exit 134
zio_clear_transform_stack 134
zio_create 134
zio_done 134
zio_null 134
zio_push_transform 134
zio_root 134
zio_wait_children_done 134
zfs_acl_free 151
zfs_acl_alloc 152
zap_leaf_chunk_alloc 165
mze_compare 168
arc_buf_remove_ref 170
remove_reference 170
arc_released 173
zfs_ace_access 184
dmu_tx_hold_bonus 187
arc_buf_add_ref 206
add_reference 207
arc_access_and_exit 207
dmu_tx_willuse_space 229
dnode_willuse_space 229
dsl_dir_willuse_space 229
dnode_setdirty 233
dbuf_add_ref 256
zio_wait_for_children 267
dmu_tx_count_dnode 276
spa_get_asize 287
dsl_dataset_block_freeable 296
dmu_buf_hold 309
dmu_tx_hold_object_impl 333
zfs_set_ace 336
vdev_lookup_top 345
zio_next_stage 401
dbuf_will_dirty 424
buf_hash 453
dsl_dataset_prev_snap_txg 477
dnode_rele 580
dbuf_dirty 588
dnode_hold 604
dnode_hold_impl 664
dsl_dir_dirty 817
txg_list_add 886
dmu_buf_get_user 907
dbuf_hold 975
dbuf_whichblock 978
dbuf_rele 1099
dbuf_hold_impl 1103
dbuf_find 1113
dbuf_hash 1116
parent_delta 1119
dsl_dir_estimated_space 1120
dbuf_update_data 1222
ldv 1301
dbuf_read 1487
av wall clock
dnode_allocate 185006
zap_table_load 193232
dbuf_will_dirty 231433
zap_entry_create 255027
zap_lockdir 258613
dmu_zfetch_find 259608
dmu_zfetch 269971
dnode_setdirty 309299
dmu_zfetch_fetch 367224
dmu_buf_hold 368589
dmu_zfetch_dofetch 376191
dbuf_dirty 386128
dnode_hold_impl 535814
dnode_hold 566143
dmu_tx_hold_bonus 617734
dmu_tx_hold_object_impl 663582
dmu_tx_add_new_object 673663
dmu_tx_hold_zap 706378
dnode_new_blkid 761775
fzap_lookup 782656
dmu_bonus_hold 791209
zap_get_leaf_byblk 841362
zap_deref_leaf 973677
zap_lookup 1085355
zap_add 1837000
fzap_add_cd 1916822
fzap_add 1932422
zfs_link_create 2353198
dmu_object_alloc 403070558
zfs_mknode 417633571
av cpu
dmu_tx_add_new_object 14083
dbuf_will_dirty 14796
dmu_buf_hold 17393
dmu_tx_count_write 19129
zfs_acl_chmod 19416
dbuf_dirty 19525
dmu_tx_try_assign 20067
zap_lockdir 20951
zap_table_load 21232
dmu_tx_assign 21946
zfs_perm_init 23304
zap_increment_num_entries 25355
dnode_allocate 25526
zap_entry_create 28102
dmu_zfetch_find 28629
dmu_zfetch 30387
dmu_bonus_hold 37614
dmu_zfetch_fetch 38222
dmu_zfetch_dofetch 39842
zap_get_leaf_byblk 40255
fzap_lookup 40518
zap_deref_leaf 54850
zap_lookup 65315
dmu_tx_hold_zap 86567
zap_add 157876
fzap_add_cd 158310
fzap_add 160400
zfs_link_create 191034
dmu_object_alloc 401750326
zfs_mknode 416439036
_______________________________________________
zfs-discuss mailing list
zfs-discuss@xxxxxxxxxxxxxxx
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|