Any software engineer who has worked on large projects - industry or community - can tell you the importance of solid change control processes. So… since everything for the build had to be checked into clearcase… yes, we checked Oracle 10g into the repo. A 2G tarball. And whenever there was a 1K patch to the oracle install? A brand new 2G tarball. The clearcase guys loved me.

And that was how I started thinking about an automated build process. I don’t need to check B24792-01_1of5.zip into the repository because it’s straight off edelivery. I can skip p7150622_10203_Linux-x86-64.zip since it’s direct from metalink. The only thing missing was a solid, simple, flexible program for automating the oracle install, patchset, CPUs and oneoffs - taking Oracle’s official bits as input.

Anyone else out there who could use a program like this? How about for rapid provisioning of servers? (Like all that grid buzz.) Grid Control does some basic stuff (if you can make it work) and there are advanced kits for the big data centers - but there have to be more people than just me who would love to have this program.

Proposing orainstall

That’s why I’m proposing to write a bit of software called orainstall. It would be script-based and cross-platform and intended for community use. I have some ideas for a design but I’m really interested to hear how you might use something like this and what features might be useful to you. Do you think it’s a good design? I’d also be interested in hearing if you’d be interested to use it or to help test it.

The program would be launched by running the main executable “orainstall”. The syntax could look something like this:

Command line syntax:
  ./orainstall <stepfile> [-oh <ORACLE_HOME>] [-tmp <ORAINST_TMP>]

Example:
  ./orainstall ora10203.sf -oh /u01/app/oracle/product/10.2.0/db_1 -tmp /u01/app/oracle/tmp

Return Value:
  0 - success
  1 - invalid stepfile
  2 - installation error
  3 - patching error

The Stepfile

What I’m thinking is that the install process is defined in an input file called the stepfile. The stepfile needs to accomodate a large variety of organizational requirements. Here’s my first draft for what a stepfile would look like:

# This stepfile instructs the oracle auto-installer how to setup the oracle
# home. Each line is executed in order from top to bottom.
#
# There are ten defined directives for this stepfile:
#   ORACLE_HOME - required
#     This will cause a search-replace of each response file to explicitly
#     define the oracle home. It must be defined before any open, install or
#     patch directives and it can only be defined once per stepfile.
#       Order of precedence:
#         1. command line
#         2. stepfile
#         3. environment variable
#   ORAINST_TMP - optional
#     Top-level temp directory for extracting contents of Oracle installers
#     and patches. Each installer or patch will be extracted into a
#     subdirectory of this temp location. It must be defined before any
#     open or patch directives. It can be defined multiple times in a
#     stepfile.
#       Order of precedence:
#         1. command line
#         2. stepfile
#         3. environment variable
#         4. "/tmp/orainstall"
#   OPEN /path/to/zip
#     Path to a zip file with database installer or major patchset which
#     requires use if OUI. The file is unzipped into a temporary location.
#     If several INSTALL directives are encountered in a row then they are
#     all unzipped into the same location; if there is another directive in
#     between then the next INSTALL is unzipped into a new temporary
#     location.
#   INSTALLPATH Disk1/runInstaller
#   INSTALLRSP install.rsp
#      "runInstaller -waitForCompletion -silent -responseFile <install.rsp>"
#     INSTALLPATH is optional - if not specified then the script will search
#     for it in the most recently created temp location. If there is more
#     than one runInstaller then it will use the first one it finds; search
#     order is undefined.
#     If no path is specified for the response file then orainstall will
#     search for it in this order:
#       1. current working directory (pwd) when orainstall was launched
#       2. same directory as orainstall binary (dirname $0)
#   OPATCH/path/p8000.zip
#     Path to special "OPatch" update - contents are unzipped directly to
#     ORACLE_HOME.
#   PATCH /path/p2345.zip
#     Path to a patch. Contents are unzipped to a temporary location then
#     applied by calling "opatch" from the ORACLE_HOME.
#   NPATCH /path/p1234.zip
#     Path to a patchset with multiple subpatches which need to be applied
#     with "napply" (such as a CPU). Contents are unzipped to temp location
#     then applied properly to the ORACLE_HOME.
#   CLEAN_TMP
#     Deletes all files from the current top-level temp directory.
#   REMOVE /path/to/file
#     Runs "rm -rf $ORACLE_HOME/path/to/file" - use carefully! Some sites
#     may require  removal of components for security requirements.
#
# If there are any errors (install or patch) then the process will fail.
#
#oracle_home /u01/app/oracle/product/10.2.0/db_1
orainst_tmp /u01/app/oracle/installers.tmp
open /net/rh5lab12/u10/stage/ora10201/B24792-01_1of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_2of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_3of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_4of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_5of5.zip
installpath database/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/database10201.rsp
installpath companion/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/companion10201.rsp
open /net/rh5lab12/u10/stage/ora10203/p5337014_10203_Linux-x86-64.zip
installrsp /net/rh5lab12/u10/stage/rsp/patchset10203.rsp
clean_tmp
orainst_tmp /u01/app/oracle/patches
opatch /net/rh5lab12/u10/stage/patch10203/p6880880_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p5892355_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p6455161_10203_Linux-x86-64.zip
npatch /net/rh5lab12/u10/stage/patch10203/p7150622_10203_Linux-x86-64.zip
remove /htmldb

Program Flow

The program flow would be pretty basic; it would validate the stepfile then execute it one line at a time. I thought of six things to validate before starting execution:

1. can read stepfile
2. one oracle_home, defined before open/install/patch, can create/write
3. orainst_tmp defined before open/install/patch, can create/write each
4. clean_tmp not defined before orainst_tmp
5. can read all open/install/patch input files
6. free space in oracle_home is greater than size of input files

Feedback

Maybe there aren’t as many people as I think who would use this. After all - most DBAs don’t install the database software very often; they just want to keep it running! But please let me know if it interests you. I’ll factor in the feedback I receive as I’m developing it.

Everybody knows that asmlib was very useful when it was first introduced with Oracle 10.1 to simplify a host of issues on Linux: direct async device access without raw devices, file permissions & ownership without custom code, and persistent device naming without devlabel.

But I’m now involved in setting some standards to be used across a large organization for Oracle 10.2 on RHEL5 and I’m wondering if there’s still a case for using asmlib. So I did a little trolling for info - which was suprisingly sparse. Had a hard time finding much, but after a lot of digging I think I’ve compiled a useful bit of information about benefits and drawbacks.

Benefits

It seems to me that the ASMLIB API was originally introduced to do more than just simplify file permissions - sounds like it was an alternative I/O API to the standard unix one, allowing ASM to access the underlying storage more efficiently and completely. I don’t think it’s just an “extra layer” - it’s an alternative code path to the std unix I/O libs. Like an ODM for block devices - and the idea was that there could be additional vendor implementations. And Oracle released an initial generic implementation on Linux under the GPL.

Some theoretical benefits of ASMLIB API:

• always uses direct, async i/o
• solves persistent device naming, even if underlying device moves across reboots
• solves file permissions and ownership
• reduced user mode to kernel mode context switches during I/O, possibly reducing CPU usage
• reduced file handle usage
• pass metadata such as I/O prioritization to storage device (don’t think this is implemented in the Linux version)

Drawbacks

However from what I can tell, the ASMLIB API didn’t quite catch on. Wim recommends it (as of Apr ‘07) but Closson doesn’t really talk about it on his blog but has one disparaging off-hand reference about pitying people who use it. There have been no vendor implementations that I could find. (But I wonder what’s going on now with exadata and “iDB” which has I/O prioritization and predicate pre-filtering - could they have used the API here?) Wim’s post on Oracle Forums says that there isn’t “much of an io performance benefit” added by the linux ASMLIB implementation. And the OTN page hasn’t been updated for almost two years.

A RedHat consultant I’m working with recommended against using it and one of his reasons was that it introduces an unnecessary additional layer and dependency in the kernel. There have been a small number of past bugs in asmlib and it doesn’t seem to be frequently updated (once this year according to the changelogs)… but then it’s very simple code (mostly in a single small file).

Conclusions

After all of this, I’m not convinced that it’s worth installing ASMLIB if you’re comfortable with udev. On Linux, udev is definitely a better solution to handle persistent device naming and permissions and I don’t see enough benefits in asmlib to outweigh the very slight additional overhead.

So right now my preference would be to only install asmlib on old systems - Oracle 10.1 or Linux 2.4 - but to use udev and block devices for 10.2 and newer on 2.6 kernels. But the case isn’t closed and I’d love to hear from anyone who agrees or disagrees… thoughts?

References:

• Wim’s brief remarks on Oracle Forums - http://forums.oracle.com/forums/thread.jspa?threadID=498215
• In-depth information from ASM reference book - http://books.google.co.uk/books?id=HB453L86Q6AC&pg=RA1-PA142&lpg=RA1-PA142&dq=asmlib+api&source=web&ots=bEP4epFbp5&sig=47aLUWW9o73Snr1tcnvz_1h3prU&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PRA1-PA142,M1
• Uses submit_bio() call - source code - http://oss.oracle.com/viewvc/oracleasm/trunk/kernel/oracleasm.c?view=markup
• cdos newsgroup thread with speculation about performance - http://groups.google.com/group/comp.databases.oracle.server/browse_thread/thread/905296d0aba00b84?pli=1
• tim hall was going to investigate though he didn’t publish results - http://www.oracle-base.com/blog/2006/05/01/asm-with-asmlib-or-raw-devices/
• kevin closson doesn’t say much but makes this off-hand remark - http://kevinclosson.wordpress.com/2006/11/02/dbwr-efficiency-aio-io-libraries-with-asm/
• short anonymous comment possibly by closson - http://archives.devshed.com/forums/databases-124/asmlib-1157660.html
• OTN official site for ASMLIB - http://www.oracle.com/technology/tech/linux/asmlib/index.html

First was a panel debate “To RAC or Not To RAC: What’s Best for HA.” Dan Norris invited me to participate in this panel along with Alex Gorbachev (Pythian), Neil Greene (Predictive Technologies) and Matt Zito (GridApp) - I certainly felt privileged to take part. Here are a few of the things that I took away from the debate… feel free to discuss:

• There are no silver bullets. And if there were, RAC wouldn’t be one. :) Many RAC implementation problems can be traced to poorly set expectations; knowing your requirements and RAC’s capabilities will go a long way towards making any project successful.

• RAC really shines when solving scalability problems. The simple act of moving an application from single-instance to RAC is more likely to improve throughput than service time. (Of course, improvement isn’t guaranteed: as Dan pointed out during our conversation it depends on where the bottlenecks are.)

• There are two major application design areas that need to be addressed when moving to RAC: (1) scalability and (2) connectivity. (There are other issues too but these are probably the biggest.) Regarding scalability, small inefficiencies will become major bottlenecks when you move your app to a cluster. Look out for any points of contention or serialization whether they’re logical (like an ID field that must be sequential) or physical (like free space management). On the connectivity front, there is both initial connection and load balancing to worry about.

• When it comes to HA, I personally think that failover clusters should get more serious consideration then they do. I think that there are essentially two differences between a failover cluster and RAC: (1) a few minutes of downtown each year, and (2) the prices for licenses - probably tens or hundreds of thousands of dollars minimum. You can even still use TAF with a failover cluster - it’s not much different than using it with Data Guard. The few apps that need extremely fast failover often can’t even wait a few minutes - they commonly run parallel systems in tandem to avoid failover altogether. Most other apps can tolerate an extra 60 seconds during failover, if we’re honest.

At any rate, I thought that the debate was informative and that Alex, Neil and Matt were each very well spoken. Alex also has a good write-up on his Pythian blog.

After the debate panel I slipped over to hear Alex’s 11g New Features presentation (which was actually Christo’s presentation but Alex delivered it since Christo wasn’t able to fly over from Dubai). The presentation went great even though Alex hadn’t written it himself and I snapped this pic with my cell phone while I was there.

Alex’s session was the last one of the conference, so afterwards we grabbed a bite to eat and spent some time catching up. Dan joined us a bit later and then we made our way to the airport to have dinner and catch our respective flights. Thanks for the great company guys!

RAC 11g/VMware Lab

One last thing: on Wednesday we had a lab session where we went through the step-by-step process of setting up 11g RAC in a VMware Server virtual environment with OEL5 and ASM. I have uploaded the class to the publications section of this website.

If you have a 2Ghz processor, 2GB of memory and 20G of disk space, then you can download all of the software for free and try out this lab for yourself:

• VMware Server - http://www.vmware.com/download/server/
• Oracle Enterprise Linux (OEL) 5 - http://edelivery.oracle.com/linux
• Oracle Database Enterprise Edition 11g - http://www.oracle.com/technology/software/products/database/index.html

Next week sometime I expect to upload the instructions from today’s Hands-on Lab (11g RAC on VMware with ASM and OEL5). I want to clean it up a bit first. For anyone who didn’t hear the story, I heard on Sunday evening that they had a room with about 50 computers which was going to sit empty during a few timeslots. And to me that seemed like a tragedy - how often do I wish I could have a chance to try something new with a bit of guidance and without worrying about hosing my laptop?! So I decided to write a hands-on lab for 11gRAC/VMware. I pretty much spent all day yesterday putting it together but it came out great!

Also I’m thinking about repeating that RAC/VMware lab around Chicago sometime… is there anyone around the Chicago area who might be interested in something like this?

Guess I should also mention that I’ve been rather enjoying myself at Collaborate so far too! (Even though I spent pretty much all day yesterday making that Hands-on Lab…) Got a chance to meet Peter Scott Monday night - that was fun! Somehow he spotted my name badge and then I got to finally put a face to someone I’d only known as a blogger. :)

But before getting into the results, a quick overview of our configuration: it was 11g RAC on Red Hat Enterprise Linux 5; Dell servers with four dual-core Opteron chips each. The RAC cluster initially had four nodes but will grow to at least eight as the data is migrated. The system has 4G QLogic cards, a McData switch and a 3Par SAN (which is blazing fast). ASM (no CFS) and dedicated Oracle Homes. The first spec had an InfiniBand interconnect but after a teleconference with Alex from Pythian discussing the project’s specific requirements, the spec was updated to use redundant Gigabit Ethernet.

Picking up where I left off: the default limit set by the Linux qla2xxx driver for concurrent I/O requests on QLogic cards (32 per LUN) is conservative. So can I increase performance by increasing this limit? The best way to answer a question like this is simply to try it.

Tweaking the HBA Queue Depth

With QLogic HBA’s on Linux the queue depth is configured through the ql2xmaxqdepth module option. I want to run an experiment where I vary this parameter and measure the I/O performance. To start out, I’d like to compare the default queue size of 32 with an increased setting of 64. But how can I effectively measure I/O performance? I’m specifically interested in how Oracle’s RDBMS will perform, so I think that the best tool is Oracle’s Orion benchmarking tool which is designed to simulate database I/O patterns and measure the result.

Now there are two ways to measure I/O performance: IOPS and MBPS.

1. When you measure IOPS you’re usually investigating small I/O operations and putting stress on the overhead associated with a single read or write. Throughput is not the main concern when you measure IOPS. This is most relevant on transactional systems.

2. When you measure MBPS you’re usually investigating large I/O operations and putting stress on the overall throughput. Latency is not the main concern when you measure MBPS. This is most relevant on warehouse and analytical systems.

For more reading, James Koopmann just wrote a few good articles over at Database Journal about getting IOPS/MBPS measurements from existing databases and the relationship between IOPS/MBPS and vendor-supplied disk specs.

Now this particular project’s database will back several high-volume websites and the traffic is probably only 10-20% writes and 80-90% reads. However much of the data changes somewhat frequently and it is definitely an OLTP workload - almost entirely index-based reads. On the AWR report from a current production database, scattered reads almost didn’t even register while sequential reads were by far the most significant event. This means that we need to optimize this storage system - and our benchmark - for IOPS rather than raw throughput.

Orion Results

The basic idea behind Orion is pretty simple. Orion is designed to simulate a mixed workload between single-block reads and multi-block reads. You give it a bunch of parameters that describe your environment - and then it runs its test over and over again, varying the balance between single-block and multi-block reads while measuring the IOPS, latency and MBPS. One important point: Orion varies the balance by changing the number of threads that are concurrently doing reads/writes. Unlike swingbench or hammerora, there is no think time - each thread constantly tries to do I/O.

Since the database for this project is nearly 100% single-block reads, I decided to just run a “basic” matrix - which doesn’t test mixed workloads. It runs about 45 tests with varying levels of concurrency between 1 thread and 500 threads for single-block reads and the same for multiblock-reads. (I just ignored the multi-block results.) I instructed Orion to do 15% write operations and 85% read operations.

#
# ./orion_linux_em64t -run advanced -testname simple -num_disks 100 -write=15 -matrix=basic -verbose
#
# 

I ran this benchmark four times in a row. Before each run I changed the queue depth and rebooted the server. I alternated between 32 and 64, using each queue depth twice. The result was a consistent 7% improvement in IOPS and latency by doubling the queue depth.

What’s the Best Setting for Queue Depth?

That depends on how many clients are accessing the storage device. Do not max out the queue depth on all your servers based only on this article. Remember from the first article that the Storage Device’s FC Port can concurrently process a limited number of requests. If you have a large number of devices accessing the same storage array and you increase the queue depth on all of them, then you will start seeing the dreaded “QUEUE FULL SCSI” errors! However if there are only a handful of clients and you know that you won’t be adding more then you can certainly increase this parameter and get the associated performance boost at peak workload.

To get the optimal value you need to consult the manuals or support channels for your storage system to find out its queue depth. Factor in the number of clients you have accessing this array plus some buffer for safety and then you can determine optimal values for each server.

Almost exactly one year ago I wrote about using udev on 2.6 kernels to set the proper permissions for Oracle RAC. Two weeks after that post (March 14) Red Hat Enterprise Linux 5 was released and changed everything.

In my original post, I demonstrated how to create a PERMISSIONS file that udev would use when creating the device nodes. This worked on RHEL4 and SLES9. However this week I’ve been helping a client deploy 11g RAC on a RHEL5-based cluster - and I remembered that the PERMISSIONS facility was removed from udev in RH5. Seems like I remember reading something about having a single source of configuration for udev, which makes sense… so maybe they picked the RULES. (You’ll remember from my previous post that RULES are processes right before PERMISSIONS.) This is just as well since RULES are actually quite a bit more powerful than PERMISSIONS.

So on RHEL5 and OEL5 - in order to conform to Linux Best Practices - we now have to set correct RAC file permissions using udev RULES. To get started, we need to review how RULES work. The udev manual page gives a good overview of rules processing. But of course there are plenty of great tutorials that go deeper if you’re looking for more.

Block Devices and Raw Devices

Now of course I’m not the first person to notice that there’s no permissions.d directory on RHEL5 and OEL5. Last September, Grégory Guillou from Pythian blogged about installing on Red Hat 5 and referenced a post on his French blog that shows how to setup a RULES file for SCSI block devices. I don’t speak French but I was able to copy the text of his RULES files:

# Oracle Configuration Registry
KERNEL=="sdb[8-9]", OWNER="root", GROUP="oinstall", MODE="640"
# Voting Disks
KERNEL=="sdb1[0-2]", OWNER="oracle", GROUP="oinstall", MODE="640"
# ASM Devices
KERNEL=="sdb[5-7]", OWNER="oracle", GROUP="dba", MODE="660"

Many people are still configuring raw devices for their voting disks and cluster registries even though this is not necessary. This Oracle Forums thread gives an sample of using a RULES file to set permissions for raw devices. It says to create a file called /etc/udev/rules.d/65-raw-permissions.rules with these contents:

# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"

In fact there’s a great Oracle Whitepaper on the udev and multipathing that was published all the way back in June of 2007. It gives another sample configuration for raw devices on Red Hat 5.

Linux Multipath (Device Mapper) Devices

That’s all great. However… I’m doing an implementation on RHEL5 right now and we’re using the device mapper to multipath connections to the SAN. And the Oracle white paper - which goes into marvelous depth about multipath, udev and RH5 - never tells us anything about configuring multipath, udev and RH5 all together! So I have to figure this one out on my own. No harm done; it provided an interesting challenge for the day. :)

Multipath devices are a bit tricky. Usually if you’re using multipath then you don’t want to assume that devices will always be discovered in the same order. (This is the purpose of assigning friendly names by WWN in the multipath.conf file with the alias directive.) In fact, on our cluster the devices were assigned in different orders on different nodes - the ocr was dm-9 on one node and dm-10 on another. How do you write a RULES entry if you don’t know what the name of the device is?

I finally did get the multipath rules file to gave the proper permissions to my OCR and Voting Disks based on alias. However I was only able to change the /dev/dm-* files and not the /dev/mapper/* nodes. (Those nodes are not created by udev.) Therefor I had to use the aliases in /dev/mpath - not the aliases in /dev/mapper - when running the Oracle installer. How does it work? The symlinks in /dev/mpath are created by udev RULES - so all I had to do was piggyback on the udev config that created them and modify the permissions of the underlying devices.

This client used the aliases vote1, vote2, vote3, ocr1 and ocr2. Here’s the config file /etc/udev/rules.d/40-multipath.rules with my changes in bold:

# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", ACTION=="add", PROGRAM=="/bin/bash -c '/sbin/lsmod | /bin/grep \
    ^dm_multipath'", RUN+="/sbin/multipath -v0 %M:%m"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
ACTION=="add", RUN+="/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p p' \
    -j %M -m %m"
PROGRAM=="/sbin/dmsetup ls --target multipath --exec /bin/basename -j %M -m %m", \
    RESULT=="?*", NAME="%k", SYMLINK="mpath/%c", GOTO="check_cluster_devs"
PROGRAM!="/bin/bash -c '/sbin/dmsetup info -c --noheadings -j %M -m %m | /bin/grep \
    -q .*:.*:.*:.*:.*:.*:.*:part[0-9]*-mpath-'", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup ls --target linear --exec /bin/basename -j %M -m %m", NAME="%k", \
    RESULT=="?*", SYMLINK="mpath/%c", OPTIONS="last_rule"
GOTO="end_mpath"
LABEL="check_cluster_devs"
RESULT=="ocr*", GROUP="dba", MODE="640"
RESULT=="vote*", OWNER="oracle", GROUP="dba", MODE="640"
OPTIONS="last_rule"
LABEL="end_mpath"

This post is just another interesting case study from a customer I’m working with right now. We were looking at various queues in the I/O stream and wondering how we might be able to tweak them. I was mainly investigating the host HBA’s - but in the process of digging into this I also learned about few other queues and Linux internals in general as it relates to Oracle.

Direct Async I/O

It’s pretty well documented that the most efficient configuration for Oracle is direct asynchronous I/O.

Direct I/O means to avoid unnecessarily copying bits from one memory location to another between the hardware and the SGA. Every flavor of Unix implements some form of caching for block devices and filesystems; in Linux the buffer cache (for block devices) or page cache (for filesystem data) can cache database blocks. Most applications benefit from this caching - however Oracle does not. There are two primary reasons this caching is bad for Oracle:

1. Oracle caches this data itself in the SGA and it does a far better job of predicting what blocks will be best retained in memory. When the OS caches data you essentially end up with two copies of every data block - one that’s really unused. You waste half the memory in your server with almost no performance benefit.

2. The data has to be copied extra times before arriving in the SGA - and these extra “copy” operations slow down your read operations.

Asynchronous I/O means to multitask your read and write operations. Remember how in MSDOS you could only run one program at a time? Believe it or not, this is how most Unix filesystems do I/O by default - one thing at a time. Using Asynchronous is equivalent to upgrading to an Operating System that allows you to run more than one program at a time. It allows Oracle to issue multiple read and write operations concurrently - this is obviously more efficient.

There are different ways of implementing Direct I/O and Asynchronous I/O in different environments (CIO, QIO, ODM, etc) - but it’s always best to enable them and then shift memory from the OS caches to the SGA.

The Linux I/O Path

I should start out by saying that I’m not a kernel expert. After some digging through source code and mailing lists I think that I’ve got a handle on the main concepts here - but I’m open to correction since this is complicated stuff and I easily could miss some details of how it’s all implemented! Let me know if you have anything to add!

On Linux, asynchronous I/O requests are submitted through the io_submit() call. (Kevin Closson once wrote about tracing this call.) This function essentially puts the I/O request into a queue which is managed by the Linux kernel. I think that this workqueue is a generic kernel object which you can’t and shouldn’t need to tweak. The I/O request is subsequently picked up by one of several kernel AIO background processes and serviced.

What the kernel thread does depends on the underlying device. FC cards support asynchronous requests to the target (it’s part of the SCSI spec) - so in this case the kernel thread will issue a number of SCSI read or write requests in parallel. This is where a second queue comes in! And this queue, unlike the kernel workqueues, can be tweaked.

The target device (Symetrix, DSxxxx, 3Par, etc) also has an FC port which has its own queue for I/O requests that are being serviced. On a 3par device these ports only have between 500 and 2000 slots for concurrent I/O requests depending on the model. This means that all the servers accessing any LUNs over that port cannot issue more than 500-2000 concurrent I/O requests - or you will start to see “QUEUE FULL SCSI” errors.

My best guess: this means that the default limit set by the Linux qla2xxx driver for concurrent I/O requests on QLogic cards (32 per LUN) is conservative. The driver sets this limit by limiting the size of the queue. So naturally… my next question was if I could increase performance by increasing this limit…

Tweaking the HBA Queue Depth

I did run a few tests - but I think I’ll save the results for a second post since this one has already gotten pretty long. Any guesses about what the results were?

http://www.ardentperf.com/2007/09/27/two-days-until-asia-trip-update/

One common piece of advice for gc buffer busy waits is to treat them like regular buffer busy waits. Because essentially that’s what they are - a buffer busy wait on a remote instance. So another avenue of investigation is to look at what might be causing buffer busy waits across the cluster.

Some people may remember that back in the days before YAPP and the wait interface, latches were usually where the purported “experts” looked when you had performance problems. Particularly those two infamous latches cache buffers chains and library cache. And of course today these are still an important part of any in-depth investigation and V$LATCH even includes wait time so you can take a time-based approach to analysis. I spent some time yesterday having a look at the latching in this RAC system and it yielded some results that I thought might be interesting to post. So here goes…

Cache Buffers Chains Latch

Now it was apparent to me pretty quickly that the cache buffers chains latch was the busiest on the system. And of course it’s normal for there to be some contention on this latch. You’ll notice from the screenshot the enormous difference in gets, spin_gets and sleeps. So that gave me a bit of a head start - but you could certainly get the same information from the AWR (that blessed new feature) as well. In fact you can do all of this from the AWR - although look out since latch information is not gathered by default!

So these guys weren’t gathering latch info - and so I couldn’t use the AWR. Instead I just setup my own temporary table to hold the statistics.

create table jschneider.waits as
select 1 snap, systimestamp timestamp, inst_id
,      CHILD#
,      ADDR
,      GETS
,      MISSES
,      SLEEPS
from gv$latch_children
where name = 'cache buffers chains';

Table created.

At this point I worked on something else for awhile. After an hour or so I took a second snapshot.

insert into jschneider.waits
select 2, systimestamp, inst_id
,      CHILD#
,      ADDR
,      GETS
,      MISSES
,      SLEEPS
from gv$latch_children
where name = 'cache buffers chains';

393216 rows created.

commit;

Commit complete.

I guess I should briefly explain why I grabbed that information. The cache buffers chains latch is actually made up of a large number of child latches. When Oracle needs to access the buffer cache it hashes some of the block’s information to discover which child it needs to use. That way each child latch only has a short list of blocks that it is managing. What I aim to do is find out which child latches are the busiest and then see which segments have blocks protected by those latches. Not only that but we’ll be able to see which blocks are the busiest (in real time).

So the next step is to find the child latches suffering from the most contention. In order to do this we’ll look for latches that frequently cause processes to sleep (relinquish the processor) while waiting.

with subq as (
select t2.inst_id i, t2.child#, t2.addr,
  t2.gets-t1.gets gets,
  t2.misses-t1.misses misses,
  t2.sleeps-t1.sleeps sleeps
from jschneider.waits6 t1, jschneider.waits6 t2
where t1.child#=t2.child#
  and t1.inst_id=t2.inst_id
  and t1.snap=1 and t2.snap=2
order by sleeps desc
)
select * from subq
where rownum < 40;

         I     CHILD# ADDR                   GETS     MISSES     SLEEPS
---------- ---------- ---------------- ---------- ---------- ----------
         3      32664 00000005F9EA6780     349526       8642        168
         2      32664 00000005F9EA6780     352351       9850        167
         3      21135 00000005F7D9FC80     109896       1284        146
         3      39340 00000005FBA3AD10      58127       1100        142
         5      13536 00000005A8F41F40    2305516      63418        126
         3      57464 0000000608DD4200      45361        909        112
         3      59297 000000060799DF68      31056        690         94
         3      14549 00000005F9C30B08      46444        624         90
         2      53157 00000006078C8808       4727       1129         83
         3      27008 00000005F6E27820      64685       1148         82
         3      62193 00000005FAD17808      28596        550         78
         3      24183 00000005F6DC54B8      32420        505         77
         3      35290 00000005F7F8BE38      65142       1115         75
         3      46376 0000000606C9CA30      29474        653         74
         3       8428 00000005F6BA1900      67146        303         73
         3      15740 00000005F7CE4388      94209        338         72
         3      30375 00000005F7EE1040       4962        663         69
         2      57464 0000000608DD4200      59033        887         68
         3      21180 00000005F7DA1588      65043        743         68
         3      38233 00000005F6FADC08      31475        516         66
         3      25742 00000005F8DE1520      57828        928         65
         3      46198 00000005FAAEB6D0      61915        751         65
         3       6648 00000005F6B63AE0      32022        462         62
         5      18908 00000005A2E51A30    1480518      39246         62
         3      59340 00000005FBCF2210      77557        307         61
         6      63812 00000005A0CD70E8    1071392      63118         61
         2       2526 00000005F5B16B90     339617        303         60
         1      32664 00000005F9EA6780     234869       5319         57
         2      14849 00000005F9C3B1E8      30541        248         57
         3      11539 00000005F9BC80B8     141915        220         57
         2      16944 00000005F9C83F40       3931        961         56
         4      18475 00000005F7D434E0      87239        477         56
         3      33009 00000005F9EB2768      66883        793         56
         2      59438 00000005FACB7B90      27733        742         52
         2      63600 00000005FBD863B0      10316        188         51
         3      44988 00000005FAAC15C0      28087        208         51
         2      52350 00000005FBBFF1E0     570855       3252         50
         3       1388 00000005F6AACD00     203545        163         50
         3       8972 00000005F8B9A4D0      28815        205         49

Exploring the Buffer Cache

Notice that I’ve also queried all latches across the entire cluster. So this is giving me a system-wide picture of cache buffer chains latches. Interestingly, the top child is the same one on two different instances (2 and 3). So next let’s login locally to instance 3 and see what that latch child is protecting! Also, I’m going to grab the current SCN - but only the base - from dbms_flashback.

select mod(dbms_flashback.get_system_change_number,power(2,32)) cur_scn_base from dual;

CUR_SCN_BASE
------------
   279813816

col object format a55
col state format a5
select /*+rule*/ *
from (
select o.owner||'.'||o.object_name||decode(o.subobject_name,NULL,'','.')||
  o.subobject_name||' ['||o.object_type||']' object, sq.*
from (
select
  x.obj,
  x.file#,x.dbablk,
  x.tch,
  decode(x.state,0,'FREE',1,'XCUR',2,'SCUR',3,'CR',4,'READ',
    5,'MREC',6,'IREC',7,'WRITE',8,'PI',9,'MEMORY',10,'MWRITE',
    11,'DONATED',x.state) state,
  decode(x.state,3,cr_scn_bas,NULL) scn_bas
from
  sys.v$latch_children  l,
  sys.x$bh  x
where
  x.hladdr = l.addr and
  x.obj < power(2,22) and
  x.hladdr  = '00000005F9EA6780'
) sq, dba_objects o
where
  o.data_object_id=sq.obj
  order by sq.tch desc, file#, dbablk, scn_bas
) where rownum<40;

OBJECT                                                         OBJ      FILE#     DBABLK        TCH STATE    SCN_BAS
------------------------------------------------------- ---------- ---------- ---------- ---------- ----- ----------
JSCHDER.SPOT_ACTIVITY [TABLE]                               903892        214     110918          3 SCUR
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          2 CR     279622172
JSCHDER.BIGTABLE_ORDERS_MF_AND_PARTS [INDEX]                  3136        994     121460          1 SCUR
JSCHDER.BIGTABLE.P_PARTS_APPROVED [TABLE PARTITION]        3064309       1156      33703          1 SCUR
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          1 CR     279619002
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          1 CR     279619202
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          1 CR     279621063
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          1 CR     279618998
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          1 PI
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]           3208618       1209      16393          1 PI
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTI    3208620       1542       4537          1 SCUR
TION]

JSCHDER.PERSISTENT_QUERY [TABLE]                           1455110         10      99668          0 SCUR
JSCHDER.GENERATED_REPORT [TABLE]                           1455107         13      22427          0 SCUR
JSCHDER.SPOT_ACTIVITY_URL_DETAIL [TABLE]                   2684382        287      51469          0 SCUR
JSCHDER.GENERATED_REPORT [TABLE]                           1455107        476     103892          0 SCUR
JSCHDER.BIGTABLE_TOTALS [TABLE]                            1746552        524     187199          0 SCUR
JSCHDER.PERSISTENT_QUERY [TABLE]                           1455110       1476      21915          0 SCUR

17 rows selected.

Now this query has done more than show me the blocks protected by this latch child. The “TCH” field is telling me the Touch Count for the buffer - an indication of how hot the block is. This counter is incremented most of the time that Oracle accesses the block and if I remember right it is decremented automatically every 3 seconds or so. So this will show me - in real time - what blocks are being accessed.

First of all I noticed that there are only 17 blocks on this latch - which should be great for concurrency! These blocks must have been hammered to have so many sleeps! Secondly, I noticed that there’s one block that has seven different versions in this instance alone. I was immediately suspicious about that block and decided to check something…

select header_file, header_block from dba_segments
where owner='JSCHDER' and segment_name='BIGTABLE_LOG' and PARTITION_NAME='P_2007_09';

HEADER_FILE HEADER_BLOCK
----------- ------------
       1209        16393

Looks like that’s the header block!! Interesting… so I wonder how many copies of this block are spread around the cluster? That’s pretty easy to check too.

select inst_id, status, dirty, stale from gv$bh
where file#=1209 and block#=16393 order by 1, 2;

   INST_ID STATUS  D S
---------- ------- - -
         1 cr      N Y
         1 cr      N N
         1 cr      N N
         1 cr      N N
         1 cr      N N
         1 pi      Y N
         2 cr      N N
         2 cr      N N
         2 cr      N Y
         2 cr      N N
         2 pi      Y N
         3 cr      N N
         3 cr      N N
         3 cr      N Y
         3 cr      N N
         3 cr      N N
         3 pi      Y N
         4 cr      N N
         4 cr      N N
         4 cr      N N
         4 cr      N N
         4 cr      N N
         4 pi      Y N
         4 xcur    N N

Quite a few. In fact I discovered that if I repeated that query even a few moments apart the “xcur” block (which is the one being updated) is constantly moving between instances. Normal operation for RAC… but interesting to watch nonetheless!

What Does it All Mean?

So what’s the point? Well we’ve shored up the theory that BIGTABLE_LOG is the main culprit in this system. The header block is where the master freelist is and that would be bounced around a lot without freelist groups. After switching to ASSM or adding freelist groups I would expect to see fewer copies of the header block in the global cache. Hopefully they’ll let me know what happens so I can post an update! (Sounds like they’re planning to move the partition to ASSM during the downtime window this weekend!)

Appendix: Quickly Mapping file#/block# to an Object

I’m sure that everyone, at some point, has had a block number and wanted to know the object. And if you’ve been in this situation with a large database then you realize that there’s just no good way to do it. You have to query the dreaded DBA_EXTENTS view… which can take years to finish.

If you didn’t notice, I actually just demonstrated a potential alternative - maybe. If the block you’re checking is in the buffer cache (on any instance in RAC) then you can just read from v$bh instead of waiting around for DBA_EXTENTS! Just use a query something like this:

col object_name format a20
col owner format a10
col subobject_name format a20
select distinct o.owner, o.object_name, o.subobject_name, o.object_type
from dba_objects o, gv$bh b
where o.data_object_id=b.objd and b.objd < power(2,22) and status != 'free'
  and b.file#=1209
  and b.block#=16393
;

OWNER      OBJECT_NAME          SUBOBJECT_NAME       OBJECT_TYPE
---------- -------------------- -------------------- -------------------
JSCHDER    BIGTABLE_LOG         P_2007_09            TABLE PARTITION

Very handy shortcut in my opinion. If you’re not on RAC then this will go even faster.

Happy hacking!

Update 9/14/07 - updated queries against dba_objects/v$bh to eliminate rollback and temp segs. also fixed last query in the post to use data_object_id.

More Resources

Yas left a comment pointing me to a post from Lewis - and that led me to a few other useful posts that seem worth linking here.

PS - if you haven’t checked out my Links page in awhile then have another look - I’ve added quite a few links over the past few weeks!

Now you’ll recall that this event simply means that we’re waiting for another instance who has the block. But generally if you see lots of these then it’s an indication of contention across the cluster. So here’s how I got to the bottom of a problem on a pretty active 6-node cluster here in NYC.

Using the ASH

I’ll show two different ways here to arrive at the same conclusion. First, we’ll look a the ASH to see what the sampled sessions today were waiting on. Second, we’ll look at the segment statistics captured by the AWR.

First of all some setup. I already knew what the wait events looked like from looking at dbconsole but here’s a quick snapshot using the ASH data from today:

select min(begin_interval_time) min, max(end_interval_time) max
from dba_hist_snapshot
where snap_id between 12831 and 12838;

MIN                            MAX
------------------------------ ------------------------------
12-SEP-07 09.00.17.451 AM      12-SEP-07 05.00.03.683 PM

This is the window I’m going to use; 9am to 5pm today.

select wait_class_id, wait_class, count(*) cnt
from dba_hist_active_sess_history
where snap_id between 12831 and 12838
group by wait_class_id, wait_class
order by 3;

WAIT_CLASS_ID WAIT_CLASS                            CNT
------------- ------------------------------ ----------
   3290255840 Configuration                         169
   2000153315 Network                               934
   4108307767 System I/O                           7199
   3386400367 Commit                               7809
   4217450380 Application                         12248
   3875070507 Concurrency                         14754
   1893977003 Other                               35499
                                                  97762
   3871361733 Cluster                            104810
   1740759767 User I/O                           121999

You can see that there were a very large number of cluster events recorded in the ASH. Let’s look a little closer.

select event_id, event, count(*) cnt from dba_hist_active_sess_history
where snap_id between 12831 and 12838 and wait_class_id=3871361733
group by event_id, event
order by 3;

  EVENT_ID EVENT                                           CNT
---------- ---------------------------------------- ----------
3905407295 gc current request                                4
3785617759 gc current block congested                       10
2705335821 gc cr block congested                            15
 512320954 gc cr request                                    16
3794703642 gc cr grant congested                            17
3897775868 gc current multi block request                   17
1742950045 gc current retry                                 18
1445598276 gc cr disk read                                 148
1457266432 gc current split                                229
2685450749 gc current grant 2-way                          290
 957917679 gc current block lost                           579
 737661873 gc cr block 2-way                               699
2277737081 gc current grant busy                           991
3570184881 gc current block 3-way                         1190
3151901526 gc cr block lost                               1951
 111015833 gc current block 2-way                         2078
3046984244 gc cr block 3-way                              2107
 661121159 gc cr multi block request                      4092
3201690383 gc cr grant 2-way                              4129
1520064534 gc cr block busy                               4576
2701629120 gc current block busy                         14379
1478861578 gc buffer busy                                67275

Notice the huge gap between the number of buffer busy waits and everything else. Other statistics I checked also confirmed that this wait event was the most significant on the cluster. So now we’ve got an event and we know that 67,275 sessions were waiting on it during ASH snapshots between 9am and 5pm today. Let’s see what SQL these sessions were executing when they got snapped. In fact lets even include the “gc current block busy” events since there was a bit of a gap for them too.

select sql_id, count(*) cnt from dba_hist_active_sess_history
where snap_id between 12831 and 12838
and event_id in (2701629120, 1478861578)
group by sql_id
having count(*)>1000
order by 2;

SQL_ID               CNT
------------- ----------
6kk6ydpp3u8xw       1011
2hvs3mpab5j0w       1022
292jxfuggtsqh       1168
3mcxaqffnzgfw       1226
a36pf34c87x7s       1328
4vs8wgvpfm87w       1390
22ggtj4z9ak3a       1574
gsqhbt5a6d4uv       1744
cyt90uk11a22c       2240
39dtqqpr7ygcw       4251
8v3b2m405atgy      42292

Wow - another big leap - 4,000 to 42,000! Clearly there’s one SQL statement which is the primary culprit. What’s the statement?

select sql_text from dba_hist_sqltext where sql_id='8v3b2m405atgy';

SQL_TEXT
---------------------------------------------------------------------------
insert into bigtable(id, version, client, cl_business_id, cl_order_id, desc

I’ve changed the table and field names so you can’t guess who my client might be. :) But it gets the idea across - an insert statement. Hmmm. Any guesses yet about what the problem might be? Well an insert statement could access a whole host of objects (partitions and indexes)… and even more in this case since there are a good number of triggers on this table. Conveniently, the ASH in 10g records what object is being waited on so we can drill down even to that level.

select count(distinct(current_obj#)) from dba_hist_active_sess_history
where snap_id between 12831 and 12838
and event_id=1478861578 and sql_id='8v3b2m405atgy';

COUNT(DISTINCT(CURRENT_OBJ#))
-----------------------------
                           14

select current_obj#, count(*) cnt from dba_hist_active_sess_history
where snap_id between 12831 and 12838
and event_id=1478861578 and sql_id='8v3b2m405atgy'
group by current_obj#
order by 2;

CURRENT_OBJ#        CNT
------------ ----------
     3122841          1
     3122868          3
     3173166          4
     3324924          5
     3325122          8
     3064307          8
          -1         10
     3064369        331
           0        511
     3122795        617
     3064433        880
     3208619       3913
     3208620       5411
     3208618      22215

Well a trend is emerging. Another very clear outlier - less than a thousand sessions waiting on most objects but the last one is over twenty-two thousand. Let’s have a look at all three of the biggest ones.

select object_id, owner, object_name, subobject_name, object_type from dba_objects
where object_id in (3208618, 3208619, 3208620);

 OBJECT_ID OWNER      OBJECT_NAME                    SUBOBJECT_NAME                 OBJECT_TYPE
---------- ---------- ------------------------------ ------------------------------ -------------------
   3208618 JSCHDER    BIGTABLE_LOG                   P_2007_09                      TABLE PARTITION
   3208619 JSCHDER    BIGTABL_LG_X_ID                P_2007_09                      INDEX PARTITION
   3208620 JSCHDER    BIGTABL_LG_X_CHANGE_DATE       P_2007_09                      INDEX PARTITION

Now wait just a moment… this isn’t even the object we’re updating!! Well I’ll spare you the details but one of the triggers logs every change to BIGTABLE with about 7 inserts into this one. It’s all PL/SQL so we get bind variables and everything - it’s just the sheer number of accesses that is causing all the contention.

One further thing we can do is actually see which blocks are getting most contended for - the ASH records this too. (Isn’t the ASH great?)

select current_file#, current_block#, count(*) cnt
from dba_hist_active_sess_history
where snap_id between 12831 and 12838
and event_id=1478861578 and sql_id='8v3b2m405atgy'
and current_obj# in (3208618, 3208619, 3208620)
group by current_file#, current_block#
having count(*)>50
order by 3;

CURRENT_FILE# CURRENT_BLOCK#        CNT
------------- -------------- ----------
         1330         238073         51
         1542          22645         55
         1487         237914         56
         1330         238724         61
         1330         244129         76
         1487         233206        120

One thing that I immediately noticed is that there does not seem to be a single hot block!!! (What?) Out of 40,000 sessions accessing these three objects no more than 120 ever tried to hit the same block. Let’s quickly check if any of these are header blocks on the segments.

select segment_name, header_file, header_block
from dba_segments where owner='JHEIDER' and partition_name='P_2007_09'
and segment_name in ('PLACEMENTS_LOG','PLCMNTS_LG_X_ID',
  'PLCMNTS_LG_X_CHANGE_DATE');

SEGMENT_NAME                   HEADER_FILE HEADER_BLOCK
------------------------------ ----------- ------------
BIGTABL_LG_X_CHANGE_DATE              1207       204809
BIGTABL_LG_X_ID                       1207       196617
BIGTABLE_LOG                          1209        16393

No - all seem to be data blocks. Why so much contention? Maybe the RAC and OPS experts out there already have some guesses… but first let’s explore one alternative method to check the same thing and see of the numbers line up.

AWR Segment Statistics

Here’s a handy little query I made up the other day to quickly digest any of the segment statistics from the AWR and grab the top objects for the cluster, reporting on each instance. I’m not going to explain the whole thing but I’ll just copy it verbatim - feel free to use it but you’ll have to figure it out yourself. :)

col object format a60
col i format 99
select * from (
select o.owner||'.'||o.object_name||decode(o.subobject_name,NULL,'','.')||
  o.subobject_name||' ['||o.object_type||']' object,
  instance_number i, stat
from (
  select obj#||'.'||dataobj# obj#, instance_number, sum(
GC_BUFFER_BUSY_DELTA
) stat
  from dba_hist_seg_stat
  where (snap_id between 12831 and 12838)
  and (instance_number between 1 and 6)
  group by rollup(obj#||'.'||dataobj#, instance_number)
  having obj#||'.'||dataobj# is not null
) s, dba_hist_seg_stat_obj o
where o.dataobj#||'.'||o.obj#=s.obj#
order by max(stat) over (partition by s.obj#) desc,
  o.owner||o.object_name||o.subobject_name, nvl(instance_number,0)
) where rownum<=40;

OBJECT                                                         I       STAT
------------------------------------------------------------ --- ----------
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]                    2529540
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]               1     228292
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]               2     309684
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]               3     289147
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]               4     224155
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]               5    1136822
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION]               6     341440
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]        2270221
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]   1     220094
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]   2     313038
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]   3     299509
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]   4     217489
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]   5     940827
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION]   6     279264
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                      1793931
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                 1     427482
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                 2     352305
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                 3     398699
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                 4     268045
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                 5     269230
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION]                 6      78170
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION]                  771060
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION]            1     162296
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION]            2     231141
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION]            3     220573
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION]            4     157050
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                        393663
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                  1      66277
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                  2      10364
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                  3       6930
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                  4       3484
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                  5     266722
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION]                  6      39886
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]                 276637
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]           1      13750
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]           2      12207
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]           3      23522
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]           4      28336
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]           5      99704
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION]           6      99118

40 rows selected.

As an aside, there is a line in the middle that says “GC_BUFFER_BUSY_DELTA”. You can replace that line with any of these values to see the top objects for the corresponding waits during the reporting period:

LOGICAL_READS_DELTA
BUFFER_BUSY_WAITS_DELTA
DB_BLOCK_CHANGES_DELTA
PHYSICAL_READS_DELTA
PHYSICAL_WRITES_DELTA
PHYSICAL_READS_DIRECT_DELTA
PHYSICAL_WRITES_DIRECT_DELTA
ITL_WAITS_DELTA
ROW_LOCK_WAITS_DELTA
GC_CR_BLOCKS_SERVED_DELTA
GC_CU_BLOCKS_SERVED_DELTA
GC_BUFFER_BUSY_DELTA
GC_CR_BLOCKS_RECEIVED_DELTA
GC_CU_BLOCKS_RECEIVED_DELTA
SPACE_USED_DELTA
SPACE_ALLOCATED_DELTA
TABLE_SCANS_DELTA

Now as you can see, these statistics confirm what we observed from the ASH: the top waits in the system are for the BIGTABLE_LOG table. However this also reveals something the ASH didn’t - that the date-based index on the same table is a close second.

The Real Culprit

Any time you see heavy concurrency problems during inserts on table data blocks there should always be one first place to look: space management. Since ancient versions of OPS it has been a well-known fact that freelists are the enemy of concurrency. In this case, that was exactly the culprit.

select distinct tablespace_name from dba_tab_partitions
where table_name='BIGTABLE_LOG';

TABLESPACE_NAME
------------------------------
BIGTABLE_LOG_DATA

select extent_management, allocation_type, segment_space_management
from dba_tablespaces where tablespace_name='BIGTABLE_LOG_DATA';

EXTENT_MAN ALLOCATIO SEGMEN
---------- --------- ------
LOCAL      USER      MANUAL

SQL> select distinct freelists, freelist_groups from dba_tab_partitions
  2  where table_name='BIGTABLE_LOG';

 FREELISTS FREELIST_GROUPS
---------- ---------------
         1               1

And there you have it. The busiest table on their 6-node OLTP RAC system is using MSSM with a single freelist group. I’m pretty sure this could cause contention problems! But in this case it wasn’t quite what I expected. It looks to me like the single freelist itself wasn’t the point of contention - but it was pointing all of the nodes to the same small number of blocks for inserts and these data blocks were getting fought over. But they were probably filling up quickly and so no single block had a large number of waits reported in the ASH. I haven’t proven that but it’s my current theory. :) If anyone has another idea then leave a comment and let me know!