Why Make Charts

I’ve heard Kyle Hailey speak on a few different occasions, and more than once he’s talked about the power of visualizing data. (In fact Kyle was a key person behind Grid Control’s performance screens.)

I couldn’t agree more. I regularly visualize data when I’m working. Two reasons come immediately to mind:

1. It helps me to better understand what’s happening. There have been times when I’ve had an “aha” moment very quickly after seeing the picture.
2. It helps others more easily understand what i’m trying to communicate. It’s great for management reports and such – not because it’s fluff, but because it’s a good communication tool.

Last week, I made a few quick charts as illustrations for a performance report. The process really isn’t that complicated, but I thought I’d put the steps into a blog post… for myself to reference in the future and for anyone else who might find this helpful. :)

Making Simple Charts

This demonstration will use data from the AWR to build graphs. Note that if you run these queries, Oracle legally requires you to purchase the extra-cost “diagnostic pack” license. But similar queries could be written from free statspack or S-ASH tables.

You need multiple data points to make a graph. For this demo, The AWR was configured to take snapshotsd every 30 minutes and I’m looking at a query which ran for about 10 hours. Also, it was the only query running in the instance for most of that time – so I will also look at some instance-wide statistics.

1. The first step is to get any needed parameters for pulling performance data. In the case of the AWR, I will need the INSTANCE_NUMBER, the SQL_ID and the first/last SNAP_ID.

   It’s pretty easy to get this information from Grid Control or from Database Console. But if you don’t have access to the web console then you can still get the info from SQLPlus.

   Here’s a useful query to get an overview of the SNAP_IDs:

   SQL> select to_char(BEGIN_INTERVAL_TIME,'MON YYYY') month,
          min(snap_id) min_snap, max(snap_id) max_snap
        from dba_hist_snapshot
        where instance_number=4
        group by to_char(BEGIN_INTERVAL_TIME,'MON YYYY');
   
   MONTH        MIN_SNAP     MAX_SNAP
   -------- ------------ ------------
   APR 2009        10239        10240
   DEC 2009        28752        31924
   FEB 2010        38115        40939
   MAY 2010        47498        48783
   AUG 2010        54975        55013
   NOV 2010        60979        61986
   DEC 2010        61987        64218
   JAN 2011        64219        66448
   FEB 2011        66449        67803
   MAR 2011        67804        69291
   APR 2011        69292        70731
   MAY 2011        70732        72219
   JUN 2011        72220        73655
   JUL 2011        73656        75139
   AUG 2011        75140        76608
   SEP 2011        76609        78048
   OCT 2011        78049        79536
   NOV 2011        79537        80338
   
   18 rows selected.

   Something similar to this might also be useful:

   SQL> select snap_id,instance_number,begin_interval_time,snap_level
        from dba_hist_snapshot
        where begin_interval_time between '11-nov-11 17:30' and '11-nov-11 19:00'
          and instance_number=4
        order by snap_id, instance_number;

   For this demo I’m going to use INSTANCE_NUMBER 4 and SQL_ID 8suhywrkmpj5c between snaps 80298 and 80318.

2. Now create a new spreadsheet in your office suite. I use the free OpenOffice spreadsheet application, but Excel or iWork Numbers should work pretty much the same.

   In the second row of the new spreadsheet, enter the time of the first snapshot you’re going to analyze. In the third row, enter this formula:

   = A2 + 1/24/60 * [minutes between snaps]

   Select several rows below this formula and select Edit > Fill > Down to copy the formula to the following rows. Repeat this until you have reached the end of your analysis window.

   

3. Open a SQLPlus session. We will copy-and-paste directly from SQLPlus into the spreadsheet.

   Use a SQL like this to retrieve data for one system statistic:

   set pagesize 999
   col value format 999999999999999
   
   select value from dba_hist_sysstat
    where instance_number=4 and snap_id between 80298 and 80318
      and stat_name='transaction tables consistent read rollbacks'
    order by snap_id

   You can copy the statistic name directly from an AWR report of there’s a certain stat you’re interested in. You can find more information about system statistics in Oracle’s docs.

   Now move right to the next empty column. First, copy the name of this statistic into the first row. Then, in the second box, enter a formula to find the difference between its left peer and the left upper peer. For cell C3, the formula is B3-B2. Choose Edit > Fill > Down again, as before.

   = B3 - B2

   

   You can repeat this step to access further system statistics. You can also create another column where you divide or multiply each other.

   

4. The previous SQL statement retrieved system statistics. Another easy query runs against historical V$SQLSTAT snapshots. (This only works for long-running queries.)

   SQL> select BUFFER_GETS_TOTAL value from dba_hist_sqlstat
        where instance_number=4 and snap_id between 80298 and 80318
          and sql_id='8suhywrkmpj5c'
        order by snap_id

   Once again, you can read about the available fields and statistics in the oracle docs. You can repeat this step to quickly get additional statistics for a particular SQL, and you can then combine some stats for better graphs.

   

That’s it! I know, really not that complicated. I hope it’s helpful. :)

Post Script

For anyone who’s curious, the charts in this article are related to a SQL report which was recently discussed on the mailing list.

Our project was very successful – the key changes had dramatic effects and we are now running the same set of reports much faster.  (Some reports went from many hours to less than 30 minutes.)  My final project summary included a small section where I passed along a few suggestions specifically for these developers – but looking over the suggestions now, I think they might actually be good tips for a wider audience.  At any rate it might be interesting to put them out for comments!

Of course these tips aren’t my own original work; in fact I think they’re pretty widely accepted in the Oracle professional community.  We’ve been hearing these ideas for years now at conferences and user groups.  I just picked a few that could make the biggest impact for this team right now – and I put the ideas in my own words.  These were my top three:

• Aggressively search for opportunities to do less computational work.
  • Fewer switches between SQL, PL/SQL and Java.
  • Java is slowest, PL/SQL is faster, C and SQL are fastest.
  • Operate in bulk whenever possible.
    • Avoid using any kind of loop in your code whenever possible. If loops are required, operate on large chunks at a time (for example, by increasing fetch sizes).
    • Avoid coding in multiple steps whenever possible. If multiple steps are faster, ask how you’re code works and why Oracle isn’t automatically doing the same.
• Always utilize tracing/logging when coding and tuning. Create logs at the most granular level possible, then summarize the results by time (aka profiling). Calibrate everything in terms of wall clock elapsed time for entire job.
  • Utilize database 10046 log files for any jobs where the database is an important component. There are several profilers available with various feature levels and price points (starting with free); utilize one of these to summarize the logs. These logs can safely be generated in production.
  • Use the Oracle PL/SQL Profiler to generate profiles on critical path PL/SQL code.
• When tuning SQL, follow these steps:
  1. Remove all hints.
  2. Use dbms_xplan, 10046 and 10053 traces – in that order – to determine where the CBO went wrong. Compare CBO row count estimates with actual row counts before time-consuming, in-depth analysis of 10053 logs.
  3. Add hints only with justification about why the hint is preferable to fixing underlying cause. Generally speaking, hints should be viewed as band-aids.
  4. When a good plan is found and underlying causes cannot be addressed, use DBMS_XPLAN functions with the ADVANCED flag to retrieve a full set of hints. Use this to lock in the good plan and document the situation with SQL comments. (11g will offer new options for plan management, but this method will be sufficient for 10g.)

I’m sure that these tips aren’t perfect – so please share your thoughts!  How could I better word them?  Do they apply to your development team?

Post-Script

I’m definitely no guru of Oracle performance yet, but I have been involved in a number of performance-oriented projects. My experience has been that it’s pretty common to have certain assumptions built in to the project through scoping and goal statements before I arrive. Often, project sponsors have already decided that one of the most important items in a performance project is that someone with experience in storage and system configuration for Oracle (this is usually me) will review settings and statistics for things like database parameters, AWR reports, operating system kernel settings, networking, I/O, etc.  The contractual “statement of work” might say something like “review and assess configuration and architecture” with a list of specific areas of configuration.

I’m not opposed to periodic configuration audits or having a second set of eyes look over the setup.  But this very customer was a perfect example of how these assumptions about performance tuning through system review really are not the best way to work.  Due to the many parties involved in this contract and various expectations, it was important for me to review the system configuration – so I did spend time on this review.  However I was lucky that the team and managers who I directly worked with were sharp and open-minded. After some spirited discussion we slightly altered our approach: starting at the top of the stack and working down rather than the other way around.

Our first two weeks were focused on the most important business reports and only during the last two weeks did I take an in-depth look at storage and system configuration.  Can you guess what happened?  Looking at the top business problems led us straight to the most important bottlenecks impacting reports across the board.  It also showed me which areas of system configuration could have the biggest impact on the business.  By the time I started doing a system review, I already knew which areas to focus on.  Furthermore, I could already make some estimations about best-case impact of system changes on overall report run-times.  Not surprisingly, I knew that no system change was going to give us the improvement we wanted – but tuning a few small pieces of code that were used widely gave us even more improvement than we had thought possible.

Again, this isn’t new… it’s stuff we’ve been hearing at conferences and user groups for years now.  But it seems to me that it’s still worth repeating.

However: please notice the Security Addendum at the end of this article.

Of course if you’re lucky enough that your developers use certain tools from Oracle, then there are some slick 3rd party plugins that will help download and manage tracefiles for you. But what if your developers don’t want to add a whole new environment to their already memory-bound workstations? What if there are corporate policies making this difficult – such as a time-consuming review and approval process for any new software installs?

Wouldn’t it be nice of Oracle had a system view that your developers could just QUERY to find tracefiles? Maybe something like this (which of course displayed data in real time, reflecting the current status of the filesystem):

SQL> desc all_trace_files
 Name                          Null?    Type
 ----------------------------- -------- --------------------
 DIRECTORY                              CHAR(5)
 FILE_NAME                              VARCHAR2(400)
 FILE_TIME                              DATE
 FILE_BYTES                             NUMBER
 FILE_TRACEID                           VARCHAR2(100)
 FIRST_TIME                             DATE
 FIRST_ACTION                           VARCHAR2(100)
 FIRST_MODULE                           VARCHAR2(100)
 FIRST_SERVICE                          VARCHAR2(100)
 SID                                    NUMBER
 SERIAL                                 NUMBER
 ERROR                                  VARCHAR2(400)

Sure, this looks nice – maybe in the next version of Oracle. When you upgrade to it, 10 years from now. But what about NOW – your old 10g database running on a huge mainframe?

We all know how nice it would be if your developers run a query like this… today… on almost any database:

SQL> select directory, file_name, file_traceid, first_action from all_trace_files
  2  where first_action is not null;

DIREC FILE_NAME                                FILE_TRACE FIRST_ACTION
----- ---------------------------------------- ---------- ----------------------------------------
UDUMP ardentp1_ora_12386502.trc                           Test Window - New
UDUMP ardentp1_ora_22347974.trc                           Test Window - New
UDUMP ardentp1_ora_22675498.trc                           SQL Window - xml_file_writing_te
UDUMP ardentp1_ora_24248536.trc                           SQL Window - test_cases.sql
UDUMP ardentp1_ora_24641726.trc                           DEQ
UDUMP ardentp1_ora_2621852.trc                            Test Window - New
UDUMP ardentp1_ora_3342766.trc                            problemTbsp
UDUMP ardentp1_ora_4063428.trc                            SQL Window - New
UDUMP ardentp1_ora_45350944.trc                           Debug Test Window - New
UDUMP ardentp1_ora_45547538.trc                           Main session
UDUMP ardentp1_ora_48103606.trc                           SQL Window - SELECT * FROM ext.e
UDUMP ardentp1_ora_50593842.trc                           SQL Window - test_cases.sql
UDUMP ardentp1_ora_50790578.trc                           SQL Window - New
UDUMP ardentp1_ora_51380240.trc                           Debug Test Window - New
UDUMP ardentp1_ora_52429026.trc                           UserBlock
UDUMP ardentp1_ora_52691004.trc                           SQL Window - New
UDUMP ardentp1_ora_60817584.trc                           Test Window - New
UDUMP ardentp1_ora_61538330.trc                           Test Window - New
UDUMP ardentp1_ora_62783512_schn_02.trc        schn_02    Test Window - New
UDUMP ardentp1_ora_62783512_schn_03.trc        schn_03    Test Window - New
UDUMP ardentp1_ora_6291778.trc                            Test Window - New
UDUMP ardentp1_ora_63176754.trc                           Test Window - New
UDUMP ardentp1_ora_66388158.trc                           Test Window - New
UDUMP ardentp1_ora_8192480.trc                            Test Window - New
UDUMP ardentp1_ora_8651144.trc                            SQL Window - test_cases.sql
BDUMP ardentp1_m000_12517418.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_20119784.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_24248408.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_45154330.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_mmnl_52560064.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_mmon_52756580.trc                          Monitor Tablespace Thresholds

31 rows selected.

And of course – last but not least, you’d like your developers to actually be able to download the tracefile themselves, right? Actually, this has already been done – Dion Cho blogged some sample code about two and a half years ago. He has a very elegant solution using a pipelined PL/SQL function and UTL_FILE.

http://dioncho.wordpress.com/2009/03/19/another-way-to-use-trace-file/

Just one minor tweak would be needed, to grab files from more than one directory. (In addition to your session traces, you might need to get logs from parallel query slaves or DBMS_JOB/SCHEDULER processes.)

Not only can you write queries to your heart’s content on this, but just about every developer environment on the planet can easily take this query output and save it to a file – then they can run the profiler of their choice on the tracefile. That would be cool, right?!

SQL> select * from table(textfile('ardentp1_ora_52691004.trc','UDUMP')) where rownum<20;

COLUMN_VALUE
-------------------------------------------------------------------------------------------
/u0001/app/oracle/admin/ardentp/udump/ardentp1_ora_52691004.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u0001/app/oracle/product/db/10.2.0/ardentp
System name:    AIX
Node name:      ardent4
Release:        1
Version:        6
Machine:        00F66BE22C00
Instance name: ardentp1
Redo thread mounted by this instance: 1
Oracle process number: 59
Unix process pid: 52691004, image: oracle@ardent4

*** ACTION NAME:(SQL Window - New) 2011-08-15 16:35:24.791
*** MODULE NAME:(PL/SQL Developer) 2011-08-15 16:35:24.791
*** SERVICE NAME:(ardentp_dba) 2011-08-15 16:35:24.791
*** SESSION ID:(572.24734) 2011-08-15 16:35:24.791

19 rows selected.

Well I AGREE that this would be cool! And as it happens, the customer I’m working with now would find it very useful. And honestly… it’s really not that complicated to code something like this. So last night I stayed late a few hours to write it up and test it. It’s mainly build on a pipelined java function, and of course a minor tweak on Dion’s code for getting the file.

Basically, every time you query the view it scans the first 30 lines of all files in the bdump and udump directories for timestamps and identification information (module, action, service, sid, serial). This information is combined with basic file stats (name, size, last-modified-date) and the tracefile_identifier is calculated from the filename. Any errors encountered are reported in the error column (file permissions, etc).

You can download the entire thing at my github repo (specifically, the files tracefile_find.txt and tracefile_get.txt). All examples in this blog post are from a real system – nothing here is cooked, except that I changed database and host names to protect the innocent.

Enjoy!

Security Addendum

As pointed out in the comments after I published this post, there are some important security considerations with allowing access to trace files. A good explanation can be found in Pete Finnigan’s blog article: Is it possible to steal data with just ALTER SESSION? To summarize: if a clever hacker is able to access tracefiles then it is possible for them to see things that the database normally protects. This certainly includes table data, and it may even include passwords for other database accounts – sometimes even in cleartext. This is very important if you’re considering any tool that grants access to tracefiles – including the scripts I’ve published here. Although he generally disapproves of any tracefile access, Pete’s comment suggested a way to reduce the risk. The second table (which actually gives tracefile contents) can be limited to DBAs and then developers can get a view which only shows files that the DBAs explicitly grant. I think that the most useful part of my program is the first view anyway, which scans tracefiles and lets you use SQL to search on module/action/service.

One further quick postscript… lest you think I was terribly irresponsible with this client, we did consider security implications before endorsing this script. Admittedly I didn’t consider every scenario that Pete has written about, but I do still think we made an appropriate decision. (Also… the script wasn’t yet installed into production when I left the client, because it had to go through an approval process itself. ) In this particular case, the small group of developers getting access to tracefiles were people who already had full access to all data in the database and they had passwords for all key application accounts. Although they didn’t have SYSDBA passwords or shell access, they essentially had everything else – including all the data. Furthermore, they were in a stage of performance troubleshooting where their productivity could be significantly increased with the ability to run many traces and quickly access the tracefiles (think Method R). And finally, these developers were on the same small, tight team as the DBAs who took care of backups and patching, all sitting together in one corner of the office building. The DBAs were involved in the decision and seemed to find their reduced workload a very positive thing – especially since they know all the developers who would get access.

One could certainly comment on the security policies of this client, and their decisions about balancing trust and productivity. But I do not think that we were circumventing the company security policies in this case. And certainly my scripts are not intended to support such activity. Nonetheless, I do think there are cases where this is a very useful script. It’s much more convenient for developers who – even if they can get shell access – may not know much about navigating unix and are more comfortable working in their SQL development environment.

• 100% class met expectations, would recommend to others

• 66% easy to follow, could use skills in working environment

• 100% already familiar with oracle, 90% use oracle daily

• 0 negative reviews of instructor (phew!)

• 1 negative review of curriculum: said practice exercises weren’t relevant but training manual was still above average.

• 0 negative “comments”

There were several positive comments such as this: “I would recommend this class to others. This setup is perfect to pick up new skills and expose what ifs w/out worrying about pressing the wrong button.”

There were also some suggestions for improvements which I’ve taken note of. The most common suggestion was this: “More time to practice / explore new concepts.”

What’s Next

The RAC Attack curriculum has been around for over three years now – and I’m still getting requests to put it on.  Seems that a number of people have felt that it’s a decent quality class and handbook.  I talked to several other Oracle instructors and speakers at Collaborate this year about the RAC Attack curriculum – including an in-depth conversation with Arup Nanda.

As a result of these conversations, I did some careful thinking and I’ve been very busy for the past few weeks since Collaborate.  And I now have three major announcements about RAC Attack:

1. Last week, I published an updated RAC Attack Lab Handbook.  The handbook is now completely self-contained for home use.  I’ve added significant new content:
   • Detailed hardware requirements (taking into consideration optimal I/O and memory configuration)
   • How to install and setup VMware environment (including networking and storage)
   • How to create the RAC Attack Classroom DVD (this is actually scripted now)

This means that you can now (finally) setup RAC Attack on your own computer – without needing any external assistance!

2. Although I created most of the labs for RAC Attack, some labs were also contributed by Dan Norris and Parto Jalili.  Last week I also obtained their permission to move all RAC Attack content under a Creative Commons Attribution-ShareAlike license – which you will see in the latest handbook.The RAC Attack lab handbook has now been donated to the Oracle professional community with a solid copyleft license.
   • New DBAs are invited to use this material at home.
   • Experienced DBAs are invited to make corrections and enhancements to the textbook.  Also you can add your own labs which work on the RAC Attack “platform”.
   • Instructors are invited to use this material partially or completely in their own classes, as they wish. You can even run your own class based solely on this material – without needing anyone’s permission. Also, you can charge for these classes – there isn’t a “non-commercial” limitation on the terms of our license.

On a related note, all code related to RAC Attack (including the jumpstart scripts and supplemental DVD build script) is now under the GPL and has been published on github for easy downloading, copying and branching.

3. Finally… how are you supposed to make corrections, enhancements and additions if you’ve only got a PDF version?  Right now the textbook sources are in OpenOffice format (with a master document and about 50 lab documents).  Although this is very well suited to print media, it’s difficult to coordinate with very many authors or editors.  I’ve decided that the best way to truly encourage collaboration with other teachers & instructors is to use the best platform on the web for collaborative open-content textbook projects: the RAC Attack lab textbook has a new home on wikibooks.

   http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home

   I’m just getting started now – but I’m hoping to have all existing RAC Attack content on this wiki textbook by the end of the month.  Please have a look at the project page and consider getting involved!

Feedback

I consider these to be some major changes.  I’ve put some careful thought into them, and I’m excited about the new direction for this class.  Let me know what you think.

Everything started with this short, innocent-seeming instant message:

there is something wrong with a 10g server config. connecting to it from windows works, the linux instant client returns ORA-28547. the same linux client can connect to 9g installation though. google is not helpful, does anyone have an idea?

Seems like a simple question! But we ended up chatting on-and-off throughout the whole day as I helped troubleshoot the problem. We finally found the problem at the end of the day – and I promise you that it would have taken Oracle Support weeks (or even months) to find this problem. Yikes!

Troubleshooting Summary

The client was php on a linux box. This client was connecting to the database server over a network connection which includes a VPN tunnel. The database is being upgraded from 9i to 10g – and the new database is already running on a new server. The client can connect to the 9i database but it can’t connect to the 10g database. The developer also has a windows box with an oracle client; this windows box can connect to the 10g database over the network link just fine. TNSNAMES is not used, and all connect strings are identical except for the IP addresses. The 11g instant client for Linux is being used.

Our investigation process involved a lot of trial-and-error. We repeated this cycle: gather data, postulate a theory, suggest an action. We knew that we were close because often the error messages would change – but they didn’t go away! For posterity (and google), here are the errors we encountered:

• ORA-28547: connection to server failed, probable Oracle Net admin error
• ORA-12170: TNS:Connect timeout occurred
• ORA-03113: end-of-file on communication channel
• WARNING: inbound connection timed out (ORA-3136)
• Fatal NI connect error 12170.
• TNS-12535: TNS:operation timed out / ns secondary err code: 12606
• ntt2err:Read unexpected EOF ERROR on 7
• ORA-12537 or TNS-12537: TNS: connection closed / ns2: 12560

We started with simple changes before looking into deep logs or tracefiles. I didn’t have any access to the actual system, just a chat connection to the developer. And of course I was hoping for a quick fix… though it didn’t end up happening. But FWIW, here’s everything we tried:

• Double-check that TNSNAMES or connection string is correct
• If RAC, then check REMOTE SERVER connect string in LSNRCTL output (this comes from local_listener/remote_listener settings)
• Add server hostname to local client DNS. (Connect string used IP, but server might pass back hostname for re-connect during connection process.)
• Check if DB server is behind a firewall, client seeing IP address different than server – see metalink note 270096.1 (can possibly address on 10g+ with DIRECT_HANDOFF_TTC_ in listener.ora or USE_SHARED_SOCKET in windows registry)
• Attempt to disable shared-socket/direct-handoff. (Trace revealed that the working 9i connection didn’t bequeath its connection but broken 10g connection did.)
• Downgraded instant client from 11g to 10g (to match target database)
• Check if there’s a software firewall active on the database server

To gather data, of course we first just captured error messages from the client. Later, we checked the server sqlnet.log, alert log, bdump/udump, and windows OS event log for messages. The developer I was working with was very savvy, so I actually had him generate some client trace files fairly early on – from both working and non-working connections.

We heavily leveraged the two working connections (linux->9i and windows->10g) by comparing tracefiles. In the end, these tracefiles tipped us off to the real problem.

Infuriating

I expected a quick, easy answer – but it wasn’t. It drove me crazy all day.

From the beginning it was obviously some kind of networking problem. The database was running on a windows server, so my first thought was that it was related to the re-connect that happens when the listener hands off the connection to the database process. But the trace file revealed that 10g was actually bequeathing (or rather, sharing) the connection – shame on me for not keeping up to speed with Oracle/Windows developments!! (I usually work with unix environments.)

Generating a client trace is easy – you just edit (or create) the text file sqlnet.ora and add two lines TRACE_DIRECTORY_CLIENT and TRACE_LEVEL_CLIENT (detailed description). If you’re using the instant client then you can set the environment variable TNS_ADMIN to the directory containing this file.

Here are a few lines from the client trace we generated:

2010-09-07 20:41:00.451918 : nscall:connecting...
2010-09-07 20:41:00.452854 : nsopen:opening transport...
2010-09-07 20:41:00.453086 : nttcni:Tcp conn timeout = 60000 (ms)
2010-09-07 20:41:00.504377 : nttcni:connected on ipaddr 10.10.0.97
2010-09-07 20:41:00.506948 : nscon:doing connect handshake...
2010-09-07 20:41:00.506979 : nscon:sending NSPTCN packet
2010-09-07 20:41:00.507113 : nttwr:socket 7 had bytes written=233
2010-09-07 20:41:00.567169 : nttrd:socket 7 had bytes read=8
2010-09-07 20:41:00.567434 : nscon:got NSPTRS packet
2010-09-07 20:41:00.567692 : nscon:sending NSPTCN packet
2010-09-07 20:41:00.567817 : nttwr:socket 7 had bytes written=233
2010-09-07 20:41:00.617139 : nttrd:socket 7 had bytes read=32
2010-09-07 20:41:00.617362 : nscon:got NSPTAC packet
2010-09-07 20:41:00.617453 : nscon:connect handshake is complete
2010-09-07 20:41:00.620093 : nsdofls:sending NSPTDA packet
2010-09-07 20:41:00.620253 : nttwr:socket 7 had bytes written=156
2010-09-07 20:41:00.667129 : nttrd:socket 7 had bytes read=127
2010-09-07 20:41:00.667359 : nsrdr:got NSPTDA packet
2010-09-07 20:41:00.670112 : nttfpwr:socket 7 had bytes written=47
2010-09-07 20:41:00.717287 : nttfprd:socket 7 had bytes read=181
2010-09-07 20:41:00.719612 : nttfpwr:socket 7 had bytes written=2011
2010-09-07 20:41:00.720401 : nttfpwr:socket 7 had bytes written=666
2010-09-07 20:42:01.577217 : ntt2err:Read unexpected EOF ERROR on 7

The presence of NSPTRS indicates that the connection is being bequeathed/shared on the server. Metalink notes 270096.1 and 1007807.1 are very helpful for understanding this handshake process. Here’s a brief outline of what’s happening:

One other important piece of information we eventually unearthed was this error in sqlnet.log on the server:

Fatal NI connect error 12170.
...
  Tns error struct:
    ns main err code: 12535
    TNS-12535: TNS:operation timed out
    ns secondary err code: 12606

Both the server and client were seeing the connection spontaneously die! Yet the client connects fine to a 9i database, and a different client connects fine to this database!! Right about now, I was thinking of two possible root causes:

1. Rogue firewall somewhere that’s killing the connection for some unknown reason. But the developer is pretty sure that there’s no firewall killing connections.
2. Bug?!

Breakthrough

At this point I asked the developer to run a level 16 client trace (which includes packet dumps) and I started reading through the dump. Then I noticed two important things.

1. There is a 60 second pause before the client detects the dead connection. This is equal to the TCP connection timeout. I suddenly realized that the client “EOF” error was not from the server terminating the connection – the client was timing out while waiting for the server. And then I realized that the server was doing the same thing – waiting for the client!
2. I looked a lot closer at a certain section of the tracefile that immediately precedes this timeout. In particular, I looked at the packet contents and sizes – and I compared with the working 9i connection.

Here’s a snip from the diff between the working 9i connection and the broken 10g connection:

 nttfpwr:socket 7 had bytes written=47
-nttfprd:socket 7 had bytes read=164
-nscontrol:Vect I/O support: 02010-09-07 20:48:06.447554 : nscontrol:cmd=44, lcl=0x0
-nttfpwr:socket 7 had bytes written=1080
-nttfprd:socket 7 had bytes read=949
-nttfpwr:socket 7 had bytes written=213
-nttfprd:socket 7 had bytes read=22
+nttfprd:socket 7 had bytes read=181
+nscontrol:Vect I/O support: 02010-09-07 20:52:12.267498 : nscontrol:cmd=44, lcl=0x0
+nttfpwr:socket 7 had bytes written=2011
+nttfpwr:socket 7 had bytes written=666
+ntt2err:Read unexpected EOF ERROR on 7

When I looked at the packet contents for the broken connection, it was immediately evident that those TWO packets (2011 bytes and 666 bytes) were ONE message. The message was too large for one packet so it had been fragmented.

However the first big message sent to the 9i database is 1080 bytes. It gets sent in a single packet.

I asked the developer, what’s the max packet size on the Linux box? He replied that the MTU was 1456.

WILD guess… I wondered, was the big packet getting dropped somewhere? (Windows client trace shows the large packet getting through without any problem, but still…)

Oracle is Better than MySQL

…ok, maybe the standard MySQL client has this functionality too – I actually don’t know. I just wrote that title because it’s provocative!! But when it comes to tweaks, Oracle really does shine. It has more switches and knobs than you’d believe.

It’s very simple to reduce the max packet size in the Oracle client: just add “(SDU=####)” to the connect string. The default is 2048 – which explains the fragment sizes we observed.

Here’s what transpired next in my chat (please excuse the language):

f**k me
just tried with 1024 and it worked

Seems that we identified the problem! And most importantly, we have a working connection so that this developer can get back to what’s important – coding.

Of course, we all know what happened after that – virtualization.  It didn’t take long before my home-built test clusters were running on VMware.  (Personally, I think that virtualization really started because of those NES and SNES emulators.  Most great achievements start with a geek who wants to play more video games.)  There are lots of people now who run RAC on virtual environments and it’s easy to find tutorials on the web for many different OS and VM combinations.

Low-Memory Linux

Something I haven’t seen many other people do is RAC with a very small memory configuration.  Like 760M of memory per server. (!)  Of course you’d only do this for a hobby setup – never on a system where you want any kind of support.  But I’m kinda cheap… and running RAC on these small VMs means that I don’t have to go buy an expensive new home computer.  My current gateway laptop with Vista Home does the job quite nicely!

10.2 and 11.1 RAC will install and run on servers with 760M of memory. But things were a little unstable at first. Now I’m the curious type… I like to fiddle with things… so I investigated a little bit.

Basic Unix Investigation

There are two basic investigation scenarios:

In this particular case, I noticed pretty quickly from the top utility that one process was consuming over 30% of the system’s memory!  (Note: in top, you can press the ‘<’ and ‘>’ keys to move the sort column left and right.  The initial sort column is %CPU.  I moved it one column to the right, sorting by %MEM.)

top - 18:41:13 up  5:28,  3 users,  load average: 0.04, 0.35, 0.60
Tasks: 180 total,   2 running, 178 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  3.7%sy,  0.0%ni, 89.4%id,  6.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    767020k total,   754296k used,    12724k free,     7084k buffers
Swap:  1540088k total,   654696k used,   885392k free,   361800k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17435 oracle    RT   0  230m 229m  31m S  0.3 30.6   0:26.21 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
 7765 oracle    15   0  464m 111m 102m S  0.0 14.9   0:04.52 ora_smon_RAC1
 7743 oracle    -2   0  445m  90m  83m S  0.0 12.1   0:12.01 ora_lms0_RAC1
 7783 oracle    15   0  440m  70m  66m S  0.0  9.4   0:05.35 ora_mmon_RAC1
20321 oracle    15   0  438m  48m  45m S  0.0  6.5   0:00.88 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8676 oracle    16   0  437m  46m  45m S  0.0  6.2   0:02.69 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8801 oracle    15   0  440m  44m  41m S  0.0  6.0   0:04.57 ora_cjq0_RAC1

This process (ocssd) is Oracle’s Cluster Synchronization Services Daemon.  It’s the process that sends and receives heartbeats from other nodes. Any delays sending or receiving those heartbeats can cause node evictions (a.k.a. server reboots) – so it’s a pretty important process! That’s why it runs with realtime (RT) scheduling priority, as you can see in the above output from top.

I was surprised that CSS uses so much physical memory – usually Linux is very good at memory management. In top, the VIRT column shows how much total memory each process is using, while the RES column shows how much actual physical memory Linux has allocated to it. It’s clear that Linux is pretty actively managing the physical memory for other processes.

A quick glance at vmstat shows that although we are actively swapping, it seems under control. This is about what I’d expect when we’re idle and all of the processes except CSS are sharing only 500M of memory:

collabn1:/home/oracle[RAC1]$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 604144  14024  34948 300632    0    0    32    86 1044 1797  1  8 84  7  0
 2  1 603912  11224  34968 303968   87    0   763    81 1083 1902  3 11 56 30  0
 0  0 604840  12660  34912 303904   25    0   253    38 1050 1975  2 11 69 18  0
 0  0 604840  12728  34924 303900    0    0    68   106 1043 1975  1  9 82  8  0
 0  0 604784  14536  34936 304140   19    0    89    71 1043 2042  1  5 81 13  0
 1  0 604752  14168  34944 304496    6    0   119    21 1040 2016  2  8 80 11  0
 0  1 604736  18732  35020 306212    0    0   384    73 1050 2489  1 10 70 19  0
 1  1 604736  11144  35232 311252  104    0  1209   128 1074 2024  2 12 38 47  0
 3  1 607900   8056  30788 307352   77 1500   504  1661 1055 2642  9 16 56 19  0
 0  0 607900   9836  30800 307360    0    0    71    70 1031 1798  1  6 85  8  0
 1  0 607884  10536  30812 307400    8    0    44    93 1031 1832  1  4 87  8  0

The SO column tells us when memory is written to disk (and removed from physical). The SI column tells us when memory is read from disk (and put back in physical). On a side note, remember that on a healthy Unix system the free memory is always small. Sometimes this is confusing at first.

Linux Process Memory Investigation

Nonetheless, I’m not happy that CSS is using 30% of my physical memory in this highly-constrained hobby environment. Why is Linux allowing this? The first clue comes simply from the output of the familiar unix ps utility:

collabn1:/home/oracle[RAC1]$ ps v -C ocssd.bin
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
17435 ?        SLl    0:08      7   588 235331 234840 30.6 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin

On Linux, the “v” flag tells ps to give information relevant to virtual memory. TRS and DRS tell me how much physical (resident) memory is used for machine executable code (text) and data, respectively. But more importantly – the STAT column gives some informative BSD-style flags about the process. That capital-L indicates that CSS has some pages that are locked into physical memory. Bingo.

If I have root access, then I can get a very detailed report on process memory usage with the pmap command. The output was a little long, so I’ve abbreviated it here:

[root@collabn1 ~]# pmap -x 17435
17435:   /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
Address   Kbytes     RSS    Anon  Locked Mode   Mapping
00110000     656       -       -       - r-x--  libhasgen11.so
001b4000       8       -       -       - rwx--  libhasgen11.so
  ... 37 more library blocks
02a0d000     100       -       -       - rwx--    [ anon ]
02a26000       4       -       -       - --x--    [ anon ]
02a27000   10240       -       -       - rwx--    [ anon ]
03427000       4       -       -       - --x--    [ anon ]
03428000   10240       -       -       - rwx--    [ anon ]
  ... 10 more anonymous blocks, half are 10240K
08048000     592       -       -       - r-x--  ocssd.bin
080dc000       4       -       -       - rwx--  ocssd.bin
  ... 30 more anonymous blocks, half are 10240K
bfe40000     148       -       -       - rwx--    [ stack ]
bfe65000       8       -       -       - rw---    [ anon ]
-------- ------- ------- ------- -------
total kB  235920       -       -       -

Interestingly, the linux pmap utility does not indicate any locked memory! I don’t know whether that output column is non-functional or if it refers only to some particular kind of locking. But at any rate, I know something is locked. I couldn’t think of anything better, so the next place I looked was in the Linux /proc pseudo-filesystem.

collabn1:/home/oracle[RAC1]$ grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Now we’re talking. The process has a total of 235920 kB of memory – and it’s ALL locked. On a normal RAC system you’d want this. Generally, important realtime processes should be locked so that they are never delayed by paging or swapping. (Remember how that could cause node reboots?)

But I personally doubt that all of the memory really NEEDS to be locked, and I think that Linux will actually do a decent job of not swapping the most important parts. And my highly constrained hobby environment will probably run much smoother if Linux has more flexibility when managing a measly 760M of memory.

Unlocking Linux Process Memory

But is it actually possible to unlock the process memory? As far as I know, Oracle provides no option to disable CSS memory locking. (For good reason.) There is the system call munlockall() – which unlocks all of a particular processes’ memory. But the CSS process itself would have to call this function. And of course it will not. Or will it?

If you’ve got root, then there’s a hacker-back-door way of doing this. Remember, you’d be crazy to try this anywhere besides a dark closet at home. And if you type too slow then CSS could reboot your machine.

But watch this…

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

[root@collabn1 ~]# gdb -p 17435 <<EOF
> call munlockall()
> quit
> EOF

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Attaching to process 17435
Reading symbols from /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin...done.
  ... 17 more Reading/Loading symbols.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f629f0 (LWP 17435)]
  ... 19 more New Threads.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libnsl.so.1...done.
  ... 13 more Loaded/Reading symbols.
0x008e7402 in __kernel_vsyscall ()
(gdb) $1 = 0
(gdb) The program is running.  Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
Detaching from program: /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin, process 17435

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:         0 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Ha. This is something you won’t find on metalink. After generating some activity on the system, the top utility shows me that Linux has significantly reduced the physical memory used by CSS.

These days I’ve actually scripted this for my home and classroom VM environments. I haven’t done a careful comparison or analysis, but it really has seemed to me that my low-memory Linux systems run noticeably smoother.

DBCA is not missing by mistake – this is a change in Oracle 11g Release 2.  There is a new server-based java GUI tool for ASM management: ASMCA.

A Few ASM-related 11gR2 Changes

You might remember that there haven’t been major changes in Oracle clustering since version 10gR1 – when Oracle introduced their Clusterware package. But 11gR2 brings big changes and big new features to database clustering.

Here are a few that I found very noticeable when installing a cluster:

New Tool: ASMCA

• You can’t defer storage configuration until after CRS installation.  Furthermore, raw/block storage is no longer supported.  Before you begin the CRS install you must either (1) install and configure a Cluster Filesystem or (2) prepare storage for ASM.
• The CRS install includes ASM.  This means that the bare-minimum space requirements are higher by about 3 or 4 GB.  (I used to run ASM and DB from a single home for VMware test/educational environments.)
• You can’t do any ASM operations when connected as sysdba!  A new role called “sysasm” was introduced in 11gR1… it’s now the only way to manage ASM.  Connect / as sysasm.
• There’s a new tool called ASMCA to manage ASM.  DBCA isn’t even installed in the ASM/Grid home.  This tool is pretty slick – it also manages the new ASM dynamic volumes and ASM cluster filesystems.

I noticed that there wasn’t much on google about ASM and DBCA.  Hopefully this will save you a few minutes of head-scratching if you’re trying out 11g Release 2 for the first time.  Happy testing!

The Thursday and Friday before OpenWorld (September 16-17), some kind folks in Michigan are bringing many of the best and brightest minds (see them here and here) to the Midwest to share their experience.  It’s like attending the best sessions from San Francisco – for less than a third of the cost.  The event is being called the Michigan OakTable Symposium or MOTS.

If anyone lives around the midwest and won’t be at OpenWorld, then I think this would be especially good.  It’s only a two day event!  But of course I’m sure there’ll also be folks who go to the Michigan Symposium and still catch a last-minute flight to San Francisco…

I’ve always been very enthusiastic about local user groups and independent talent, so this is really my kind of event.  I’m pretty sure that any enthusiasm you hear will be based solely on proven real-world usefulness – with no influence from sales commissions (direct or indirect).  It’ll probably be just as much relational as it is technical.  You can bring an especially hard problem that you can’t solve – maybe someone will have a useful tip or new approach.  But taking home an “answer” is insignificant compared with taking home a better problem-solving process.  If I were you, I’d come to these sessions looking for that.

Anyone in Illinois who is heading over for this?  I think it would be worth the drive (only 4 hours from Chicago).  Maybe we could even arrange for Illinois folks to hang out together at some point.

Reminder – seating is limited and prices go up Sunday.  If you are interested, then you might want to try for management sign-off and event registration by tomorrow.

What?!! You call this a volume manager and it doesn’t support a hot spare disk?! That’s it, we’re going back to filesystems!

Not so fast.

You’ll remember that the purpose of a hot spare disk is to restore protection in the event of a disk failure. When the RAID controller or volume manager detects a failed disk (R.I.P.), it uses the spare and reproduces a copy of the complete now-missing disk (R.I.P.) by reading the mirror or parity data.

Restoring Protection Without A Hot Spare Disk

In fact, ASM does even better. Instead of a hot spare disk, it has hot spare capacity – spread across all of the disks. There are two immediate effects of this: (1) no loss of potential performance from a usable disk head sitting idle and (2) the hot spare capacity is available for an emergency situation (like the archivelog volume filling up).

How is this possible? The key of course is understanding how ASM mirrors data – by extent rather than disk. So if I allocate three extents on disk 1, the first could be mirrored on disk 10, the second on disk 11 and the third on disk 12.

Now if disk 1 fails, I don’t need to rebuild the disk – I only need to ensure that each extent is mirrored. For example: lets copy the first extent to disk 11, the second to disk 12, and the third to disk 10. Once everything has two copies, my data is protected – even though the first disk was never rebuilt!

You’ll also notice that the I/O for the rebuild operation is spread across all of the disks. Faster and better.

So there actually is a “hot spare” of sorts – spare capacity instead of a spare disk. And unlike RAID, this hot spare isn’t optional: ASM will always reserve enough spare capacity to bring the diskgroup back to its proper level of redundancy after a failure.

Failure Groups Mean A Little Less Usable Space

WAIT WAIT WAIT!! I know you’re excited, but before you press enter and execute that migration script, I need to tell you about one more very important detail. How does ASM know that amount of spare capacity to reserve? The size of the largest disk, right? NO!!!

This is, in fact, what failure groups are for!! (You always wondered, right?!) Because ASM doesn’t mirror across disks – it mirrors across failure groups. And accordingly, ASM will always reserve enough spare capacity to restore protection after losing the largest failure group.

Now it’s important to understand how ASM will report this. Lets look quickly at asmcmd’s lsdg command. This command (11gR1) gives us four important numbers: Total_MB, Free_MB, Req_mir_free_MB and Usable_file_MB. The 11gR2 Storage Administrator’s Guide gives good descriptions of the output:

Putting it all together, we see that Req_mir_free_MB should be equal to the largest failure group. Secondly, Usable_file_MB is the most important number — that’s how much space we can actually use. And with normal (2-way) mirroring, it should always be half of the available raw free space. And available raw free space is total raw free space minus Req_mir_free_MB.

Example

So let’s see if we can observe this on a system in the wild. We’ll revisit one that I briefly described last year.

The system had 46 local disks (500M each). The disks were attached to six controllers: two controllers had six disks each, the other four controllers had eight disks each. We configured ASM so that each controller is a failure group.

First lets get the output of lsdg, which we’ll verify step-by-step:

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  NORMAL  N         512   4096  1048576  18041706  18041326          3137688         7451819              0  DATA01/

Now – to begin our verification – lets look at the exact size of a single disk:

# sfdisk -suM /dev/sdav1
401624968

The Linux kernel is reporting KB and we need MB – so we’ll divide by 1024.

401624968 /1024 = 392211.8828125 M

ASM uses “allocation units” of 1M and can’t use a partial AU – so we truncate that to 392211. Now there were 46 disks – so the total amount of space available should be 392211 * 46 = 18041706. And this perfectly matches Total_MB from the lsdg output! So far, so good!

Next, lets look at the largest failure group. The four large failure groups were each the same size with eight disks: 392211 * 8 = 3137688. This perfectly matches Req_mir_free_MB from the lsdg output! Lovely!

Finally, lets see how much space I can actually use for datafiles and such – accounting for mirroring. The available raw free space should be Total_MB minus our calculated Req_mir_free_MB: 18041326 – 3137688 = 14903638. I’m using normal mirroring, so my available/usable free space should be half of that: 14903638 / 2 = 7451819. Perfectly matches Usable_file_MB from the lsdg output. Wow.

The Big Caveat

Sounds amazing, right? Well there is one very important caveat. I alluded to it earlier: Oracle will allow you to use this reserved “hot spare” capacity. This is very different from a traditional “hot spare” disk. What happens if you use the space? (1) The “usable free space” reports a negative number!!! (2) If a disk fails, then you will not lose any data – but ASM cannot automatically restore protection until you replace the disk or give ASM additional storage somewhere else.

That sound serious! What’s worse, you can get into this situation without even realizing it. If you create a datafile that’s bigger than the available space then ASM won’t warn you – it will just create the datafile and you lose your protection from double-failure.

You get alerts when your traditional filesystems are filling up. (How many times has your inbox been flooded because of those stupid archivelogs?!) If you deploy ASM, then MAKE SURE YOU HAVE ADEQUATE ALERTING. Oracle’s management tools can help out, and there are other solutions too. Don’t overlook it!

Conclusion

At this point, it should be evident that anyone with storage background needs to be careful around ASM. It doesn’t work the same as a traditional volume manager or RAID controller. Oracle Corp really does love SUPER EXCITING NEW WORDS (aka marketing buzz)… but this one isn’t just a matter of terminology. There is some real difference in the underlying concepts.

A few key takeaways:

• To automatically restore protection after a failure, ASM reserves spare capacity instead of a spare disk.

• In V$ASM_DISKGROUP and asmcmd’s lsdg command, Total_MB and Free_MB report raw capacity while Usable_file_MB reports usable capacity based on the default mirroring level.*

• ASM will allow you to use the spare capacity, after which it will report negative free space and won’t tolerate a double-failure. Make sure you have adequate reporting so that this is always caught and addressed quickly.

*Note: Various objects can have different mirroring levels in ASM.

Well I think that this should serve as a good primer on ASM and mirroring. Let me know if I’ve left anything out!

A Short History

We first ran the RAC Attack last year at Collaborate in Denver. On the first day of Collaborate we heard that one of the computer labs was going to be sitting empty for several hours on Wednesday. Being such a fervent believer in hands-on learning, this sounded like a terrible waste! So at the last minute we arranged to deliver the RAC Attack hands-on lab during the open slot. If I remember right, we had something like 20 people and everyone seemed to really come away having learned something new. The lab focused on getting a running system.

In August when we ran RAC Attack as a standalone workshop in Chicago. In addition to having several excellent speakers, we dramatically expanded the scope of the lab to include topics like backup & recovery, services, and clusterware. During the full day allocated, only a few people completed every possible exercise.

Five Reasons To Sign Up

This year at Collaborate we’re doing it again. The lab will again be expanded and improved when we bring it to Orlando. Furthermore, I’m taking requests – if there’s a topic you’d like to learn about then leave a comment and I’ll try to get it included! I’m not sure how quickly the session will fill, but I know that there are a limited number of seats – so you should sign up soon.

Here are five reasons I think you should come:

1. Get exposure to and practice on all of the latest software, including VMware Server 2.0.1, Oracle Enterprise Linux (same as RedHat) 5.3, and Oracle RAC 11.1.0.7

2. There are hands-on challenges for every experience level. Meticulous step-by-step guide for beginners (with the choice to skip difficult steps), start-from-scratch or instant running environment for masters.

3. Learn from professionals with years of RAC experience. Several of us will be on-hand to answer questions and assist with the labs. If you have never touched a cluster database then we are here to help you complete the labs and understand what you’re doing!

4. Choose your own adventure. ASM or OCFS2. Parallel SQL or clusterware callouts. Got something specific you want to learn about? Leave a comment and I’ll try to get it included.

5. Walk out with a lab binder that tells you how to do it all on your own laptop after you get home. It won’t be possible to finish all the labs during the timeslot we have at Collaborate, but you can finish them later!

Finances are tight and many companies are cutting training budgets – but remember that generally speaking, Collaborate is one of the best value opportunities for Oracle training that’s available in the United States. You have world-class educational sessions and networking opportunities, complimented by top notch deep-dive university seminars such as RAC Attack. (Well… I’m a little biased about this session!) And with a slightly greater degree of independence you won’t just hear the corporate messages – it’ll be a little easier to find real-life users telling the great, the good, the bad and the ugly.

Hope to see you there!