Why Make Charts

I’ve heard Kyle Hailey speak on a few different occasions, and more than once he’s talked about the power of visualizing data. (In fact Kyle was a key person behind Grid Control’s performance screens.)

I couldn’t agree more. I regularly visualize data when I’m working. Two reasons come immediately to mind:

1. It helps me to better understand what’s happening. There have been times when I’ve had an “aha” moment very quickly after seeing the picture.
2. It helps others more easily understand what i’m trying to communicate. It’s great for management reports and such – not because it’s fluff, but because it’s a good communication tool.

Last week, I made a few quick charts as illustrations for a performance report. The process really isn’t that complicated, but I thought I’d put the steps into a blog post… for myself to reference in the future and for anyone else who might find this helpful. :)

Making Simple Charts

This demonstration will use data from the AWR to build graphs. Note that if you run these queries, Oracle legally requires you to purchase the extra-cost “diagnostic pack” license. But similar queries could be written from free statspack or S-ASH tables.

You need multiple data points to make a graph. For this demo, The AWR was configured to take snapshotsd every 30 minutes and I’m looking at a query which ran for about 10 hours. Also, it was the only query running in the instance for most of that time – so I will also look at some instance-wide statistics.

1. The first step is to get any needed parameters for pulling performance data. In the case of the AWR, I will need the INSTANCE_NUMBER, the SQL_ID and the first/last SNAP_ID.

   It’s pretty easy to get this information from Grid Control or from Database Console. But if you don’t have access to the web console then you can still get the info from SQLPlus.

   Here’s a useful query to get an overview of the SNAP_IDs:

   SQL> select to_char(BEGIN_INTERVAL_TIME,'MON YYYY') month,
          min(snap_id) min_snap, max(snap_id) max_snap
        from dba_hist_snapshot
        where instance_number=4
        group by to_char(BEGIN_INTERVAL_TIME,'MON YYYY');
   
   MONTH        MIN_SNAP     MAX_SNAP
   -------- ------------ ------------
   APR 2009        10239        10240
   DEC 2009        28752        31924
   FEB 2010        38115        40939
   MAY 2010        47498        48783
   AUG 2010        54975        55013
   NOV 2010        60979        61986
   DEC 2010        61987        64218
   JAN 2011        64219        66448
   FEB 2011        66449        67803
   MAR 2011        67804        69291
   APR 2011        69292        70731
   MAY 2011        70732        72219
   JUN 2011        72220        73655
   JUL 2011        73656        75139
   AUG 2011        75140        76608
   SEP 2011        76609        78048
   OCT 2011        78049        79536
   NOV 2011        79537        80338
   
   18 rows selected.

   Something similar to this might also be useful:

   SQL> select snap_id,instance_number,begin_interval_time,snap_level
        from dba_hist_snapshot
        where begin_interval_time between '11-nov-11 17:30' and '11-nov-11 19:00'
          and instance_number=4
        order by snap_id, instance_number;

   For this demo I’m going to use INSTANCE_NUMBER 4 and SQL_ID 8suhywrkmpj5c between snaps 80298 and 80318.

2. Now create a new spreadsheet in your office suite. I use the free OpenOffice spreadsheet application, but Excel or iWork Numbers should work pretty much the same.

   In the second row of the new spreadsheet, enter the time of the first snapshot you’re going to analyze. In the third row, enter this formula:

   = A2 + 1/24/60 * [minutes between snaps]

   Select several rows below this formula and select Edit > Fill > Down to copy the formula to the following rows. Repeat this until you have reached the end of your analysis window.

   

3. Open a SQLPlus session. We will copy-and-paste directly from SQLPlus into the spreadsheet.

   Use a SQL like this to retrieve data for one system statistic:

   set pagesize 999
   col value format 999999999999999
   
   select value from dba_hist_sysstat
    where instance_number=4 and snap_id between 80298 and 80318
      and stat_name='transaction tables consistent read rollbacks'
    order by snap_id

   You can copy the statistic name directly from an AWR report of there’s a certain stat you’re interested in. You can find more information about system statistics in Oracle’s docs.

   Now move right to the next empty column. First, copy the name of this statistic into the first row. Then, in the second box, enter a formula to find the difference between its left peer and the left upper peer. For cell C3, the formula is B3-B2. Choose Edit > Fill > Down again, as before.

   = B3 - B2

   

   You can repeat this step to access further system statistics. You can also create another column where you divide or multiply each other.

   

4. The previous SQL statement retrieved system statistics. Another easy query runs against historical V$SQLSTAT snapshots. (This only works for long-running queries.)

   SQL> select BUFFER_GETS_TOTAL value from dba_hist_sqlstat
        where instance_number=4 and snap_id between 80298 and 80318
          and sql_id='8suhywrkmpj5c'
        order by snap_id

   Once again, you can read about the available fields and statistics in the oracle docs. You can repeat this step to quickly get additional statistics for a particular SQL, and you can then combine some stats for better graphs.

   

That’s it! I know, really not that complicated. I hope it’s helpful. :)

Post Script

For anyone who’s curious, the charts in this article are related to a SQL report which was recently discussed on the mailing list.

If you’ll be at OpenWorld – in just 11 days – then the IOUG RAC SIG is putting together a special event for you!  (You might have already heard about this on Twitter or from Justin at the OTN Blog.)

Every day from 9am to 1pm, find our table in the OTN Lounge (on Howard Street) and we’ll help you get an 11gR2 RAC cluster database running inside virtual machines on your own windows-based laptop. You can experiment boldly – if you make a mistake then you won’t have to start over; we can easily “reset” your virtual machines to any point.

The basic idea is to facilitate some community-driven direct mentoring.  We’ll have experienced RAC DBAs on hand to answer any questions you come up with.  We’ve run limited-seat classes (always full) at several past Collaborate conventions including this year in Orlando; but we’re changing the format at OpenWorld so that more people can participate.

It’s not required, but to have the best experience we recommend bringing a USB-powered external hard drive with 100G of free space – you can buy one online for about $40.   Detailed hardware requirements are in the online lab handbook.  But come visit our table and say hello regardless – besides, who knows what other interesting stuff we might be hacking on…  :)

Hope to see you there!

http://racattack.org/e

Post-Script

If know a DBA who already has experience with RAC, then they can help us meet and mentor newer DBAs.  The last 5 volunteer slots are still open.  (Each volunteer can sign up for 2 max.)  Pass along the word.  Anyone can sign up for a volunteer slot by directly editing the wiki page at http://racattack.org/e and emailing me.

Our project was very successful – the key changes had dramatic effects and we are now running the same set of reports much faster.  (Some reports went from many hours to less than 30 minutes.)  My final project summary included a small section where I passed along a few suggestions specifically for these developers – but looking over the suggestions now, I think they might actually be good tips for a wider audience.  At any rate it might be interesting to put them out for comments!

Of course these tips aren’t my own original work; in fact I think they’re pretty widely accepted in the Oracle professional community.  We’ve been hearing these ideas for years now at conferences and user groups.  I just picked a few that could make the biggest impact for this team right now – and I put the ideas in my own words.  These were my top three:

• Aggressively search for opportunities to do less computational work.
  • Fewer switches between SQL, PL/SQL and Java.
  • Java is slowest, PL/SQL is faster, C and SQL are fastest.
  • Operate in bulk whenever possible.
    • Avoid using any kind of loop in your code whenever possible. If loops are required, operate on large chunks at a time (for example, by increasing fetch sizes).
    • Avoid coding in multiple steps whenever possible. If multiple steps are faster, ask how you’re code works and why Oracle isn’t automatically doing the same.
• Always utilize tracing/logging when coding and tuning. Create logs at the most granular level possible, then summarize the results by time (aka profiling). Calibrate everything in terms of wall clock elapsed time for entire job.
  • Utilize database 10046 log files for any jobs where the database is an important component. There are several profilers available with various feature levels and price points (starting with free); utilize one of these to summarize the logs. These logs can safely be generated in production.
  • Use the Oracle PL/SQL Profiler to generate profiles on critical path PL/SQL code.
• When tuning SQL, follow these steps:
  1. Remove all hints.
  2. Use dbms_xplan, 10046 and 10053 traces – in that order – to determine where the CBO went wrong. Compare CBO row count estimates with actual row counts before time-consuming, in-depth analysis of 10053 logs.
  3. Add hints only with justification about why the hint is preferable to fixing underlying cause. Generally speaking, hints should be viewed as band-aids.
  4. When a good plan is found and underlying causes cannot be addressed, use DBMS_XPLAN functions with the ADVANCED flag to retrieve a full set of hints. Use this to lock in the good plan and document the situation with SQL comments. (11g will offer new options for plan management, but this method will be sufficient for 10g.)

I’m sure that these tips aren’t perfect – so please share your thoughts!  How could I better word them?  Do they apply to your development team?

Post-Script

I’m definitely no guru of Oracle performance yet, but I have been involved in a number of performance-oriented projects. My experience has been that it’s pretty common to have certain assumptions built in to the project through scoping and goal statements before I arrive. Often, project sponsors have already decided that one of the most important items in a performance project is that someone with experience in storage and system configuration for Oracle (this is usually me) will review settings and statistics for things like database parameters, AWR reports, operating system kernel settings, networking, I/O, etc.  The contractual “statement of work” might say something like “review and assess configuration and architecture” with a list of specific areas of configuration.

I’m not opposed to periodic configuration audits or having a second set of eyes look over the setup.  But this very customer was a perfect example of how these assumptions about performance tuning through system review really are not the best way to work.  Due to the many parties involved in this contract and various expectations, it was important for me to review the system configuration – so I did spend time on this review.  However I was lucky that the team and managers who I directly worked with were sharp and open-minded. After some spirited discussion we slightly altered our approach: starting at the top of the stack and working down rather than the other way around.

Our first two weeks were focused on the most important business reports and only during the last two weeks did I take an in-depth look at storage and system configuration.  Can you guess what happened?  Looking at the top business problems led us straight to the most important bottlenecks impacting reports across the board.  It also showed me which areas of system configuration could have the biggest impact on the business.  By the time I started doing a system review, I already knew which areas to focus on.  Furthermore, I could already make some estimations about best-case impact of system changes on overall report run-times.  Not surprisingly, I knew that no system change was going to give us the improvement we wanted – but tuning a few small pieces of code that were used widely gave us even more improvement than we had thought possible.

Again, this isn’t new… it’s stuff we’ve been hearing at conferences and user groups for years now.  But it seems to me that it’s still worth repeating.

However: please notice the Security Addendum at the end of this article.

Of course if you’re lucky enough that your developers use certain tools from Oracle, then there are some slick 3rd party plugins that will help download and manage tracefiles for you. But what if your developers don’t want to add a whole new environment to their already memory-bound workstations? What if there are corporate policies making this difficult – such as a time-consuming review and approval process for any new software installs?

Wouldn’t it be nice of Oracle had a system view that your developers could just QUERY to find tracefiles? Maybe something like this (which of course displayed data in real time, reflecting the current status of the filesystem):

SQL> desc all_trace_files
 Name                          Null?    Type
 ----------------------------- -------- --------------------
 DIRECTORY                              CHAR(5)
 FILE_NAME                              VARCHAR2(400)
 FILE_TIME                              DATE
 FILE_BYTES                             NUMBER
 FILE_TRACEID                           VARCHAR2(100)
 FIRST_TIME                             DATE
 FIRST_ACTION                           VARCHAR2(100)
 FIRST_MODULE                           VARCHAR2(100)
 FIRST_SERVICE                          VARCHAR2(100)
 SID                                    NUMBER
 SERIAL                                 NUMBER
 ERROR                                  VARCHAR2(400)

Sure, this looks nice – maybe in the next version of Oracle. When you upgrade to it, 10 years from now. But what about NOW – your old 10g database running on a huge mainframe?

We all know how nice it would be if your developers run a query like this… today… on almost any database:

SQL> select directory, file_name, file_traceid, first_action from all_trace_files
  2  where first_action is not null;

DIREC FILE_NAME                                FILE_TRACE FIRST_ACTION
----- ---------------------------------------- ---------- ----------------------------------------
UDUMP ardentp1_ora_12386502.trc                           Test Window - New
UDUMP ardentp1_ora_22347974.trc                           Test Window - New
UDUMP ardentp1_ora_22675498.trc                           SQL Window - xml_file_writing_te
UDUMP ardentp1_ora_24248536.trc                           SQL Window - test_cases.sql
UDUMP ardentp1_ora_24641726.trc                           DEQ
UDUMP ardentp1_ora_2621852.trc                            Test Window - New
UDUMP ardentp1_ora_3342766.trc                            problemTbsp
UDUMP ardentp1_ora_4063428.trc                            SQL Window - New
UDUMP ardentp1_ora_45350944.trc                           Debug Test Window - New
UDUMP ardentp1_ora_45547538.trc                           Main session
UDUMP ardentp1_ora_48103606.trc                           SQL Window - SELECT * FROM ext.e
UDUMP ardentp1_ora_50593842.trc                           SQL Window - test_cases.sql
UDUMP ardentp1_ora_50790578.trc                           SQL Window - New
UDUMP ardentp1_ora_51380240.trc                           Debug Test Window - New
UDUMP ardentp1_ora_52429026.trc                           UserBlock
UDUMP ardentp1_ora_52691004.trc                           SQL Window - New
UDUMP ardentp1_ora_60817584.trc                           Test Window - New
UDUMP ardentp1_ora_61538330.trc                           Test Window - New
UDUMP ardentp1_ora_62783512_schn_02.trc        schn_02    Test Window - New
UDUMP ardentp1_ora_62783512_schn_03.trc        schn_03    Test Window - New
UDUMP ardentp1_ora_6291778.trc                            Test Window - New
UDUMP ardentp1_ora_63176754.trc                           Test Window - New
UDUMP ardentp1_ora_66388158.trc                           Test Window - New
UDUMP ardentp1_ora_8192480.trc                            Test Window - New
UDUMP ardentp1_ora_8651144.trc                            SQL Window - test_cases.sql
BDUMP ardentp1_m000_12517418.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_20119784.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_24248408.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_45154330.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_mmnl_52560064.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_mmon_52756580.trc                          Monitor Tablespace Thresholds

31 rows selected.

And of course – last but not least, you’d like your developers to actually be able to download the tracefile themselves, right? Actually, this has already been done – Dion Cho blogged some sample code about two and a half years ago. He has a very elegant solution using a pipelined PL/SQL function and UTL_FILE.

http://dioncho.wordpress.com/2009/03/19/another-way-to-use-trace-file/

Just one minor tweak would be needed, to grab files from more than one directory. (In addition to your session traces, you might need to get logs from parallel query slaves or DBMS_JOB/SCHEDULER processes.)

Not only can you write queries to your heart’s content on this, but just about every developer environment on the planet can easily take this query output and save it to a file – then they can run the profiler of their choice on the tracefile. That would be cool, right?!

SQL> select * from table(textfile('ardentp1_ora_52691004.trc','UDUMP')) where rownum<20;

COLUMN_VALUE
-------------------------------------------------------------------------------------------
/u0001/app/oracle/admin/ardentp/udump/ardentp1_ora_52691004.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u0001/app/oracle/product/db/10.2.0/ardentp
System name:    AIX
Node name:      ardent4
Release:        1
Version:        6
Machine:        00F66BE22C00
Instance name: ardentp1
Redo thread mounted by this instance: 1
Oracle process number: 59
Unix process pid: 52691004, image: oracle@ardent4

*** ACTION NAME:(SQL Window - New) 2011-08-15 16:35:24.791
*** MODULE NAME:(PL/SQL Developer) 2011-08-15 16:35:24.791
*** SERVICE NAME:(ardentp_dba) 2011-08-15 16:35:24.791
*** SESSION ID:(572.24734) 2011-08-15 16:35:24.791

19 rows selected.

Well I AGREE that this would be cool! And as it happens, the customer I’m working with now would find it very useful. And honestly… it’s really not that complicated to code something like this. So last night I stayed late a few hours to write it up and test it. It’s mainly build on a pipelined java function, and of course a minor tweak on Dion’s code for getting the file.

Basically, every time you query the view it scans the first 30 lines of all files in the bdump and udump directories for timestamps and identification information (module, action, service, sid, serial). This information is combined with basic file stats (name, size, last-modified-date) and the tracefile_identifier is calculated from the filename. Any errors encountered are reported in the error column (file permissions, etc).

You can download the entire thing at my github repo (specifically, the files tracefile_find.txt and tracefile_get.txt). All examples in this blog post are from a real system – nothing here is cooked, except that I changed database and host names to protect the innocent.

Enjoy!

Security Addendum

As pointed out in the comments after I published this post, there are some important security considerations with allowing access to trace files. A good explanation can be found in Pete Finnigan’s blog article: Is it possible to steal data with just ALTER SESSION? To summarize: if a clever hacker is able to access tracefiles then it is possible for them to see things that the database normally protects. This certainly includes table data, and it may even include passwords for other database accounts – sometimes even in cleartext. This is very important if you’re considering any tool that grants access to tracefiles – including the scripts I’ve published here. Although he generally disapproves of any tracefile access, Pete’s comment suggested a way to reduce the risk. The second table (which actually gives tracefile contents) can be limited to DBAs and then developers can get a view which only shows files that the DBAs explicitly grant. I think that the most useful part of my program is the first view anyway, which scans tracefiles and lets you use SQL to search on module/action/service.

One further quick postscript… lest you think I was terribly irresponsible with this client, we did consider security implications before endorsing this script. Admittedly I didn’t consider every scenario that Pete has written about, but I do still think we made an appropriate decision. (Also… the script wasn’t yet installed into production when I left the client, because it had to go through an approval process itself. ) In this particular case, the small group of developers getting access to tracefiles were people who already had full access to all data in the database and they had passwords for all key application accounts. Although they didn’t have SYSDBA passwords or shell access, they essentially had everything else – including all the data. Furthermore, they were in a stage of performance troubleshooting where their productivity could be significantly increased with the ability to run many traces and quickly access the tracefiles (think Method R). And finally, these developers were on the same small, tight team as the DBAs who took care of backups and patching, all sitting together in one corner of the office building. The DBAs were involved in the decision and seemed to find their reduced workload a very positive thing – especially since they know all the developers who would get access.

One could certainly comment on the security policies of this client, and their decisions about balancing trust and productivity. But I do not think that we were circumventing the company security policies in this case. And certainly my scripts are not intended to support such activity. Nonetheless, I do think there are cases where this is a very useful script. It’s much more convenient for developers who – even if they can get shell access – may not know much about navigating unix and are more comfortable working in their SQL development environment.

[24 May 11 09:30] * cheboygan: glad to see that someone read the
  blog post though.  at least i know one person read it.
[24 May 11 09:30] * rizzo: it got re-tweeted a lot last week
[24 May 11 09:30] * rizzo: get yourself on twitter, man
[24 May 11 09:30] * cheboygan: i've been resisting...
[24 May 11 09:30] * rizzo: give in
[24 May 11 09:30] * rizzo: it calls to you
[24 May 11 09:31] * cheboygan is still fighting it
[24 May 11 09:31] * rizzo: resistance is futile
[24 May 11 09:31] * cheboygan will never be assimilated
[24 May 11 09:32] * cheboygan: is there a point to having a twitter
  acct  if it's not tied to some kind of mobile device?
[24 May 11 09:34] * rizzo: cheboygan: sure, most of my twitter
  stuff is done at my desk
[24 May 11 09:34] * rizzo: just reading and posting updates
[24 May 11 09:35] * cheboygan: hm.  i just did a search and it
  turns out there's a twitter plugin for my chat program.  might
  give that a try actually.  wow, really didn't think i'd be so
  easily persuaded.
[24 May 11 09:49] * rizzo: Dan Norris will thank me for this
[24 May 11 09:49] * cheboygan: have to admit that i did hold out
  for an impressively long time :)

[24 May 11 09:50] * rizzo: what's your twitter handle?
[24 May 11 09:50] * cheboygan: haven't setup the account quite
  yet... give me a few more minutes :)
[24 May 11 09:50] * rizzo: heh ok
[24 May 11 09:50] * rizzo refreshes madly

[24 May 11 09:56] * rizzo: bieber4life?
[24 May 11 09:57] * cheboygan: are you kidding?  bieber4life is
  definitely taken.

[24 May 11 10:04] * cheboygan: the problem with holding out on
  something like twitter, is that the longer you wait the more
  impossible it is to have any readable kind of handle.
[24 May 11 10:06] * cheboygan: maybe i should write a blog post
  and elicit suggestions or something.  this is ridiculous.  in
  fact, even the word ridiculous is taken.
[24 May 11 10:10] * Ramen: use l33t
[24 May 11 10:11] * cheboygan: just dicovered underscore is ok.
  new possibilities.

Just a few minutes after this, I finally caved in and signed up for Twitter.  My account is @jer_s – feel free to find me over there while I try to figure out what you’re supposed to do with this Twitter thing.

• 100% class met expectations, would recommend to others

• 66% easy to follow, could use skills in working environment

• 100% already familiar with oracle, 90% use oracle daily

• 0 negative reviews of instructor (phew!)

• 1 negative review of curriculum: said practice exercises weren’t relevant but training manual was still above average.

• 0 negative “comments”

There were several positive comments such as this: “I would recommend this class to others. This setup is perfect to pick up new skills and expose what ifs w/out worrying about pressing the wrong button.”

There were also some suggestions for improvements which I’ve taken note of. The most common suggestion was this: “More time to practice / explore new concepts.”

What’s Next

The RAC Attack curriculum has been around for over three years now – and I’m still getting requests to put it on.  Seems that a number of people have felt that it’s a decent quality class and handbook.  I talked to several other Oracle instructors and speakers at Collaborate this year about the RAC Attack curriculum – including an in-depth conversation with Arup Nanda.

As a result of these conversations, I did some careful thinking and I’ve been very busy for the past few weeks since Collaborate.  And I now have three major announcements about RAC Attack:

1. Last week, I published an updated RAC Attack Lab Handbook.  The handbook is now completely self-contained for home use.  I’ve added significant new content:
   • Detailed hardware requirements (taking into consideration optimal I/O and memory configuration)
   • How to install and setup VMware environment (including networking and storage)
   • How to create the RAC Attack Classroom DVD (this is actually scripted now)

This means that you can now (finally) setup RAC Attack on your own computer – without needing any external assistance!

2. Although I created most of the labs for RAC Attack, some labs were also contributed by Dan Norris and Parto Jalili.  Last week I also obtained their permission to move all RAC Attack content under a Creative Commons Attribution-ShareAlike license – which you will see in the latest handbook.The RAC Attack lab handbook has now been donated to the Oracle professional community with a solid copyleft license.
   • New DBAs are invited to use this material at home.
   • Experienced DBAs are invited to make corrections and enhancements to the textbook.  Also you can add your own labs which work on the RAC Attack “platform”.
   • Instructors are invited to use this material partially or completely in their own classes, as they wish. You can even run your own class based solely on this material – without needing anyone’s permission. Also, you can charge for these classes – there isn’t a “non-commercial” limitation on the terms of our license.

On a related note, all code related to RAC Attack (including the jumpstart scripts and supplemental DVD build script) is now under the GPL and has been published on github for easy downloading, copying and branching.

3. Finally… how are you supposed to make corrections, enhancements and additions if you’ve only got a PDF version?  Right now the textbook sources are in OpenOffice format (with a master document and about 50 lab documents).  Although this is very well suited to print media, it’s difficult to coordinate with very many authors or editors.  I’ve decided that the best way to truly encourage collaboration with other teachers & instructors is to use the best platform on the web for collaborative open-content textbook projects: the RAC Attack lab textbook has a new home on wikibooks.

   http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home

   I’m just getting started now – but I’m hoping to have all existing RAC Attack content on this wiki textbook by the end of the month.  Please have a look at the project page and consider getting involved!

Feedback

I consider these to be some major changes.  I’ve put some careful thought into them, and I’m excited about the new direction for this class.  Let me know what you think.

Cheap Lab Environment

RAC on EC2 would be useful. (1) If you’ve never done much with RAC, then you can have a sandbox to play in – try the installer, try some parallel queries, try anything else. (2) You can setup a cluster with any version, any patch level – and see if your pl/sql or query works. (3) You could give each individual student in a class their own cluster.

And of course, the cost is very reasonable:

These numbers are based on small instances. In my observation so far the network connection is the bottleneck — and quite likely the 20 node cluster wouldn’t actually work. It would be about $25/hr to run a 10 node cluster with 4 SCSI targets on EC2 “cluster compute” nodes (w/10G network)… probably a little steep but maybe someday I’ll try running a benchmark for an hour or two just to see how fast it is. :)

Progress

I’m convinced that this will work. After a few days of working on it in between diapers, I haven’t quite gotten all the way – but I’m very close. I’ve got shared iSCSI-based block storage for ASM and two private networks which I think will support VIPs and multicast. The only two pre-checks that fail are swap space and an optional flag to NTP. The installer completes. I just need iron the kinks out of the post-install/root script execution.

I was initially trying to install 11.2.0.2 but I wimped out and reverted to 11.2.0.1 after wrestling with it for a while. I only have so much time in between diapers… :)

Everything is scripted. When you’re paying by the hour, you don’t want to waste time building the environment! Also, you don’t want to leave the environment running for days after you’ve set it up. Much better if you can start the RAC environment on-demand when you want it, and shut it down if you’re not using it.

Here’s what I’ve got so far:

jeremy@joshua:~/rac-ec2$ ./run-cluster 3 2
EC2-RAC-BUILDER Copyright 2010 Jeremy Schneider

    This program comes with ABSOLUTELY NO WARRANTY.
    This is free software, and you are welcome to redistribute it
    under certain conditions.  See included gpl.txt for details.

Creating cluster: ID #3
1 storage node(s)
2 compute node(s)

time      storage             network             compute
-----     --------            --------            --------
18:21:42  starting            starting            starting
18:22:18  ec2 load instance   ec2 load instance   ec2 load instance
18:23:44  boot O.S.
18:23:50  setup iscsi
18:23:55                      boot O.S.
18:24:10  finished
18:24:36                                          wait strg/ntwrk
18:25:27                      configure switch
18:25:37                      reboot O.S.
18:27:22                      finished
18:27:28                                          boot O.S.
18:27:48                                          configure O.S.
18:33:17                                          finished

username root/oracle
password ec2rac
hostname ec2-50-16-23-197.compute-1.amazonaws.com

This part is working pretty well. Right now I’m working on the script to actually install the oracle bits. Like I said – I’m still working out the kinks with the root script… but here’s what it currently looks like:

jeremy@joshua:~/rac-ec2$ ./install-oracle 3 2 >/tmp/ora-inst.out
19:57:19  downloading grid software from Oracle support
19:58:16  extracting grid software (/tmp/oracle/grid)
19:59:10  running cluvfy
20:00:33  installing grid infrastructure
20:28:11  running root scripts
20:30:38  downloading database software from Oracle support
20:36:16  extracting database software (/tmp/oracle/database)
20:38:02  finished!

Try It Yourself

If you’re interested in running RAC on EC2 then I’d love your help with this. You can download these programs and try them out for yourself. And let me know if you finish the install, and what steps you took. I’d really love to have a public toolkit that anyone can download, which will build RAC environments in EC2 on demand.

We had a single-instance database running on a failover cluster (RHCS). A database link existed for a related database and the connection had to pass through a firewall. The problem was the firewall: it had a rule which only allowed connections from the VIP.

The database server has two IP addresses – a system IP and a VIP. Is there any way to bind the dblink to one specific interface? (Note: we would still like the system IP to be used for other traffic.)

I couldn’t think of a way for Oracle to do that. But we did find a workaround, of sorts (though not perfect) – by using the operating system route command.

Sometimes there’s a misconception that the purpose of OS routing is just to set the default gateway. For example, in the windows networking GUI that function is visible and it’s rare to need much else. However the command-line OS route command is in fact much more powerful (on all platforms).

Here’s a detailed manual page for the linux version of route:
http://www.linuxmanpages.com/man8/route.8.php

First, in addition to setting the default gateway, the route command can also determine which physical interface to use. Second, in addition to configuration for a subnet, the route command can also do configuration for only one specific host. So the syntax we’d want is something like this:

route add -host [targetIP] dev [interface/ethX]

Of course it’s important to check that there’s nothing else connecting from the database cluster to the target host IP. Also, you’d want to link this command with the cluster VIP resource so that it follows the VIP around the cluster.

But it does seem to be one possible solution to the dilemma.

This will be my first year attending UKOUG – and I will be giving a new presentation called “Large Scale ASM Adoptions and Lessons Learned.”  I was personally involved in a very large ASM adoption project and I’m also talking to a few other acquaintances with similar experiences.  I will be summarizing our collective stories and lessons learned in this presentation.  My session will be Monday morning (Nov 29) at 10:25 am – please stop by!

However there’s another way you are invited to participate in my UKOUG session.  Do you know anybody who has been involved in an ASM adoption?  Have you been involved in one?  I have created a web survey with the questions that I’m asking my acquaintances about their experiences with ASM.

You can find the survey here:  http://www.ardentperf.com/asm-survey

If you can find 20 or 30 minutes then please share your own experiences and feedback!  I would like to gather as many stories as possible and summarize them all in a single presentation.

Do me a favor and help spread the word to anyone who’s been involved in an ASM adoption!   (Especially at large companies!)

Also, try to send me your feedback by next Tuesday so that I can include it when I finalize my slides on Wednesday.  And – of course – I will send you a link to the slides after UKOUG.  Thanks!

It’s nothing spectacular, but I wrote this short Haiku (poem) on Sunday…

OpenWorld: crush, splat…
Brain worked overtime last week!
Still catching up sleep.

Haha… ok… so I think that I’m finally caught up on sleep now that it’s Thursday. But it was a busy week!

OpenWorld was worth the plane ticket, for both learning and networking opportunities. My two favorite sessions were at oracle closed world and the unconference, respectively… but there were plenty of great “official” openworld sessions too!

It was great to see lots of old friends, to make some new friends and to meet people who I’d only known online. In particular, I ran into Martin Nash, George Woods, Karl Arao and Gwen Shapira quite a few times. Definitely looking forward to seeing everyone again soon!

Play-By-Play

Oracle decided to hand out plenty of “blogger” passes this year and lots of people have already written excellent day-by-day summaries of OpenWorld.  If you’ve read quite a bit about OpenWorld already then I’m not offended if you just skip the rest of this article.

For future OpenWorld attendees, I’d stress one point: “who” is more important than “what”.  Same philosophy I used to have for college classes.  A moderately interesting subject from a good speaker is a lot better than a good subject from a poor speaker.  Sure, hit a few important-sounding topics.  But mainly you should find the smart guys and follow them around!

Here’s the action last week from my perspective:

Sunday

• MySQL – Keynote, High Availability, Idiosyncrasies That Bite: Everyone sitting near me at the keynote was an Oracle DBA who wanted to learn more about MySQL.  They sat us all down and told us that MySQL is great software.  I liked the session on HA since I could compare it to RAC.  And “Idiosyncracies” was also very relevant – showing the key differences between MySQL and Oracle RDBMS.
• IOUG – SQL Server to 11g Cluster (Dylan Kucera), Oracle RAC Pack Panel: I’ve worked with Dylan before and he really knows a lot about migrating from SQL Server to Oracle.  Nice to reconnect – and he’s definitely got his technical ducks in a row.  He also has rare insight on navigating the soft (sometimes political) side of things.  The RAC Pack are also fun to hear – these guys have loads of experience and inside knowledge on RAC.
• Larry’s first keynote: blah, blah, cloud, blah, amazon, salesforce, exalogic, blah, blah…  exalogic?
• ACE Dinner: Awesome! Reconnected with lots of people. Also, got a fleece. It’s not bad. (Apparently they don’t want to repeat the Stanley episode.)

Monday

• Lunch: Spent the morning hanging out with my non-technical friends who I stayed with for the week.  Got to Moscone in time for lunch.  Sat at a table with two bearded guys.  Turns out they were not AIX sysadmins or linux hackers, they were database guys working for the government in Oman.  Also, the other guy at the table was the CIO of some Israili organization.  Small world.
• Closed World (Tanel Poder, Arup Nanda, Karl Arao, Me): Tanel, Arup and Karl were all quality. I always especially enjoy hearing Tanel – in fact just today I was telling someone on IRC about Tanel’s rlwrap bonus tips (remembering screen text and wordfile autocomplete).  After everyone finished their presentations, I also jumped up to the microphone and gave a spontaneous 5-minute summary of my trick for running Oracle RAC on low-memory Linux.  Hope I didn’t bore people too much.  Good times!
• OTN Night, Thirsty Bear: After leaving Closed World, I ran into Karl near Moscone.  We had dinner together with a few of his friends.  After dinner we dropped by OTN Night – just as it was wrapping up – and we recruited Martin and George to hit the Thirsty Bear with us.  Great way to close up the day!

Dan Norris

Tuesday

• Linux Roadmap (Wim Coekaerts), Exadata Lessons Learned (Dan Norris): Got a late start again on Tuesday, but I still caught most of Wim’s presentation. This Linux announcement is a big deal – so I was glad I showed up. Also, I finally had a chance to ask Wim a few questions about OCFS2. One interesting speculation… Oracle probably does still have a larger OCFS2 customer base than Novell. ACFS is compelling (especially with a single cluster manager) but it’s still new and there’s still a strong production OCFS2 install-base. In the ExaData session, Aaron Werman had some cool things to share about their implementation at Bank of America – but I have to admit that I was disappointed that Dan only gave a short intro and didn’t have a chance to share much else!  Oh well…
• Closed World (Martin Nash, Kyle Hailey): Kyle is one of the guys who helped design Grid Control – and he has an excellent presentation on visual SQL tuning. He talked a bit about his new product at Embarcadero too, which looks pretty slick. Unfortunately I missed most of Martin’s presentation because I was getting introduced to Mogens Noorgard. It was also nice running into Mike Swing from TruTek again and we chatted a bit about what makes good educational/training courses.
• Unconference (Kevin Closson): This was one of the highlights of the week for me – just having a chance to throw any old question at Kevin. We chatted about memory hierarchies, chip architectures, blogging and more. It was a small group of people and the atmosphere was decidedly casual and conversational – a marked difference from most presentations (where larger groups don’t really allow this).
• The Zone, Java Posse Party: After hanging out with Kevin for awhile I dropped into the Zone (aka OTN Tent), where I ran into Martin and Brian again. We then decided to sneak into the Java Posse party. We’re not technically java guys, but I think we managed to fool them because we stayed the whole night without getting kicked out.  :)

Arup Nanda at The Zone

Wednesday

• OLTP and FC: Oracle Schedule Builder claims I was scanned as an attendee here. Interestingly, I don’t remember it at all. Heh.
• Unconference (Craig Shallahamer): I was familiar with Craig’s writing but I’d never heard him speak. The session was more like normal conference than unconference, but very good all the same. Craig strikes me as similar to Cary Millsap (indeed they worked very closely for many years) – but a bit more theoretical. His presentation feels a little heavier on the modeling and mathematics than Cary’s presentations usually are.
• Neil Greene: After spending quite awhile getting to know Craig, I trekked up to the Fairmont to meet Neil Greene from Predictive Technologies Inc.  We’d been chatting in cyberspace quite a bit and I was eager to hang out in person.  Had a great time learning about what he’s been up to with PTI.
• Blogger’s Meetup: After coffee with Neil, it was time for the long-awaited bloggers’ meetup! Pythian organized this party, where everyone got a tshirt and there was a contest to get the most signatures of other bloggers! I was a late arrival and didn’t win the contest, but I had a blast hanging out with lots of familiar faces!
• The Big Party: And who can forget Oracle’s big party?  A festival in itself – lots of food, carnival games and amazing live music.  (Black-Eyed Peas, Don Henley of the Eagles, Steve Miller Band, Montgomery Gentry, etc…)  In reality, the party was fun but it was too big for me.  The crowd was so huge that I only ran into three familiar faces.  (Dylan, Neil, and a local friend who got a ticket.) Next year I might just skip it and get a group together for dinner or something.

Thursday

• Oracle Linux (Todd Sheetz), Query Transformations (Joze Senegacnik), RAC Performance (Arup Nanda), Clouds (Alex Gorbachev): Funny. I attended more official OpenWorld sessions on Thursday than any other day. Todd is a great guy and I enjoyed his session on Linux. Joze, of course, is amazing.  This session is a must-see. Arup draws a huge audience but I think that sometimes he has to pitch a little slower so everyone can keep up. (He went a lot more in-depth at Closed World.) And Alex did a great presentation on clouds – especially considering that Larry Ellison stole half of his slides for the OpenWorld keynote. So Alex spent Wednesday night re-writing this presentation.
• Small Party: And last but not least, there was the closing party. I managed to find Martin and Karl again, and the three of us hit the wrap-up party together.  It was perfect.  There’s nothing like kicking your feet up and listening to Bob Dylan and Bruce Springsteen when your brain is totally fried.

OpenWorld officially ended with the wrap-up party, but my trip continued a little bit longer.  Thursday night, Martin and I crashed a couchsurfer event for dinner. On the train Friday, I sat next to an Iranian guy from Iowa who also attended OpenWorld, then I had lunch with my father-in-law and then I had ice cream with another local friend.  And in the airport I discovered that I was taking the same flight home as Steven Feuerstein!

In conclusion, it was a busy but worthwhile trip.  I’m glad to be home – with life getting back to a normal pace.  But I’m looking forward to seeing everyone again.  Next year at OpenWorld?  Collaborate?  UKOUG?  I’m sure we’ll run into each other soon!