Before I completely forget the details from my work in Africa I want to wrap up my article about high-level lessons learned earlier this week. By the way, I’m not just stretching obscure aspects of my work in Africa to get stuff that sounds good. I view these cultural lessons which we learned together as the most central and most important technical aspect of my work at the hospital. And it might be surprising, but it’s true: the same cultural adjustments are important and oft-missed here in corporate America.

The first two lessons were to (1) understand the fundamentals and (2) avoid unjustifiable complexity. The remaining two lessons I want to talk about are slightly less technical but equally important.

3. People first, Technology second

   I’m using the word “people” here to sum up three major components of our accomplishments at the hospital: organizational policy, user education and technical training.

   First, a fun technical story:

   ---

   On day 1 after our arrival at the hospital, several issues required immediate attention – I discussed one example in the previous article. A second issue involved the network links to the outside world. In particular, sites like gmail were often completely unreachable. I worked hard on this one. Eventually I got the reproducible test case by connecting to an AWS instance very close to the other end of their link. I compared low-level packet traces from both sides to see what was happening.

   I could initiate a very large download and throttle the connection on our end, causing TCP window full messages to cascade all the way up to the AWS source server – normal behavior for throttling connections. But I did notice that the source server sent a very large chunk of data before it started throttling to the same rate that I was demanding.

   Next I opened a second connection with a different protocol on a different port. The throttled connection wasn’t even using a fraction of our contracted bandwidth – but every single packet on the second port took a consistent 60 seconds or so to get through. If I killed the download then things would zoom at normal high speeds again.

   The killer was that the TLS handshake in HTTPS connections needed a response to the initial packet within 30 seconds in order to continue. Small file downloads had no impact – but if a download was large enough, then SSL quit working completely – although HTTP would still (slowly) connect.

   My best guess was that this network provider (not an African company BTW) was running some enormous cache in the middle combined with a single packet queue per customer – no fair queuing by TCP connection. It doesn’t make complete sense but I haven’t yet thought of anything better. Maybe those other packets were just stuck in line on a huge cache throttled by packets in front of them? I actually didn’t think it was possible to configure network equipment to be this stupid… but maybe? Whatever the cause, the end result was that a large download – even throttled – generally trashed our whole network connection.

   I did pursue a ticket with the network provider, but it was a lot of effort to keep the issue moving. We discussed technical solutions on our end. Block large downloads? Sometimes they’re needed for the business. I spent quite a bit of time researching various QOS solutions – and this was when I learned a very important lesson. I especially learned from and ebook called How To Accelerate Your Internet (with several good African case studies) and some slides from Christian Benvenuti (International Centre for Theoretical Physics).

   Communication and people can do things that technology can’t.

   ---

   We weren’t able to resolve this issue in a “technical” way – so how did we solve it? We had something that I’ll call our PEOPLE strategy – it addressed this challenge and many others too. Here are a few elements of the strategy:

   1. We wrote a computer policy for the hospital, approved and enforcable at the top.
   2. Made sure a few tools were in place to identify violations.
   3. Setup a new wiki for the entire hospital community to use for any purposes.
   4. Designed the network so that every new device was redirected to a wiki page with the policy.
   5. Held extensive discussions with all staff and compound residents on several technical subjects.
   6. Hand-picked a few users for special training on several topics.

   We kept the policy short and simple and easy to remember – it was three rules that I could recite on my fingers. But it covered our needs well, protecting us from bandwidth problems and from malware.

   I worked to catalyze broad user education. Lots of conversations with non-technical people. Basic conversations, not fancy ones! It began to cultivate a new culture around technology.

   I briefly mentioned a new wiki up there – this was also a big part of our user education strategy. Except for a small restricted section, the wiki was 100% open to be updated by anybody. Part of our cultural change included training everyone at the hospital to use this wiki as a central repository for processes and information. With a high frequency of arrivals and departures, with people often rotating into and out of well-defined functions, the usefulness of the wiki was immediately apparent.

   The third important part of our people strategy was the special training. There were three unique qualities of this training:

   • Documentation: Much like the work I’ve done for Oracle RAC Attack, we worked very hard to boil down tough concepts and processes into extremely verbose step-by-step instructions on the wiki. Whenever possible, I taught people to start practicing step-by-step documentation for everything they do.
   • Reproducibility: Enabled by the growing documentation library on the wiki, we expected that everyone who received training would be able to train someone else on the same thing. When we switched to a new internet access system, I trained a few people on setting up accounts and had those few people handle the rest of the compound.
   • Functional Selection (rather than technical selection): also enabled by the documentation library, we started choosing the most logically positioned people for jobs instead of the most technical people. As one example, several non-IT people helped setup the new internet accounts. As a second example, the special projects coordinator – despite not being at IT person at all – followed extensive documentation to practice a bare-metal server recovery… and built our new server in the process! As a previous director, she was logically positioned for training which could potentially provide access to any data at the hospital.

   In Corporate America, we typically have more financial resources, more technology at our disposal, and much longer staff retention. Even if you have the same problems that we had in Africa, your situation will require different strategies. But the main point here is universally, absolutely crucial: there are two sides to every project, the technical side and the people side.

   Even if you’re working with outside partners to provide technical expertise, there is an in-house “people” component to your project. Even if it’s being called an appliance, it’s gonna be your baby to use & maintain & retire someday. Even if it’s a cloud-based service, you have to deal with the upgrades & functionality changes & workarounds for buggy, uncommon use cases. Every new technology you start working with will require somebody in-house to start learning about it. Never underestimate the people side of technology! It’s always there and it’s always important.

   I think that we easily get immersed in the technology side of our projects (myself included) and we often need a reminder to keep the people side in view.

4. Adapt the Process instead of Customizing the Product

   Finally, I have one quick word about processes and products. Now I don’t want to overstate this point; obviously there’s a cost-benefit analysis that happens for each case – and often changing business processes is not an easy thing. But I do believe that as technologists we have a tendency to favor customization a little more than we should. And in general (technologists and managers and executives) I think that we tend to not investigate many possible process changes because we think that it’s not really plausible. I think that if we start asking, we might be surprised how willing people sometimes are to change the way they work in order to have the business as a whole work better. If we’re getting some good new tools then people could totally understand changes to better accommodate those tools.

   When I was working at the hospital, there were two specific places where this discussion happened. The first was around a new pharmacy system that we were putting in place. The present system is heavily paper-based and for many reasons the hospital is pushing ahead with a new computerized system. It’s the classic conversation about software customization and frankly I didn’t do an awful lot besides convince hospital staff to work more closely with the company who writes the pharmacy software. (Great company – and eager to help – and they totally understand this conversation.)

   The second conversation was a little more unusual: server administration. Specifically, around NTFS file permissions and administrator access to user home directories. One requirement of the hospital was the ability to audit contents of redirected user home directories. By default these directories are created so that even Administrators cannot peer into them.

   Of course none of us were experienced windows server administrators. With some fiddling we managed to get folder redirection to work. I had found that many folder redirections on the old network didn’t work because of incorrect permissions on home directory folders. We could have done more testing to figure out what permissions worked – but we were running short on time. So this was where I encouraged the staff to “go light on changing things”. In the end we found another very good solution which didn’t require any NTFS permission changes.

   I often aim for environments to be as close as possible to the engineers who are building the software. I’ll ask people from my vendors, “what do your engineers mostly develop on?” With some companies it’s hard to get a straight answer, but I like to ask!

   When it comes to customization in Oracle databases, of course underscore parameters come to mind right away. Those little buggers can be life-savers sometimes… but their indiscriminately global effects can also be killers. Be careful!

Well I hope you can see a bit of why I enjoyed the work in Africa. Even though I wasn’t working for a fortune 500 company with millions to invest in bleeding-edge technology, there were still interesting and challenging projects. It was a great opportunity to hone my skills at helping people find the best ways to use technology in a real life business. And to be honest, I think that’s the one thing I’m most passionate about.

I hope that I’ve challenged your thinking a little bit around how technology serves your company. I hope that these new ideas push you to the next level in your professional career. And with any luck I might have even convinced a few people to help out in the non-profit world.

First, my family was living in Africa for three months earlier this year while I did some tech work at an NGO hospital. Second, upon our return I decided to join the good people at Pythian. I’m not moving to Canada, although I will travel a decent bit as part of the company’s consulting group.

If you’re interested in the Africa trip, look at the Africa page. I wasn’t working with Oracle technology but it was still a very interesting, challenging and engaging project.

I thought I’d briefly share a few high-level insights. You might be surprised how well these lessons apply almost anywhere (even Oracle-related projects)!

Four Lessons from IT in Rural Africa

1. Understand the Fundamentals

   Two fun and important projects at the hospital in Africa:

   • protecting assets (both physical equipment and their data) from very unreliable electricity, and
   • doing some field testing to see how radio signals from wifi routers would behave in a large compound with concrete walls and tin roofs.

   How much more fundamental does it get than copper wires and radio signals? And in our industry, it doesn’t matter what you do – these are also the fundamentals underneath what you’re building.

   In the days of [everything]-as-a-service and engineered-[anything]-appliances, we’re building at a higher level than ever before. You might think that building on clouds (or any abstracted/virtualized platform) means we can leave the platform implementation details to specialists.

   But smart companies and experienced engineers still pay a lot of attention to the fundamentals. There’s no magic or voodoo in computer systems. An experienced engineer can understand how a particular stack works from top to bottom – and you should be wary of anyone who won’t explain at some level of detail how their piece works.

   Do you remember that youtube video where the data center guy shouts into a bank of hard drives? Even if you’re buying pre-engineered, pre-packaged systems that come by the rack and fill half a room, you still need to ask the same basic environmental questions that you would with any other deployment into your datacenter.

   Bottom line: everything generally comes down to the same few basic things – for example processors and I/O and memory/storage hierarchies. Ninety percent of what you need to know, even for very complex systems, is in chapter one of my computer systems college textbook. Know your fundamentals and find them even in your complex systems.

2. Avoid Unjustifiable Complexity

   When I arrived in Africa, there were a number of issues crying out for immediate attention. For example: the head of finance couldn’t login to his workstation unless it was unplugged from the network. Every morning, this friendly Canadian guy unplugged his desktop, logged in to his domain account, then re-connected the network cable so he could access his network shares.

   This problem had existed almost a year. A sudden power outage had corrupted a virtual server running as a domain controller. To get systems back up, overseas IT support had directed local employees to restore a previous backup image of the VM. This did restore that server’s file shares but caused havoc among the multi-master DC setup – which was never totally resolved.

   A simple workaround beyond the unplug-and-login trick was not forthcoming. But now – after discussion with the director of the Hospital – we made a key decision and we changed course. A major upgrade was on the wish list… so rather than diving into complex debug & repair operation on multi-master windows domain, we put all our effort on the upgrade. And we re-architected the system so that this particular problem could never happen again.

   When I say “re-architected” … I mean we started at the very beginning. Did you pay attention during the requirements engineering section of your software engineering college class?

   • Functional Requirements: What does this hospital use IT for in the first place? How would we prioritize these functions? … (e.g. communication/email, collaboration/file sharing/printing, internet access for guests & residences, specialized programs like accounting & pharm…)
   • Non-Functional Requirements: Assuming that we must have IT, what specific qualities are important to us? How would we prioritize these qualities? … (e.g. data protection after >24hr power loss or theft or emergency evacuation, data retention after mistakes or theft or evacuation or catastrophe, ability for short-term IT staff to become productive quickly, can be supported remotely in lieu of local IT staff, protection from malware and viruses, etc.)

   Asking these simple questions led to two very important findings:

   1. There’s high turnover in the nonprofit world. We might get people for a few months or a year… but a lot of the IT support has to be do-able from overseas.
   2. There is no need for 99.99% uptime. If there’s a serious problem and we’re offline for a day then that’s really not a big deal at all.

   We realized that it was much better to have a simple system which would nonetheless prove reliable and easy to maintain, instead of a complex system which was hard to fix if something ever went wrong. The result was – as I wrote in my technical summary of the trip – we completely got rid of virtualization and the multi-master domain controller “cluster”. We migrated from four operating systems on two servers to a single server with a single operating system. But we also added a few things: on-disk encryption, RAID mirroring, improved & thoroughly tested backups, a printed & tested DR plan, a test server/domain exactly like the production server/domain, and a wiki.

   The most important word here is actually not complexity but rather unjustifiable. We justified everything in the architecture by showing tracability to this organization’s unique requirements.

   …

   Here in the States, I spend a lot of time working with clusters and I really enjoy it. But I can’t count how many times I’ve thought someone forgot these simple questions. Do you know your requirements?

   Maybe you’re buying pre-engineered or pre-packaged systems that come by the rack and fill half a room. You already learned from my first lesson and you have engineers who understand the fundamentals for these systems. But the most critical part is this step, the second one: now you need to justify those fundamental architectural characteristics by connecting them to the unique requirements of your organisation.

   Big job? Yes. Somebody else’s job? I don’t buy it. No matter where you are in your organisation, you can start asking questions and learning. Don’t assume it’s not your problem just because you’re not the decision-maker. If you become an expert on both your business and your technology, then before long everyone will be specifically asking for your input!

   If it’s worthwhile for a non-profit hospital in the middle of Africa, then how much more will it be beneficial for you?

…

I do have four lessons, but I think I’ll save the other two for another article. (This got long. <g>) Hope it’s thought-provoking and helpful.

Update: Lessons from Africa, Part 2

Why Make Charts

I’ve heard Kyle Hailey speak on a few different occasions, and more than once he’s talked about the power of visualizing data. (In fact Kyle was a key person behind Grid Control’s performance screens.)

I couldn’t agree more. I regularly visualize data when I’m working. Two reasons come immediately to mind:

1. It helps me to better understand what’s happening. There have been times when I’ve had an “aha” moment very quickly after seeing the picture.
2. It helps others more easily understand what i’m trying to communicate. It’s great for management reports and such – not because it’s fluff, but because it’s a good communication tool.

Last week, I made a few quick charts as illustrations for a performance report. The process really isn’t that complicated, but I thought I’d put the steps into a blog post… for myself to reference in the future and for anyone else who might find this helpful. :)

Making Simple Charts

This demonstration will use data from the AWR to build graphs. Note that if you run these queries, Oracle legally requires you to purchase the extra-cost “diagnostic pack” license. But similar queries could be written from free statspack or S-ASH tables.

You need multiple data points to make a graph. For this demo, The AWR was configured to take snapshotsd every 30 minutes and I’m looking at a query which ran for about 10 hours. Also, it was the only query running in the instance for most of that time – so I will also look at some instance-wide statistics.

1. The first step is to get any needed parameters for pulling performance data. In the case of the AWR, I will need the INSTANCE_NUMBER, the SQL_ID and the first/last SNAP_ID.

   It’s pretty easy to get this information from Grid Control or from Database Console. But if you don’t have access to the web console then you can still get the info from SQLPlus.

   Here’s a useful query to get an overview of the SNAP_IDs:

   SQL> select to_char(BEGIN_INTERVAL_TIME,'MON YYYY') month, 
          min(snap_id) min_snap, max(snap_id) max_snap
        from dba_hist_snapshot 
        where instance_number=4
        group by to_char(BEGIN_INTERVAL_TIME,'MON YYYY');
   
   MONTH        MIN_SNAP     MAX_SNAP
   -------- ------------ ------------
   APR 2009        10239        10240
   DEC 2009        28752        31924
   FEB 2010        38115        40939
   MAY 2010        47498        48783
   AUG 2010        54975        55013
   NOV 2010        60979        61986
   DEC 2010        61987        64218
   JAN 2011        64219        66448
   FEB 2011        66449        67803
   MAR 2011        67804        69291
   APR 2011        69292        70731
   MAY 2011        70732        72219
   JUN 2011        72220        73655
   JUL 2011        73656        75139
   AUG 2011        75140        76608
   SEP 2011        76609        78048
   OCT 2011        78049        79536
   NOV 2011        79537        80338
   
   18 rows selected.

   Something similar to this might also be useful:

   SQL> select snap_id,instance_number,begin_interval_time,snap_level 
        from dba_hist_snapshot
        where begin_interval_time between '11-nov-11 17:30' and '11-nov-11 19:00'
          and instance_number=4
        order by snap_id, instance_number;

   For this demo I’m going to use INSTANCE_NUMBER 4 and SQL_ID 8suhywrkmpj5c between snaps 80298 and 80318.

2. Now create a new spreadsheet in your office suite. I use the free OpenOffice spreadsheet application, but Excel or iWork Numbers should work pretty much the same.

   In the second row of the new spreadsheet, enter the time of the first snapshot you’re going to analyze. In the third row, enter this formula:

   = A2 + 1/24/60 * [minutes between snaps]

   Select several rows below this formula and select Edit > Fill > Down to copy the formula to the following rows. Repeat this until you have reached the end of your analysis window.

   

3. Open a SQLPlus session. We will copy-and-paste directly from SQLPlus into the spreadsheet.

   Use a SQL like this to retrieve data for one system statistic:

   set pagesize 999
   col value format 999999999999999
   
   select value from dba_hist_sysstat
    where instance_number=4 and snap_id between 80298 and 80318
      and stat_name='transaction tables consistent read rollbacks'
    order by snap_id

   You can copy the statistic name directly from an AWR report of there’s a certain stat you’re interested in. You can find more information about system statistics in Oracle’s docs.

   Now move right to the next empty column. First, copy the name of this statistic into the first row. Then, in the second box, enter a formula to find the difference between its left peer and the left upper peer. For cell C3, the formula is B3-B2. Choose Edit > Fill > Down again, as before.

   = B3 - B2

   

   You can repeat this step to access further system statistics. You can also create another column where you divide or multiply each other.

   

4. The previous SQL statement retrieved system statistics. Another easy query runs against historical V$SQLSTAT snapshots. (This only works for long-running queries.)

   SQL> select BUFFER_GETS_TOTAL value from dba_hist_sqlstat
        where instance_number=4 and snap_id between 80298 and 80318
          and sql_id='8suhywrkmpj5c'
        order by snap_id

   Once again, you can read about the available fields and statistics in the oracle docs. You can repeat this step to quickly get additional statistics for a particular SQL, and you can then combine some stats for better graphs.

   

That’s it! I know, really not that complicated. I hope it’s helpful. :)

Post Script

For anyone who’s curious, the charts in this article are related to a SQL report which was recently discussed on the mailing list.

If you’ll be at OpenWorld – in just 11 days – then the IOUG RAC SIG is putting together a special event for you!  (You might have already heard about this on Twitter or from Justin at the OTN Blog.)

Every day from 9am to 1pm, find our table in the OTN Lounge (on Howard Street) and we’ll help you get an 11gR2 RAC cluster database running inside virtual machines on your own windows-based laptop. You can experiment boldly – if you make a mistake then you won’t have to start over; we can easily “reset” your virtual machines to any point.

The basic idea is to facilitate some community-driven direct mentoring.  We’ll have experienced RAC DBAs on hand to answer any questions you come up with.  We’ve run limited-seat classes (always full) at several past Collaborate conventions including this year in Orlando; but we’re changing the format at OpenWorld so that more people can participate.

It’s not required, but to have the best experience we recommend bringing a USB-powered external hard drive with 100G of free space – you can buy one online for about $40.   Detailed hardware requirements are in the online lab handbook.  But come visit our table and say hello regardless – besides, who knows what other interesting stuff we might be hacking on…  :)

Hope to see you there!

http://racattack.org/e

Post-Script

If know a DBA who already has experience with RAC, then they can help us meet and mentor newer DBAs.  The last 5 volunteer slots are still open.  (Each volunteer can sign up for 2 max.)  Pass along the word.  Anyone can sign up for a volunteer slot by directly editing the wiki page at http://racattack.org/e and emailing me.

Our project was very successful – the key changes had dramatic effects and we are now running the same set of reports much faster.  (Some reports went from many hours to less than 30 minutes.)  My final project summary included a small section where I passed along a few suggestions specifically for these developers – but looking over the suggestions now, I think they might actually be good tips for a wider audience.  At any rate it might be interesting to put them out for comments!

Of course these tips aren’t my own original work; in fact I think they’re pretty widely accepted in the Oracle professional community.  We’ve been hearing these ideas for years now at conferences and user groups.  I just picked a few that could make the biggest impact for this team right now – and I put the ideas in my own words.  These were my top three:

• Aggressively search for opportunities to do less computational work.
  • Fewer switches between SQL, PL/SQL and Java.
  • Java is slowest, PL/SQL is faster, C and SQL are fastest.
  • Operate in bulk whenever possible.
    • Avoid using any kind of loop in your code whenever possible. If loops are required, operate on large chunks at a time (for example, by increasing fetch sizes).
    • Avoid coding in multiple steps whenever possible. If multiple steps are faster, ask how you’re code works and why Oracle isn’t automatically doing the same.
• Always utilize tracing/logging when coding and tuning. Create logs at the most granular level possible, then summarize the results by time (aka profiling). Calibrate everything in terms of wall clock elapsed time for entire job.
  • Utilize database 10046 log files for any jobs where the database is an important component. There are several profilers available with various feature levels and price points (starting with free); utilize one of these to summarize the logs. These logs can safely be generated in production.
  • Use the Oracle PL/SQL Profiler to generate profiles on critical path PL/SQL code.
• When tuning SQL, follow these steps:
  1. Remove all hints.
  2. Use dbms_xplan, 10046 and 10053 traces – in that order – to determine where the CBO went wrong. Compare CBO row count estimates with actual row counts before time-consuming, in-depth analysis of 10053 logs.
  3. Add hints only with justification about why the hint is preferable to fixing underlying cause. Generally speaking, hints should be viewed as band-aids.
  4. When a good plan is found and underlying causes cannot be addressed, use DBMS_XPLAN functions with the ADVANCED flag to retrieve a full set of hints. Use this to lock in the good plan and document the situation with SQL comments. (11g will offer new options for plan management, but this method will be sufficient for 10g.)

I’m sure that these tips aren’t perfect – so please share your thoughts!  How could I better word them?  Do they apply to your development team?

Post-Script

I’m definitely no guru of Oracle performance yet, but I have been involved in a number of performance-oriented projects. My experience has been that it’s pretty common to have certain assumptions built in to the project through scoping and goal statements before I arrive. Often, project sponsors have already decided that one of the most important items in a performance project is that someone with experience in storage and system configuration for Oracle (this is usually me) will review settings and statistics for things like database parameters, AWR reports, operating system kernel settings, networking, I/O, etc.  The contractual “statement of work” might say something like “review and assess configuration and architecture” with a list of specific areas of configuration.

I’m not opposed to periodic configuration audits or having a second set of eyes look over the setup.  But this very customer was a perfect example of how these assumptions about performance tuning through system review really are not the best way to work.  Due to the many parties involved in this contract and various expectations, it was important for me to review the system configuration – so I did spend time on this review.  However I was lucky that the team and managers who I directly worked with were sharp and open-minded. After some spirited discussion we slightly altered our approach: starting at the top of the stack and working down rather than the other way around.

Our first two weeks were focused on the most important business reports and only during the last two weeks did I take an in-depth look at storage and system configuration.  Can you guess what happened?  Looking at the top business problems led us straight to the most important bottlenecks impacting reports across the board.  It also showed me which areas of system configuration could have the biggest impact on the business.  By the time I started doing a system review, I already knew which areas to focus on.  Furthermore, I could already make some estimations about best-case impact of system changes on overall report run-times.  Not surprisingly, I knew that no system change was going to give us the improvement we wanted – but tuning a few small pieces of code that were used widely gave us even more improvement than we had thought possible.

Again, this isn’t new… it’s stuff we’ve been hearing at conferences and user groups for years now.  But it seems to me that it’s still worth repeating.

However: please notice the Security Addendum at the end of this article.

Of course if you’re lucky enough that your developers use certain tools from Oracle, then there are some slick 3rd party plugins that will help download and manage tracefiles for you. But what if your developers don’t want to add a whole new environment to their already memory-bound workstations? What if there are corporate policies making this difficult – such as a time-consuming review and approval process for any new software installs?

Wouldn’t it be nice of Oracle had a system view that your developers could just QUERY to find tracefiles? Maybe something like this (which of course displayed data in real time, reflecting the current status of the filesystem):

SQL> desc all_trace_files
 Name                          Null?    Type
 ----------------------------- -------- --------------------
 DIRECTORY                              CHAR(5)
 FILE_NAME                              VARCHAR2(400)
 FILE_TIME                              DATE
 FILE_BYTES                             NUMBER
 FILE_TRACEID                           VARCHAR2(100)
 FIRST_TIME                             DATE
 FIRST_ACTION                           VARCHAR2(100)
 FIRST_MODULE                           VARCHAR2(100)
 FIRST_SERVICE                          VARCHAR2(100)
 SID                                    NUMBER
 SERIAL                                 NUMBER
 ERROR                                  VARCHAR2(400)

Sure, this looks nice – maybe in the next version of Oracle. When you upgrade to it, 10 years from now. But what about NOW – your old 10g database running on a huge mainframe?

We all know how nice it would be if your developers run a query like this… today… on almost any database:

SQL> select directory, file_name, file_traceid, first_action from all_trace_files 
  2  where first_action is not null;

DIREC FILE_NAME                                FILE_TRACE FIRST_ACTION
----- ---------------------------------------- ---------- ----------------------------------------
UDUMP ardentp1_ora_12386502.trc                           Test Window - New
UDUMP ardentp1_ora_22347974.trc                           Test Window - New
UDUMP ardentp1_ora_22675498.trc                           SQL Window - xml_file_writing_te
UDUMP ardentp1_ora_24248536.trc                           SQL Window - test_cases.sql
UDUMP ardentp1_ora_24641726.trc                           DEQ
UDUMP ardentp1_ora_2621852.trc                            Test Window - New
UDUMP ardentp1_ora_3342766.trc                            problemTbsp
UDUMP ardentp1_ora_4063428.trc                            SQL Window - New
UDUMP ardentp1_ora_45350944.trc                           Debug Test Window - New
UDUMP ardentp1_ora_45547538.trc                           Main session
UDUMP ardentp1_ora_48103606.trc                           SQL Window - SELECT * FROM ext.e
UDUMP ardentp1_ora_50593842.trc                           SQL Window - test_cases.sql
UDUMP ardentp1_ora_50790578.trc                           SQL Window - New
UDUMP ardentp1_ora_51380240.trc                           Debug Test Window - New
UDUMP ardentp1_ora_52429026.trc                           UserBlock
UDUMP ardentp1_ora_52691004.trc                           SQL Window - New
UDUMP ardentp1_ora_60817584.trc                           Test Window - New
UDUMP ardentp1_ora_61538330.trc                           Test Window - New
UDUMP ardentp1_ora_62783512_schn_02.trc        schn_02    Test Window - New
UDUMP ardentp1_ora_62783512_schn_03.trc        schn_03    Test Window - New
UDUMP ardentp1_ora_6291778.trc                            Test Window - New
UDUMP ardentp1_ora_63176754.trc                           Test Window - New
UDUMP ardentp1_ora_66388158.trc                           Test Window - New
UDUMP ardentp1_ora_8192480.trc                            Test Window - New
UDUMP ardentp1_ora_8651144.trc                            SQL Window - test_cases.sql
BDUMP ardentp1_m000_12517418.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_20119784.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_24248408.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_m000_45154330.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_mmnl_52560064.trc                          Monitor Tablespace Thresholds
BDUMP ardentp1_mmon_52756580.trc                          Monitor Tablespace Thresholds

31 rows selected.

And of course – last but not least, you’d like your developers to actually be able to download the tracefile themselves, right? Actually, this has already been done – Dion Cho blogged some sample code about two and a half years ago. He has a very elegant solution using a pipelined PL/SQL function and UTL_FILE.

http://dioncho.wordpress.com/2009/03/19/another-way-to-use-trace-file/

Just one minor tweak would be needed, to grab files from more than one directory. (In addition to your session traces, you might need to get logs from parallel query slaves or DBMS_JOB/SCHEDULER processes.)

Not only can you write queries to your heart’s content on this, but just about every developer environment on the planet can easily take this query output and save it to a file – then they can run the profiler of their choice on the tracefile. That would be cool, right?!

SQL> select * from table(textfile('ardentp1_ora_52691004.trc','UDUMP')) where rownum<20;

COLUMN_VALUE
-------------------------------------------------------------------------------------------
/u0001/app/oracle/admin/ardentp/udump/ardentp1_ora_52691004.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u0001/app/oracle/product/db/10.2.0/ardentp
System name:    AIX
Node name:      ardent4
Release:        1
Version:        6
Machine:        00F66BE22C00
Instance name: ardentp1
Redo thread mounted by this instance: 1
Oracle process number: 59
Unix process pid: 52691004, image: oracle@ardent4

*** ACTION NAME:(SQL Window - New) 2011-08-15 16:35:24.791
*** MODULE NAME:(PL/SQL Developer) 2011-08-15 16:35:24.791
*** SERVICE NAME:(ardentp_dba) 2011-08-15 16:35:24.791
*** SESSION ID:(572.24734) 2011-08-15 16:35:24.791

19 rows selected.

Well I AGREE that this would be cool! And as it happens, the customer I’m working with now would find it very useful. And honestly… it’s really not that complicated to code something like this. So last night I stayed late a few hours to write it up and test it. It’s mainly build on a pipelined java function, and of course a minor tweak on Dion’s code for getting the file.

Basically, every time you query the view it scans the first 30 lines of all files in the bdump and udump directories for timestamps and identification information (module, action, service, sid, serial). This information is combined with basic file stats (name, size, last-modified-date) and the tracefile_identifier is calculated from the filename. Any errors encountered are reported in the error column (file permissions, etc).

You can download the entire thing at my github repo (specifically, the files tracefile_find.txt and tracefile_get.txt). All examples in this blog post are from a real system – nothing here is cooked, except that I changed database and host names to protect the innocent.

Enjoy!

Security Addendum

As pointed out in the comments after I published this post, there are some important security considerations with allowing access to trace files. A good explanation can be found in Pete Finnigan’s blog article: Is it possible to steal data with just ALTER SESSION? To summarize: if a clever hacker is able to access tracefiles then it is possible for them to see things that the database normally protects. This certainly includes table data, and it may even include passwords for other database accounts – sometimes even in cleartext. This is very important if you’re considering any tool that grants access to tracefiles – including the scripts I’ve published here. Although he generally disapproves of any tracefile access, Pete’s comment suggested a way to reduce the risk. The second table (which actually gives tracefile contents) can be limited to DBAs and then developers can get a view which only shows files that the DBAs explicitly grant. I think that the most useful part of my program is the first view anyway, which scans tracefiles and lets you use SQL to search on module/action/service.

One further quick postscript… lest you think I was terribly irresponsible with this client, we did consider security implications before endorsing this script. Admittedly I didn’t consider every scenario that Pete has written about, but I do still think we made an appropriate decision. (Also… the script wasn’t yet installed into production when I left the client, because it had to go through an approval process itself. ) In this particular case, the small group of developers getting access to tracefiles were people who already had full access to all data in the database and they had passwords for all key application accounts. Although they didn’t have SYSDBA passwords or shell access, they essentially had everything else – including all the data. Furthermore, they were in a stage of performance troubleshooting where their productivity could be significantly increased with the ability to run many traces and quickly access the tracefiles (think Method R). And finally, these developers were on the same small, tight team as the DBAs who took care of backups and patching, all sitting together in one corner of the office building. The DBAs were involved in the decision and seemed to find their reduced workload a very positive thing – especially since they know all the developers who would get access.

One could certainly comment on the security policies of this client, and their decisions about balancing trust and productivity. But I do not think that we were circumventing the company security policies in this case. And certainly my scripts are not intended to support such activity. Nonetheless, I do think there are cases where this is a very useful script. It’s much more convenient for developers who – even if they can get shell access – may not know much about navigating unix and are more comfortable working in their SQL development environment.

[24 May 11 09:30] * cheboygan: glad to see that someone read the
  blog post though.  at least i know one person read it.
[24 May 11 09:30] * rizzo: it got re-tweeted a lot last week
[24 May 11 09:30] * rizzo: get yourself on twitter, man
[24 May 11 09:30] * cheboygan: i've been resisting...
[24 May 11 09:30] * rizzo: give in
[24 May 11 09:30] * rizzo: it calls to you
[24 May 11 09:31] * cheboygan is still fighting it
[24 May 11 09:31] * rizzo: resistance is futile
[24 May 11 09:31] * cheboygan will never be assimilated
[24 May 11 09:32] * cheboygan: is there a point to having a twitter
  acct  if it's not tied to some kind of mobile device?
[24 May 11 09:34] * rizzo: cheboygan: sure, most of my twitter
  stuff is done at my desk
[24 May 11 09:34] * rizzo: just reading and posting updates
[24 May 11 09:35] * cheboygan: hm.  i just did a search and it 
  turns out there's a twitter plugin for my chat program.  might
  give that a try actually.  wow, really didn't think i'd be so
  easily persuaded.
[24 May 11 09:49] * rizzo: Dan Norris will thank me for this
[24 May 11 09:49] * cheboygan: have to admit that i did hold out
  for an impressively long time :)

[24 May 11 09:50] * rizzo: what's your twitter handle?
[24 May 11 09:50] * cheboygan: haven't setup the account quite
  yet... give me a few more minutes :)
[24 May 11 09:50] * rizzo: heh ok
[24 May 11 09:50] * rizzo refreshes madly

[24 May 11 09:56] * rizzo: bieber4life?
[24 May 11 09:57] * cheboygan: are you kidding?  bieber4life is
  definitely taken.

[24 May 11 10:04] * cheboygan: the problem with holding out on
  something like twitter, is that the longer you wait the more
  impossible it is to have any readable kind of handle.
[24 May 11 10:06] * cheboygan: maybe i should write a blog post
  and elicit suggestions or something.  this is ridiculous.  in
  fact, even the word ridiculous is taken.
[24 May 11 10:10] * Ramen: use l33t
[24 May 11 10:11] * cheboygan: just dicovered underscore is ok.
  new possibilities.

Just a few minutes after this, I finally caved in and signed up for Twitter.  My account is @jer_s – feel free to find me over there while I try to figure out what you’re supposed to do with this Twitter thing.

• 100% class met expectations, would recommend to others

• 66% easy to follow, could use skills in working environment

• 100% already familiar with oracle, 90% use oracle daily

• 0 negative reviews of instructor (phew!)

• 1 negative review of curriculum: said practice exercises weren’t relevant but training manual was still above average.

• 0 negative “comments”

There were several positive comments such as this: “I would recommend this class to others. This setup is perfect to pick up new skills and expose what ifs w/out worrying about pressing the wrong button.”

There were also some suggestions for improvements which I’ve taken note of. The most common suggestion was this: “More time to practice / explore new concepts.”

What’s Next

The RAC Attack curriculum has been around for over three years now – and I’m still getting requests to put it on.  Seems that a number of people have felt that it’s a decent quality class and handbook.  I talked to several other Oracle instructors and speakers at Collaborate this year about the RAC Attack curriculum – including an in-depth conversation with Arup Nanda.

As a result of these conversations, I did some careful thinking and I’ve been very busy for the past few weeks since Collaborate.  And I now have three major announcements about RAC Attack:

1. Last week, I published an updated RAC Attack Lab Handbook.  The handbook is now completely self-contained for home use.  I’ve added significant new content:
   • Detailed hardware requirements (taking into consideration optimal I/O and memory configuration)
   • How to install and setup VMware environment (including networking and storage)
   • How to create the RAC Attack Classroom DVD (this is actually scripted now)

This means that you can now (finally) setup RAC Attack on your own computer – without needing any external assistance!

2. Although I created most of the labs for RAC Attack, some labs were also contributed by Dan Norris and Parto Jalili.  Last week I also obtained their permission to move all RAC Attack content under a Creative Commons Attribution-ShareAlike license – which you will see in the latest handbook.The RAC Attack lab handbook has now been donated to the Oracle professional community with a solid copyleft license.
   • New DBAs are invited to use this material at home.
   • Experienced DBAs are invited to make corrections and enhancements to the textbook.  Also you can add your own labs which work on the RAC Attack “platform”.
   • Instructors are invited to use this material partially or completely in their own classes, as they wish. You can even run your own class based solely on this material – without needing anyone’s permission. Also, you can charge for these classes – there isn’t a “non-commercial” limitation on the terms of our license.

On a related note, all code related to RAC Attack (including the jumpstart scripts and supplemental DVD build script) is now under the GPL and has been published on github for easy downloading, copying and branching.

3. Finally… how are you supposed to make corrections, enhancements and additions if you’ve only got a PDF version?  Right now the textbook sources are in OpenOffice format (with a master document and about 50 lab documents).  Although this is very well suited to print media, it’s difficult to coordinate with very many authors or editors.  I’ve decided that the best way to truly encourage collaboration with other teachers & instructors is to use the best platform on the web for collaborative open-content textbook projects: the RAC Attack lab textbook has a new home on wikibooks.

   http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home

   I’m just getting started now – but I’m hoping to have all existing RAC Attack content on this wiki textbook by the end of the month.  Please have a look at the project page and consider getting involved!

Feedback

I consider these to be some major changes.  I’ve put some careful thought into them, and I’m excited about the new direction for this class.  Let me know what you think.

Cheap Lab Environment

RAC on EC2 would be useful. (1) If you’ve never done much with RAC, then you can have a sandbox to play in – try the installer, try some parallel queries, try anything else. (2) You can setup a cluster with any version, any patch level – and see if your pl/sql or query works. (3) You could give each individual student in a class their own cluster.

And of course, the cost is very reasonable:

These numbers are based on small instances. In my observation so far the network connection is the bottleneck — and quite likely the 20 node cluster wouldn’t actually work. It would be about $25/hr to run a 10 node cluster with 4 SCSI targets on EC2 “cluster compute” nodes (w/10G network)… probably a little steep but maybe someday I’ll try running a benchmark for an hour or two just to see how fast it is. :)

Progress

I’m convinced that this will work. After a few days of working on it in between diapers, I haven’t quite gotten all the way – but I’m very close. I’ve got shared iSCSI-based block storage for ASM and two private networks which I think will support VIPs and multicast. The only two pre-checks that fail are swap space and an optional flag to NTP. The installer completes. I just need iron the kinks out of the post-install/root script execution.

I was initially trying to install 11.2.0.2 but I wimped out and reverted to 11.2.0.1 after wrestling with it for a while. I only have so much time in between diapers… :)

Everything is scripted. When you’re paying by the hour, you don’t want to waste time building the environment! Also, you don’t want to leave the environment running for days after you’ve set it up. Much better if you can start the RAC environment on-demand when you want it, and shut it down if you’re not using it.

Here’s what I’ve got so far:

jeremy@joshua:~/rac-ec2$ ./run-cluster 3 2
EC2-RAC-BUILDER Copyright 2010 Jeremy Schneider

    This program comes with ABSOLUTELY NO WARRANTY.
    This is free software, and you are welcome to redistribute it
    under certain conditions.  See included gpl.txt for details.

Creating cluster: ID #3
1 storage node(s)
2 compute node(s)

time      storage             network             compute
-----     --------            --------            --------
18:21:42  starting            starting            starting
18:22:18  ec2 load instance   ec2 load instance   ec2 load instance
18:23:44  boot O.S.
18:23:50  setup iscsi
18:23:55                      boot O.S.
18:24:10  finished
18:24:36                                          wait strg/ntwrk
18:25:27                      configure switch
18:25:37                      reboot O.S.
18:27:22                      finished
18:27:28                                          boot O.S.
18:27:48                                          configure O.S.
18:33:17                                          finished

username root/oracle
password ec2rac
hostname ec2-50-16-23-197.compute-1.amazonaws.com

This part is working pretty well. Right now I’m working on the script to actually install the oracle bits. Like I said – I’m still working out the kinks with the root script… but here’s what it currently looks like:

jeremy@joshua:~/rac-ec2$ ./install-oracle 3 2 >/tmp/ora-inst.out
19:57:19  downloading grid software from Oracle support
19:58:16  extracting grid software (/tmp/oracle/grid)
19:59:10  running cluvfy
20:00:33  installing grid infrastructure
20:28:11  running root scripts
20:30:38  downloading database software from Oracle support
20:36:16  extracting database software (/tmp/oracle/database)
20:38:02  finished!

Try It Yourself

If you’re interested in running RAC on EC2 then I’d love your help with this. You can download these programs and try them out for yourself. And let me know if you finish the install, and what steps you took. I’d really love to have a public toolkit that anyone can download, which will build RAC environments in EC2 on demand.

We had a single-instance database running on a failover cluster (RHCS). A database link existed for a related database and the connection had to pass through a firewall. The problem was the firewall: it had a rule which only allowed connections from the VIP.

The database server has two IP addresses – a system IP and a VIP. Is there any way to bind the dblink to one specific interface? (Note: we would still like the system IP to be used for other traffic.)

I couldn’t think of a way for Oracle to do that. But we did find a workaround, of sorts (though not perfect) – by using the operating system route command.

Sometimes there’s a misconception that the purpose of OS routing is just to set the default gateway. For example, in the windows networking GUI that function is visible and it’s rare to need much else. However the command-line OS route command is in fact much more powerful (on all platforms).

Here’s a detailed manual page for the linux version of route:
http://www.linuxmanpages.com/man8/route.8.php

First, in addition to setting the default gateway, the route command can also determine which physical interface to use. Second, in addition to configuration for a subnet, the route command can also do configuration for only one specific host. So the syntax we’d want is something like this:

route add -host [targetIP] dev [interface/ethX]

Of course it’s important to check that there’s nothing else connecting from the database cluster to the target host IP. Also, you’d want to link this command with the cluster VIP resource so that it follows the VIP around the cluster.

But it does seem to be one possible solution to the dilemma.