Of course, we all know what happened after that – virtualization.  It didn’t take long before my home-built test clusters were running on VMware.  (Personally, I think that virtualization really started because of those NES and SNES emulators.  Most great achievements start with a geek who wants to play more video games.)  There are lots of people now who run RAC on virtual environments and it’s easy to find tutorials on the web for many different OS and VM combinations.

Low-Memory Linux

Something I haven’t seen many other people do is RAC with a very small memory configuration.  Like 760M of memory per server. (!)  Of course you’d only do this for a hobby setup – never on a system where you want any kind of support.  But I’m kinda cheap… and running RAC on these small VMs means that I don’t have to go buy an expensive new home computer.  My current gateway laptop with Vista Home does the job quite nicely!

10.2 and 11.1 RAC will install and run on servers with 760M of memory. But things were a little unstable at first. Now I’m the curious type… I like to fiddle with things… so I investigated a little bit.

Basic Unix Investigation

There are two basic investigation scenarios:

In this particular case, I noticed pretty quickly from the top utility that one process was consuming over 30% of the system’s memory!  (Note: in top, you can press the ‘<’ and ‘>’ keys to move the sort column left and right.  The initial sort column is %CPU.  I moved it one column to the right, sorting by %MEM.)

top - 18:41:13 up  5:28,  3 users,  load average: 0.04, 0.35, 0.60
Tasks: 180 total,   2 running, 178 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  3.7%sy,  0.0%ni, 89.4%id,  6.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    767020k total,   754296k used,    12724k free,     7084k buffers
Swap:  1540088k total,   654696k used,   885392k free,   361800k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17435 oracle    RT   0  230m 229m  31m S  0.3 30.6   0:26.21 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
 7765 oracle    15   0  464m 111m 102m S  0.0 14.9   0:04.52 ora_smon_RAC1
 7743 oracle    -2   0  445m  90m  83m S  0.0 12.1   0:12.01 ora_lms0_RAC1
 7783 oracle    15   0  440m  70m  66m S  0.0  9.4   0:05.35 ora_mmon_RAC1
20321 oracle    15   0  438m  48m  45m S  0.0  6.5   0:00.88 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8676 oracle    16   0  437m  46m  45m S  0.0  6.2   0:02.69 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8801 oracle    15   0  440m  44m  41m S  0.0  6.0   0:04.57 ora_cjq0_RAC1

This process (ocssd) is Oracle’s Cluster Synchronization Services Daemon.  It’s the process that sends and receives heartbeats from other nodes. Any delays sending or receiving those heartbeats can cause node evictions (a.k.a. server reboots) – so it’s a pretty important process! That’s why it runs with realtime (RT) scheduling priority, as you can see in the above output from top.

I was surprised that CSS uses so much physical memory – usually Linux is very good at memory management. In top, the VIRT column shows how much total memory each process is using, while the RES column shows how much actual physical memory Linux has allocated to it. It’s clear that Linux is pretty actively managing the physical memory for other processes.

A quick glance at vmstat shows that although we are actively swapping, it seems under control. This is about what I’d expect when we’re idle and all of the processes except CSS are sharing only 500M of memory:

collabn1:/home/oracle[RAC1]$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 604144  14024  34948 300632    0    0    32    86 1044 1797  1  8 84  7  0
 2  1 603912  11224  34968 303968   87    0   763    81 1083 1902  3 11 56 30  0
 0  0 604840  12660  34912 303904   25    0   253    38 1050 1975  2 11 69 18  0
 0  0 604840  12728  34924 303900    0    0    68   106 1043 1975  1  9 82  8  0
 0  0 604784  14536  34936 304140   19    0    89    71 1043 2042  1  5 81 13  0
 1  0 604752  14168  34944 304496    6    0   119    21 1040 2016  2  8 80 11  0
 0  1 604736  18732  35020 306212    0    0   384    73 1050 2489  1 10 70 19  0
 1  1 604736  11144  35232 311252  104    0  1209   128 1074 2024  2 12 38 47  0
 3  1 607900   8056  30788 307352   77 1500   504  1661 1055 2642  9 16 56 19  0
 0  0 607900   9836  30800 307360    0    0    71    70 1031 1798  1  6 85  8  0
 1  0 607884  10536  30812 307400    8    0    44    93 1031 1832  1  4 87  8  0

The SO column tells us when memory is written to disk (and removed from physical). The SI column tells us when memory is read from disk (and put back in physical). On a side note, remember that on a healthy Unix system the free memory is always small. Sometimes this is confusing at first.

Linux Process Memory Investigation

Nonetheless, I’m not happy that CSS is using 30% of my physical memory in this highly-constrained hobby environment. Why is Linux allowing this? The first clue comes simply from the output of the familiar unix ps utility:

collabn1:/home/oracle[RAC1]$ ps v -C ocssd.bin
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
17435 ?        SLl    0:08      7   588 235331 234840 30.6 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin

On Linux, the “v” flag tells ps to give information relevant to virtual memory. TRS and DRS tell me how much physical (resident) memory is used for machine executable code (text) and data, respectively. But more importantly – the STAT column gives some informative BSD-style flags about the process. That capital-L indicates that CSS has some pages that are locked into physical memory. Bingo.

If I have root access, then I can get a very detailed report on process memory usage with the pmap command. The output was a little long, so I’ve abbreviated it here:

[root@collabn1 ~]# pmap -x 17435
17435:   /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
Address   Kbytes     RSS    Anon  Locked Mode   Mapping
00110000     656       -       -       - r-x--  libhasgen11.so
001b4000       8       -       -       - rwx--  libhasgen11.so
  ... 37 more library blocks
02a0d000     100       -       -       - rwx--    [ anon ]
02a26000       4       -       -       - --x--    [ anon ]
02a27000   10240       -       -       - rwx--    [ anon ]
03427000       4       -       -       - --x--    [ anon ]
03428000   10240       -       -       - rwx--    [ anon ]
  ... 10 more anonymous blocks, half are 10240K
08048000     592       -       -       - r-x--  ocssd.bin
080dc000       4       -       -       - rwx--  ocssd.bin
  ... 30 more anonymous blocks, half are 10240K
bfe40000     148       -       -       - rwx--    [ stack ]
bfe65000       8       -       -       - rw---    [ anon ]
-------- ------- ------- ------- -------
total kB  235920       -       -       -

Interestingly, the linux pmap utility does not indicate any locked memory! I don’t know whether that output column is non-functional or if it refers only to some particular kind of locking. But at any rate, I know something is locked. I couldn’t think of anything better, so the next place I looked was in the Linux /proc pseudo-filesystem.

collabn1:/home/oracle[RAC1]$ grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Now we’re talking. The process has a total of 235920 kB of memory – and it’s ALL locked. On a normal RAC system you’d want this. Generally, important realtime processes should be locked so that they are never delayed by paging or swapping. (Remember how that could cause node reboots?)

But I personally doubt that all of the memory really NEEDS to be locked, and I think that Linux will actually do a decent job of not swapping the most important parts. And my highly constrained hobby environment will probably run much smoother if Linux has more flexibility when managing a measly 760M of memory.

Unlocking Linux Process Memory

But is it actually possible to unlock the process memory? As far as I know, Oracle provides no option to disable CSS memory locking. (For good reason.) There is the system call munlockall() – which unlocks all of a particular processes’ memory. But the CSS process itself would have to call this function. And of course it will not. Or will it?

If you’ve got root, then there’s a hacker-back-door way of doing this. Remember, you’d be crazy to try this anywhere besides a dark closet at home. And if you type too slow then CSS could reboot your machine.

But watch this…

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

[root@collabn1 ~]# gdb -p 17435 <<EOF
> call munlockall()
> quit
> EOF

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Attaching to process 17435
Reading symbols from /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin...done.
  ... 17 more Reading/Loading symbols.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f629f0 (LWP 17435)]
  ... 19 more New Threads.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libnsl.so.1...done.
  ... 13 more Loaded/Reading symbols.
0x008e7402 in __kernel_vsyscall ()
(gdb) $1 = 0
(gdb) The program is running.  Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
Detaching from program: /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin, process 17435

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:         0 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Ha. This is something you won’t find on metalink. After generating some activity on the system, the top utility shows me that Linux has significantly reduced the physical memory used by CSS.

These days I’ve actually scripted this for my home and classroom VM environments. I haven’t done a careful comparison or analysis, but it really has seemed to me that my low-memory Linux systems run noticeably smoother.

DBCA is not missing by mistake – this is a change in Oracle 11g Release 2.  There is a new server-based java GUI tool for ASM management: ASMCA.

A Few ASM-related 11gR2 Changes

You might remember that there haven’t been major changes in Oracle clustering since version 10gR1 – when Oracle introduced their Clusterware package. But 11gR2 brings big changes and big new features to database clustering.

Here are a few that I found very noticeable when installing a cluster:

New Tool: ASMCA

• You can’t defer storage configuration until after CRS installation.  Furthermore, raw/block storage is no longer supported.  Before you begin the CRS install you must either (1) install and configure a Cluster Filesystem or (2) prepare storage for ASM.
• The CRS install includes ASM.  This means that the bare-minimum space requirements are higher by about 3 or 4 GB.  (I used to run ASM and DB from a single home for VMware test/educational environments.)
• You can’t do any ASM operations when connected as sysdba!  A new role called “sysasm” was introduced in 11gR1… it’s now the only way to manage ASM.  Connect / as sysasm.
• There’s a new tool called ASMCA to manage ASM.  DBCA isn’t even installed in the ASM/Grid home.  This tool is pretty slick – it also manages the new ASM dynamic volumes and ASM cluster filesystems.

I noticed that there wasn’t much on google about ASM and DBCA.  Hopefully this will save you a few minutes of head-scratching if you’re trying out 11g Release 2 for the first time.  Happy testing!

The Thursday and Friday before OpenWorld (September 16-17), some kind folks in Michigan are bringing many of the best and brightest minds (see them here and here) to the Midwest to share their experience.  It’s like attending the best sessions from San Francisco – for less than a third of the cost.  The event is being called the Michigan OakTable Symposium or MOTS.

If anyone lives around the midwest and won’t be at OpenWorld, then I think this would be especially good.  It’s only a two day event!  But of course I’m sure there’ll also be folks who go to the Michigan Symposium and still catch a last-minute flight to San Francisco…

I’ve always been very enthusiastic about local user groups and independent talent, so this is really my kind of event.  I’m pretty sure that any enthusiasm you hear will be based solely on proven real-world usefulness – with no influence from sales commissions (direct or indirect).  It’ll probably be just as much relational as it is technical.  You can bring an especially hard problem that you can’t solve – maybe someone will have a useful tip or new approach.  But taking home an “answer” is insignificant compared with taking home a better problem-solving process.  If I were you, I’d come to these sessions looking for that.

Anyone in Illinois who is heading over for this?  I think it would be worth the drive (only 4 hours from Chicago).  Maybe we could even arrange for Illinois folks to hang out together at some point.

Reminder – seating is limited and prices go up Sunday.  If you are interested, then you might want to try for management sign-off and event registration by tomorrow.

What?!! You call this a volume manager and it doesn’t support a hot spare disk?! That’s it, we’re going back to filesystems!

Not so fast.

You’ll remember that the purpose of a hot spare disk is to restore protection in the event of a disk failure. When the RAID controller or volume manager detects a failed disk (R.I.P.), it uses the spare and reproduces a copy of the complete now-missing disk (R.I.P.) by reading the mirror or parity data.

Restoring Protection Without A Hot Spare Disk

In fact, ASM does even better. Instead of a hot spare disk, it has hot spare capacity – spread across all of the disks. There are two immediate effects of this: (1) no loss of potential performance from a usable disk head sitting idle and (2) the hot spare capacity is available for an emergency situation (like the archivelog volume filling up).

How is this possible? The key of course is understanding how ASM mirrors data – by extent rather than disk. So if I allocate three extents on disk 1, the first could be mirrored on disk 10, the second on disk 11 and the third on disk 12.

Now if disk 1 fails, I don’t need to rebuild the disk – I only need to ensure that each extent is mirrored. For example: lets copy the first extent to disk 11, the second to disk 12, and the third to disk 10. Once everything has two copies, my data is protected – even though the first disk was never rebuilt!

You’ll also notice that the I/O for the rebuild operation is spread across all of the disks. Faster and better.

So there actually is a “hot spare” of sorts – spare capacity instead of a spare disk. And unlike RAID, this hot spare isn’t optional: ASM will always reserve enough spare capacity to bring the diskgroup back to its proper level of redundancy after a failure.

Failure Groups Mean A Little Less Usable Space

WAIT WAIT WAIT!! I know you’re excited, but before you press enter and execute that migration script, I need to tell you about one more very important detail. How does ASM know that amount of spare capacity to reserve? The size of the largest disk, right? NO!!!

This is, in fact, what failure groups are for!! (You always wondered, right?!) Because ASM doesn’t mirror across disks – it mirrors across failure groups. And accordingly, ASM will always reserve enough spare capacity to restore protection after losing the largest failure group.

Now it’s important to understand how ASM will report this. Lets look quickly at asmcmd’s lsdg command. This command (11gR1) gives us four important numbers: Total_MB, Free_MB, Req_mir_free_MB and Usable_file_MB. The 11gR2 Storage Administrator’s Guide gives good descriptions of the output:

Putting it all together, we see that Req_mir_free_MB should be equal to the largest failure group. Secondly, Usable_file_MB is the most important number — that’s how much space we can actually use. And with normal (2-way) mirroring, it should always be half of the available raw free space. And available raw free space is total raw free space minus Req_mir_free_MB.

Example

So let’s see if we can observe this on a system in the wild. We’ll revisit one that I briefly described last year.

The system had 46 local disks (500M each). The disks were attached to six controllers: two controllers had six disks each, the other four controllers had eight disks each. We configured ASM so that each controller is a failure group.

First lets get the output of lsdg, which we’ll verify step-by-step:

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  NORMAL  N         512   4096  1048576  18041706  18041326          3137688         7451819              0  DATA01/

Now – to begin our verification – lets look at the exact size of a single disk:

# sfdisk -suM /dev/sdav1
401624968

The Linux kernel is reporting KB and we need MB – so we’ll divide by 1024.

401624968 /1024 = 392211.8828125 M

ASM uses “allocation units” of 1M and can’t use a partial AU – so we truncate that to 392211. Now there were 46 disks – so the total amount of space available should be 392211 * 46 = 18041706. And this perfectly matches Total_MB from the lsdg output! So far, so good!

Next, lets look at the largest failure group. The four large failure groups were each the same size with eight disks: 392211 * 8 = 3137688. This perfectly matches Req_mir_free_MB from the lsdg output! Lovely!

Finally, lets see how much space I can actually use for datafiles and such – accounting for mirroring. The available raw free space should be Total_MB minus our calculated Req_mir_free_MB: 18041326 – 3137688 = 14903638. I’m using normal mirroring, so my available/usable free space should be half of that: 14903638 / 2 = 7451819. Perfectly matches Usable_file_MB from the lsdg output. Wow.

The Big Caveat

Sounds amazing, right? Well there is one very important caveat. I alluded to it earlier: Oracle will allow you to use this reserved “hot spare” capacity. This is very different from a traditional “hot spare” disk. What happens if you use the space? (1) The “usable free space” reports a negative number!!! (2) If a disk fails, then you will not lose any data – but ASM cannot automatically restore protection until you replace the disk or give ASM additional storage somewhere else.

That sound serious! What’s worse, you can get into this situation without even realizing it. If you create a datafile that’s bigger than the available space then ASM won’t warn you – it will just create the datafile and you lose your protection from double-failure.

You get alerts when your traditional filesystems are filling up. (How many times has your inbox been flooded because of those stupid archivelogs?!) If you deploy ASM, then MAKE SURE YOU HAVE ADEQUATE ALERTING. Oracle’s management tools can help out, and there are other solutions too. Don’t overlook it!

Conclusion

At this point, it should be evident that anyone with storage background needs to be careful around ASM. It doesn’t work the same as a traditional volume manager or RAID controller. Oracle Corp really does love SUPER EXCITING NEW WORDS (aka marketing buzz)… but this one isn’t just a matter of terminology. There is some real difference in the underlying concepts.

A few key takeaways:

• To automatically restore protection after a failure, ASM reserves spare capacity instead of a spare disk.

• In V$ASM_DISKGROUP and asmcmd’s lsdg command, Total_MB and Free_MB report raw capacity while Usable_file_MB reports usable capacity based on the default mirroring level.*

• ASM will allow you to use the spare capacity, after which it will report negative free space and won’t tolerate a double-failure. Make sure you have adequate reporting so that this is always caught and addressed quickly.

*Note: Various objects can have different mirroring levels in ASM.

Well I think that this should serve as a good primer on ASM and mirroring. Let me know if I’ve left anything out!

A Short History

We first ran the RAC Attack last year at Collaborate in Denver. On the first day of Collaborate we heard that one of the computer labs was going to be sitting empty for several hours on Wednesday. Being such a fervent believer in hands-on learning, this sounded like a terrible waste! So at the last minute we arranged to deliver the RAC Attack hands-on lab during the open slot. If I remember right, we had something like 20 people and everyone seemed to really come away having learned something new. The lab focused on getting a running system.

In August when we ran RAC Attack as a standalone workshop in Chicago. In addition to having several excellent speakers, we dramatically expanded the scope of the lab to include topics like backup & recovery, services, and clusterware. During the full day allocated, only a few people completed every possible exercise.

Five Reasons To Sign Up

This year at Collaborate we’re doing it again. The lab will again be expanded and improved when we bring it to Orlando. Furthermore, I’m taking requests – if there’s a topic you’d like to learn about then leave a comment and I’ll try to get it included! I’m not sure how quickly the session will fill, but I know that there are a limited number of seats – so you should sign up soon.

Here are five reasons I think you should come:

1. Get exposure to and practice on all of the latest software, including VMware Server 2.0.1, Oracle Enterprise Linux (same as RedHat) 5.3, and Oracle RAC 11.1.0.7

2. There are hands-on challenges for every experience level. Meticulous step-by-step guide for beginners (with the choice to skip difficult steps), start-from-scratch or instant running environment for masters.

3. Learn from professionals with years of RAC experience. Several of us will be on-hand to answer questions and assist with the labs. If you have never touched a cluster database then we are here to help you complete the labs and understand what you’re doing!

4. Choose your own adventure. ASM or OCFS2. Parallel SQL or clusterware callouts. Got something specific you want to learn about? Leave a comment and I’ll try to get it included.

5. Walk out with a lab binder that tells you how to do it all on your own laptop after you get home. It won’t be possible to finish all the labs during the timeslot we have at Collaborate, but you can finish them later!

Finances are tight and many companies are cutting training budgets – but remember that generally speaking, Collaborate is one of the best value opportunities for Oracle training that’s available in the United States. You have world-class educational sessions and networking opportunities, complimented by top notch deep-dive university seminars such as RAC Attack. (Well… I’m a little biased about this session!) And with a slightly greater degree of independence you won’t just hear the corporate messages – it’ll be a little easier to find real-life users telling the great, the good, the bad and the ugly.

Hope to see you there!

But it turns out that ASM actually handles this quite nicely – the “usable space” is used only for display and alerting. If a datafile needs to autoextend or if an archivelog needs to be created then ASM will keep using space until it completely exhausts everything physically available. It will go ahead and use the “reserved space” and just report a negative amount of free space! Here’s the output of my test confirming this behavior:

SQL> alter database datafile
2>  '+TEST01/js1_rh5lab20/datafile/filler.256.681667743' resize 18G;

Database altered.

.

rh5lab20$ asmcmd dgs
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB
MOUNTED  NORMAL  N         512   4096  1048576     38561     1529

  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
             7844           -3157              0  TEST01/

.

rh5lab20$ asmdf
Filesystem        SizeKB     UsedKB    AvailKB    Use% Mounted on
+TEST01         15727104   18959872   -3232768 120.56% TEST01

There is also a second way to get negative usable space in ASM. If you do experience a failure, then ASM will restore your redundancy level making new copies of the data that was on the failed disk(s). It will use the “reserved space” on your disks. After the rebuild is complete, ASM will recalculate the amount of reserved space it needs to protect from a second failure. If you were using most of the space before the first failure then ASM might report a negative amount of usable space after the rebuild.

It’s important for you to remember that negative free space doesn’t mean your data isn’t safe – you just won’t be able to restore redundancy after the failure. In other words, if you have negative free space then you cannot physically survive two subsequent failures without data loss (or three with high redundancy/triple mirroring). But ASM allows the negative numbers because it’s smart enough not to hang your database when it can be avoided (for example if archivelogs fill up the reported usable space).

This is an worked-through example of how ASM reports free space, taking into consideration “hot spare capacity” that is reserved to restore redundancy after the largest failure group dies.

ASM has 46 local disks attached to 6 controllers
- two controllers have 6 disks, other four have 8
- ASM has failure groups by controller

===========================================================================

Each disk is partitioned. Get the size of the ASM partition according to
the Linux kernel. All partitions are the same size.

# sfdisk -s /dev/sdav1
401624968

Now get the size info from ASM.

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB
MOUNTED  NORMAL  N         512   4096  1048576  18041706  18041326
Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
        3137688         7451819              0  DATA01/

===========================================================================

-- size of individual disk in MB is kernel-reported KB divided by 1024
401624968 /1024 = 392211.8828125 M

-- ASM can't use partial AU's so truncate and we get Total_MB
392211 * 46 = 18041706

-- largest failure group [controller] has 8 disks so get Req_mir_free_MB
     (amount of "hot spare" capacity that is reserved to restore redundancy
     after a controller failure)
392211 * 8 = 3137688

-- asm reports that some space is used; use "Free_MB" to see total free space
= 18041326

-- with normal-mirroring, usable space is (totalfree-spare)/2
(18041326 -  3137688) / 2 = 7451819

We’re internally debating whether or not we should include OCFS2 in this design right now, and I’m curious if anyone has arguments one way or the other to share. Our standard design on Solaris does utilize a cluster filesystem and we would welcome a similar design, but there are some concerns about the readiness, stability and future of OCFS2.

OCFS2 is being considered for these four use cases:

• database binaries (vs local files or NFS)
• diag top (11g) or admin tree (10g) (vs local files or NFS)
• archived logs
• backups

Other files will be stored in ASM.

I have seen mention in blogs such as http://bigdaveroberts.wordpress.com/ of something called ASMFS in 11gR2 and I’m wondering – will this feature (if included) have any impact on Oracle’s commitment to OCFS2 development? Could Oracle conceivably develop a whole new cluster filesystem and put their full weight behind it as they did for ASM storage, leaving OCFS2 as a lower priority for new features and improvements? Has Oracle demonstrated significant commitment to OCFS2 development and support in the past, and is this a mature enough technology for wide-scale deployment?

Just looking for opinions. :)

ASM Extents

However the manual states that the opposite is true – striping based on AU instead of data extent – in two different places on that page. First, just before the illustration:

The ASM coarse striping is always equal to the disk group AU size, but fine striping size always remains 128KB in any configuration (not shown in the figure). The AU size is determined at creation time with the allocation unit size (AU_SIZE) disk group attribute. The values can be 1, 2, 4, 8, 16, 32, and 64 MB.

And again right after it:

To stripe data, ASM separates files into stripes and spreads data evenly across all of the disks in a disk group. The stripes are equal in size to the effective AU. The coarse-grained stripe size is always equal to the AU size. The fine-grained stripe size always equals 128 KB; this provides lower I/O latency for small I/O operations such as redo log writes.

As far as I can tell, it seems to be a contradiction in the manual. I commented on the documentation page and I’ve opened an SR; I’ll post an update when I get a reply.

Update 4:45pm:

The SR got answered very quickly (less than an hour) – and said that striping is based on the data extent size. The usage of the term “allocation unit” was what I found confusing (even in the SR resolution)… but at least we have an answer. :)

Suppose you have a diskgroup with 1MB AU size. After a particular file has 20000 extents, the allocation unit for that file becomes 8MB.

allocation unit = extent size
40000>extent>20000 => FILE_AU=DISKGROUP_AU*8.

Here’s the definitive quote from the SR: “[After 20000 extents] the 8meg AU will be striped, with the first 8meg AU on disk one, the second 8meg AU on disk 2 and so on.”

This would apply for both striping and mirroring. My gut feeling was right; I was just a bit confused by the definitions of extents and allocation units in the manual (which are incorrect).

Actually it’s part of a bigger question of how we develop elegant software – other related design issues in this project are clear end-to-end traceability (requirement -> source code version -> released binary package), solid change control, integrated testing framework and readiness for automation (build and test). But the one I’m interested in right now is this: how can I develop a robust version numbering strategy?

First of all, there really isn’t a one-size-fits-all “correct” version numbering scheme. The best way to choose version numbers is influenced by your software development philosophy (and resultant development lifecycle), project management approach and development team. (Globally distributed informal volunteer network or local engineers on payroll?) So the best solution for my current project might not be the best solution for my next project.

But lets take a closer look at this project. First, a little background on the art of version numbering. In the open-source world, many projects follow a three-digit version: MAJOR.MINOR.PATCH. Also, it’s clear that the version number is closely tied to the development lifecycle. You can find plenty of example discussion from open-source projects such as the Linux kernel, Linux desktop distros, Eclipse, Apache APR and FarCry CMS. Before the 2.6 series the linux kernel used an even-odd scheme to distinguish between unstable and stable releases but that has now been abandoned and most people seem to regard this as a good thing.

The best overview of the topic that I could find was the wikipedia article on Software Versioning. The section about pre-release versions does a good job of describing one requirement of my current project. I also found the section about internal version numbers interesting; I’ve worked on projects in the past with that strategy.

I found an article on linux.com written by Nathan Willis a few years ago offering some principles. I agree with his first and third suggestions: “Pick a numbering scheme, then don’t change it. Ever.” and “Make friends with infinity.” Best to think this thing out at the front end of a project (before 1.1) and you shouldn’t be bothered by big version numbers. However I don’t buy his second point about the meaning of the decimal; I think most everyone today is used to the meaning of the decimal in version numbers.

For some alternate viewpoints I came across something Zack Weinberg wrote back in 2002. He states strongly that the decimal point isn’t mathematical, which I agree with. However contrary to his article, in our development lifecycle the test releases do need version numbers. But he has an interesting suggesting of using date-based numbers for development snapshots – that’s an interesting idea which I’ll have to think about. His list if ways to do it wrong has a good point about distinguishing between version x and version x.0 although I’m not in full agreement with the other list items. I actually find the GNU suggestion for numbering test versions very interesting – I’ll have to give that some thouht. (Tests versions for 4.6 are numbered 4.5.90 – 4.5.99)

Ultimately I don’t think either of these two sets of guidelines is robust enough for me – especially for repackaging the oracle database binaries. Some of these might seem obvious, but I thought a good place to start might be by listing what I would consider to be requirements for an elegant and robust version numbering scheme:

1. elegant, clear, understandable version numbers that are as simple as possible
2. support for single-digit or double-digit versions (1,2,3,4 or 1.0, 1.1, 1.2 ,2.0)
3. versions for pre-release builds such as test builds (for the quality process) and early-access beta builds.
4. very clear difference between pre-release and release versions, so someone doesn’t mistakenly install a pre-release build when they want a production build.
5. names need to sort correctly – pre-release before release. it’s critical that RPM and RHN sort them right; it would be nice if unix “ls” did too.
6. support for continuing development. after release 1.0, the 2.0-beta version should sort after release 1.0 and before release 2.0
7. support for special branch versions – version 1.0+patch12345, where patch12345 is not included in version 2.0 which has different patches
8. clear difference between pre-release and release versions on special branches.

I have to be honest – this is tough. I’ll be giving it some thought over the next few weeks. Let me know if you have any suggestions. :)