Oracle Server - Storage (EMC) layout advice..

Author

Storage (EMC) layout advice..

NetComrade

2005-07-26, 8:23 pm

Hello,

We have recently received an EMC CX500, which is going to be dedicated
exclusively for database use. The storage array is going to be
attached to 3 v40Z Opteron servers, which will be running
Red Hat AS 4.0
Oracle 9i
Veritas Cluster/DMP 4.1 (to be released Aug 1st)
no RAC, databases on different hosts support different (but often
similar) applications.

Our EMC setup involves 4x15drive shelves with 73G 15K drives (for a
total of 60 drives)

If you know EMC, you might know that the first 4-6 disks in EMC array
are semi-reserved for array needs (~12G of each disk is unusable).
Therefore, the first 6 disks we'll have either 3x3 RAID 10, or 3x
RAID1s (please advice, in the past, when using JBODs under VxVM, I've
noticed that striping does slow down log writes slightly, however, 3
disks would even out IO better). We'll carve a few small LUNs out of
these to present to host for redo log placement

Then we plan to use the next 8 disks in the first shelve also in RAID
10 configuration for archived log files. EMC recommends that one
disables cacheing for archived logs, so that cache is not 'saturated'
with unneeded data.

This leaves 1 disk left on shelve one (which we'll leave as hot
spare), and then pretty much 3 empty shelves for
data/indexes/system/rbs. Since EMC does not support more than 16
drives per LUN, we are thinking to use MetaLUNS in order to utilize
most disks and spready the IO evenly across the rest of the drives.
with about 45 disks, we were thinking of creating nine or ten 2x2 RAID
10 groups, and striping across them.. (using another 36 or 40 drives
respectfully). However, this creates a stripe size which I am not that
comfortable with. With default 'stripe unit size' or (stripe element
size, as EMC calls it) of 64K, a stripe size of a 2x2 RAID10 LUN
becomes 128K (2x64K), and an 8 x 2x2 RAID10 MetaLUN would have a
stripe size of 1M, while adding 9th 2x2 LUN to MetaLUN will increment
that by 128K.

So, question is, should I worry about using 1M + 128K stripe (metalun
stripe) size, or even 1M + 256K metalun stripe size? Theoretically, it
shouldn't matter much, provided max io size could be higher than 1M
on OS, database, file system and HBA.

I am pretty sure Oracle can do more than 1M, so can Veritas File
System. I couldn't verify if Linux can do more than 1M (but I think it
can), and couldn't find any info on Emulex LP9802-E HBA.

Your thoughts?

BTW, the reason we thought 4 disk RAID10's striped across are good, is
b/c it would be easy to either add or substruct from a MetaLUN, but
we'd be happy to hear your opinions as well..

I have also yet to completely understand how "element size multiplier"
affects all of this. It's not clear from the docs I've reviewed so
far.

Any advice, pointers is greately appreciated.

Thanks.

........
We use Oracle 8.1.7.4 and 9.2.0.5 on Solaris 2.7 boxes
remove NSPAM to email

Sybrand Bakker

2005-07-26, 8:23 pm

On Tue, 26 Jul 2005 21:42:12 GMT, netcomradeNSPAM@book
exchange.net
(NetComrade) wrote:

>Your thoughts?

The budget didn't include a consultant to set this up?
Because obviously this is way beyond the scope of a newsgroup.

--
Sybrand Bakker, Senior Oracle DBA

NetComrade

2005-07-26, 8:23 pm

On Tue, 26 Jul 2005 23:56:51 +0200, Sybrand Bakker
<postbus@sybrandb.demon.nl> wrote:

>On Tue, 26 Jul 2005 21:42:12 GMT, netcomradeNSPAM@book
exchange.net
>(NetComrade) wrote:
>
>
>The budget didn't include a consultant to set this up?
>Because obviously this is way beyond the scope of a newsgroup.
>

Consultant or not.. I always try to get another opinion and full
understanding .. the EMC white papers talk about separating indexes,
data, etc.. I am semi-sure, that once we get to the consultant part,
we'll hear the same thing..

Besides EMC specific question, I also had a question on non-standard
stripe sizes of 1M + 128K and/or 1M + 256K, and was curious if anyone
had an opinion on it, which really depends (in my opinion at least),
on all components involved (but might have other sideaffects/drawbacks
which I was seeking to receive).
........
We use Oracle 8.1.7.4 and 9.2.0.5 on Solaris 2.7 boxes
remove NSPAM to email

bdbafh@gmail.com

2005-07-26, 8:23 pm

I'm partial to 8 disk RAID 10 volumes for datafiles.
how about if you create one on each tray for datafiles and stripe
across those?
you could then use the remaining 7 drives per tray for 4 disk RAID 10
and 2 disk RAID 1 vols with a hot spare.
Aim for a 1 MB stripe on the 8 disk RAID 10 vols, as your OS supports
that for a max IO size and based upon your db_block_size (or the block
size for each tablespace) and the dbfmbrc oracle should be able to
issue 1 MB read requests - in theory.

kinda cool having the volume sizes increase by powers of 2.

Are you planning on allocating any LUNs to ASM?
Are you planning on reserving any LUNs for backup sets?

kudos on planning to run a new version of clusterware that hasn't even
been released yet - you are most brave. Are you going to run 10g R2 on
that as well?

-bdbafh

NetComrade

2005-07-27, 1:24 pm

On 26 Jul 2005 17:04:05 -0700, bdbafh@gmail.com wrote:

>I'm partial to 8 disk RAID 10 volumes for datafiles.
>how about if you create one on each tray for datafiles and stripe
>across those?
>you could then use the remaining 7 drives per tray for 4 disk RAID 10
>and 2 disk RAID 1 vols with a hot spare.
>Aim for a 1 MB stripe on the 8 disk RAID 10 vols, as your OS supports
>that for a max IO size and based upon your db_block_size (or the block
>size for each tablespace) and the dbfmbrc oracle should be able to
>issue 1 MB read requests - in theory.

Do you know if OS supports higher than 1M?
db_block_multi_block
_read_count can be set to any number.. that
doesn't mean OS will support it.

>kinda cool having the volume sizes increase by powers of 2.
>
>Are you planning on allocating any LUNs to ASM?

No.. and it'll be running 9i..

>Are you planning on reserving any LUNs for backup sets?

No, I don't believe in backups running on the same SAN.

>kudos on planning to run a new version of clusterware that hasn't even
>been released yet - you are most brave. Are you going to run 10g R2 on
>that as well?

This release is primarily to support AMD x64 chips in 64bit mode, I
don't think there are any new significant features.. Additionally, we
don't really use all of functionality. I just wish all these vendors
worked better together... AMD servers came out more than a year ago,
Veritas is only coming around now.. redhat 4 came out a number of
months ago, Oracle and EMC are just coming around..

-a

........
We use Oracle 8.1.7.4 and 9.2.0.5 on Solaris 2.7 boxes
remove NSPAM to email

Matthias Hoys

2005-07-27, 8:23 pm

"NetComrade" < netcomradeNSPAM@book
exchange.net> wrote in message
news:42e7c437.700706828@localhost...
> On 26 Jul 2005 17:04:05 -0700, bdbafh@gmail.com wrote:
>
>
> Do you know if OS supports higher than 1M?
> db_block_multi_block
_read_count can be set to any number.. that
> doesn't mean OS will support it.
>

And higher values for DMBRC might "cheapen" the use of full-table scans
instead of indexes so that could be a non-optimal side-efffect.

Matthias

NetComrade

2005-07-27, 8:23 pm

We actually most likely won't even set db_block_multi_block
_read_count
high in regular operation (this is a transactional database), but at
night we might set it high for batch jobs on per-session level.

During the day the parameter will equal to stripe unit (element) size,
most likely, to discourage FTS.

-A

On Wed, 27 Jul 2005 21:31:03 +0200, "Matthias Hoys"
< idmwarpzone_NOSPAM_@
yahoo.com> wrote:

>
>"NetComrade" < netcomradeNSPAM@book
exchange.net> wrote in message
>news:42e7c437.700706828@localhost...
>
>And higher values for DMBRC might "cheapen" the use of full-table scans
>instead of indexes so that could be a non-optimal side-efffect.
>
>
>Matthias
>
>

........
We use Oracle 8.1.7.4 and 9.2.0.5 on Solaris 2.7 boxes
remove NSPAM to email

Sybrand Bakker

2005-07-27, 8:23 pm

On Wed, 27 Jul 2005 20:33:42 GMT, netcomradeNSPAM@book
exchange.net
(NetComrade) wrote:

>We actually most likely won't even set db_block_multi_block
_read_count
>high in regular operation (this is a transactional database), but at
>night we might set it high for batch jobs on per-session level.
>
>During the day the parameter will equal to stripe unit (element) size,
>most likely, to discourage FTS.
>
>-A

Could you *PLEASE* stop top-posting and wasting bandwith?

--
Sybrand Bakker, Senior Oracle DBA

NetComrade

2005-07-29, 1:23 pm

On Wed, 27 Jul 2005 23:48:26 +0200, Sybrand Bakker
<postbus@sybrandb.demon.nl> wrote:

>On Wed, 27 Jul 2005 20:33:42 GMT, netcomradeNSPAM@book
exchange.net
>(NetComrade) wrote:
>
>
>Could you *PLEASE* stop top-posting and wasting bandwith?

Err.. my apologies.. i keep forgetting that this is irritating, and
can't change the default behavior of my news client.
........
We use Oracle 8.1.7.4 and 9.2.0.5 on Solaris 2.7 boxes
remove NSPAM to email

Paul

2005-07-30, 7:23 am

netcomradeNSPAM@book
exchange.net (NetComrade) wrote:

>Err.. my apologies.. i keep forgetting that this is irritating, and
>can't change the default behavior of my news client.

What is there about your newsgroup client that stops you from hitting
Control-End?

I, like you, use FreeAgent (an excellent piece of software) and have
no problems not top-posting.

Paul...

--

plinehan __at__ yahoo __dot__ __com__

XP Pro, SP 2,

Oracle, 9.2.0.1.0 (Enterprise Ed.)
Interbase 6.0.1.0;

When asking database related questions, please give other posters
some clues, like operating system, version of db being used and DDL.
The exact text and/or number of error messages is useful (!= "it didn't work!").
Thanks.

Furthermore, as a courtesy to those who spend
time analysing and attempting to help, please
do not top post.

hpuxrac

2005-07-30, 9:23 am

Sorry net I am a little late on this posting. My systems sound pretty
similar.

You don't want to think about any other stripe size than 1M in my
opinion.

The DBMRC I have set at ( forget ) either 8 or 16 with an 8k block
size. Tom Kyte has a good section as I believe Steve Adams does also
about evaulating this parameter even if you set it higher and the
system can handle it doesn't mean you are better off performance wise.

Personally I would be inclined to not worry so much about what disks
that EMC carves out little portions for the sym to do it's own internal
management thing. I would recommend more on deciding what LUN size you
want and why. If you go with 8+gig LUN size you have less flexibility
in configuring what goes where but also less admin.

One of the big picture things you want to think about is BCV space and
how much you need as these can be very handy for an instant (almost)
backup and restore capability.

Sure you can break apart the data and index into different tablespaces
if it makes you feel better about it.

Some people still do carve out separate disks for redo in an EMC
environment but many do not. If you have time to setup and test both
ways do some benchmarking and if you never ( I don't ) see much in
terms of wait events for the logs ... that's where you want to be.

Sponsored Links