PostgreSQL Performance - Re: [PATCHES] COPY FROM performance improvements

Author

Re: [PATCHES] COPY FROM performance improvements

Mark Wong

2005-07-22, 8:24 pm

On a single spindle:

$ time dd if=/dev/zero of=bigfile bs=8k count=2000000
2000000+0 records in
2000000+0 records out

real 2m8.569s
user 0m0.725s
sys 0m19.633s

None of my drives are partitioned big enough for me to create 2x RAM
sized files on a single disk. I have 16MB RAM and only 36GB drives.
But here are some number for my 12-disk lvm2 striped volume.

$ time dd if=/dev/zero of=bigfile3 bs=8k count=4000000
4000000+0 records in
4000000+0 records out

real 1m17.059s
user 0m1.479s
sys 0m41.293s

Mark

On Thu, 21 Jul 2005 16:14:47 -0700
"Luke Lonergan" <llonergan@greenplum.com> wrote:

> Cool!
>
> At what rate does your disk setup write sequential data, e.g.:
> time dd if=/dev/zero of=bigfile bs=8k count=500000
>
> (sized for 2x RAM on a system with 2GB)
>
> BTW - the Compaq smartarray controllers are pretty broken on Linux from a
> performance standpoint in our experience. We've had disastrously bad
> results from the SmartArray 5i and 6 controllers on kernels from 2.4 ->
> 2.6.10, on the order of 20MB/s.
>
> For comparison, the results on our dual opteron with a single LSI SCSI
> controller with software RAID0 on a 2.6.10 kernel:
>
> & #91;llonergan@stinge
r4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k
> count=500000
> 500000+0 records in
> 500000+0 records out
>
> real 0m24.702s
> user 0m0.077s
> sys 0m8.794s
>
> Which calculates out to about 161MB/s.
>
> - Luke
>
>
> On 7/21/05 2:55 PM, "Mark Wong" <markw@osdl.org> wrote:
>
>

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql
.org so that your
message can get through to the mailing list cleanly

Luke Lonergan

2005-07-23, 8:38 pm

Mark,

On 7/22/05 12:47 PM, "Mark Wong" <markw@osdl.org> wrote:

> On a single spindle:
>
> $ time dd if=/dev/zero of=bigfile bs=8k count=2000000
> 2000000+0 records in
> 2000000+0 records out
>
> real 2m8.569s
> user 0m0.725s
> sys 0m19.633s

This is super fast! 124MB/s seems too fast for true write performance on a
single spindle.

> But here are some number for my 12-disk lvm2 striped volume.

So, software striping on how many controllers?

> $ time dd if=/dev/zero of=bigfile3 bs=8k count=4000000
> 4000000+0 records in
> 4000000+0 records out
>
> real 1m17.059s
> user 0m1.479s
> sys 0m41.293s

Again - super fast at 416MB/s. How many controllers?

When we had our problems with the cciss driver and the smartarray 5i/6
controllers, we found the only way to get any performance out of them was to
run them in JBOD mode and software stripe. However, when we did so the CPU
usage skyrocketed and the performance of simple SCSI adapters was 50% faster
with less CPU consumption. These numbers show 100*(42.8/67) = 64% CPU
consumption, I'd expect less with 2 simple U320 SCSI controllers.

- Luke

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Sponsored Links