PostgreSQL Hacks - Libpq optimization

Author

Libpq optimization

Alon Goldshuv

2005-10-27, 8:14 am

In the libpq COPY interface function PQputCopyData():

/*
* Check for NOTICE messages coming back from the server. Since the
* server might generate multiple notices during the COPY, we have to
* consume those in a reasonably prompt fashion to prevent the comm
* buffers from filling up and possibly blocking the server.
*/
if (!PQconsumeInput(con
n))
return -1; /* I/O failure */
parseInput(conn);

I moved it to a different location, just a bit further, after the check for
"is output buffer full and we are ready to flush?" in the same function

if ((conn->outBufSize - conn->outCount - 5) < nbytes)
{
<here>
....
....
}

As the code comment suggests, it is extremely important to consume incoming
messages from the server to prevent deadlock. However we should only worry
about it before sending data out. Most calls to PQputCopyData don't actually
send any data but just place it in the out buffer and return. Therefore we
can perform the consumeinput/parseinput right before flushing, instead of
reading from the server every time we call PQputCopyData and not send
anything (which happens probably in 99% of the time).

Right? Or am I missing something.

This change improves COPY performance.

thx
Alon.

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Tom Lane

2005-10-27, 8:14 am

"Alon Goldshuv" <agoldshuv@greenplum.com> writes:
> As the code comment suggests, it is extremely important to consume incoming
> messages from the server to prevent deadlock. However we should only worry
> about it before sending data out.

And, unfortunately, you've broken it. The pqFlush call visible in that
routine is not the only place that may try to send data (see also
pqPutMsgEnd).

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Alon Goldshuv

2005-10-27, 8:14 am

Tom,

> And, unfortunately, you've broken it. The pqFlush call visible in that
> routine is not the only place that may try to send data (see also
> pqPutMsgEnd).

You are right, thanks for pointing that out.

Still, in pqPutMsgEnd we will be sending data only after 8K is reached,
which is about once in 80 for a 100 byte row size...

Alon.

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql
.org so that your
message can get through to the mailing list cleanly

Sponsored Links