Slony1 PostgreSQL Replication - sequencesetvalue() fails after power outage

Author

sequencesetvalue() fails after power outage

Chris Mount

2006-04-03, 11:31 am

2 node cluster, PG 8.1.3 /slony 1.1.5
Node2 writes logshipping files

After a power failure over the weekend. Postgres comes up fine, but when
starting slony node2 it quickly dies with the following in the error log.
node1 starts fine. I am kind of a newbie, and would really appreciate some
help in how to fix this. I did restart again with log 4 to see if any more
detail.

Thanks,

Chris

2006-04-03 10:09:09 CDT DEBUG4 remoteHelperThread_1
_1: send DONE/ERROR line to
worker
2006-04-03 10:09:09 CDT DEBUG3 remoteHelperThread_1
_1: waiting for workgroup
to finish
2006-04-03 10:09:09 CDT DEBUG3 remoteWorkerThread_1
: helper 1 finished
2006-04-03 10:09:09 CDT DEBUG4 remoteWorkerThread_1
: returning lines to pool
2006-04-03 10:09:09 CDT DEBUG3 remoteWorkerThread_1
: all helpers done.
2006-04-03 10:09:09 CDT DEBUG4 remoteWorkerThread_1
: changing helper 1 to IDLE
2006-04-03 10:09:09 CDT DEBUG4 remoteWorkerThread_1
: cleanup
2006-04-03 10:09:09 CDT DEBUG4 remoteHelperThread_1
_1: waiting for work
2006-04-03 10:09:09 CDT ERROR remoteWorkerThread_1
: "select
"_cpc_replic". sequenceSetValue(111
,1,'106076','380399'
);
" PGRES_FATAL_ERROR PANIC: right sibling's left-link doesn't match
CONTEXT: SQL statement "insert into "_cpc_replic".sl_seqlog (seql_seqid,
seql_origin, seql_ev_seqno, seql_last_value)
values ( $1 , $2 , $3 , $4 )"
PL/pgSQL function "sequencesetvalue" line 28 at SQL statement
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2006-04-03 10:09:09 CDT ERROR remoteWorkerThread_1
: "rollback transaction"
PGRES_FATAL_ERROR 2006-04-03 10:09:09 CDT
DEBUG1 slon: shutdown requested
2006-04-03 10:09:09 CDT DEBUG2 slon: notify worker process to shutdown
2006-04-03 10:09:09 CDT DEBUG2 slon: wait for worker process to shutdown
2006-04-03 10:09:09 CDT DEBUG1 cleanupThread: thread done
2006-04-03 10:09:09 CDT ERROR remoteListenThread_1
: "select ev_origin,
ev_seqno, ev_timestamp, ev_minxid, ev_m
axxid, ev_xip, ev_type, ev_data1, ev_data2, ev_data3,
ev_data4, ev_data5, ev_data6,
ev_data7, ev_data8 from "_cpc_replic".sl_event e where (e.ev_origin = '1' and
e.ev_seqno > '106208') order by e.ev_or
igin, e.ev_seqno" - server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2006-04-03 10:09:09 CDT DEBUG1 remoteListenThread_1
: thread done
2006-04-03 10:09:09 CDT DEBUG1 syncThread: thread done
2006-04-03 10:09:09 CDT DEBUG1 localListenThread: thread done
2006-04-03 10:09:09 CDT DEBUG1 main: scheduler mainloop returned
2006-04-03 10:09:09 CDT DEBUG2 main: wait for remote threads

Christopher Browne

2006-04-03, 8:27 pm

Chris Mount <chris- 6kfppi0h84dBDgjK7y7T
UQ@public.gmane.org> writes:
> 2006-04-03 10:09:09 CDT ERROR remoteWorkerThread_1
: "select
> "_cpc_replic". sequenceSetValue(111
,1,'106076','380399'
);
> " PGRES_FATAL_ERROR PANIC: right sibling's left-link doesn't match
> CONTEXT: SQL statement "insert into "_cpc_replic".sl_seqlog (seql_seqid,
> seql_origin, seql_ev_seqno, seql_last_value)
> values ( $1 , $2 , $3 , $4 )"

Searching for "right sibling's left-link doesn't match", I find
suggestions from Tom Lane to reindex tables.

It sounds to me as though the failure corrupted an index on one of the
tables, probably _cpc_replic.sl_seqlog.

Try reindexing it on the failing node, and see if that helps.

Of course, where there's one corruption, there could be more...

I'd suggest doing a vacuum of the whole thing via "VACUUM VERBOSE"; if
that runs into problems, you'll know...
--
output = reverse("ofni.sailifa.ac" "@" "enworbbc")
<http://dba2.int.libertyrms.com/>
Christopher Browne
(416) 673-4124 (land)

Chris Mount

2006-04-04, 9:32 am

That seems to have taken care of it. Thanks so much.

I ran the test_slony_state-dbi.pl and the sync age has cleared up.

On Monday 03 April 2006 16:22, Christopher Browne wrote:
> Chris Mount <chris- 6kfppi0h84dBDgjK7y7T
UQ@public.gmane.org> writes:
>
> Searching for "right sibling's left-link doesn't match", I find
> suggestions from Tom Lane to reindex tables.
>
> It sounds to me as though the failure corrupted an index on one of the
> tables, probably _cpc_replic.sl_seqlog.
>
> Try reindexing it on the failing node, and see if that helps.
>
> Of course, where there's one corruption, there could be more...
>
> I'd suggest doing a vacuum of the whole thing via "VACUUM VERBOSE"; if
> that runs into problems, you'll know...

Sponsored Links