Slony1 PostgreSQL Replication

Author

Banner day :-)

Christopher Browne

2006-02-25, 9:50 am

Jan submitted an update for log switching (switch periodically between
sl_log_1, sl_log_2, truncating the empty one), I have a patch for the
EXECUTE SCRIPT bug, and Darcy submitted updates for improving
interaction between Slony-I and pg_autovacuum.

That's a pretty good day!

Those are, if I recall properly, the big things between us and 1.2, so
while they're not all *widely* tested, we're making progress :-).

There are some bugs out there to be looked at, mostly a bit cosmetic,
but nonetheless there, and now that we have at least 'drafty' versions
of everything we want added for now, it makes sense to start stomping those.

I'm a bit less "scarily busy" right now, so I'll certainly be looking
into the outstanding bugs next week.

The more that can be stomped, now the less data has to be taken over by
hand to pgFoundry :-).

Rod Taylor

2006-02-25, 9:50 am

On Fri, 2006-02-24 at 17:22 -0500, Christopher Browne wrote:
> Jan submitted an update for log switching (switch periodically between
> sl_log_1, sl_log_2, truncating the empty one), I have a patch for the
> EXECUTE SCRIPT bug, and Darcy submitted updates for improving
> interaction between Slony-I and pg_autovacuum.

Slick.

Will the sl_log_[12] truncation handle insanely long transactions? That
is, if Slony is supposed to switch log tables after 1 million
transactions, what happens if a single transaction crosses 2 million
transactions and is still active precluding either structure from being
truncated (or deleted from)?

--

Christopher Browne

2006-02-25, 9:51 am

Rod Taylor wrote:

>On Fri, 2006-02-24 at 17:22 -0500, Christopher Browne wrote:
>
>
>
>Slick.
>
>Will the sl_log_[12] truncation handle insanely long transactions? That
>is, if Slony is supposed to switch log tables after 1 million
>transactions, what happens if a single transaction crosses 2 million
>transactions and is still active precluding either structure from being
>truncated (or deleted from)?
>
>
>
Just to be picky, the trouble is actually with 2 billion transactions...

Supposing you have a system *so* busy that you go through 2 billion
transactions per week, but you also have a single transaction that runs
for an entire week, that's the case that'll be troublesome.

- The long running transaction will Cause Problems for any tables that
are updated frequently because you'll have a week's worth of (billions
of updates per week!!!) dead tuples not vacuuming out.

- Up until 1.2, Slony-I would have keeled over, in that scenario,
because pg_listener would have a zillion dead tuples in it.

- This case points to my paranoia about TRUNCATE (expressed this
afternoon on IRC) having some merit...

Jan, I think the case Rod describes is a case that makes it mandatory
for logswitch_finish() to exit without truncating if it sees any locks
outstanding on the sl_log_n table in question; you could have the
situation where a long running transaction has 1M tuples, not yet
committed, sitting in sl_log_1. They aren't visible as being live
tuples, but they are NOT dead...

I'm not certain what ultimately happens, now, with the "insanely long
transactions" case (on an otherwise also-busy server). We now have
fewer (I'm not sure about "no") reasons for Slony-I to punish you for
long running transactions (LRTs :-)), so a replicated DB might now
survive traffic + LRT.

So I guess I'm not sure now :-). I'm glad you brought this up; if
several more people do a bit of wondering, we may come up with an answer
:-).

Jim Archer

2006-02-25, 9:51 am

--On Friday, February 24, 2006 6:38 PM -0500 Christopher Browne
<cbbrowne-swQf4SbcV9C7WVzo/KQ3Mw@public.gmane.org> wrote:
[color=darkred]

Hi All...

I have been away from Slony for a while and I was just wondering if the
large record issue that killed slons will be addressed in 1.2?

Thanks!

Jim

cbbrowne-swQf4SbcV9C7WVzo/KQ3Mw@public.gmane.org

2006-02-25, 9:51 am

> --On Friday, February 24, 2006 6:38 PM -0500 Christopher Browne
> <cbbrowne-swQf4SbcV9C7WVzo/KQ3Mw@public.gmane.org> wrote:
>
>
>
> Hi All...
>
> I have been away from Slony for a while and I was just wondering if the
> large record issue that killed slons will be addressed in 1.2?

You're referring to the trouble where if you have 50MB records, and
possibly 100 in a row, the slon might chew up 10GB of memory, I believe?

Jan put a patch for that in place a little while back for 1.2, so yes,
that should indeed be addressed.

Come to think of it, it probably starts making sense to start assembling
the "new feature list." It's fairly large :-). Win32 is one of the
items...

Marko Kreen

2006-02-25, 9:51 am

On 2/25/06, Christopher Browne <cbbrowne-swQf4SbcV9C7WVzo/KQ3Mw@public.gmane.org> wrote:
> - This case points to my paranoia about TRUNCATE (expressed this
> afternoon on IRC) having some merit...
>
> Jan, I think the case Rod describes is a case that makes it mandatory
> for logswitch_finish() to exit without truncating if it sees any locks
> outstanding on the sl_log_n table in question; you could have the
> situation where a long running transaction has 1M tuples, not yet
> committed, sitting in sl_log_1. They aren't visible as being live
> tuples, but they are NOT dead...

You can see if there are long-running queries from snapshot.

You can even tell in which sl_log table they can be with a bit
of accounting.

--
marko

Jan Wieck

2006-02-28, 8:34 pm

On 2/24/2006 5:22 PM, Christopher Browne wrote:
> Jan submitted an update for log switching (switch periodically between
> sl_log_1, sl_log_2, truncating the empty one), I have a patch for the
> EXECUTE SCRIPT bug, and Darcy submitted updates for improving
> interaction between Slony-I and pg_autovacuum.

I discovered over the weekend that log switching isn't complete yet.
Forwarded log is written to sl_log_1 unconditional ... I'm at it ...

>
> That's a pretty good day!
>
> Those are, if I recall properly, the big things between us and 1.2, so
> while they're not all *widely* tested, we're making progress :-).
>
> There are some bugs out there to be looked at, mostly a bit cosmetic,
> but nonetheless there, and now that we have at least 'drafty' versions
> of everything we want added for now, it makes sense to start stomping those.

There is still the rewrite of rebuild_listen.

>
> I'm a bit less "scarily busy" right now, so I'll certainly be looking
> into the outstanding bugs next week.
>
> The more that can be stomped, now the less data has to be taken over by
> hand to pgFoundry :-).

Don't you think moving to pgfoundry before 1.2 is a bit disruptive?

Jan

> ____________________
____________________
_______
> Slony1-general mailing list
> Slony1-general- AuKwsB3Fm+ugFIWk8tvy
RWD2FQJk+8+b@public.gmane.org
> http://gborg.postgresql.org/mailman.../slony1-general

--
#===================
====================
====================
===========#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#===================
====================
=========== JanWieck- bwPqjjyvM7QAvxtiuMwx
3w@public.gmane.org #

Jan Wieck

2006-02-28, 8:34 pm

On 2/24/2006 6:38 PM, Christopher Browne wrote:

> Rod Taylor wrote:
>
> Just to be picky, the trouble is actually with 2 billion transactions...
>
> Supposing you have a system *so* busy that you go through 2 billion
> transactions per week, but you also have a single transaction that runs
> for an entire week, that's the case that'll be troublesome.

I think Postgres itself will be pretty unhappy with this particular
situation.

Jan

>
> - The long running transaction will Cause Problems for any tables that
> are updated frequently because you'll have a week's worth of (billions
> of updates per week!!!) dead tuples not vacuuming out.
>
> - Up until 1.2, Slony-I would have keeled over, in that scenario,
> because pg_listener would have a zillion dead tuples in it.
>
> - This case points to my paranoia about TRUNCATE (expressed this
> afternoon on IRC) having some merit...
>
> Jan, I think the case Rod describes is a case that makes it mandatory
> for logswitch_finish() to exit without truncating if it sees any locks
> outstanding on the sl_log_n table in question; you could have the
> situation where a long running transaction has 1M tuples, not yet
> committed, sitting in sl_log_1. They aren't visible as being live
> tuples, but they are NOT dead...
>
> I'm not certain what ultimately happens, now, with the "insanely long
> transactions" case (on an otherwise also-busy server). We now have
> fewer (I'm not sure about "no") reasons for Slony-I to punish you for
> long running transactions (LRTs :-)), so a replicated DB might now
> survive traffic + LRT.
>
> So I guess I'm not sure now :-). I'm glad you brought this up; if
> several more people do a bit of wondering, we may come up with an answer
> :-).
> ____________________
____________________
_______
> Slony1-general mailing list
> Slony1-general- AuKwsB3Fm+ugFIWk8tvy
RWD2FQJk+8+b@public.gmane.org
> http://gborg.postgresql.org/mailman.../slony1-general

--
#===================
====================
====================
===========#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#===================
====================
=========== JanWieck- bwPqjjyvM7QAvxtiuMwx
3w@public.gmane.org #

Sponsored Links