Slony1 PostgreSQL Replication - Upcoming Things...

Author

Upcoming Things...

Christopher Browne

2005-06-29, 11:24 am

With the release of 1.1.0, this now frees us to start looking at what
should start going into version 1.2. A number of activities have been
awaiting this:

1. Known enhancements where concrete changes are forthcoming:
a) Rotating between sl_log_1 and sl_log_2
b) Improving performance of COPY_SET by dropping/recreating indices
c) Win32 support
d) Major revision to test suite

2. Plausible Changes (if we can come up with strategies/time):
a) Reduce numbers of redundant changes submitted in sl_seqlog
b) Reduce numbers of pg_listener entries that need to be destroyed
c) Migrate documentation from DocBook/SGML to DocBook/XML
d) Invalidate cached SPI plans so that when Slony-I is deinstalled
from a node, queries don't fail due to pointing to the now-invalid
query plans
e) Separate fetch/put into their own threads so updates (on subscriber)
can be concurrent with the fetchs (on provider)

3. Migrate to pgFoundry

Which offers a fair bit more functionality than GBorg...

This might include such things as:
- Moving from CVS to Subversion (if possible; it's not clear how
supported Subversion is at pgFoundry at this time)
- Separating out some of the components to separate repositories
(e.g. - the "Admin Guide" isn't really part of the replication
engine, so we almost certainl shouldn't pretend that it is...)
Likely repositories would include:
a) 'minimal' "replication engine" repository
b) Admin guide
c) Perhaps multiple repositories for different groups of
tools/scripts including:
- 'altperl'
- Log Shipping samples/helpers
- Maintenance jobs (e.g. - generate_syncs.sh)
- Monitoring scripts
These are presently somewhat jumbled together...

There already is a Slony-I project at pgFoundry.org; it is likely
that the migration will take place over the span of at least a
month, so that things are thought through, and done coherently.

Supposing we could accomplish merely the items in 1) and 3) between
now and (say) September, that would provide a quite compelling 1.2
release, as those changes should lead to a quite appreciable
improvement in performance, combined with the addition of Windows support.

In the short term, by all means, please continue to report bugs and
requests for features at GBorg
<http://gborg.postgresql.org/project/slony1/>.

Those that hang out on IRC should consider the above a pretty fair
"target for discussion." :-). I'm sure this quasi-plan will change
somewhat when others get to react to it!

Hannu Krosing

2005-06-29, 11:24 am

On K, 2005-06-29 at 11:11 -0400, Christopher Browne wrote:
> With the release of 1.1.0, this now frees us to start looking at what
> should start going into version 1.2. A number of activities have been
> awaiting this:
>
> 1. Known enhancements where concrete changes are forthcoming:
> a) Rotating between sl_log_1 and sl_log_2
> b) Improving performance of COPY_SET by dropping/recreating indices
> c) Win32 support
> d) Major revision to test suite

e) enable slon to run in polling mode on heavy-use scenarios to avoid
needless LISTEN/NOTIFY overhead and also not to grow a huge pg_listener
when there are long transactions.

--
Hannu Krosing <hannu-7C/ iILuz2RdeoWH0uzbU5w@
public.gmane.org>

Christopher Browne

2005-06-29, 1:25 pm

Hannu Krosing wrote:

>On K, 2005-06-29 at 11:11 -0400, Christopher Browne wrote:
>
>
>
> e) enable slon to run in polling mode on heavy-use scenarios to avoid
>needless LISTEN/NOTIFY overhead and also not to grow a huge pg_listener
>when there are long transactions.
>
>
All of the #1 items are ones for which a clear strategy is available,
where either a patch already exists, or someone has at least prototyped
the change, and has verified feasibility.

If you have a patch, or something near to one, then this can fall into
category #1.

If not, then it has to stay at the "wishful" stage for now.

I'm not at all sure that changing to some sort of "polling mode" would
address anything anyways. The problems with pg_listener growth are...

1. A lot of events are generated and deleted. Changing the polling
interval won't change this; it merely shifts the lifecycle.

2. The problem tends to be on the provider node; the only way I can see
to avoid the problem of dead tuples is if, during the "long running
transaction," we altogether *turn off* replication, which introduces a
different risk, namely that subscribers would have to wait to start
replicating until after that transaction, which probably doesn't fit
with the notion of subscribers being kept pretty well up to date.

Or perhaps I am misunderstanding the change you have in mind.

Darcy Buskermolen

2005-06-29, 8:25 pm

On Wednesday 29 June 2005 08:11, Christopher Browne wrote:
> With the release of 1.1.0, this now frees us to start looking at what
> should start going into version 1.2. A number of activities have been
> awaiting this:
>
> 1. Known enhancements where concrete changes are forthcoming:
> a) Rotating between sl_log_1 and sl_log_2
> b) Improving performance of COPY_SET by dropping/recreating indices
> c) Win32 support
> d) Major revision to test suite

Another item that should be someware reasonably high up in this list is
demonizing code, I knowe there was a patch sent to -general a long while ago
that was rejected on the grounds of being non portable, which we should
revamp and include.

>
> 2. Plausible Changes (if we can come up with strategies/time):
> a) Reduce numbers of redundant changes submitted in sl_seqlog
> b) Reduce numbers of pg_listener entries that need to be destroyed
> c) Migrate documentation from DocBook/SGML to DocBook/XML
> d) Invalidate cached SPI plans so that when Slony-I is deinstalled
> from a node, queries don't fail due to pointing to the now-invalid
> query plans
> e) Separate fetch/put into their own threads so updates (on subscriber)
> can be concurrent with the fetchs (on provider)
>
> 3. Migrate to pgFoundry
>
> Which offers a fair bit more functionality than GBorg...
>
> This might include such things as:
> - Moving from CVS to Subversion (if possible; it's not clear how
> supported Subversion is at pgFoundry at this time)
> - Separating out some of the components to separate repositories
> (e.g. - the "Admin Guide" isn't really part of the replication
> engine, so we almost certainl shouldn't pretend that it is...)
> Likely repositories would include:
> a) 'minimal' "replication engine" repository
> b) Admin guide
> c) Perhaps multiple repositories for different groups of
> tools/scripts including:
> - 'altperl'
> - Log Shipping samples/helpers
> - Maintenance jobs (e.g. - generate_syncs.sh)
> - Monitoring scripts
> These are presently somewhat jumbled together...
>
> There already is a Slony-I project at pgFoundry.org; it is likely
> that the migration will take place over the span of at least a
> month, so that things are thought through, and done coherently.
>
> Supposing we could accomplish merely the items in 1) and 3) between
> now and (say) September, that would provide a quite compelling 1.2
> release, as those changes should lead to a quite appreciable
> improvement in performance, combined with the addition of Windows support.
>
> In the short term, by all means, please continue to report bugs and
> requests for features at GBorg
> <http://gborg.postgresql.org/project/slony1/>.
>
> Those that hang out on IRC should consider the above a pretty fair
> "target for discussion." :-). I'm sure this quasi-plan will change
> somewhat when others get to react to it!
> ____________________
____________________
_______
> Slony1-general mailing list
> Slony1-general- AuKwsB3Fm+ugFIWk8tvy
RWD2FQJk+8+b@public.gmane.org
> http://gborg.postgresql.org/mailman.../slony1-general

--
Darcy Buskermolen
Wavefire Technologies Corp.

http://www.wavefire.com
ph: 250.717.0200
fx: 250.763.1759

Dave Page

2005-06-30, 3:24 am

=20

> -----Original Message-----
> From: slony1-general-bounces- AuKwsB3Fm+ugFIWk8tvy
RWD2FQJk+8+b@public.gmane.org=20
> [mailto:slony1-general-bounces- AuKwsB3Fm+ugFIWk8tvy
RWD2FQJk+8+b@public.gmane.org] On=20
> Behalf Of Christopher Browne
> Sent: 29 June 2005 16:12
> To: Slony Mailing List
> Subject: [Slony1-general] Upcoming Things...
>=20
> - Moving from CVS to Subversion (if possible; it's not clear how
> supported Subversion is at pgFoundry at this time)

It isn't, though it is supported by Gforge so should just be a case of
persuading Andrew Dunstan or one of the other maintainers to install it.

Regards, Dave.

David Fetter

2005-06-30, 9:24 am

On Thu, Jun 30, 2005 at 08:55:43AM +0100, Dave Page wrote:
>
>
>
> It isn't, though it is supported by Gforge so should just be a case
> of persuading Andrew Dunstan or one of the other maintainers to
> install it.

Going to Gforge 4.0 is already on the pgFoundry maintainers' agenda.
I believe that Tom Copeland is taking point on this, as he's already
done this successfully with Rubyforge.

Cheers,
D
--
David Fetter david- iTBeR5ZXhegdnm+yROfE
0A@public.gmane.org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

Christopher Browne

2005-06-30, 11:24 am

David Fetter wrote:

>On Thu, Jun 30, 2005 at 08:55:43AM +0100, Dave Page wrote:
>
>
>
>Going to Gforge 4.0 is already on the pgFoundry maintainers' agenda.
>I believe that Tom Copeland is taking point on this, as he's already
>done this successfully with Rubyforge.
>
>Cheers,
>D
>
>
Is there something of a schedule expected on this?

Like, if it's likely to be 6 months, then we might as well "deep 6" any
notion of moving to Subversion for now, and see about migrating CVS over
directly.

On the other hand, if we're likely to see GForge "subversion-enabled" in
a month, then it's worth putting off migrating code until that happens...

David Fetter

2005-06-30, 11:24 am

On Thu, Jun 30, 2005 at 12:27:07PM -0400, Christopher Browne wrote:
> David Fetter wrote:
>
> Is there something of a schedule expected on this?
>
> Like, if it's likely to be 6 months, then we might as well "deep 6" any
> notion of moving to Subversion for now, and see about migrating CVS over
> directly.
>
> On the other hand, if we're likely to see GForge "subversion-enabled" in
> a month, then it's worth putting off migrating code until that happens...

Chris,

I'm cc:ing this to the pgfoundry list.

Folks, are we going ahead with the upgrade? If so, do we have an ETA
and/or dependencies we need to think about?

Cheers,
D
--
David Fetter david- iTBeR5ZXhegdnm+yROfE
0A@public.gmane.org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

Sponsored Links