Slony1 PostgreSQL Replication - RFC - Migration to pgFoundry

Author

RFC - Migration to pgFoundry

Chris Browne

2006-02-25, 9:50 am

If you watch the pgsql.hackers list, you'll see that there is some
intent to deprecate gBorg in favor of pgFoundry. Which doesn't
generally seem a bad thing...

- gBorg apparently doesn't allow deleting old files without great
trouble; downloadable file management at pgFoundry is much better

- The problem ticketing system at pgFoundry is a fair bit more
sophisticated and customizable to boot, definitely more suitable

- There is extra backup hardware available for pgFoundry; nobody is
too interested in contributing such for gBorg

- It *looks* as though we'd be in better shape to have multiple CVS
modules

- It looks as though plain ordinary project admin users have ssh
access, at pgFoundry, not the case with gBorg, which covers all
manner of problems...

There were discussions back in the summer about moving over to
pgFoundry; the thing which we were holding off over was that there
were hopes that the next release of the gForge software (what
pgFoundry runs on) would have Subversion support. Discussions on
pgsql.hackers indicate that this particular hope is likely to be
forlorn, as configuring that on gForge is apparently rather
troublesome.

Mind you, SVN *is* installed, and available, at the shell level,
which, I think, means that we could have an SVN repository; we'd lose
any "web admin" capabilities (e.g. - gForge, like gBorg, includes the
ability to browse the CVS repository using a "WebCVS" client). In
effect, the pgFoundry software would be unaware of the existence of
the SVN repository.

In any case, it seems to me we should start thinking about planning
migration to pgFoundry.

- There is already stuff there; I think Devrim is stowing .rpm files
there.

- I think I started configuring the bug tracker; what to add warrants
some discussion, and what to do about managing the switchover warrants
discussion.

It seems to me this could work via taking all still-open items and
migrating them, one by one, to pgFoundry, and then marking them
closed at gBorg, with reference to the new ticket...

- We could open mailing lists there at any time; how transparent we'd
want to make that transition is an open question... It would be
possible to automatically migrate all users over; I think that would
mandate announcing it well ahead of time...

- Copying CVS over would need to take place at a well-announced point
in time.

There may be other things I'm not thinking of; that's why this is a
Request For Comments :-).
--
let name="cbbrowne" and tld="ntlug.org" in name ^ "@" ^ tld;;
http://www.ntlug.org/~cbbrowne/internet.html
"How should I know if it works? That's what beta testers are for. I
only coded it." (Attributed to Linus Torvalds, somewhere in a
posting)

Vivek Khera

2006-02-25, 9:50 am

On Feb 23, 2006, at 10:49 AM, Chris Browne wrote:

> There may be other things I'm not thinking of; that's why this is a
> Request For Comments :-).

Well, is moving to svn such a critical thing? It seems that unless
we're doing many branches and many vendor drops and such, cvs is
sufficient for now.

The mailing lists could move at any time. It would probably be a
good opportunity to get rid of the cruft on the list by having
everyone re-subscribe. No matter how good a mailing list manager is,
there are always dead addresses that accumulate after a while... If
we could arrange it that every existing member got an invitation to
just click a link that would be ideal.

Moving over the history of the bug tracker DB is important. We don't
want to lose any institutional memory. /me wonders if it could be
scripted with WWW::Mechanize....

Jim C. Nasby

2006-02-25, 9:50 am

On Thu, Feb 23, 2006 at 12:16:10PM -0500, Vivek Khera wrote:
>
> On Feb 23, 2006, at 10:49 AM, Chris Browne wrote:
>
>
> Well, is moving to svn such a critical thing? It seems that unless
> we're doing many branches and many vendor drops and such, cvs is
> sufficient for now.
>
> The mailing lists could move at any time. It would probably be a
> good opportunity to get rid of the cruft on the list by having
> everyone re-subscribe. No matter how good a mailing list manager is,
> there are always dead addresses that accumulate after a while... If
> we could arrange it that every existing member got an invitation to
> just click a link that would be ideal.

I disagree. First, if there are invalid emails subscribed, who cares?
Second, by creating new lists, you'd lose the archives of everything
that's been discussed, which seems like it would be a huge loss.

> Moving over the history of the bug tracker DB is important. We don't
> want to lose any institutional memory. /me wonders if it could be
> scripted with WWW::Mechanize....

I believe that a script to do that exists. I've been pushing for the
creation of a project on pgfoundry for the migration, so that stuff can
get posted and people can help.

If there is such a script then I suspect we might actually be close to
being able to seamlessly migrate projects. There shouldn't be any issue
with moving mailing lists, SVN exists (and worst-case a seperate install
of viewcvs could be setup until there's native gforge support for SVN),
and I believe everything else is in the database.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby-D/ iDPWeZeLdl57MIdRCFDg
@public.gmane.org
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Darcy Buskermolen

2006-02-25, 9:50 am

On Thursday 23 February 2006 13:07, Jim C. Nasby wrote:
> On Thu, Feb 23, 2006 at 12:16:10PM -0500, Vivek Khera wrote:
>
> I disagree. First, if there are invalid emails subscribed, who cares?
> Second, by creating new lists, you'd lose the archives of everything
> that's been discussed, which seems like it would be a huge loss.
>
>
> I believe that a script to do that exists. I've been pushing for the
> creation of a project on pgfoundry for the migration, so that stuff can
> get posted and people can help.

There is a foundery project for this, (but I'll be darned if I can remember
what it is called)

>
> If there is such a script then I suspect we might actually be close to
> being able to seamlessly migrate projects. There shouldn't be any issue
> with moving mailing lists, SVN exists (and worst-case a seperate install
> of viewcvs could be setup until there's native gforge support for SVN),
> and I believe everything else is in the database.

--
Darcy Buskermolen
Wavefire Technologies Corp.

http://www.wavefire.com
ph: 250.717.0200
fx: 250.763.1759

Christopher Browne

2006-02-25, 9:50 am

Let me preface everything with the comment that there is a checklist
already partially prepared on this...

http://slony-wiki.dbitech.ca/index....h
ecklist

Jim C. Nasby wrote:

>On Thu, Feb 23, 2006 at 12:16:10PM -0500, Vivek Khera wrote:
>
>
>
>I disagree. First, if there are invalid emails subscribed, who cares?
>
>
It'll irritate them.

This is a nice way of eliminating cruft; those that ARE interested are
free to subscribe to the new list; those that aren't can be silently
dropped, which is probably better for all.

>Second, by creating new lists, you'd lose the archives of everything
>that's been discussed, which seems like it would be a huge loss.
>
>
That's a separate issue that is a good point.

Addition to the checklist:

"Draw a copy of the existing archives; if nothing else, they may be
added, as a set of files, to pgFoundry so that old email archives are
not lost"

Actually, taking a peek at this, "wget -r" is your friend...

The command "wget -r
http://gborg.postgresql.org/pipermail/slony1-general/" pulls the
archives fairly successfully in TWO forms:
1. As the set of HTML files,
2. As a set of "mbox-style" data.

I'll bet this is pretty easy to redeploy on pgFoundry...

>
>
>
>I believe that a script to do that exists. I've been pushing for the
>creation of a project on pgfoundry for the migration, so that stuff can
>get posted and people can help.
>
>
I'm not convinced that automating bug tracker transfer is *all* that
necessary.

We only forcibly need to bring over the items that aren't closed, and
there are only 64 of those. If four of us split them up, we can
doubtless handle it in an afternoon.

And note that we'd use considerably *better* statuses; I don't think
that an automated conversion would be all that successful

>If there is such a script then I suspect we might actually be close to
>being able to seamlessly migrate projects. There shouldn't be any issue
>with moving mailing lists, SVN exists (and worst-case a seperate install
>of viewcvs could be setup until there's native gforge support for SVN),
>and I believe everything else is in the database.
>
>
Aside from "general popularity," it is not evident that we Really
Absolutely need to migrate to SVN.

If we stay with CVS, then we could get the "bonus" that we could fairly
much copy the files verbatim to pgFoundry, and not lose anything at all...

Jim C. Nasby

2006-02-25, 9:50 am

On Thu, Feb 23, 2006 at 05:33:40PM -0500, Christopher Browne wrote:
> It'll irritate them.
>
> This is a nice way of eliminating cruft; those that ARE interested are
> free to subscribe to the new list; those that aren't can be silently
> dropped, which is probably better for all.

Call me cold-hearted, but I have absolutely no pity for anyone who can't
figure out how to unsubscribe. And I'd argue that by now they've either
setup a filter or have gone insane with the unwanted email; problem
solved in either case. :)

> That's a separate issue that is a good point.
>
> Addition to the checklist:
>
> "Draw a copy of the existing archives; if nothing else, they may be
> added, as a set of files, to pgFoundry so that old email archives are
> not lost"
>
> Actually, taking a peek at this, "wget -r" is your friend...
>
> The command "wget -r
> http://gborg.postgresql.org/pipermail/slony1-general/" pulls the
> archives fairly successfully in TWO forms:
> 1. As the set of HTML files,
> 2. As a set of "mbox-style" data.
>
> I'll bet this is pretty easy to redeploy on pgFoundry...

Maybe, but far less easy than just moving the list over in mailman.

> I'm not convinced that automating bug tracker transfer is *all* that
> necessary.
>
> We only forcibly need to bring over the items that aren't closed, and
> there are only 64 of those. If four of us split them up, we can
> doubtless handle it in an afternoon.
>
> And note that we'd use considerably *better* statuses; I don't think
> that an automated conversion would be all that successful

I guess maybe we just have different views of the usefulness of keeping
older info around. Of course, if there isn't a script available, then
just moving stuff over manually might well make more sense.

> Aside from "general popularity," it is not evident that we Really
> Absolutely need to migrate to SVN.
>
> If we stay with CVS, then we could get the "bonus" that we could fairly
> much copy the files verbatim to pgFoundry, and not lose anything at all...

Oh, I thought slony was already using SVN. If not then it's an
orthogonal issue; there's a CVS to SVN migration tool that afaik will
remain supported for the indefinite future, so the move to SVN could
always be done some other time.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby-D/ iDPWeZeLdl57MIdRCFDg
@public.gmane.org
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Marc G. Fournier

2006-02-25, 9:50 am

On Thu, 23 Feb 2006, Christopher Browne wrote:

> That's a separate issue that is a good point.
>
> Addition to the checklist:
>
> "Draw a copy of the existing archives; if nothing else, they may be
> added, as a set of files, to pgFoundry so that old email archives are
> not lost"
>
> Actually, taking a peek at this, "wget -r" is your friend...
>
> The command "wget -r
> http://gborg.postgresql.org/pipermail/slony1-general/" pulls the
> archives fairly successfully in TWO forms:
> 1. As the set of HTML files,
> 2. As a set of "mbox-style" data.
>
> I'll bet this is pretty easy to redeploy on pgFoundry...

Yes, we've done this before ... there is a 'regen' script in mailman that
allows you to move the mbox format over and regen the html archives from
that ...

Marc G. Fournier

2006-02-25, 9:50 am

On Thu, 23 Feb 2006, Jim C. Nasby wrote:

> Maybe, but far less easy than just moving the list over in mailman.

Actually, far easier ... I believe, but am not 100% certain (I do
majordomo2 for everything, so am by no means a mailman expert), that it
hard codes domain information its binary config files ...

but someone that knows mailman would know better ...

Andrew Sullivan

2006-02-25, 9:50 am

On Thu, Feb 23, 2006 at 05:33:40PM -0500, Christopher Browne wrote:
>
> This is a nice way of eliminating cruft; those that ARE interested are
> free to subscribe to the new list; those that aren't can be silently
> dropped, which is probably better for all.

I really strongly disagree with this. There are lists that I follow
only sporadically. I put the messages into their own folder with
procmail. Perhaps once a month or so I read the messages.

Moving the list without moving the subscriber list and their settings
is just bad list management. If there are invalid list members,
they're already getting annoying mail. They can unsubscribe now.

A

--
Andrew Sullivan | ajs-oaT0K0jot5/q2IAV+ODieA@public.gmane.org
This work was visionary and imaginative, and goes to show that visionary
and imaginative work need not end up well.
--Dennis Ritchie

Jim C. Nasby

2006-02-25, 9:51 am

On Thu, Feb 23, 2006 at 09:14:35PM -0400, Marc G. Fournier wrote:
> On Thu, 23 Feb 2006, Jim C. Nasby wrote:
>
>
> Actually, far easier ... I believe, but am not 100% certain (I do
> majordomo2 for everything, so am by no means a mailman expert), that it
> hard codes domain information its binary config files ...
>
> but someone that knows mailman would know better ...

http://www.nabble.com/how-to-move-m...t-t1111610.html
--
Jim C. Nasby, Sr. Engineering Consultant jnasby-D/ iDPWeZeLdl57MIdRCFDg
@public.gmane.org
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Sponsored Links