|
| Hi,
I have a problem since upgrading from 1.1.0 to 1.1.5 and adding 3 more nodes
to my set. I am getting too many connections. Bug ID: 1383 (closed) seems to
be very similar. I use the altperl scripts to manage everything. All nodes
were set up w/parent in conf file.
If I use slon_kill script all connections will drop in about 10 seconds.
Replication speed seems to be fine. I can restart slons with script and will
receive several lines of "NOTICE: Slony-I: cleanup stale sl_nodelock entry
for pid=9984" for the master.
My original setup was running for about 4-6 months no real problems. It was
only in a test environment. Cluster is master + 5 nodes, different sets for
various nodes.
Master- SuSe Enterprise Server 9 , PG 7.4.8 distro install, slony1-1.1.0
source
Nodes- SuSe 9.3 Prof, PG 7.4.9 from source, slony1-1.1.0 source
Then I upgraded to 1.1.5 release (not rc versions) and added 3 nodes in the
same day. Since then after a few hours of replicating ps ax shows way too
many connections. In postgresql.conf, max connections is set to 100.
I am also running pg_autovacuum that ends up stopping after the build up of
connections.
Referencing the question in the Bug ID 1383 concenring the network setup,
nothing has changed and all nodes are connected on VPN, the subnetmasks are
the same except one.
Thanks for any recommendations.
14419 ? S 0:04 /usr/bin/postmaster -D /var/lib/pgsql/data
14423 ? S 0:12 postgres: stats buffer process
14425 ? S 1:28 postgres: stats collector process
15131 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.10.10.15 dbname=attest user=slony
15141 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15142 ? Sl 2:17 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.10.10.15 dbname=attest user=slony
15152 ? S 35:45 postgres: slony attest 10.10.10.15 async_notify
waiting
15163 ? S 29:03 postgres: slony attest 10.10.10.15 COMMIT waiting
15176 ? S 29:15 postgres: slony attest 10.10.10.15 COMMIT waiting
15177 ? S 30:14 postgres: slony attest 10.10.10.15 COMMIT waiting
15178 ? S 29:00 postgres: slony attest 10.10.10.15 COMMIT waiting
15179 ? S 30:34 postgres: slony attest 10.10.10.15 COMMIT waiting
15180 ? S 29:20 postgres: slony attest 10.10.10.15 COMMIT waiting
15181 ? S 30:30 postgres: slony attest 10.10.10.15 COMMIT waiting
15182 ? S 30:52 postgres: slony attest 10.10.10.15 COMMIT waiting
15183 ? S 29:26 postgres: slony attest 10.10.10.15 COMMIT waiting
15184 ? S 0:46 postgres: slony attest 10.10.10.15 idle
15185 ? S 9:47 postgres: slony attest 10.10.10.15 idle
15197 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.1.175 dbname=attest user=slon
15206 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15216 ? Sl 0:24 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.1.175 dbname=attest user=slon
15231 ? S 60:19 postgres: slony attest 10.10.10.15 async_notify
waiting
15233 ? S 0:09 postgres: slony attest 10.10.10.15 idle
15245 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.3.175 dbname=attest user=slon
15254 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15264 ? Sl 0:24 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.3.175 dbname=attest user=slon
15279 ? S 60:39 postgres: slony attest 10.10.10.15 async_notify
waiting
15292 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.4.175 dbname=attest user=slon
15301 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15311 ? S 0:10 postgres: slony attest 10.10.10.15 idle
15321 ? Sl 0:24 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.4.175 dbname=attest user=slon
15343 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15412 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.12.175 dbname=attest user=slo
15421 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15431 ? Sl 0:25 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.12.175 dbname=attest user=slo
15433 ? S 61:05 postgres: slony attest 10.10.10.15 async_notify
waiting
15455 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15541 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.29.175 dbname=attest user=slo
15550 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15561 ? S 60:26 postgres: slony attest 10.10.10.15 async_notify
waiting
15571 ? Sl 0:25 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.29.175 dbname=attest user=slo
15573 ? S 0:10 postgres: slony attest 10.10.10.15 idle
15574 ? S 60:13 postgres: slony attest 10.10.10.15 async_notify
waiting
15586 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.30.175 dbname=attest user=slo
15595 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15634 ? Sl 0:25 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.30.175 dbname=attest user=slo
15666 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=192.168.1.175 dbname=attest user=slo
15675 ? S 0:03 /usr/bin/perl /usr/local/bin/slon_watchdog
--config=/usr/local/etc/ cpc_attest_slon_tool
s.co
15698 ? Sl 0:24 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=192.168.1.175 dbname=attest user=slo
15736 ? S 60:43 postgres: slony attest 10.10.10.15 async_notify
waiting
15747 ? S 0:11 postgres: slony attest 10.10.10.15 idle
15785 ? S 60:55 postgres: slony attest 10.10.10.15 async_notify
waiting
15796 ? S 0:10 postgres: slony attest 10.10.10.15 idle
15863 ? S 0:09 postgres: slony attest 10.10.10.15 idle
514 ? S 4:37 postgres: slony attest 10.10.10.15 async_notify
waiting
991 ? S 4:27 postgres: slony attest 10.10.10.15 async_notify
waiting
1398 ? S 4:19 postgres: slony attest 10.10.10.15 async_notify
waiting
1426 ? S 4:18 postgres: slony attest 10.10.10.15 async_notify
waiting
1427 ? S 4:14 postgres: slony attest 10.10.10.15 async_notify
waiting
1491 ? S 4:15 postgres: slony attest 10.10.10.15 async_notify
waiting
1510 ? S 4:16 postgres: slony attest 10.10.10.15 async_notify
waiting
1774 ? S 4:09 postgres: slony attest 10.10.10.15 async_notify
waiting
1985 ? S 4:05 postgres: slony attest 10.10.10.15 async_notify
waiting
1995 ? S 4:06 postgres: slony attest 10.10.10.15 async_notify
waiting
2104 ? S 4:04 postgres: slony attest 10.10.10.15 async_notify
waiting
2448 ? S 4:04 postgres: slony attest 10.10.10.15 async_notify
waiting
2449 ? S 4:01 postgres: slony attest 10.10.10.15 async_notify
waiting
2502 ? S 4:00 postgres: slony attest 10.10.10.15 async_notify
waiting
2512 ? S 3:59 postgres: slony attest 10.10.10.15 async_notify
waiting
2810 ? S 3:54 postgres: slony attest 10.10.10.15 async_notify
waiting
2982 ? S 3:53 postgres: slony attest 10.10.10.15 async_notify
waiting
2983 ? S 3:57 postgres: slony attest 10.10.10.15 async_notify
waiting
2984 ? S 3:52 postgres: slony attest 10.10.10.15 async_notify
waiting
3157 ? S 3:51 postgres: slony attest 10.10.10.15 async_notify
waiting
3484 ? S 3:46 postgres: slony attest 10.10.10.15 async_notify
waiting
3512 ? S 3:53 postgres: slony attest 10.10.10.15 async_notify
waiting
3540 ? S 3:49 postgres: slony attest 10.10.10.15 async_notify
waiting
3550 ? S 3:50 postgres: slony attest 10.10.10.15 async_notify
waiting
3859 ? S 3:45 postgres: slony attest 10.10.10.15 async_notify
waiting
4022 ? S 3:43 postgres: slony attest 10.10.10.15 async_notify
waiting
4023 ? S 3:44 postgres: slony attest 10.10.10.15 async_notify
waiting
4024 ? S 3:46 postgres: slony attest 10.10.10.15 async_notify
waiting
4188 ? S 3:43 postgres: slony attest 10.10.10.15 async_notify
waiting
4486 ? S 3:42 postgres: slony attest 10.10.10.15 async_notify
waiting
4524 ? S 3:43 postgres: slony attest 10.10.10.15 async_notify
waiting
4543 ? S 3:40 postgres: slony attest 10.10.10.15 async_notify
waiting
4553 ? S 3:39 postgres: slony attest 10.10.10.15 async_notify
waiting
4842 ? S 3:36 postgres: slony attest 10.10.10.15 async_notify
waiting
5077 ? S 3:37 postgres: slony attest 10.10.10.15 async_notify
waiting
5087 ? S 3:34 postgres: slony attest 10.10.10.15 async_notify
waiting
5088 ? S 3:37 postgres: slony attest 10.10.10.15 async_notify
waiting
5252 ? S 3:40 postgres: slony attest 10.10.10.15 async_notify
waiting
5592 ? S 3:36 postgres: slony attest 10.10.10.15 async_notify
waiting
5620 ? S 3:36 postgres: slony attest 10.10.10.15 async_notify
waiting
5621 ? S 3:35 postgres: slony attest 10.10.10.15 async_notify
waiting
5631 ? S 3:34 postgres: slony attest 10.10.10.15 async_notify
waiting
5987 ? S 3:34 postgres: slony attest 10.10.10.15 async_notify
waiting
6162 ? S 3:27 postgres: slony attest 10.10.10.15 async_notify
waiting
6163 ? S 3:29 postgres: slony attest 10.10.10.15 async_notify
waiting
6164 ? S 3:30 postgres: slony attest 10.10.10.15 async_notify
waiting
6336 ? S 3:31 postgres: slony attest 10.10.10.15 async_notify
waiting
6680 ? S 3:30 postgres: slony attest 10.10.10.15 async_notify
waiting
6681 ? S 3:25 postgres: slony attest 10.10.10.15 async_notify
waiting
6682 ? S 3:29 postgres: slony attest 10.10.10.15 async_notify
waiting
6720 ? S 3:26 postgres: slony attest 10.10.10.15 async_notify
waiting
7121 ? S 3:27 postgres: slony attest 10.10.10.15 async_notify
waiting
7308 ? S 3:27 postgres: slony attest 10.10.10.15 async_notify
waiting
7309 ? S 3:24 postgres: slony attest 10.10.10.15 async_notify
waiting
7310 ? S 3:26 postgres: slony attest 10.10.10.15 async_notify
waiting
7464 ? S 3:22 postgres: slony attest 10.10.10.15 async_notify
waiting
7808 ? S 3:24 postgres: slony attest 10.10.10.15 async_notify
waiting
7809 ? S 3:19 postgres: slony attest 10.10.10.15 async_notify
waiting
7810 ? S 3:24 postgres: slony attest 10.10.10.15 async_notify
waiting
7856 ? S 3:21 postgres: slony attest 10.10.10.15 async_notify
waiting
8198 ? S 3:21 postgres: slony attest 10.10.10.15 async_notify
waiting
8397 ? S 3:21 postgres: slony attest 10.10.10.15 async_notify
waiting
8398 ? S 3:22 postgres: slony attest 10.10.10.15 async_notify
waiting
8399 ? S 3:20 postgres: slony attest 10.10.10.15 async_notify
waiting
8571 ? S 3:18 postgres: slony attest 10.10.10.15 async_notify
waiting
8932 ? S 3:20 postgres: slony attest 10.10.10.15 async_notify
waiting
8933 ? S 3:19 postgres: slony attest 10.10.10.15 async_notify
waiting
8943 ? S 3:16 postgres: slony attest 10.10.10.15 async_notify
waiting
8980 ? S 3:21 postgres: slony attest 10.10.10.15 async_notify
waiting
9334 ? R 3:15 postgres: slony attest 10.10.10.15 async_notify
9516 ? S 3:15 postgres: slony attest 10.10.10.15 async_notify
waiting
9517 ? S 3:17 postgres: slony attest 10.10.10.15 async_notify
waiting
9527 ? S 3:16 postgres: slony attest 10.10.10.15 async_notify
waiting
13569 ? S 3:03 postgres: slony attest 10.10.10.15 async_notify
waiting
13966 ? S 3:02 postgres: slony attest 10.10.10.15 async_notify
waiting
28086 ? S 0:00 trivial-rewrite -n rewrite -t unix -u
22888 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.18.175 dbname=attest user=slo
22896 ? Sl 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.18.175 dbname=attest user=slo
23501 ? S 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.6.175 dbname=attest user=slon
23708 ? Sl 0:00 /usr/bin//slon -c 0 -s 1000 -d2 cpc_attest_replic
host=10.200.6.175 dbname=attest user=slon
[2006-02-09 02:30:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_confirm"
[2006-02-09 02:40:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_event"
[2006-02-09 02:50:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_confirm"
[2006-02-09 03:00:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_event"
[2006-02-09 03:00:11 AM] Performing: VACUUM ANALYZE
"_cpc_attest_replic"."sl_seqlog"
[2006-02-09 03:10:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_confirm"
[2006-02-09 03:20:11 AM] Performing: VACUUM ANALYZE
"_cpc_attest_replic"."sl_event"
[2006-02-09 03:20:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_seqlog"
[2006-02-09 03:30:11 AM] Performing: ANALYZE "_cpc_attest_replic"."sl_confirm"
[2006-02-09 03:40:11 AM] Failed connection to database template1 with error:
FATAL: sorry, too many clients already
..
[2006-02-09 03:40:11 AM] Error: Cannot connect to template1, exiting.
Waiting for data... (interrupt to abort)
CUSTDB:/home/cmount # NOTICE: Slony-I: cleanup stale sl_nodelock entry for
pid=9983
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9984
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9985
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=10499
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=10613
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=10969
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11024
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11025
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11035
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11054
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11055
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11056
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11507
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=11626
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12017
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12072
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12073
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12074
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12165
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12166
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12167
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12592
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=12712
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13066
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13112
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13113
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13114
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13249
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13250
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13251
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13731
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=13877
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14193
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14309
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14310
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14320
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14443
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14444
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14445
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=14880
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15052
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15387
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15487
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15488
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15489
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15607
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15608
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15609
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=16020
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6207
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6296
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6297
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6578
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6579
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6596
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6580
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6627
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6582
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6584
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6644
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6656
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6595
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=6675
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7182
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7192
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7302
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7339
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7646
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7710
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7711
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7721
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7722
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7723
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=7724
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8168
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8322
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8773
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8844
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8845
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8848
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8862
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8863
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=8865
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9371
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9534
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9843
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9917
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9918
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=9919
|
|