|
Home > Archive > EAserver > June 2005 > Application Partitioning - multiple EAServers
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Application Partitioning - multiple EAServers
|
|
|
| Hello,
Our estimations shows that we need to partion our workload into serveral
EAServers. This means that most of our compononents
will be running on the main EAServer instance and that some components will
be running on the secondary server. So far soo god.
The problem is that we have some common components that must be running on
both servers.
Has anyone set-up a similar configuration and can share some insights?
/T
| |
| EdHolderman 2005-06-07, 1:24 pm |
| The way we've done it is to first cluster multiple servers
all set up as name servers so that you can fail over if any
of the machines die. This requires some sort of failover
technique from the client side so that you list all of the
available name servers to which you should try to connect.
Then we deploy the same code to all servers. We let our
client-facing code get served up round-robin by the cluster.
This is the most fault tolerant. We configure our
background service processes to run on any particular server
by using properties files on each server. However, we have
all services turned on for all servers and have written our
code to be aware of other threads running on other servers.
This is also for fault tolerance. You really want multiple
identical servers set up in a cluster instead of load
balancing by picking and choosing which components to run on
each server. Atleast, that's my experience.
Ed
> Hello,
>
> Our estimations shows that we need to partion our workload
> into serveral EAServers. This means that most of our
> compononents will be running on the main EAServer instance
> and that some components will be running on the secondary
> server. So far soo god. The problem is that we have some
> common components that must be running on both servers.
>
> Has anyone set-up a similar configuration and can share
> some insights?
>
> /T
>
>
>
>
>
>
>
>
>
>
| |
| Adam Simmonds [TeamSybase] 2005-06-08, 3:24 am |
| EdHolderman wrote:
> The way we've done it is to first cluster multiple servers
> all set up as name servers
I wouldnt recommend to everyone that they set all machines as name
servers without first checking what impact this has on performance (
esepcially if you also reduce the heartbeat )
a.
so that you can fail over if any[color=darkred]
> of the machines die. This requires some sort of failover
> technique from the client side so that you list all of the
> available name servers to which you should try to connect.
> Then we deploy the same code to all servers. We let our
> client-facing code get served up round-robin by the cluster.
> This is the most fault tolerant. We configure our
> background service processes to run on any particular server
> by using properties files on each server. However, we have
> all services turned on for all servers and have written our
> code to be aware of other threads running on other servers.
> This is also for fault tolerance. You really want multiple
> identical servers set up in a cluster instead of load
> balancing by picking and choosing which components to run on
> each server. Atleast, that's my experience.
>
> Ed
>
>
| |
| EdHolderman 2005-06-09, 1:24 pm |
| Good point, sorry about that. I should have mentioned that
we use that on a two-server cluster for failover reasons.
Ed
> I wouldnt recommend to everyone that they set all machines
> as name servers without first checking what impact this
> has on performance ( esepcially if you also reduce the
> heartbeat )
| |
| Adam Simmonds [TeamSybase] 2005-06-09, 8:24 pm |
| EdHolderman wrote:
> Good point, sorry about that. I should have mentioned that
> we use that on a two-server cluster for failover reasons.
Thats fair enough if you only have two servers, the rule I follow is to
have at least half my servers as name servers with 2 machines being the
exception.
2 machines, 2 would be NS
3 machines, 2 would be NS
4 machines, 2 would be NS
5 machines, 3 would be NS.
etc.
a.
[color=darkred]
> Ed
>
>
| |
| Marty Jones 2005-06-10, 7:24 am |
| Adam,
Is there that much of a overhead penalty for running all servers in the
cluster as Name Servers?
We are running a 3 machine cluster with all 3 as name servers and have not
seen any huge impact.
"Adam Simmonds [TeamSybase]" <asimmond@remove.mail.usyd.edu.au> wrote in
message news:42a8d4ff$1@foru
ms-1-dub...[color=darkred]
> EdHolderman wrote:
>
> Thats fair enough if you only have two servers, the rule I follow is to
> have at least half my servers as name servers with 2 machines being the
> exception.
>
> 2 machines, 2 would be NS
> 3 machines, 2 would be NS
> 4 machines, 2 would be NS
> 5 machines, 3 would be NS.
> etc.
>
> a.
>
>
>
| |
| Jonathan Baker [Sybase] 2005-06-10, 11:24 am |
| You won't see real impact from the back-and-forth messaging until you
get above 5 servers or so. Name servers swap information between each
other, so each name server creates 2^(n-1) messages every several
seconds. For example, 2 servers create 2 messages (one each direction)
every several seconds, 3 creates 4 messages, 4 creates 8, 5 create 16.
Jonathan
Marty Jones wrote:
> Adam,
>
> Is there that much of a overhead penalty for running all servers in the
> cluster as Name Servers?
> We are running a 3 machine cluster with all 3 as name servers and have not
> seen any huge impact.
>
>
> "Adam Simmonds [TeamSybase]" <asimmond@remove.mail.usyd.edu.au> wrote in
> message news:42a8d4ff$1@foru
ms-1-dub...
>
>
>
>
| |
| Adam Simmonds 2005-06-10, 8:24 pm |
| Marty Jones wrote:
> Adam,
>
> Is there that much of a overhead penalty for running all servers in the
> cluster as Name Servers?
> We are running a 3 machine cluster with all 3 as name servers and have not
> seen any huge impact.
Everyone is different. I found that in a cluster of 4 machines, having
all as name servers had inverse impact during load testing. Other apps
may be fine with 4/4. I was just offering a general rule I use, take it
or throw it away :)
a.
>
> "Adam Simmonds [TeamSybase]" <asimmond@remove.mail.usyd.edu.au> wrote in
> message news:42a8d4ff$1@foru
ms-1-dub...
>
>
>
>
| |
| Dean Jones 2005-06-21, 8:24 pm |
| You also have to look at how you connect.
It's mentioned failover, but high availability plays in with how you call
the name server. Then a bunch of EAS logic jumps in.
Example. If you always list your name servers in the same order, the first
name server is going to take a lions share of the load. That is the name
server first in the list is a available so it will use it. It wont go to the
next name server unless you change the order of the name server. Next if a
given name server has the component you are looking for it will put its self
first in the list. EAServer says, hey the client already has a socket open
to me, might as well put my self first in the list. So most project create a
cluster, set the load to random or round robin, and can't understand why one
server is running 90% and the other is at 5%.
I also believe their might be some logic that cause the HA to use a the
local server if its in the list. So if I'm running a web server and it's
running in EAS and you are looking up a component the name server used will
be a local name server.
is this a web application running in EAServer? If yes; you should balance
the HTTP before getting to EAS. If this is a PB client shuffle the order of
name servers before doing the lookup. If the web server is Tomcat shuffle
the name servers before doing a lookup.
If you have multiple EAS and only one is running http you need to make sure
it does not take load of web, nameserver and components. there is an ORB
property that should tell it EAServer local (i think setting this to false)
will make it treat the local eas as if its not local. not 100%. Have not
proven it yet.
--
Dean Jones [TeamSybase]
CEO
Certified PowerBuilder Developer
www.powerobjects.com
(612) 339-3355 ext 112
"Marty Jones" < marty_jones@dailyacc
ess.com> wrote in message
news:42a98cc4$1@foru
ms-2-dub...
> Adam,
>
> Is there that much of a overhead penalty for running all servers in the
> cluster as Name Servers?
> We are running a 3 machine cluster with all 3 as name servers and have not
> seen any huge impact.
>
>
> "Adam Simmonds [TeamSybase]" <asimmond@remove.mail.usyd.edu.au> wrote in
> message news:42a8d4ff$1@foru
ms-1-dub...
>
>
| |
| Adam Simmonds [TeamSybase] 2005-06-21, 8:24 pm |
| Dean Jones wrote:
> You also have to look at how you connect.
>
> It's mentioned failover, but high availability plays in with how you call
> the name server. Then a bunch of EAS logic jumps in.
>
> Example. If you always list your name servers in the same order, the first
> name server is going to take a lions share of the load. That is the name
> server first in the list is a available so it will use it.
Are you sure about this? I have used round robin, random and adaptive
and all of them spread the load fairly evenly and I have never seen one
machine have 85% more load than another when under normal operating
load. The socket reuse will govern how often a client hits the same
machine but even then I have performed cluster testing with one client
and can see all machines being utilised at some point in the testing.
Note: I have 4 machines with 2 as name servers in production. I can
validate the load is being spread via monitoring the number of no wait
DB connection requests and also the network stats on the IIOP listener.
a.
It wont go to the
> next name server unless you change the order of the name server. Next if a
> given name server has the component you are looking for it will put its self
> first in the list. EAServer says, hey the client already has a socket open
> to me, might as well put my self first in the list. So most project create a
> cluster, set the load to random or round robin, and can't understand why one
> server is running 90% and the other is at 5%.
> I also believe their might be some logic that cause the HA to use a the
> local server if its in the list. So if I'm running a web server and it's
> running in EAS and you are looking up a component the name server used will
> be a local name server.
>
> is this a web application running in EAServer? If yes; you should balance
> the HTTP before getting to EAS. If this is a PB client shuffle the order of
> name servers before doing the lookup. If the web server is Tomcat shuffle
> the name servers before doing a lookup.
>
> If you have multiple EAS and only one is running http you need to make sure
> it does not take load of web, nameserver and components. there is an ORB
> property that should tell it EAServer local (i think setting this to false)
> will make it treat the local eas as if its not local. not 100%. Have not
> proven it yet.
>
| |
| Dave Wolf 2005-06-22, 11:24 am |
| Dean Jones wrote:
> You also have to look at how you connect.
>
> It's mentioned failover, but high availability plays in with how you call
> the name server. Then a bunch of EAS logic jumps in.
>
> Example. If you always list your name servers in the same order, the first
> name server is going to take a lions share of the load. That is the name
> server first in the list is a available so it will use it. It wont go to the
> next name server unless you change the order of the name server. Next if a
> given name server has the component you are looking for it will put its self
> first in the list. EAServer says, hey the client already has a socket open
> to me, might as well put my self first in the list. So most project create a
> cluster, set the load to random or round robin, and can't understand why one
> server is running 90% and the other is at 5%.
Not at all true. The client side ORB randomizes the name server order.
internsally. Look over my prior Techwave sessions on EAServer internals.
You cannot control the order (it is actually just psuedorandom) like
you can load, but it is not static and fixed.
However once the ORB makes a connection, it will cache that connection
and use that connection from that point on. There is an ORB property
socket_reuse_limit whihc tells the client side ORB to drop cached
sockets every N method calls. Once dropped it will find the next random
name server, connect to it, lookup the next component, and viola.
Having a low socket_reuse_limit helps even balancing out, but increases
load on the name servers.
Its all a balancing act.
>
> I also believe their might be some logic that cause the HA to use a the
> local server if its in the list. So if I'm running a web server and it's
> running in EAS and you are looking up a component the name server used will
> be a local name server.
True.
>
> is this a web application running in EAServer? If yes; you should balance
> the HTTP before getting to EAS. If this is a PB client shuffle the order of
> name servers before doing the lookup. If the web server is Tomcat shuffle
> the name servers before doing a lookup.
No need to do this. The ORB does it for you.
>
Dave Wolf
Cynergy Systems
http://www.cynergysystems.com
> If you have multiple EAS and only one is running http you need to make sure
> it does not take load of web, nameserver and components. there is an ORB
> property that should tell it EAServer local (i think setting this to false)
> will make it treat the local eas as if its not local. not 100%. Have not
> proven it yet.
>
| |
| Dave Wolf 2005-06-22, 11:24 am |
| Adam Simmonds [TeamSybase] wrote:
> Dean Jones wrote:
>
>
>
> Are you sure about this? I have used round robin, random and adaptive
> and all of them spread the load fairly evenly and I have never seen one
> machine have 85% more load than another when under normal operating
> load. The socket reuse will govern how often a client hits the same
> machine but even then I have performed cluster testing with one client
> and can see all machines being utilised at some point in the testing.
>
> Note: I have 4 machines with 2 as name servers in production. I can
> validate the load is being spread via monitoring the number of no wait
> DB connection requests and also the network stats on the IIOP listener.
>
> a.
>
Yup. Right on!
Dave Wolf
Cynergy Systems
http://www.cynergysystems.com
[color=darkred]
>
> It wont go to the
>
>
>
| |
| Adam Simmonds 2005-06-23, 9:24 am |
| Dave Wolf wrote:
> Adam Simmonds [TeamSybase] wrote:
>
>
> Yup. Right on!
>
cheers Dave, at least I know I am not going mad in
my old age :)
| |
| Dean Jones 2005-06-23, 11:24 am |
| Not sure what you are saying Dave.
Explain how the HA name server is used? The load balancing will not have
anything to do with the client doing the lookup. So if you have multiple
name server in a list example iiop://jag1:9000;iiop://jag1:9000. Your saying
the client will not use jag1 to do the lookup? I have tested this many
times. I have worked with Evan on this (a while back) I believe there was an
issue and the client did not randomly select a name server. To fix this we
simple "shuffle" the list before doing a lookup.
Are you saying that a name server will not have a preference to itself?
Example jag1 is a name server and has component "A". I look up component "A"
using jag1, Are you saying EAServer does not have a preference for itself in
this case?
--
Dean Jones [TeamSybase]
CEO
Certified PowerBuilder Developer
www.powerobjects.com
(612) 339-3355 ext 112
"Adam Simmonds" <asimmond@mail.usyd.edu.au> wrote in message
news:42bab763$1@foru
ms-2-dub...
> Dave Wolf wrote:
>
>
> cheers Dave, at least I know I am not going mad in my old age :)
| |
| Dean Jones 2005-06-23, 1:24 pm |
| See below.
--
Dean Jones [TeamSybase]
CEO
Certified PowerBuilder Developer
www.powerobjects.com
(612) 339-3355 ext 112
"Dave Wolf" <dave.wolf@cynergysystems.com_nospam> wrote in message
news:42b9819d$1@foru
ms-1-dub...
> Dean Jones wrote:
>
> Not at all true. The client side ORB randomizes the name server order.
> internsally. Look over my prior Techwave sessions on EAServer internals.
> You cannot control the order (it is actually just psuedorandom) like you
> can load, but it is not static and fixed.
I understand this is a property of the ORB init. We have seen this behavior
many time. I pretty sure this is an issue I showed Evan two years ago and he
did find a location in the code that the ORB was not initialized correctly.
>
> However once the ORB makes a connection, it will cache that connection and
> use that connection from that point on. There is an ORB property
> socket_reuse_limit whihc tells the client side ORB to drop cached sockets
> every N method calls. Once dropped it will find the next random name
> server, connect to it, lookup the next component, and viola. Having a low
> socket_reuse_limit helps even balancing out, but increases load on the
> name servers.
>
> Its all a balancing act.
>
I fully understand the socket reuse limit and how it can be used. I also
agree its a balancing act performance to load balancing. One thing I know
for sure I get a better balance if I randomize my URL's for my name servers.
>
> True.
>
If it perfers the local server, how can it be a random list as stated above?
So if the server is host http, is a name server and serving up components,
it will get hammered even if its in a cluster?
My point to the post is to simply point out you can setup a cluster turn on
load balancing and expect the load to be balanced.
>
> No need to do this. The ORB does it for you.
How will the ORB balance HTTP traffic?
Again I understand the ORB init will randomize the URL's of the name
servers, but it does have preference for itself. Also we saw using EAServer
the load go primarily to the first server in the list. This could have been
a bug that was corrected, or might still exist.
I'm not disagreing with anything your saying. I'm just stating what I have
seen at clients. I'm also stating that my things affect how a load is
balanced.
[color=darkred]
>
>
> Dave Wolf
> Cynergy Systems
> http://www.cynergysystems.com
>
>
| |
| Dave Wolf 2005-06-23, 8:24 pm |
| In-Reply-To: <42bae5f5$1@forums-2-dub>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
NNTP-Posting-Host: h-66-166-186-122.mclnva23.covad.net
X-Original-NNTP-Posting-Host: h-66-166-186-122.mclnva23.covad.net
Message-ID: <42bb0ef7$1@forums-1-dub>
Date: 23 Jun 2005 12:35:19 -0700
X-Trace: forums-1-dub 1119555319 66.166.186.122 (23 Jun 2005 12:35:19 -0700)
X-Original-Trace: 23 Jun 2005 12:35:19 -0700, h-66-166-186-122.mclnva23.covad.net
Lines: 51
X-Authenticated-User: usecase
Path: forums-2-dub!forums-master!forums-1-dub!not-for-mail
Xref: forums-2-dub sybase.public.easerver.general:24756
Dean Jones wrote:
> Not sure what you are saying Dave.
>
> Explain how the HA name server is used? The load balancing will not have
> anything to do with the client doing the lookup. So if you have multiple
> name server in a list example iiop://jag1:9000;iiop://jag1:9000. Your saying
> the client will not use jag1 to do the lookup? I have tested this many
> times. I have worked with Evan on this (a while back) I believe there was an
> issue and the client did not randomly select a name server. To fix this we
> simple "shuffle" the list before doing a lookup.
>
What I am saying is that the client orb is supposed to randomize the
list you pass in of possible name server endpoints. So if you pass in
iiop://server1:9000;iiop://server2:9000
The client ORB should be scrambling those for you and choosing one at
random. If it is not, then something has changed because this was the
way it worked. I know it did work this way, because I did all the
benchmarks at PC Labs. Its been years since I had access to the code,
but this is my firm memory.
You should not have to scramble them youself. The ORB should do that.
Now, once it chooses one, it will keep going there until either the
connection fails, or the socket is closed by say a socket_reuse_limit.
> Are you saying that a name server will not have a preference to itself?
> Example jag1 is a name server and has component "A". I look up component "A"
> using jag1, Are you saying EAServer does not have a preference for itself in
> this case?
>
It should not specifically have a preference to find components on
itself. The name server returns an IOR to the client with multiple
profiles in it. The order of the profiles in that IOR are in the order
prescribed by the LB algorithm chosen. The client ORB simply connects
to server profiles in the order they exist within the IOR.
Do note however the ORB will not go lookup the component again at
failover time. It will simply look in its current IOR, choose the next
endpoint in the profile and move on.
If it had a preference for itself, and if it didnt scramble name servers
in the orb, then it would never LB. What would be the point?
Dave Wolf
Cynergy Systems
http://www.cynergysystems.com
|
|
|
|
|