EAserver - Regular EAServer Crashes

Author

Regular EAServer Crashes

Mark Maslow

2005-06-08, 1:24 pm

Having no luck so far solving this problem through Sybase Tech Support
after having a case open for nearly 2 months, I thought I'd see if
anybody on this newsgroup knows about something they don't.

We have a single instance of EAServer that serves up all of our database
driven dynamic content. EAS is serving as the HTTP server as well as
the servlet engine as well as hosting PowerBuilder components.

The server has been crashing crashing randomly, approximately 3-4 times
per week.

The error occurs outside the Java VM and PBVM. Typical message:

Jun 07 21:40:45 2005: An unexpected exception has been detected in
native code outside the VM.
Jun 07 21:40:45 2005: Unexpected Signal : unknown exception code (0x0)
occurred at PC=0x7C59BBF3
Jun 07 21:40:45 2005: Function=
Jun 07 21:40:45 2005: RaiseException+0x56
Jun 07 21:40:45 2005: Library=C:\WINNT\sys
tem32\KERNEL32.dll
Jun 07 21:40:45 2005:
Jun 07 21:40:45 2005: Current Java thread:
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.JaguarConnection.sendError(Native Method)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ResponseImpl.sendErrorMessage
(ResponseImpl.java:1217)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ResponseImpl. sendError(ResponseIm
pl.java:1209)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ServletEngine.processErrorPage
(ServletEngine.java:993)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ServletServiceImpl.processErrorPage
(ServletServiceImpl.java:321)
Jun 07 21:40:45 2005: at com.sybase.jaguar.servlet.
_sk_JaguarServlet_Se
rvletService.invoke
(_sk_JaguarServlet_S
ervletService.java:468)

The software is:

MS Windows 2000 5.00.2195 Service Pack 4
Jaguar CTS - Component Transaction Server/Version 4.2.4 (Build 42409)
PowerBuilder 8.0.4 Build 10726

TS has pointed the finger at several different things, including SSL
decryption and now some kind of PowerBuilder leak. We've sent a bunch
of trace files, but it's not clear we're any closer to a resolution now
than we were 2 months ago.

I'm beginning to think that the best place to invest our time is
converting everything we can to Tomcat, leaving EAServer to merely serve
PB components, until we can eventually phase those out (Spring and
Hibernate, here we come!!). I'm obviously feeling quite frustrated with
EAServer at the moment.

Doug Porter

2005-06-08, 1:24 pm

Anything interesting in the Windows event log around the times of the
crashes?

Doug Porter
DailyAccess Corporation

"Mark Maslow" <mark.maslow@sierraclub.org> wrote in message
news:MPG. 1d10c70edc9b44379898
08@forums.sybase.com...
> Having no luck so far solving this problem through Sybase Tech Support
> after having a case open for nearly 2 months, I thought I'd see if
> anybody on this newsgroup knows about something they don't.
>
> We have a single instance of EAServer that serves up all of our database
> driven dynamic content. EAS is serving as the HTTP server as well as
> the servlet engine as well as hosting PowerBuilder components.
>
> The server has been crashing crashing randomly, approximately 3-4 times
> per week.
>
> The error occurs outside the Java VM and PBVM. Typical message:
>
> Jun 07 21:40:45 2005: An unexpected exception has been detected in
> native code outside the VM.
> Jun 07 21:40:45 2005: Unexpected Signal : unknown exception code (0x0)
> occurred at PC=0x7C59BBF3
> Jun 07 21:40:45 2005: Function=
> Jun 07 21:40:45 2005: RaiseException+0x56
> Jun 07 21:40:45 2005: Library=C:\WINNT\sys
tem32\KERNEL32.dll
> Jun 07 21:40:45 2005:
> Jun 07 21:40:45 2005: Current Java thread:
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.JaguarConnection.sendError(Native Method)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ResponseImpl.sendErrorMessage
> (ResponseImpl.java:1217)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ResponseImpl. sendError(ResponseIm
pl.java:1209)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ServletEngine.processErrorPage
> (ServletEngine.java:993)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ServletServiceImpl.processErrorPage
> (ServletServiceImpl.java:321)
> Jun 07 21:40:45 2005: at com.sybase.jaguar.servlet.
> _sk_JaguarServlet_Se
rvletService.invoke
> (_sk_JaguarServlet_S
ervletService.java:468)
>
> The software is:
>
> MS Windows 2000 5.00.2195 Service Pack 4
> Jaguar CTS - Component Transaction Server/Version 4.2.4 (Build 42409)
> PowerBuilder 8.0.4 Build 10726
>
> TS has pointed the finger at several different things, including SSL
> decryption and now some kind of PowerBuilder leak. We've sent a bunch
> of trace files, but it's not clear we're any closer to a resolution now
> than we were 2 months ago.
>
> I'm beginning to think that the best place to invest our time is
> converting everything we can to Tomcat, leaving EAServer to merely serve
> PB components, until we can eventually phase those out (Spring and
> Hibernate, here we come!!). I'm obviously feeling quite frustrated with
> EAServer at the moment.
>

Mark Maslow

2005-06-08, 1:24 pm

Nada.

We got a dmp file out of Dr Watson, until we installed the debug server,
as requested by TS. Now we don't even get that.

In article <42a72b98$1@forums-2-dub>, "Doug Porter"
< doug_porterATdailyac
cessDOTnospamDOTcom> says...[color=darkred]
> Anything interesting in the Windows event log around the times of the
> crashes?
>
> Doug Porter
> DailyAccess Corporation
>
> "Mark Maslow" <mark.maslow@sierraclub.org> wrote in message
> news:MPG. 1d10c70edc9b44379898
08@forums.sybase.com...

Doug Porter

2005-06-08, 1:24 pm

Gremlins?

Doug Porter
DailyAccess Corporation

"Mark Maslow" <mark.maslow@sierraclub.org> wrote in message
news:MPG. 1d10cd1a5961488b9898
09@forums.sybase.com...[color=darkred]
> Nada.
>
> We got a dmp file out of Dr Watson, until we installed the debug server,
> as requested by TS. Now we don't even get that.
>
> In article <42a72b98$1@forums-2-dub>, "Doug Porter"
> < doug_porterATdailyac
cessDOTnospamDOTcom> says...
database[color=darkr
ed]
times[color=darkred]

com.sybase.jaguar.servlet.ResponseImpl. sendError(ResponseIm
pl. java:1209)[color=dar
kred]

Mark Maslow

2005-06-08, 1:24 pm

Gee - thanks. Does Sybase charge extra for the gremlin-free version?
Or a fee for on-going gremlin extermination?

In article <42a73091@forums-1-dub>, "Doug Porter"
< doug_porterATdailyac
cessDOTnospamDOTcom> says...[color=darkred]
> Gremlins?
>
> Doug Porter
> DailyAccess Corporation
>
> "Mark Maslow" <mark.maslow@sierraclub.org> wrote in message
> news:MPG. 1d10cd1a5961488b9898
09@forums.sybase.com...
> database
> times

Dave Wolf

2005-06-08, 1:24 pm

OK so it looks like it is falling over trying to marshall an error
message. Rather loading the default error page. Usually crashes like
this come from a null pointer. Remember EAS is a C based server, and
uses JNI to interact with the VM. Although null pointers are annoying
in Java, they arent destabilizing. In C thats not true. A peek into
the wrong address will cause a crash.

So my experience is uaully that a null pointer in Java somehow worked
its way across the JNI bridge, thus sending EAS down.

My theory is backed up a bit by this line way up in the first frame of
the stack.

Jun 07 21:40:45 2005: Current Java thread:
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.JaguarConnection.sendError(Native Method)
Jun 07 21:40:45 2005: at

Notice the "native method" part. You might tell TS to poke into
sendError in their C code and see if they check for a null pointer in
that string pointer. (I dont have source these days!)

Any change you have some log4j logs, or could add some logging to try to
figure out where in your servlet/jsp code you are when this happens?

Tomcat is a very nice servlet engine, and we combine it with EAS quite a
bit. The combination makes a lot of sense.

Spring/hibernate? Another technology du jour IMHO. Be glad to send you
to people who were pretty gung ho too until they actually used them.

I know I've also pointed out some other kind of RAD java products we
like, and they combine with EAS nicely.

Dave Wolf
Cynergy Systems
http://www.cynergysystems.com

Mark Maslow wrote:
> Having no luck so far solving this problem through Sybase Tech Support
> after having a case open for nearly 2 months, I thought I'd see if
> anybody on this newsgroup knows about something they don't.
>
> We have a single instance of EAServer that serves up all of our database
> driven dynamic content. EAS is serving as the HTTP server as well as
> the servlet engine as well as hosting PowerBuilder components.
>
> The server has been crashing crashing randomly, approximately 3-4 times
> per week.
>
> The error occurs outside the Java VM and PBVM. Typical message:
>
> Jun 07 21:40:45 2005: An unexpected exception has been detected in
> native code outside the VM.
> Jun 07 21:40:45 2005: Unexpected Signal : unknown exception code (0x0)
> occurred at PC=0x7C59BBF3
> Jun 07 21:40:45 2005: Function=
> Jun 07 21:40:45 2005: RaiseException+0x56
> Jun 07 21:40:45 2005: Library=C:\WINNT\sys
tem32\KERNEL32.dll
> Jun 07 21:40:45 2005:
> Jun 07 21:40:45 2005: Current Java thread:
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.JaguarConnection.sendError(Native Method)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ResponseImpl.sendErrorMessage
> (ResponseImpl.java:1217)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ResponseImpl. sendError(ResponseIm
pl.java:1209)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ServletEngine.processErrorPage
> (ServletEngine.java:993)
> Jun 07 21:40:45 2005: at
> com.sybase.jaguar.servlet.ServletServiceImpl.processErrorPage
> (ServletServiceImpl.java:321)
> Jun 07 21:40:45 2005: at com.sybase.jaguar.servlet.
> _sk_JaguarServlet_Se
rvletService.invoke
> (_sk_JaguarServlet_S
ervletService.java:468)
>
> The software is:
>
> MS Windows 2000 5.00.2195 Service Pack 4
> Jaguar CTS - Component Transaction Server/Version 4.2.4 (Build 42409)
> PowerBuilder 8.0.4 Build 10726
>
> TS has pointed the finger at several different things, including SSL
> decryption and now some kind of PowerBuilder leak. We've sent a bunch
> of trace files, but it's not clear we're any closer to a resolution now
> than we were 2 months ago.
>
> I'm beginning to think that the best place to invest our time is
> converting everything we can to Tomcat, leaving EAServer to merely serve
> PB components, until we can eventually phase those out (Spring and
> Hibernate, here we come!!). I'm obviously feeling quite frustrated with
> EAServer at the moment.
>

Doug Porter

2005-06-08, 1:24 pm

Just think of it as getting the gremlins at no extra charge (what a bargain)
:-)

Doug Porter
DailyAccess Corporation

"Mark Maslow" <mark.maslow@sierraclub.org> wrote in message
news:MPG. 1d10d140db3947119898
0a@forums.sybase.com...[color=darkred]
> Gee - thanks. Does Sybase charge extra for the gremlin-free version?
> Or a fee for on-going gremlin extermination?
>
> In article <42a73091@forums-1-dub>, "Doug Porter"
> < doug_porterATdailyac
cessDOTnospamDOTcom> says...
server,[color=darkre
d]
the[color=darkred]
Support[color=darkre
d]
as[color=darkred]
(0x0)[color=darkred]

Method)[color=darkre
d]

Mark Maslow

2005-06-08, 8:24 pm

Well, I can only assume that, after the case has been open for 2 months,
and escalated up 2 levels of management, that TS has been poking into
their C code. If I have to tell them to do so, I think we'd better hang
up EAServer quick!

We've turned on all kinds of logging. The error is *not* in my Java
code or PB code - it is clearly in Sybase's native code for the servlet
engine, as you point out. The fact that EAS is a C based server that is
so easily destablized is perhaps the real problem. And it's way
frustrating for us, because there's absolutely nothing we can do
ourselves.

I've already used Hibernate for 2 web apps and for those apps it has
worked exceptionally well. After using PB components for web app data
access, it is a real breath of fresh air. It really shines when doing
enhancements and modifications, since the entire object graph is already
in memory or the pieces are there just by referencing them, so I don't
have to go chasing down child objects and code translations with
separate DB calls. And the lazy loading and built-in caching seem to
make interaction with the DB quite efficient as well as easy. I'm aware
that there are people who don't like it for various reasons, but there
are a great many that do - and it has clearly influenced the EJB3 spec.
Plus it supports direct JDBC calls if that's what best suits your need
in a particular case, although I haven't yet found the need.

As for Spring - the underlying concepts are so simple that I have a hard
time imagining that it has big drawbacks - but I do not yet have enough
personal experience with it to say with any authority. I certainly
*can* say with authority what the drawbacks with EAS are.

RAD - well, I found the following rather interesting:

http://www.xebia.com/ oth_publicati... /> s_productiv
e_as_4gl.html

Peter Abbott

2005-06-08, 8:24 pm

We have experienced many different flavours of jvm aborts and crashes
while using easerver. Some were related to memory issues and other
required opening cases and getting patches. One of these patches has
even made it to the latest EBF for 4.2.3

One of the main issues when we started getting aborts was due to the
java memory sizing even though it appeared to come from the C section of
code. With the java heap even though you can specify -Xmx to a large
value you can often run into problems expecially if there is not enough
memory left for an exception to be raised.

The fisrt set we did was to specify -XX:MaxPermSize=96M as the class
garbage collection is turned off within eas and the class definitions
are stored in the permanent generatoion section of the heap. The deafult
is 64M but we found for us after modifying it it would top out at about ~80M

The next thing we did which helped prevent more crashes was tune the
rest of the java heap, but this is system dependent so cant really
suggest much but look at using the jvm options like -XX:NewSize,
-XX:MaxNewSize, - XX:MaxTenuringThresh
old, -XX:SurvivorRatio

The jvm options helped for a while but when ever we see real heavy usage
we started to get more crashes. We are now at the stage where we are
sending the details of the crashes to sybase and they are providing us
with patches.

hope this helps
Pete

PS. EAS 6 is being rewritten entirely in java (or so I have heard), so
these problems with the C code will go away.

Mark Maslow wrote:

> Well, I can only assume that, after the case has been open for 2 months,
> and escalated up 2 levels of management, that TS has been poking into
> their C code. If I have to tell them to do so, I think we'd better hang
> up EAServer quick!
>
> We've turned on all kinds of logging. The error is *not* in my Java
> code or PB code - it is clearly in Sybase's native code for the servlet
> engine, as you point out. The fact that EAS is a C based server that is
> so easily destablized is perhaps the real problem. And it's way
> frustrating for us, because there's absolutely nothing we can do
> ourselves.
>
> I've already used Hibernate for 2 web apps and for those apps it has
> worked exceptionally well. After using PB components for web app data
> access, it is a real breath of fresh air. It really shines when doing
> enhancements and modifications, since the entire object graph is already
> in memory or the pieces are there just by referencing them, so I don't
> have to go chasing down child objects and code translations with
> separate DB calls. And the lazy loading and built-in caching seem to
> make interaction with the DB quite efficient as well as easy. I'm aware
> that there are people who don't like it for various reasons, but there
> are a great many that do - and it has clearly influenced the EJB3 spec.
> Plus it supports direct JDBC calls if that's what best suits your need
> in a particular case, although I haven't yet found the need.
>
> As for Spring - the underlying concepts are so simple that I have a hard
> time imagining that it has big drawbacks - but I do not yet have enough
> personal experience with it to say with any authority. I certainly
> *can* say with authority what the drawbacks with EAS are.
>
> RAD - well, I found the following rather interesting:
>
> http://www.xebia.com/ oth_publicati... /> s_productiv
> e_as_4gl.html
>

Mark Maslow

2005-06-08, 8:24 pm

Thanks for the reply. Interesting to hear that others have had ongoing
problems with EAServer crashes.

I am rather dubious that this is a memory usage problem related to the
Java VM settings. From the logs, it appears that the crashes are
entirely outside the Java VM.

Last entry in httprequest log is a bogus request:

64.12.116.131 - - [07/Jun/2005:21:40:43 -0800] "x=88&y=17" 400 0

With tracing on, the last entries in the servlet log:

Jun 07 21:40:43 2005: HTTP_VERSION: HTTP/1.0
Jun 07 21:40:43 2005: getMethodId: 0
Jun 07 21:40:43 2005: sendError()
Jun 07 21:40:43 2005: Reset JagHttp11OutputStrea
m
Jun 07 21:40:43 2005: doErrorPage(400,)
Jun 07 21:40:43 2005: sendErrorMessage(400
,)

last entries in jaguar.log:

Jun 07 21:40:43 2005: SRVLIB Message: 16389/10/0: Read operation timed
out for process 671
Jun 07 21:40:43 2005: SRVLIB Message: 16240/10/0: Net-Library routine
net_read() failed in srv_rawread
Network error: status = 0 - Unable to determine Net-Library error

No other messages in any of the log files until the following, 2 seconds
later:

Jun 07 21:40:45 2005: An unexpected exception has been detected in
native code outside the VM.
Jun 07 21:40:45 2005: Unexpected Signal : unknown exception code (0x0)
occurred at PC=0x7C59BBF3
Jun 07 21:40:45 2005: Function=
Jun 07 21:40:45 2005: RaiseException+0x56
Jun 07 21:40:45 2005: Library=C:\WINNT\sys
tem32\KERNEL32.dll

The heap at VM abort:

Jun 07 21:40:46 2005: Heap at VM Abort:
Jun 07 21:40:46 2005:
Jun 07 21:40:46 2005: Heap
Jun 07 21:40:46 2005: def new generation
Jun 07 21:40:46 2005: total 3008K, used 796K
Jun 07 21:40:46 2005: [0x1f800000, 0x1fb40000, 0x22f80000)
Jun 07 21:40:46 2005: eden
Jun 07 21:40:46 2005: space 2688K, 21% used
Jun 07 21:40:46 2005: [0x1f800000, 0x1f88e960, 0x1faa0000)
Jun 07 21:40:46 2005: from
Jun 07 21:40:46 2005: space 320K, 70% used
Jun 07 21:40:46 2005: [0x1faa0000, 0x1fad86a0, 0x1faf0000)
Jun 07 21:40:46 2005: to
Jun 07 21:40:46 2005: space 320K, 0% used
Jun 07 21:40:46 2005: [0x1faf0000, 0x1faf0000, 0x1fb40000)
Jun 07 21:40:46 2005: tenured generation
Jun 07 21:40:46 2005: total 26016K, used 19929K
Jun 07 21:40:46 2005: [0x22f80000, 0x248e8000, 0x3ec00000)
Jun 07 21:40:46 2005: the
Jun 07 21:40:46 2005: space 26016K, 76% used
Jun 07 21:40:46 2005: [0x22f80000, 0x242f6620, 0x242f6800, 0x248e8000)
Jun 07 21:40:46 2005: compacting perm gen
Jun 07 21:40:46 2005: total 22272K, used 22024K
Jun 07 21:40:46 2005: [0x3ec00000, 0x401c0000, 0x42c00000)
Jun 07 21:40:46 2005: the
Jun 07 21:40:46 2005: space 22272K, 98% used
Jun 07 21:40:46 2005: [0x3ec00000, 0x401821b0, 0x40182200, 0x401c0000)
Jun 07 21:40:46 2005:

Still think this might be a matter of tuning the Java VM?

I'm afraid this statement is not encouraging:

> The jvm options helped for a while but when ever we see real heavy usage
> we started to get more crashes. We are now at the stage where we are
> sending the details of the crashes to sybase and they are providing us
> with patches.

But do I find the prospect of dumping the C code in favor of redoing EAS
entirely in Java encouraging.

Mark Maslow

2005-06-08, 8:24 pm

I passed your message on to TS, and got the following reply. Sounds
like our issues are different ones.

Looking into the other customer's issue, it doesn't seem related. They
basically have a memory intensive application and are pushing the memory
threashholds. Most of what they see are JVM OutOfMemory issues.
Engineering has advised they tailor their memory settings so they don't
crash the server. I think their issue is more of an application issue
and less of a EAServer issue.

Tpau

2005-06-09, 11:24 am

IMHO:
If you get a JVM error in EAServer you better prepare for solving the
problems yourself. Sybase support does not have a clue.

"Mark Maslow" <mark.maslow@sierraclub.org> skrev i meddelandet
news:MPG. 1d110fe793f631f79898
0d@forums.sybase.com...
>I passed your message on to TS, and got the following reply. Sounds
> like our issues are different ones.
>
>
> Looking into the other customer's issue, it doesn't seem related. They
> basically have a memory intensive application and are pushing the memory
> threashholds. Most of what they see are JVM OutOfMemory issues.
> Engineering has advised they tailor their memory settings so they don't
> crash the server. I think their issue is more of an application issue
> and less of a EAServer issue.
>

Mark Maslow

2005-06-09, 11:24 am

That's encouraging.

But it's not a JVM error! If you look at the error, you will see that
it's outside the JVM - in native C code written by Sybase. If they
don't have a clue about that kind of error, there really is no hope!
How is it possible to solve such a problem yourself??

In article <42a86623@forums-2-dub>, Tpau@spock.com says...
> IMHO:
> If you get a JVM error in EAServer you better prepare for solving the
> problems yourself. Sybase support does not have a clue.
>
>

Tpau

2005-06-09, 1:24 pm

It was not ment to be encouraging!

But let me rephrase myself:

IMHO:
If you get any kind of error in EAServer you better prepare for solving the
problems yourself. Sybase support does not have a clue.

"Mark Maslow" <mark.maslow@sierraclub.org> skrev i meddelandet
news:MPG. 1d120d326a9cde149898
0e@forums.sybase.com...[color=darkred]
> That's encouraging.
>
> But it's not a JVM error! If you look at the error, you will see that
> it's outside the JVM - in native C code written by Sybase. If they
> don't have a clue about that kind of error, there really is no hope!
> How is it possible to solve such a problem yourself??
>
> In article <42a86623@forums-2-dub>, Tpau@spock.com says...

Mark Maslow

2005-06-09, 1:24 pm

If what you say is true, then we should run away from EAServer as fast
as possible, since much of the core functionality is implemented in
proprietary C code that we have no access to.

Anyone else have any comments on this matter??

You gonna let this one stand Jonathan?

Stability of the server and quality of support when things go wrong are
absolutely vital if there is to be a future for the product.

In article <42a87d27$1@forums-2-dub>, Tpau@spock.com says...
> It was not ment to be encouraging!
>
> But let me rephrase myself:
>
> IMHO:
> If you get any kind of error in EAServer you better prepare for solving the
> problems yourself. Sybase support does not have a clue.
>
>
> "Mark Maslow" <mark.maslow@sierraclub.org> skrev i meddelandet
> news:MPG. 1d120d326a9cde149898
0e@forums.sybase.com...
>
>
>

Peter Abbott

2005-06-09, 8:24 pm

I would have to disagree with that comment. Sybase support have been
bending over bakcwards to help us solve our vm aborts, similar to the
types of errors what Mark has been experiencing.

The main problem as with any error is that you have to be able to
reproduce the error so anyone who tries to fix it knows where to look.
We are partly guilty in that area because we would only see the vm
aborts in the production system which isnt very useful when trying to
demonstrate the error.

I take my hat off to all the help that sybase support have given us in
the last few months in helping us sort out or stability problems

Pete

Tpau wrote:
> It was not ment to be encouraging!
>
> But let me rephrase myself:
>
> IMHO:
> If you get any kind of error in EAServer you better prepare for solving the
> problems yourself. Sybase support does not have a clue.
>
>
>
>
>
>
>
>
>
>
>
> "Mark Maslow" <mark.maslow@sierraclub.org> skrev i meddelandet
> news:MPG. 1d120d326a9cde149898
0e@forums.sybase.com...
>
>
>
>

Dave Wolf

2005-06-09, 8:24 pm

Mark Maslow wrote:
> If what you say is true, then we should run away from EAServer as fast
> as possible, since much of the core functionality is implemented in
> proprietary C code that we have no access to.
>

So under the same logic you should run away from Microsoft Word,
Macromedia Flash, Adobe PDF, Quick Books, etc, all because they are
implemented in proprietary C code you dont have access to.... Of course
not. Being written in C or being proprietary isnt the issue. The issue
is of support and quality. Lets not miss that.

> Anyone else have any comments on this matter??
>

Contrary to your last post IMHO the JVM did indeed abort. It even said
so. Now did EAS cause the JVM to abort? Could have definately without
a doubt. But to be clear, the JVM did abort.

> You gonna let this one stand Jonathan?
>

Good question.

> Stability of the server and quality of support when things go wrong are
> absolutely vital if there is to be a future for the product.
>

Agreed. So does marketing and a million other things. That said, ever
opened a IBM or BEA support case? But I agree with you.

I'm trying to help.... Are you using custom error pages? Is there
nothing in any log anywhere with some consistency? I assume you have
butt-loads of log4j logs or something? Is it a time of day thing?

That said, I like your suggestion of moving the JSP layer out to tomcat.
Its the right topology from a partitioning/stability/resiliancy PoV.

Dave Wolf
Cynergy Systems
http://www.cynergysystems.com

[color=darkred]
>
> In article <42a87d27$1@forums-2-dub>, Tpau@spock.com says...
>

Mark Maslow

2005-06-09, 8:24 pm

Dave -

Thank you for your response.

I certainly did not mean to say that you should run away from anything
written in proprietary C code. I was reacting to a post that said "if
JVM aborts, you're on your own". If this is true, and you're working
with open source, you might have a chance, but with proprietary C code,
there is absolutely no hope if "you're on your own". I am in agreement
with you - the issue is of support and quality.

OK - the JVM aborted - it said so, but it also said:

An unexpected exception has been detected in native code outside the VM.

This leads me to believe that the issue is not a problem with the JVM,
but an issue with Sybase's proprietary C code. Feel free to correct me
if you believe this assumption is wrong.

We have unfortunately not been able to come up with anything about the
aborts that are consistent.

The traces that I posted indicate to me that the problem occurred
entirely within Sybase code. It doesn't appear that any of our custom
code (servlets, JSPs or components) were involved at all, since they had
finished their jobs at 21:40:39. The events leading up to the crash
occured 4 seconds later at 21:40:43, beginning with a bogus http request
that could not possibly have invoked any custom code. I copy below the
traces immediately preceeding the last crash:

Last entries in HTTP Request log:

67.183.91.39 - - [07/Jun/2005:21:40:39 -0800] "GET /action/tamain?alid=
393 HTTP/1.1" 200 37632
64.12.116.131 - - [07/Jun/2005:21:40:43 -0800] "x=88&y=17" 400 0

Last entries in the servlet log (tracing set on):

Jun 07 21:40:39 2005: JaguarOutputStream:c
lose
Jun 07 21:40:39 2005: ResponseImpl.closeResponse
Jun 07 21:40:39 2005: closeResponse: _isChunkfalse
Jun 07 21:40:39 2005: closed response
Jun 07 21:40:39 2005: ServiceEngine.service(): status: 0
Jun 07 21:40:39 2005: Returning from doService(): 0
Jun 07 21:40:43 2005: ServletServiceImpl.processErrorPage()
Jun 07 21:40:43 2005: ServletEngine.processErrorPage()
Jun 07 21:40:43 2005: contextPath=null
Jun 07 21:40:43 2005: isRequestedSecurityS
essionIdFromCookie: false
Jun 07 21:40:43 2005: getRemoteAddr()
Jun 07 21:40:43 2005: No JSESSION ID returned
Jun 07 21:40:43 2005: getProtocol()
Jun 07 21:40:43 2005: HTTP_VERSION: HTTP/1.0
Jun 07 21:40:43 2005: getMethodId: 0
Jun 07 21:40:43 2005: sendError()
Jun 07 21:40:43 2005: Reset JagHttp11OutputStrea
m
Jun 07 21:40:43 2005: doErrorPage(400,)
Jun 07 21:40:43 2005: sendErrorMessage(400
,)

last entries in jaguar.log:

Jun 07 21:40:43 2005: SRVLIB Message: 16389/10/0: Read operation timed
out for process 671
Jun 07 21:40:43 2005: SRVLIB Message: 16240/10/0: Net-Library routine
net_read() failed in srv_rawread
Network error: status = 0 - Unable to determine Net-Library error

No other messages in any of the log files until the following, 2 seconds
later:

Jun 07 21:40:45 2005: An unexpected exception has been detected in
native code outside the VM.
Jun 07 21:40:45 2005: Unexpected Signal : unknown exception code (0x0)
occurred at PC=0x7C59BBF3
Jun 07 21:40:45 2005: Function=
Jun 07 21:40:45 2005: RaiseException+0x56
Jun 07 21:40:45 2005: Library=C:\WINNT\sys
tem32\KERNEL32.dll

If you think this points to something other than Sybase code, I'd
appreciate your ideas as to where else to look.

I appreciate hearing your opinion that Tomcat is a good way to go. I
suspect that is ultimately where we'll get to. Basically, no support,
other than the community - but I've never needed any support when using
it for development. Have you heard of people having important issues
with Tomcat that were difficult to resolve?

Mark Maslow

2005-06-09, 8:24 pm

Just to complete the final abort message - there was a stack trace,
which points to the sendErrorMessage referred to in the last entry in
the servlet log:

Jun 07 21:40:45 2005: An unexpected exception has been detected in
native code outside the VM.
Jun 07 21:40:45 2005: Unexpected Signal : unknown exception code (0x0)
occurred at PC=0x7C59BBF3
Jun 07 21:40:45 2005: Function=
Jun 07 21:40:45 2005: RaiseException+0x56
Jun 07 21:40:45 2005: Library=C:\WINNT\sys
tem32\KERNEL32.dll
Jun 07 21:40:45 2005:
Jun 07 21:40:45 2005: Current Java thread:
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.JaguarConnection.sendError(Native Method)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ResponseImpl.sendErrorMessage
(ResponseImpl.java:1217)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ResponseImpl. sendError(ResponseIm
pl.java:1209)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ServletEngine.processErrorPage
(ServletEngine.java:993)
Jun 07 21:40:45 2005: at
com.sybase.jaguar.servlet.ServletServiceImpl.processErrorPage
(ServletServiceImpl.java:321)
Jun 07 21:40:45 2005: at com.sybase.jaguar.servlet.
_sk_JaguarServlet_Se
rvletService.invoke
(_sk_JaguarServlet_S
ervletService.java:468)
Jun 07 21:40:45 2005:

Jonathan Baker [Sybase]

2005-06-10, 11:24 am

Wow - go teach a class for a day, and the gauntlet is thrown! :-)

Okay, let's be honest about technical support. Assuming that they know
the entire code line, including C and Java code, and can solve any
problem within minutes is totally absurd. They have the difficult job
of recreating a problem in an environment they have never seen, doing it
all over the phone and email, and then updating you on status on a
regular basis. Oh, and they have several open cases running at the same
time... I don't want their job...

And, as an evangelist (and a former SWAT team member, with an industry
leader that I could run things by) I have seen s*** that would blow your
mind. "Performance enhancements" that are crashing things left and
right. 8 different Java VM's in a path (no kidding, I counted 8). Bad
code. Bad libraries. Poorly configured stress testing. Overloaded
servers. And the list goes on and on and on.

Tech support is a horribly difficult job. Before anyone insults them, I
should state this: every time I review a case with tech support, I find
they are doing all they can - and are often waiting for the customer for
input.

Here are my two suggestions: first, find a way to recreate the bug on
demand (or as close as possible). Transfer that test case to tech
support (basically, short cut the reproduction cycle). Then, if you
don't hear from them, just call them back. They don't mind giving
status updates - and it may help you know what is going on.

Let me know if you have any questions.

Jonathan

PS. Yeah, Dave, you owe me a beer. :-)

Tpau wrote:
> It was not ment to be encouraging!
>
> But let me rephrase myself:
>
> IMHO:
> If you get any kind of error in EAServer you better prepare for solving the
> problems yourself. Sybase support does not have a clue.
>
>
>
>
>
>
>
>
>
>
>
> "Mark Maslow" <mark.maslow@sierraclub.org> skrev i meddelandet
> news:MPG. 1d120d326a9cde149898
0e@forums.sybase.com...
>
>
>
>

Tpau

2005-06-10, 11:24 am

> Okay, let's be honest about technical support. Assuming that they know
> the entire code line, including C and Java code, and can solve any problem
> within minutes is totally absurd. They have the difficult job of
> recreating a problem in an environment they have never seen, doing it all
> over the phone and email, and then updating you on status on a regular
> basis. Oh, and they have several open cases running at the same time...
> I don't want their job...

If it was a easy task we would not pay you good money to do the job!!!!

"Jonathan Baker [Sybase]" < last_name_first_init
ial@sybase.com> skrev i
meddelandet news:42a9b19a$1@foru
ms-1-dub...[color=darkred]
> Wow - go teach a class for a day, and the gauntlet is thrown! :-)
>
> Okay, let's be honest about technical support. Assuming that they know
> the entire code line, including C and Java code, and can solve any problem
> within minutes is totally absurd. They have the difficult job of
> recreating a problem in an environment they have never seen, doing it all
> over the phone and email, and then updating you on status on a regular
> basis. Oh, and they have several open cases running at the same time...
> I don't want their job...
>
> And, as an evangelist (and a former SWAT team member, with an industry
> leader that I could run things by) I have seen s*** that would blow your
> mind. "Performance enhancements" that are crashing things left and right.
> 8 different Java VM's in a path (no kidding, I counted 8). Bad code. Bad
> libraries. Poorly configured stress testing. Overloaded servers. And the
> list goes on and on and on.
>
> Tech support is a horribly difficult job. Before anyone insults them, I
> should state this: every time I review a case with tech support, I find
> they are doing all they can - and are often waiting for the customer for
> input.
>
> Here are my two suggestions: first, find a way to recreate the bug on
> demand (or as close as possible). Transfer that test case to tech support
> (basically, short cut the reproduction cycle). Then, if you don't hear
> from them, just call them back. They don't mind giving status updates -
> and it may help you know what is going on.
>
> Let me know if you have any questions.
>
>
> Jonathan
>
>
> PS. Yeah, Dave, you owe me a beer. :-)
>
>
> Tpau wrote:

Mark Maslow

2005-06-10, 1:24 pm

Jonathan -

I actually don't have any complaints about the TS people that I have dealt
with directly, although I suspect they don't always get much help from
Engineering. I do fully understand that the issue is a tough one.

I hope you'll take a look at my last 2 postings, where I post excerpts from
the log files preceeding the last EAServer crash, and let me know if you
think there's more that I could do to help to isolate the problem.

Unfortunately, I can't even reproduce the problem in my environment on
demand. Sometimes we'll go for days without any problems. Sometimes we'll
get 3 crashes in one day. It is not clearly related to time of day or to
volume of requests. The one thing that is consistent is that the traces
show the server in the process of executing low level http server routines -
none of our custom servlets or components are executing at the time that the
server crashes.

I have patiently worked with TS for about 2 months now, trying many things
and spending many hours of my time. I'm sure you can understand why I'm
getting really frustrated. I really should be spending my time developing
applications - not attempting to debug an internal server process.

It may very well be that the best course of action is to move away from
EAServer for everything except for hosting PB components for legacy web
apps. But if anyone has any constructive suggestions that can possibly
enable us to continue using EAServer as http server and servlet engine for
production at least for a little longer without it crashing regularly, I
would appreciate hearing them.

"Jonathan Baker [Sybase]" < last_name_first_init
ial@sybase.com> wrote in
message news:42a9b19a$1@foru
ms-1-dub...[color=darkred]
> Wow - go teach a class for a day, and the gauntlet is thrown! :-)
>
> Okay, let's be honest about technical support. Assuming that they know
> the entire code line, including C and Java code, and can solve any problem
> within minutes is totally absurd. They have the difficult job of
> recreating a problem in an environment they have never seen, doing it all
> over the phone and email, and then updating you on status on a regular
> basis. Oh, and they have several open cases running at the same time...
> I don't want their job...
>
> And, as an evangelist (and a former SWAT team member, with an industry
> leader that I could run things by) I have seen s*** that would blow your
> mind. "Performance enhancements" that are crashing things left and right.
> 8 different Java VM's in a path (no kidding, I counted 8). Bad code. Bad
> libraries. Poorly configured stress testing. Overloaded servers. And the
> list goes on and on and on.
>
> Tech support is a horribly difficult job. Before anyone insults them, I
> should state this: every time I review a case with tech support, I find
> they are doing all they can - and are often waiting for the customer for
> input.
>
> Here are my two suggestions: first, find a way to recreate the bug on
> demand (or as close as possible). Transfer that test case to tech support
> (basically, short cut the reproduction cycle). Then, if you don't hear
> from them, just call them back. They don't mind giving status updates -
> and it may help you know what is going on.
>
> Let me know if you have any questions.
>
>
> Jonathan
>
>
> PS. Yeah, Dave, you owe me a beer. :-)
>
>
> Tpau wrote:

Jonathan Baker [Sybase]

2005-06-13, 8:24 pm

Good point. But, if you want it faster (and want to spend money) have
you considered bringing Sybase ProServ (or any other consulting company)
in to debug the problem for you? That would speed things up...

Jonathan

Tpau wrote:
>
>
> If it was a easy task we would not pay you good money to do the job!!!!
>
>
>
>
> "Jonathan Baker [Sybase]" < last_name_first_init
ial@sybase.com> skrev i
> meddelandet news:42a9b19a$1@foru
ms-1-dub...
>
>

Dean Jones

2005-06-21, 8:24 pm

I have to add, when we start looking into issues with the server crashing
and the client is blaming the product, 9 times out of 10 its bad code. Some
times we do find real bugs, and we get a fix from Sybase quickly.

The fact is its easier to blame a product then to blame your development
staff.

I'm in know way trying to say this issue is with the developers code.

--
Dean Jones [TeamSybase]
CEO
Certified PowerBuilder Developer
www.powerobjects.com
(612) 339-3355 ext 112

"Jonathan Baker [Sybase]" < last_name_first_init
ial@sybase.com> wrote in
message news:42ae0e0c@forums
-1-dub...[color=darkred]
> Good point. But, if you want it faster (and want to spend money) have you
> considered bringing Sybase ProServ (or any other consulting company) in to
> debug the problem for you? That would speed things up...
>
>
> Jonathan
>
>
> Tpau wrote:

Mark Maslow

2005-06-22, 11:24 am

There is no doubt plenty of bad code out there that can lead to server
crashes.

In this case, the problem was with internal EAServer code to process
http requests. The resolution took over 2 months working with TS and at
least 20-30 hours of my time. The bug was a known bug by the time we
opened the case, but it took that long for TS to realize that the known
bug was causing the crashes.

In article <42b87dc5$1@forums-2-dub>, "Dean Jones"
< dean_dot_jones_at_po
werobjects_dot_com> says...
> I have to add, when we start looking into issues with the server crashing
> and the client is blaming the product, 9 times out of 10 its bad code. Some
> times we do find real bugs, and we get a fix from Sybase quickly.
>
> The fact is its easier to blame a product then to blame your development
> staff.
>
> I'm in know way trying to say this issue is with the developers code.
>
>

Dean Jones

2005-06-22, 1:25 pm

I know your a strong developer and I assumed you had identified a bug. I
just was stating most the time we find developer errors when we are asked to
dig in and find the problem.

--
Dean Jones [TeamSybase]
CEO
Certified PowerBuilder Developer
www.powerobjects.com
(612) 339-3355 ext 112

"Mark Maslow" <mark.maslow@sierraclub.org> wrote in message
news:MPG. 1d2322827e89a1898981
8@forums.sybase.com...[color=darkred]
> There is no doubt plenty of bad code out there that can lead to server
> crashes.
>
> In this case, the problem was with internal EAServer code to process
> http requests. The resolution took over 2 months working with TS and at
> least 20-30 hours of my time. The bug was a known bug by the time we
> opened the case, but it took that long for TS to realize that the known
> bug was causing the crashes.
>
>
> In article <42b87dc5$1@forums-2-dub>, "Dean Jones"
> < dean_dot_jones_at_po
werobjects_dot_com> says...

Sponsored Links