I struggled to find out why my server broke down frequently these days. And I think I found it.
My Apache HTTP server was configured with MaxClients at 50, the default AJP connector’s connections number was about 15, and the Tomcat’s maxThreads was 150. So the bottleneck would be AJP connector, especially all my connection-heavy Comet application was serving about 20 Gtalk/MSNLive simultaneous connections. It is obviously it will break down the servers. And after switching my Gtalk/MSNLive into query modes, it still broke down the server. The reason may be the bottleneck of Apache HTTP server’s 50 maximum connections.
I updated all the configuration to 256 for maximum simultaneous connections. Hope server won’t break down for Comet application from now on.
For more about extreme maximum simultaneous connections, please read articles:
<!-- Define an AJP 1.3 Connector on port 8009 -->
maxThreads="256" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" protocol="AJP/1.3" />
Recently, my Apache HTTP server was broken down twice. The scenario was when visiting a page, it was just waiting there, blank. And I checked the server through SSH and saw that there was a lot (about 50) of inner 8009 connections. After restarting Apache HTTP Server and Tomcat Server, these connections decreased to less than 10 connections.
Client <-Kept Connection-> Apache HTTP Server <-AJP Connector (8009 Connection)-> Tomcat
When a user is connected, an AJP Connector is setup for him/her. The connection will not be tore down until the user disconnects from Gtalk or MSNLive service. So if there are more than 50 users online, there are no doubts that the server crashes.
To solve this problem, one solution would be not using Apache HTTP Server in the middle of Tomcat server and browser client. So there is no limitation on AJP connections.
And another solution would be monitoring user connections on Tomcat server side, and if there are more than 50 connections, try to switch application’s Comet-mode into Query-mode smartly.
Providing Comet applications need to solve a lot of unknown problems.
I am writing Java2Script’s Google Talk, and it uses a mode of continuation (also known as “Comet” application). In order to implement continuum data transfer between browser and server ( Tomcat in the backend ), I kept the HTTP connection blocked waiting for data to be available. In the middle, Apache httpd server is used and mod_jk is used to connect the two server.
But I met a ridiculous problem between servlet container and http server. When the browser is assigned to connect servlet directly, it can obtain data from the connection, but it can not obtain a bit when it is piped through Apache httpd server.
I tried to add “Transfer-Encoding: chunked” or “Content-length: 0” but with no helps. And I modified jk.conf with “JkOptions +FlushPackets”, still with no helps. Finally, I remove a line of “writer.flush();” from my servlet source, and things all went correctly.
The line of “writer.flush();” was located in the the line when there is no data written to the response. And I considered that when there are no data in the response, calling flush method will hang mod_jk connector up. And the whole later data transfer will be disturbed.
More details about mod_jk, I recommend you to read “Tunning apache and tomcat for web 2.0 comet application“.
Finally I found out that “JkOptions +FlushPackets” works for me. #flush method problem is a fake problem. Only after I restart both Tomcat and Apache server do “JkOptions +FlushPackets” begin to work.