Category Archives: Server

Use jmap and jhat to Detect Memory Leak

Posted on January 8, 2011 by Zhou Renjian

I noticed that my servers are running out of memory after one whole day’s service.

At first, I removed all functions that eat up memory. But still the next morning when the traffic is down, the memory is still high. So it seems I have memory leak in my server.

After making some searches and reading some blog articles, I learned that I can try to find view all memory by using jmap and jhat.

First, I made the dump:

jmap -dump:file=heap.bin ####

It was about 800M file, as my server is running at 1.5G maximum memory on 2G-memory server. I tried to use jhat to give a memory view:

jhat -J-mx1536m heap.bin

It run but failed with OutOfMemoryError, as my service was still running and it had no enough memory. Fortunately, I had another server without service running, so I uploaded the dump file to another server, and tried to start jhat there.? It run but with lots of warning message like the following:

WARNING:? Failed to resolve object id 0xac2f1690 for field key (signature L)
WARNING:? Failed to resolve object id 0xacd1c208 for field value (signature L)
WARNING:? Failed to resolve object id 0xac08df00 for field value (signature L)
WARNING:? Failed to resolve object id 0xac2f3f28 for field key (signature L)
WARNING:? Failed to resolve object id 0xac08df00 for field value (signature L)
WARNING:? Failed to resolve object id 0xac2eea70 for field key (signature L)
WARNING:? Failed to resolve object id 0xacd1c208 for field value (signature L)
WARNING:? Failed to resolve object id 0xac08df00 for field value (signature L)

And after a long time, it still gave me an OutOfMemoryError!

So I made another dump from another server which was running on its early stage with less memory being used. So the dump file was a lot smaller. And I could run jhat on the same without uploading to another. And after a very long time of waiting I finally got message:

Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

Visiting http://###.###:7000/ , I noticed that jhat was slow. But it helped! I found my memory leak.

Great tools! I think I can reduce servers but still provide more stable services now.

Posted in Java, Server | 3 Comments

Disable cron.daily Task for Server

Posted on April 23, 2010 by Zhou Renjian

I just learned the lesson that for cron.daily tasks should be disabled for Linux servers.

I were having the headache that in some certain hour of each day, my servers run into high load average. I suspected that it is my app server’s problems, and I spent tons of time on optimizing my service servers. Optimization helped a little but high system load average was still coming. I wondered why. And l installed iotop by “yum install iotop” the day before yesterday, and found that “prelink” and “updatedb” did a lot of IO reading and writing. And finally I got to know that they were the cron.daily tasks, which is located under /etc/cron.daily/ .

So disabling tasks under /etc/cron.daily/, like mlocate.cron (will invoke udpatedb), makewhatis, prelink and others, will help your servers perform a lot better.

By the way, your should also edit /etc/crontab to reschedule the time for those daily jobs. The default hour is 4 a.m. in the morning. But your servers’ timezone setting might be different or your servers’ peak traffic might be at that hour, so modifying it to your servers’ valley hour will help performance.

After these modifications, WeBuzz.IM web IM services servers will run a lot better from now on.

Posted in Server | Tagged , | Comments Off on Disable cron.daily Task for Server