Java??? Troubleshooting Guide for
HP Part Number:
Published: July 2007
Edition: 3
Java??? Troubleshooting Guide for
HP Part Number:
Published: July 2007
Edition: 3
?? Copyright 2007
Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. UNIX is a registered trademark of The Open Group.
4Table of Contents
6
List of Figures
7
8
List of Tables
9
10
About This Document
The information in this document will help application developers and support engineers debug their Java applications on
Intended Audience
This document is intended for application developers and support engineers who are debugging Java applications on
New and Changed Information in This Edition
This is the third version of this document. It contains fixes to the second version as well as a new chapter, which is a tutorial about analyzing core files.
Document Organization
This document contains four chapters:
Chapter 1: Diagnostic and Monitoring Tools and
Chapter 2: Useful System Tools for Java
Chapter 3: Getting Help from
Chapter 4: Core File
Typographic Conventions
This document uses the following typographical conventions:
11
Related Information
This document contains information specific to troubleshooting Java problems on
More information can also be found in the
Publishing History
The document printing date and part number indicate the document???s current edition. The printing date will change when a new edition is printed. Minor changes may be made at reprint without changing the printing date. The document part number will change when extensive changes are made. Document updates may be issued between editions to correct errors or document product changes. To ensure that you receive the updated or new editions, you should subscribe to the appropriate product support service. See your HP sales representative for details. The latest version of this document is available online at:
HP Encourages Your Comments
HP encourages your comments concerning this document. We are committed to providing documentation that meets your needs. Send any errors found, suggestions for improvement, or compliments to:
Include the document title, manufacturing part number, and any comment, error found, or suggestion for improvement you have concerning this document.
12 About This Document
1 Diagnostic and Monitoring Tools and Options
This chapter describes the tools and options available for postmortem diagnostics, analysis of hung/deadlocked processes, monitoring memory usage, and performance monitoring.
The tools and options are listed in tables by their respective functions in the first section of this chapter. Many of them are listed in multiple tables since they can be used for multiple functions.
The tools and options are described in detail with examples, where applicable, in the remaining sections of this chapter. All the tools and options described in this chapter are either included in the Java 2 Platform Standard Edition Development Kit (JDK 1.5+), are included with
http://www.hp.com/products1/unix/java
1.1
The tools and options are categorized into the following table groupings:
???Hung and Deadlocked Processes
???Miscellaneous Tools and Options
???JDK Tools Not Available on
1.1.1Crash Analysis Tools
Several of the options and tools described in this chapter are designed for postmortem diagnostics. These are the options and tools that can be used to obtain additional information if an application crashes. This analysis may either be done at the time of the crash or at a later time using information from the core file. In addition to these tools, many other tools have features useful for crash analysis.
Table
1.1.2 Hung and Deadlocked Processes
The following options and tools can help you debug a hung or deadlocked process:
1.1
Table
1.1.3 Fatal Error Handling
The following options are useful for retrieving more information when fatal errors occur:
Table
1.1.4 Monitoring Memory Use
The following options and tools are useful for monitoring memory usage of running applications:
Table
14 Diagnostic and Monitoring Tools and Options
Table
1.1.5 Performance Tools
The following tools are useful for identifying where the application spends its time. Some tools allow you to monitor performance in real time (dynamic analysis) and other tools allow you to analyze captured profiling data (static analysis):
Table
1.1.6 Miscellaneous Tools and Options
The following tools and options do not fall into any of the previous categories:
Table
1.1
1.1.7 JDK Tools Not Available on
Some JDK tools are not available on
Table
1.2
A thread dump is printed if the Java process receives a SIGQUIT signal. Therefore, issuing the command kill
In addition to the thread stacks, the
Following is an example of output generated when SIGQUIT is sent to a running Java process:
Full thread dump [Thu Oct 12 14:00:56 PDT 2006] (Java HotSpot(TM) Server
VM 1.5.0.03
at spec.jbb.JBButil.negativeExpDistribution(JBButil.java:795) at spec.jbb.TransactionManager.go(TransactionManager.java:234) at spec.jbb.JBBmain.run(JBBmain.java:258)
at java.lang.Thread.run(Thread.java:595)
- waiting to lock <444ba618> (a spec.jbb.Order)
at spec.jbb.DeliveryTransaction.process(DeliveryTransaction.java:213) at spec.jbb.DeliveryHandler.handleDelivery(DeliveryHandler.java:103) at spec.jbb.DeliveryTransaction.queue(DeliveryTransaction.java:363) - locked <154927e8> (a spec.jbb.DeliveryTransaction)
at spec.jbb.TransactionManager.go(TransactionManager.java:431) at spec.jbb.JBBmain.run(JBBmain.java:258)
at java.lang.Thread.run(Thread.java:595)
at spec.jbb.Warehouse.retrieveStock(Warehouse.java:307)
at spec.jbb.Orderline.validateAndProcess(Orderline.java:341) - locked <48563610> (a spec.jbb.Orderline)
at spec.jbb.Order.processLines(Order.java:289) - locked <48563128> (a spec.jbb.Order)
at spec.jbb.NewOrderTransaction.process(NewOrderTransaction.java:282) at spec.jbb.TransactionManager.go(TransactionManager.java:278)
at spec.jbb.JBBmain.run(JBBmain.java:258) at java.lang.Thread.run(Thread.java:595)
at spec.jbb.NewOrderTransaction.secondDisplay(NewOrderTransaction.java:416) - locked <154d4828> (a spec.jbb.NewOrderTransaction)
at spec.jbb.TransactionManager.go(TransactionManager.java:279) at spec.jbb.JBBmain.run(JBBmain.java:258)
16 Diagnostic and Monitoring Tools and Options
at java.lang.Thread.run(Thread.java:595)
"Low Memory Detector" daemon prio=10 tid=00778b80 nid=19 lwp_id=2669774 runnable [00000000..00000000]
"CompilerThread1" daemon prio=10 tid=00772c30 nid=17 lwp_id=2669772 waiting on condition [00000000..0a7ff728]
"CompilerThread0" daemon prio=10 tid=007703f0 nid=16 lwp_id=2669771 waiting on condition [00000000..0afff5b8]
"AdapterThread" daemon prio=10 tid=0076c8d0 nid=15 lwp_id=2669770 waiting on condition [00000000..00000000]
"Signal Dispatcher" daemon prio=10 tid=0076a2e0 nid=14 lwp_id=2669769 waiting on condition [00000000..00000000]
"Finalizer" daemon prio=10 tid=00530a60 nid=13 lwp_id=2669768 in Object.wait() [750c0000..750c0e60] at java.lang.Object.wait(Native Method)
-waiting on <11000100> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
-locked <11000100> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:197)
"Reference Handler" daemon prio=10 tid=0052de80 nid=12 lwp_id=2669767 in Object.wait() [752c0000..752c0ce0] at java.lang.Object.wait(Native Method)
-waiting on <11003dc8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123)
-locked <11003dc8> (a java.lang.ref.Reference$Lock)
"main" prio=8 tid=0047dc90 nid=1
at spec.jbb.JBButil.SecondsToSleep(JBButil.java:740)
at spec.jbb.Company.displayResultTotals(Company.java:942) at spec.jbb.JBBmain.DoARun(JBBmain.java:387)
at spec.jbb.JBBmain.DOIT(JBBmain.java:1137) at spec.jbb.JBBmain.main(JBBmain.java:1490)
"VM Thread" prio=10 tid=004ff510 nid=11 lwp_id=2669766 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=004d0520 nid=3 lwp_id=2669758 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=004d0600 nid=4 lwp_id=2669759 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=004d06e0 nid=5 lwp_id=2669760 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=004d07c0 nid=6 lwp_id=2669761 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=004d08a0 nid=7 lwp_id=2669762 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=004d0980 nid=8 lwp_id=2669763 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=004d0a60 nid=9 lwp_id=2669764 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=004d0b40 nid=10 lwp_id=2669765 runnable
"VM Periodic Task Thread" prio=8 tid=00500ad0 nid=18 lwp_id=2669773 waiting on condition
1.3 Fatal Error Log (hs_err_pid<pid>.log)
When a fatal error occurs, an error log is created in the file hs_err_pid<pid>.log, where <pid> is the process id of the process. The file is created in the working directory of the process, if possible. In the event that the file cannot be created in the working directory (for example, if there is insufficient space, a permission problem, or another issue), then the file is created in the temporary directory,/tmp. The error log contains information obtained at the time of the fatal error. This includes :
???Operating exception or signal that provoked the fatal error
???Version and configuration information
???Details on the thread that provoked the fatal error and its stack trace
???List of running threads and their states
???Summary information about the heap
???List of native libraries loaded
???
???Environment variables
???Details about the operating system and CPU
In some cases, only a subset of this information is output to the error log. This happens when a fatal error is so severe that the error handler is unable to recover and report all details.
1.4 gcore
The gcore command creates a core image of a running process. By default, the name of the core file for a
When gcore creates a core image of each specified process, the process is temporarily stopped. When the creation of the core image is complete, the process continues to execute.
This command is only available on
1.5 gdb
Java stack unwind enhancements have been added to gdb to enable it to support unwinding across Java frames and provide an effective way to examine stack traces containing mixed language frames (Java and C/C++) of both live Java processes and core files. This has been implemented by adding subcommands for Java VM debugging to gdb.
The following table shows which Java versions on
Table
*gdb version 5.3 requires SDK 1.4.2.10 and later versions or JDK 1.5.0.03 and later versions in order to use the Java VM debugging features.
In order to use this functionality, the GDB_JAVA_UNWINDLIB environment variable must be set to the path name of the Java unwind library. The default location of the Java unwind library on various systems is shown following. The examples are for SDK 1.4; if you are using JDK 1.5, substitute /opt/java1.5 for /opt/java1.4.
/opt/java1.4/jre/lib/PA_RISC/server/libjunwind.sl /opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl /opt/java1.4/jre/lib/PA_RISC2.0W/server/libjunwind.sl /opt/java1.4/jre/lib/IA64N/server/libjunwind.so /opt/java1.4/jre/lib/IA64W/server/libjunwind.so
Following are a few examples. If you are using ksh on a
export GDB_JAVA_UNWINDLIB=/opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl
18 Diagnostic and Monitoring Tools and Options
Additionally, this is how you set the environment variable on an Integrity machine for a
export GDB_JAVA_UNWINDLIB=/opt/java1.4/jre/lib/IA64N/server/libjunwind.so
If the SDK is installed in a location other than the default, substitute the
1.5.1 Java Stack Unwind Features
The Java stack unwind features are useful for troubleshooting problems in the Java VM. Following is a list of the Java stack unwind features:
???View mixed language frames information, including Java frames and C/C++ native frames, in a gdb backtrace.
???Distinguish various Java frame types including interpreted, compiled, and adapter frames.
???View Java method name, signature, and class package name for Java method frames.
Additional stack unwind features are available starting with SDK 1.4.2. These features fall into three categories: Java stack unwind enhancements, Java heap support, and Java threads support.
These additional features are available as part of the Java stack unwind enhancements:
???View Java compiled frame inlined methods.
???View Java interpreted or compiled frame specific information.
???View Java interpreted or compiled frame arguments and local variables.
???Disassemble Java method bytecodes.
???Print out the Java unwind table.
These additional features are available as part of the Java heap support:
???View Java heap parameters.
???Dump Java object.
???Print Java heap histogram.
???Find all the instances of a given Java class.
???Find all the references to a given object in the Java heap.
???Find out the object OOP
These additional features are available as part of Java threads support:
???View Java threads state information.
???View current Java thread information.
???View Java interpreted frame monitors information.
1.5.2gdb Subcommands for Java VM Debugging
To view the gdb commands that support Java VM debugging, type help java at the gdb prompt.
(gdb) help java
Java and JVM debugging commands.
List of java subcommands:
java args
java
java instances
java locals
java object
java oop
java references
1.5 gdb 19
Type "help java" followed by java subcommand name for full documentation. Command name abbreviations are allowed if unambiguous.
The following two tables list Java VM debugging commands and Java subcommands:
Table
Table
Type help java followed by the subcommand name for full documentation. Command name abbreviations are allowed if they are unambiguous.
Following are examples that illustrate the gdb
The first set of examples illustrate how to invoke gdb on a core file:
???Invoke gdb on a core file generated when running a
$ gdb /opt/java1.4/bin/IA64N/java core.java
???Invoke gdb on a core file generated when running a
$ gdb /opt/java1.4/bin/IA64W/java core.java
???Invoke gdb on a core file generated when running a
$ gdb /opt/java1.4/bin/PA_RISC2.0/java core.java
???Invoke gdb on a core file generated when running a
$ gdb /opt/java1.4/bin/PA_RISC2.0W/java core.java
When debugging a core file, it is good practice to rename the file from core to another name to avoid accidentally overwriting it.
20 Diagnostic and Monitoring Tools and Options
If the Java and system libraries used by the failed application reside in
The following example illustrate how to invoke gdb on a hung process:
???Determine the process id:
???Attach gdb to the running process:
$ gdb
HP gdb 5.0 for HP Itanium (32 or 64 bit) and target
GNU General Public License.Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support.
Reading symbols from /opt/java1.4/bin/IA64N/java...
(no debugging symbols found)...done.
Attaching to program: /opt/java1.4/bin/IA64N/java, process 23989
(no debugging symbols found)...
Reading symbols from /usr/lib/hpux32/libpthread.so.1...
(no debugging symbols found)...done.
Reading symbols from /usr/lib/hpux32/libdl.so.1...
...
NOTE: If the version of gdb on the system is older than version 4.5, it will be necessary to specify the full path of the Java executable in order to use the gdb subcommands. For example:
gdb /opt/java1.4/bin/PA_RISC2.0/java
A tutorial on gdb may be found at the following website: http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1677,00.html
1.6 HPjconfig
HPjconfig is a configuration tool for tuning your
HPjconfig can also be used to verify that your systems has all the necessary patches required for Java. The patches required for Java can be found at the following website:
http://www.hp.com/products1/unix/java/patches
HPjconfig runs on SDK 1.3.1 and later versions, SDK 1.4.x, and JDK 1.5.0.x.
For more information about HPjconfig including the download, go to:
http://www.hp.com/products1/unix/java/java2/hpjconfig/index.html
HPjconfig can be run in either graphical user interface (GUI) mode or
Following is usage information for the HPjconfig command:
usage:
HPjconfig [ options ]
HPjconfig [ options ] <object> <action>
objects:
actions:
Following are examples of invoking HPjconfig in GUI mode from the csh and the ksh:
(csh) $ setenv DISPLAY <Display's IP Address>:0.0 $ setenv PATH $PATH:/usr/sbin
$ java
(ksh) $ export DISPLAY=<Display's IP Address>:0.0 $ export PATH=$PATH:/usr/sbin
$ java
The following four figures show the System, Application, Patches, and Tunables tabs for the
HPjconfig tool:
Figure
Figure
22 Diagnostic and Monitoring Tools and Options
Figure
Figure
Following are the commands for invoking HPjconfig in
$ cd <hpjconfig_installation_dir>
$ java
Following is an example using HPjconfig in
$ java
List of missing patches:
PHSS_34201 solves problem emulating floating point conversion when running PA2.0 Java on an IPF system.. solves problem with Aries signal
handling that overlaps Java signal handling. solves problem emulating floating point conversion when running PA2.0 Java on an IPF system.. solves problem with Aries signal handling that overlaps Java signal handling.
Following is an example using HPjconfig to show the values for
$ java
Name nproc
max_thread_proc nkthread
nfile
Following is an example of using HPjconfig to display tunables that are set to values less than those recommended:
$ java
List of tunables whose values are less than the recommended values:
Following is an example log file produced by HPjconfig:
$ more HPjconfig_server1_20060915_042600.log Fri Sep 15 16:26:00 PDT 2006
HPjconfig 3.0.01 (Thu Jul 21 14:52:47 2005)
Reading required patches/tunables information from /tmp/HPjconfig.xml
Read required patches/tunables information
Reading patch list from system
Read patch list from system
List of required patches:
PHCO_30476 supports HPjmeter profiling of unbound (MxN) threads. PHKL_30192 solves kernel panic with thousands of MxN threads.
PHSS_30015 solves problem with Aries signal handling that overlaps Java sig nal handling.
PHSS_34201 solves problem emulating floating point conversion when running PA2.0 Java on an IPF system.. solves problem with Aries signal handling that ove rlaps Java signal handling. solves problem emulating floating point conversion w hen running PA2.0 Java on an IPF system.. solves problem with Aries signal handl ing that overlaps Java signal handling.
1.7 HPjmeter
With the release of HPjmeter 3.0, all previous versions of HPjmeter (1.x, 2.x) are no longer available for download and are no longer supported by HP.
If you have an old version of HPjmeter, please download HPjmeter 3.0 from:
http://www.hp.com/products1/unix/java/hpjmeter/index.html
HPjmeter can be used to identify and diagnose performance problems in Java applications running on
The following table lists the features of HPjmeter 3.0. The first two rows are static features and the remaining four rows are dynamic features.
24 Diagnostic and Monitoring Tools and Options
Table
Drill down into application profile metrics
???Graphic display of profiling data
???Call graphs with call count, or with CPU or clock time
???Per thread display of time spent
???Per thread or per process display
Integrated HPjtune functions with concurrent improvements in tool and help usability
Ability to examine Java Management Extension management beans (Mbeans) content and the Java VM internal memory configuration
Automatic problem detection and alerts
???Memory leak detection alerts with leak rate
???Thread deadlock detection
???Abnormal thread termination detection
???Expected out of memory error
???Excessive method compilation
Dynamic
???Java heap size
???Garbage collection events and percentage time spent in garbage collection
???CPU usage per method for hottest methods
Object allocation percentage by method
???Object allocation percentage by object type
???Method compilation count in the Java VM dynamic compiler
???Number of classes loaded by the Java VM
???Thrown exception statistics
???
HPjmeter can display data generated by the following Java product versions, on the specified architectures, with the specified
Table
The HPjmeter console can be run on:
???
???Integrity
???Windows XP/2000/NT
???Linux
The user's guide for HPjmeter may be found at: http://www.hp.com/products1/unix/java/hpjmeter/infolibrary/user_guide.pdf More information on HPjmeter may be found at: http://www.hp.com/products1/unix/java/hpjmeter/index.html
1.7.1 Static Data Analysis
1.7.1.1 Using HPjmeter to Analyze Profiling Data
The following steps summarize how to use HPjmeter to save and view profiling information from your applications.
1.Change the command line of your Java application to use
If you are using a Java release prior to JDK 1.5.0.04, you will need to add the
$ java
You can send the eprof output to a specified file using the file= keyword as follows: $ java
NOTE: If you are running JDK 1.5.0.04 or later, the
The SIGUSR2 signal toggles the recording of eprof data. Use the following process to gather eprof data for specific periods:
???Send SIGUSR2 to the Java VM process. The Java VM will begin recording eprof data.
???Send SIGUSR2 to the Java VM process. The Java VM will flush eprof data and close the log file.
See Profiling with Zero Preparation in the HPjmeter User's Guide for more information.
2.Run the application to create a data file.
3.Start the console from a local installation on your client workstation.
4.Click
5.A profile analysis window will open displaying a set of tabs containing summary and graphical metric data. The following screen shows an example:
26 Diagnostic and Monitoring Tools and Options
Figure
6.Click among the tabs to view available metrics. Use the Metrics or Estimate menus to select additional metrics to view. Each metric you select opens in a new tab. Mousing over each category in the cascading menu will reveal the relevant metrics for that category. The following screen shows the available metrics for the threads/locks category:
Figure
1.7.1.2 Using HPjmeter to Analyze Garbage Collection Data
The following steps summarize how to use HPjmeter to save and view garbage collection information from your applications:
1.Change the command line of your Java application to use
2.Run the application to create a data file.
3.Start the console from a local installation on your client workstation.
4.Click
5.A GC viewer window opens and displays a set of tabs containing metric data. Following is an example garbage collection analysis screen:
Figure
1.7.2 Dynamic Data Analysis
1.7.2.1 Using HPjmeter to Monitor Applications
The following steps show how to start the monitoring agent when launching the HPjmeter console. For most Java installations, linkage to the appropriate libraries is completed automatically as part of the installation process, and, therefore, the first step is not needed. Begin with the second step if you have a standard installation of the Java Runtime Environment.
1.Set the SHLIB_PATH environment variable to include the location of the HPjmeter agent library as appropriate for 32 or
Following are examples that show how to set this variable in both the csh and the ksh for the different libraries.
To select the
(csh) setenv SHLIB_PATH /opt/hpjmeter/lib/PA_RISC2.0 (ksh) export SHLIB_PATH=/opt/hpjmeter/lib/PA_RISC2.0
To select the
(csh) setenv SHLIB_PATH /opt/hpjmeter/lib/PA_RISC2.0W (ksh) export SHLIB_PATH=/opt/hpjmeter/lib/PA_RISC2.0W
To select the Integrity
28 Diagnostic and Monitoring Tools and Options
(csh) setenv SHLIB_PATH /opt/hpjmeter/lib/IA64N (ksh) export SHLIB_PATH=/opt/hpjmeter/lib/IA64N
To select the Integrity
(csh) setenv SHLIB_PATH /opt/hpjmeter/lib/IA64W (ksh) export SHLIB_PATH=/opt/hpjmeter/lib/IA64W
2.Confirm that the node agent is running. With a standard installation, the node agent should be running as a daemon on the system where it was installed. A node agent must be running before the console can connect to a managed node to discover applications and open monitoring sessions.
To verify that the node agent is running, use the following ps command:
% ps
The last output column (the args column) from ps should show the following:
$JMETER_HOME/bin/nodeagent
where JMETER_HOME=/opt/hpjmeter. The
If the node agent is not running, follow these steps to enable it:
a.Verify that you are logged in with root permissions.
b.Check that the following files exist:
???/sbin/init.d/HPjmeter_NodeAgent
???/sbin/rc3.d/S999HPjmeter_NodeAgent
c.Issue the following command to start the node agent daemon manually. Note: substitute start with stop to stop the node agent.
$ /sbin/init.d/HPjmeter_NodeAgent start
If you cannot use the node agent as a daemon or you need to set up access restrictions, start the node agent manually by issuing the following command (no root access needed):
$ /opt/hpjmeter/bin/nodeagent
By default, the node agent listens for console connections on port 9505. Use the
3.Start the Java application with the Java VM agent. For example, to start the myapp application on JDK 1.5 enter:
/opt/java1.5/bin/java
On SDK 1.4.2 versions enter:
/opt/java1.4/bin/java
This enables the myapp process to be dynamically monitored with the console.
4.Start the HPjmeter console by entering the command:
/opt/hpjmeter/bin/hpjmeter
1.7.2.2Connect to the Node Agent From the HPjmeter Console
1.Choose Connect from the File Menu or select the Connect to Server icon [
]. The following screen displays:
Figure
2.In the Connect to Server dialog box, type the host name where the Java application and corresponding node agent are running.
3.If the node agent was started on a nonstandard port, specify the port number in the Optional Port box.
4.Select Connect. The running Java VM for each application should appear in the console main window pane marked with the 
symbol.
NOTE: If there is a connection failure, the 
symbol will not be displayed. Instead the 
symbol will be displayed next to the server name to indicate the server connection failure. If this happens, verify the node agent is running on the specified server.
5.If you want to connect to several node agents, repeat the previous steps.
1.7.2.3Set Session Preferences
1.
30 Diagnostic and Monitoring Tools and Options
Figure
2.Check the default settings for metrics, filters, and alerts, and enable the settings you want to activate.
3.Click OK. The Session Preferences window will close and the newly Open Session will be visible, marked by the 
icon. Refer to the following screen for an example:
Figure
4.Wait for the console to collect metrics. The length of time depends on the application size, the load imposed on the application, and the selected preferences. Typically, the wait will be from 5 to 30 minutes. Longer collection time gives you greater accuracy in the results.
1.7HPjmeter 31
1.7.2.4Viewing Monitoring Metrics During Your Open Session
1.Click the open session or time slice to highlight the data to be viewed.
2.Use the Monitor menu on the console main window to select the desired metrics. Refer to the following screen for an example:
Figure
3.Select a metric. A metric visualizer displaying the chosen data will open. Refer to the HPjmeter User???s Guide for details on individual metrics and how to interpret the data.
1.7.2.5Running the HPjmeter Sample Programs
HPjmeter includes two sample applications you can run to see live examples of a memory leak and a thread deadlock situation. You can use the visualizers to examine data during the demonstration session.
Following are the general steps for running the sample applications:
1.Start the console.
2.Start the node agent if it is not running as a daemon.
3.Start the sample application from the command line:
$ cd $JMETER_HOME/demo
$ export LD_LIBRARY_PATH=$JMETER_HOME/lib
$ java
As a convenience, HPjmeter includes a script that sets up the library path and bootclasspath using the Java VM found at installation time. Following are instructions for using this script:
$ cd $JMETER_HOME/demo
$ ../bin/run_simple_jvmagent
Use the file name of the specific sample you want to run in place of sample_program.
4.In the console main window, select Connect and type in the host name of the machine running the sample application. If you specified a port number when starting the node agent, use the same port number. Otherwise, leave the port number box empty.
5.An icon representing the host appears in the main window. After a few moments, the console also shows the sample application as a child node of the host.
6.
7.Click OK to accept the default settings for metrics, filters, and alerts.
32 Diagnostic and Monitoring Tools and Options
1.7.2.5.1 Sample Memory Leak Application
This application demonstrates how memory leak alerts work in HPjmeter. It uses a simple program which allocates some objects. The program uses a java.util.Vector object to retain references to some of the objects. This application is configured to leak memory at the rate of about 10 MB per hour. It is available from the HPjmeter installation directory:
Source: $JMETER_HOME/demo/ML1.java
Binary: $JMETER_HOME/demo/ML1.jar
Use the class name ML1 with the run_simple_jvmagent script to start the sample. When measuring the sample application, allow considerable time for the heap to mature and stabilize, and for the Java VM agent to collect memory leak data. Eventually, you will see the following two alerts:
???Expected OutOfMemory Error Alert with the leaking rate
???Memory Leak Locations Alert with the leak location
When using the default garbage collectors and heap size for SDK 1.4.2, the detection of a memory leak for this demonstration program occurs after about 20 minutes. This time may be substantially longer when using a different Java VM or nonstandard garbage collector or heap settings. In real situations, the detection time depends on the maximum heap size, the size of the leak, the application running time, and the application and load characteristics. Typically, the detection will occur in about one hour.
Following is a memory leak alert for the sample program:
Figure
Following is the heap display:
Figure
1.7.2.5.2 Sample Thread Deadlock Application
This application demonstrates how HPjmeter detects deadlocked threads. It creates pairs of threads every 30 seconds, stopping at 50 threads, which synchronize work using shared locks. Occasionally, the program reverses the order on which locks are taken, eventually causing a deadlock, which generates a Thread Deadlock Alert.
The sample application is available from the HPjmeter installation directory:
Source: $JMETER_HOME/demo/DL1.java
Binary: $JMETER_HOME/demo/DL1.jar
Use the class name DL1 with the run_simple_jvmagent script to start the sample. Use the Thread Histogram display to view the thread activity. Deadlocked threads show a solid red bar.
Following is an example thread histogram display:
34 Diagnostic and Monitoring Tools and Options
Figure
1.8 HPjtune
NOTE: The HPjtune product has reached end of life. HP has integrated HPjtune functionality into HPjmeter 3.0 and recommends migrating to HPjmeter for the latest in bug fixes, enhancements, and support.
HPjtune is a garbage collection visualization tool for analyzing garbage collection activity in a Java program. Data files for HPjtune can be generated using
???Predefined graphs, which show the utilization of garbage collector resources and the impact of the garbage collector on application performance.
???
???Other predefined graphs, which show GC behavior pertaining to threads.
HPjtune also includes a unique feature which allows you to use the data collected with the
For more information about HPjtune and to download the tool, go to:
http://www.hp.com/products1/unix/java/java2/hpjtune/index.html
Following is an example of running Java with the
$ /opt/java1.5/bin/java
The
$ /opt/java1.5/bin/java
where <HPjtune_insdir> is the location of the HPjtune installation.
Following is an example screen shot to illustrate HPjtune's output:
Figure
1.9 hat
NOTE: Beginning with JDK 6.0, hat is replaced with jhat. For information on jhat, refer to the jhat section in this document.
The hat tool is a
Following in an example using hat. The first command generates a binary heap dump file. The second command invokes hat on the binary heap profile.
$ java
$ hat
The hat tool sets up an http server on the specified port. It can then be accessed by bringing up the default page in a web browser, for example, http://<hostname.domain>:7002. If you run hat on the same system as the browser, the server can be accessed by navigating to the URL http://<hostname.domain>:7002.
For more information on hat, refer to the following website:
For invocation details, refer to:
https://hat.dev.java.net/doc/README.html
1.10 hprof
hprof is a simple tool used for heap and CPU profiling. To start hprof, use one of the following Java command lines:
36 Diagnostic and Monitoring Tools and Options
$ java
$ java
hprof supports a number of profiling options. Use java
Following is an example hprof command to capture object data:
$ java
Load the resulting text file into HPjmeter, jhat, hat, or any editor for analysis.
Following is an example using hprof to produce a text file with summarized statistical samples taken every ten seconds during the execution of a Hello.java sample program:
$ java
For information about this tool on SDK 1.4 releases, refer to:
http://java.sun.com/j2se/1.4.2/docs/guide/jvmpi/jvmpi.html#hprof
For information about the updated version of this tool available on JDK 1.5+ releases refer to:
http://java.sun.com/developer/technicalArticles/Programming/HPROF.html
1.11 java.security.debug System Property
The java.security.debug system property controls whether the security checks in the JRE (Java Runtime Environment) print trace messages during execution. This option can be useful when trying to determine why a SecurityException is thrown by a security manager. This system property can be set to one of the following values:
???access
???jar
???policy
???scl
The access option has the following
???stack
???domain
???failure
For example, to print all checkPermission results and trace all domains in context, set java.security.debug to access,stack. To trace access failures, set it to access,failure.
Following is an example showing the output of a checkPermission failure:
$ java
access denied (java.net.SocketPermission server.foobar.com resolve
)
java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1158)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:253) at java.security.AccessController.checkPermission(AccessController.java:427)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at java.lang.SecurityManager.checkConnect(SecurityManager.java:1031) at java.net.InetAddress.getAllByName0(InetAddress.java:1117)
at java.net.InetAddress.getAllByName0(InetAddress.java:1098) at java.net.InetAddress.getAllByName(InetAddress.java:1061) at java.net.InetAddress.getByName(InetAddress.java:958)
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:124) at java.net.Socket.<init>(Socket.java:178)
at Test.main(Test.java:7)
1.12 JAVA_TOOL_OPTIONS Environment Variable
The command line used to start an application is not always readily accessible in many environments. This is especially true with applications that use embedded Java VMs or ones where the startup is deeply nested in scripts. In these environments, the JAVA_TOOL_OPTIONS
environment variable may be useful to add options to the command line when the application is run. This environment variable is primarily intended to support the initialization of tools, specifically the launching of native or Java agents using the
The JAVA_TOOL_OPTIONS environment variable is processed at the time of the invocation of the Java VM. When this environment variable is set, the JNI_CreateJavaVM() function prepends the value of the environment variable to the options supplied in its JavaVMInitArgs argument. For security reasons this option is disabled in setuid processes; that is, processes where the effective user or group ID differs from the real user or group ID.
In the following example, the environment variable is set to launch the hprof profiler when the application is started:
export
Although this environment variable is intended to support the initialization of tools, it is also useful for augmenting the command line with options for diagnostics purposes. For example, you could use it to add the
Since this environment variable is processed when JNI_CreateJavaVM() is called, it cannot be used to augment the Java launcher options. Some examples of these launcher options are the following VM selection options:
???java
???java
???java
To pass arguments to the Java launcher, set the JAVA_LAUNCHER_OPTIONS environment variable to a string containing the desired arguments.
This environment variable is fully described in the JVMTI specification at:
http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html#tooloptions
1.13 jconsole (1.5+ only)
The jconsole command launches a graphical console tool that enables you to monitor and manage Java applications on a local or remote machine.
jconsole can attach to any application that is started with the Java Management Extensions (JMX) agent. A system property defined on the command line enables the JMX agent. Once attached, jconsole can be used to display useful information such as thread usage, memory consumption, and details about class loading, runtime compilation, and the operating system.
In addition to monitoring, jconsole can be used to dynamically change several parameters in the running system. For example, the setting of the
To use jconsole:
1.Start the application with the
2.Start jconsole with the jconsole command.
3.When jconsole starts, it shows a window listing the managed Java VMs on the machine. The process id (pid) and command line arguments for each Java VM are displayed. Select one of the Java VMs, and jconsole attaches to it.
Following is an example invocation of jconsole. First the Java application must be started with the JMX agent enabled:
$ java
Now the jconsole tool can be started on the managed Java VM:
38 Diagnostic and Monitoring Tools and Options
$ /opt/java1.5/bin/jconsole 13028
The following figure shows a jconsole screen shot:
Figure
jconsole can also be run remotely. To learn more about jconsole, including remote invocation, refer to the following website:
http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
1.14 jdb
The SDK includes a
http://java.sun.com/j2se/1.5.0/docs/tooldocs/solaris/jdb.html
A jdb tutorial may be found at:
1.15 jhat
Beginning with JDK 6.0, jhat is included with the standard JDK distribution. This tool can be used for heap analysis; it is an improved version of hat. It starts a web server on a
Following in an example using jhat. The first command generates a binary heap dump file. The second command invokes jhat on the binary heap profile.
$ java
$ jhat
The jhat tool sets up an http server on the specified port. It can then be accessed by bringing up the default page in a web browser, for example, http://<hostname.domain>:7002. If you run hat on the same system as the browser, the server can be accessed by navigating to the URL http://<hostname.domain>:7002.
For more information on jhat, refer to the following website:
http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html
1.16 jps (1.5+ only)
The jps tool lists the Java VMs on the target system. The tool is limited to reporting information on Java VMs that the user has access rights to, as determined by
Following is the usage information for the jps command:
Usage: jps
jps
Description of options:
(the .hotspotrc file or the file specified by
Note: These options are subject to change or removal in the future.
Following is an example using jps:
$ /opt/java1.5/bin/jps
16641 spec.jbb.JBBmain
For more information about jps, refer to the following document:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html
1.17 jstat (1.5+ only)
The jstat utility is a statistics monitoring tool. It attaches to a Java VM and collects and logs performance statistics as specified by the
The jstat utility does not require the Java VM to be started with any special options. This utility is included in the JDK download.
The following table lists the jstat command options:
Table
40 Diagnostic and Monitoring Tools and Options
Table
A complete description of the jstat tool, including examples, can be found at:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
Following is an example jstat command which attaches to pid 27395 and takes five samples at 250 millisecond intervals. The
Following is a description of the column headings in the example:
Table
1.18 jstatd (1.5+ only)
The jstatd tool launches an RMI (remote method invocation) server that monitors the creation and termination of Java VMs and provides an interface to allow remote monitoring tools to attach to Java VMs running on the local host.
For more information, refer to the following website:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstatd.html
1.19 jvmstat Tools
The Java VM shipped with SDK 1.4.2 and later provides
As of JDK 1.5, the following subset of jvmstat tools is included with the JDK: jps (formerly jvmps) , jstat (formerly jvmstat), and jstatd (formerly perfagent). The visualgc tool is not included with JDK 1.5+, but is instead provided in the unbundled jvmstat 3.0 distribution.
For more details, refer to the following website:
http://java.sun.com/performance/jvmstat
1.20
The
1.21
The
These options along with detailed information about GC analysis and tuning, are described at Sun's GC portal site:
http://java.sun.com/developer/technicalArticles/Programming/GCPortal
The
For other GC logging options, see
1.22
The
1.23 visualgc
The visualgc tool uses jvmstat technology to provide visualization of garbage collection activity in the Java VM. The Java VM shipped with JDK 1.4.2 and later releases provides the
As of JDK 1.5+, the following subset of the jvmstat tools is included with the Java VM: jps
(formerly jvmps), jstat (formerly jvmstat), and jstatd (formerly perfagent). visualgc is not included in this set, but is instead provided in the unbundled jvmstat 3.0 distribution. The download for jvmstat 3.0 may be found at:
http://java.sun.com/performance/jvmstat
visualgc attaches to a running Java VM processs to collect and graphically display garbage collection, class loader, and Java compiler performance data.
The target Java VM is identified by its virtual machine identifier, or vmid. On
For details on visualgc usage refer to:
http://java.sun.com/performance/jvmstat/visualgc.html
When visualgc is attached to a running Java VM it opens the following windows:
42 Diagnostic and Monitoring Tools and Options
1.Application Information window
2.Graph window
3.Survivor Age Histogram window (optional)
The Survivor Age Histogram window is only available when Parallel Scavenge is in use
Following is an example visualgc Application Information window and a description of the different window areas:
Figure
The top panel of this window is labelled Application Information . This panel has an Alive/Dead indicator and the elapsed time since the start of the Java application. Following this panel there is a scrollable text area that lists miscellaneous information about the configuration of the target Java application and the Java VM. This section includes main class or jar file name, the arguments to the class's main method, arguments passed to the Java VM, and the values of certain Java properties exported as instrumentation objects.
The bottom panel shows a graphical view of the spaces that make up the generational garbage collection system. This panel is divided into three vertical sections, one for each of the generations: the Perm generation, the Old (or Tenured) generation, and the Young generation. The Young generation is comprised of three separate spaces, the Eden space, and two Survivor spaces, S0 and S1.
The screen areas representing the various spaces are sized in proportion to the maximum capacities of the spaces. The screen areas for the three GC generations are of fixed size and do not vary over time. Each space is filled with a unique color indicating the current utilization of the space relative to its maximum capacity. The unique color for each space is used consistently among this window and the other two visualgc windows (Graph and Survivor Age Histogram).
The Graph window displays the values of various statistics as a function of time. The resolution of the horizontal axis of the graph is determined by the interval
where each sample occupies two pixels of screen area. The height of each display depends on the metric being plotted. Following is an example Graph window:
Figure
Each of the GC space graphs can be displayed in one of two modes: reserved mode or committed mode; committed mode is the default. In reserved mode, the data is scaled according to the maximum capacity of the space. The background grid is painted in dark gray to represent the uncommitted portion and in green to represent the committed portion of reserved memory. In committed mode, the data is scaled according to the current capacity of the space. The mode can be toggled by
The Survivor Age Histogram window consist of two panels, the Parameters panel and the Histogram panel. The Parameters panel displays the size of the survivor spaces and the parameters that control the promotion behavior of the young generation. The Histogram panel displays a snapshot of the age distribution of objects in the active survivor space after the last Young generation collection. The display is comprised of 32 identically sized regions, one for each possible object age. Each region represents 100% of the active Survivor Space and is filled with a colored area that indicates the percentage of the survivor space occupied by objects of the given age.
Following is an example Survivor Age Histogram window:
44 Diagnostic and Monitoring Tools and Options
Figure
When the Java VM is started with the Parallel Young GC option
1.24
The
java
The
When an invalid argument is detected, the Java VM prints a message to the application console (standard output), prints the stack trace of the offending thread, and aborts the Java VM. Following is an example where a NULL is incorrectly passed to a JNI function that does not allow NULL:
FATAL ERROR in native method: Null object passed to JNI at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343) - locked <0x450b9f70> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:439) at java.net.ServerSocket.accept(ServerSocket.java:410)
at org.apache.tomcat.service.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:286) at org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:402)
at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:498) at java.lang.Thread.run(Thread.java:536)
Following is another example of output that is displayed when something other than a jfieldID is provided to a JNI function that expects a jfieldID:
FATAL ERROR in native method: Instance field not found in JNI get/set field operations at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) - locked <0xf082f290> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.bind(ServerSocket.java:318) at java.net.ServerSocket.<init>(ServerSocket.java:185) at jvm003a.<init>(jvm003.java:190)
at jvm003a.<init>(jvm003.java:151) at jvm003.run(jvm003.java:51)
at jvm003.main(jvm003.java:30)
Following are some types of problems that
???The JNI environment for the wrong thread is used
???An invalid JNI reference is used
???A reference to a
???A
???A JNI call is made with an exception pending
1.24
In general, all errors detected by
Warning: Calling other JNI functions in the scope of
Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
A JNI critical region arises when native code uses the JNI GetPrimitiveArrayCritical() or GetStringCritical() functions to obtain a reference to an array or string in the Java heap. The reference is held until the native code calls the corresponding release function. The time between the get and release is called a JNI critical section, and during that time the Java VM cannot reach a state that allows garbage collection to occur. The general recommendation is that other JNI functions should not be used when in a JNI critical section, and in particular any JNI function that blocks could potentially cause a deadlock. The warning printed by
1.25
The
The ???:help??? option prints a description of the verbosegc output format.
The ???0 | 1??? option controls the printing of help information. Specifying value ???0??? will cause the heap information to be printed after every Old Generation GC or Full GC. Specifying value ???1??? (the default) will cause the heap information to be printed after every GC.
The ???file = [stdout | stderr | <filename>]??? option specifies the output file. The default is stderr, which directs the output to the standard error stream. Alternative choices for the output file are stdout and a
java
At every garbage collection 20 fields are printed as follows:
GC: %1 %2 %3 %4 %5 %6 %7 %8 %9 %10 %11 %12 %13 %14 %15 %16 %17 %18 %19 %20
The following table contains brief descriptions of these 20 fields:
Table
1Type of GC:
???1: Scavenge (GC of New Generation only)
???2: Old Generation GC or a Full GC
???3: Complete background CMS GC
???4: Incomplete background CMS GC
???11: Ongoing CMS GC
2Additional information based on GC type in field 1.
3Program time at the beginning of the GC, in seconds.
4GC invocation. Counts of background CMS GCs and other GCs are maintained separately.
5Size of the object allocation request that forced the GC, in bytes.
6Tenuring
7Eden
46 Diagnostic and Monitoring Tools and Options
Table
8Eden
9Eden
10Survivor
11Survivor
12Survivor
13Old Generation occupied before GC.
14Old Generation occupied after GC.
15Old Generation current capacity.
16Permanent Generation (storage of Reflective Objects) occupied before GC.
17Permanent Generation (storage of Reflective Objects) occupied after GC.
18Permanent Generation (storage of Reflective Objects) current capacity.
19The total
For more details about these fields, use the :help option or refer to the Java Programmers Guide at the following website:
http://www.hp.com/products1/unix/java/infolibrary/prog_guide/index.html
To better understand how garbage collection works in the Java VM, read the article "Improving Java Application Performance and Scalability by Reducing Garbage Collection Times and Sizing Memory Using JDK 1.4.1" (November 2002) by Nagendra Nagarajayya and J. Steven Mayer at the following website:
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/#17.1
Additionally, HP recommends using the HPjtune tool, which can display graphically the information contained in a
1.26
The JDK 6.0 release contains a new option,
If this option is used with the JAVA_CORE_DESTINATION environment variable, errfilename can specify an absolute path, a relative path, or a filename placed in the JAVA_CORE_DESTINATION directory. The following list explains how the combination of the errfilename with the JAVA_CORE_DESTINATION environment variable can be used to do this:
1.If the errfilename begins with the file separator character (???/???), it specifies an absolute path. The JAVA_CORE_DESTINATION environment variable is not used for the errfilename.
2.If the errfilename contains the file separator character (???/???), but does not begin with one, it specifies a relative path ($JAVA_CORE_DESTINATION/errfilename).
3.If no file separator is found in errfilename, the fatal error log is placed in the
JAVA_CORE_DESTINATION directory.
1.26
1.27
The
The output is similar to that produced by the
With the
If _JAVA_HEAPDUMP_ONLY is set, then heap dumps are triggered by SIGVTALRM instead of SIGQUIT for this option. Only the heap dump is produced; that is, the thread and trace dump of the application to stdout is suppressed. Setting the _JAVA_BINARY_HEAPDUMP environment variable along with _JAVA_HEAPDUMP_ONLY produces a binary format heap dump when the SIGVTALRM is sent to the process instead of an ASCII one.
NOTE: A full GC is executed prior to taking the heap snapshot.
1.27.1 Other HeapDump Options
In addition to
Table
1.27.2
The
If the HP environment variable _JAVA_HEAPDUMP is set and this option is specified, then both hprof ASCII and binary dump files are created when a SIGQUIT is sent to the process. For
48 Diagnostic and Monitoring Tools and Options
example, the following file names are created: java_27298.hprof.1152743593943 and java_27298_060712_153313_heapDump.hprof.txt.
If JAVA_BINARY_HEAPDUMP is set and the
1.27.3
One known issue exists:
1.27.4
Starting with SDK 1.4.2.11 and JDK 1.5.0.05, the
The heap dump is written to a file with the following filename format: java_<pid>_<date>_<time>_heapDump.hprof.txt.
The default output format is ASCII. The output format can be changed to hprof binary format by setting the _JAVA_BINARY_HEAPDUMP environment variable. This environment variable can also be used with the
1.27.5 Using Heap Dumps to Monitor Memory Usage
By creating a series of heap dump snapshots, you can see how the number and size of objects varies over time. It is a good idea to collect at least three snapshots. The first one serves as a baseline. It should be taken after the application has finished initializing and has been running for a short time. The second snapshot should be taken after the residual heap size has grown significantly. Monitor this using
Once you have collected the snapshots, read them into HPjmeter (run with
1.28
When a fatal error occurs, the Java VM can optionally execute a
1.28
where <string> is a single command or a list of commands each separated by a semicolon. Within <string> all occurrences of ???%p??? are replaced with the current process id (pid), and all occurrences of ???%%??? are replaced by a single ???%???.
Following is an example showing how the fatal error report can be mailed to a support alias when a fatal error is encountered:
java
Following is an example that launches gdb when an unexpected error is encountered. Once launched, gdb attaches to the Java VM process:
java
1.29
In addition to
==============================================================================
Unexpected Error
SIGSEGV (0xb) at pc=0x2000000001164db1, pid=10791, tid=1026
Do you want to debug the problem?
To debug, run 'gdb /proc/10791/exe 10791'; then switch to thread 1026 Enter 'yes' to launch gdb automatically (PATH must include gdb) Otherwise, press RETURN to abort...
==============================================================================
In this case, a SIGSEGV has occurred and the user is prompted whether to launch the debugger to attach to the process. If the user enters ???y??? or ???yes??? then gdb is launched.
In the previous example, the output includes the process id (10791) and also the thread id (1026). If the debugger is launched then one of the initial steps taken in the debugger should be to select the thread and obtain its stack trace.
While waiting for a response from the process, it is possible to use other tools to obtain a crash dump or query the state of the process.
Generally,
50 Diagnostic and Monitoring Tools and Options
2 Useful System Tools for Java Troubleshooting
This chapter contains information about some system tools available on
2.1 GlancePlus
GlancePlus is a system performance monitoring and diagnostic tool. It lets you easily examine system activities, identify and resolve performance bottlenecks, and tune your system for more efficient operation. For more information on GlancePlus, refer to the following website:
http://www.managementsoftware.hp.com/products/gplus/index.html
2.2 tusc
tusc gives you another view into the system activity, in addition to Java stack traces, GlancePlus, and HPjmeter . It has many options, which you can display by entering the command tusc
2.3 Prospect
Prospect is a performance analysis tool. Beginning with Prospect revision 2.2.0, you can use Prospect to retrieve a profile of the compiled Java methods that the Java VM compiler creates in data space. In order to activate this functionality, you must have SDK 1.3.1.02 or following releases. For more information on the Prospect performance analysis tool, refer to the following website:
http://h21007.www2.hp.com/dspp/tech/tech_TechSoftwareDetailPage_IDX/1,1703,3282,00.html
2.4 HP Caliper
HP Caliper is a
2.5 sar
The sar command is a tool to report various system activities, such as CPU, I/O, context switches, interrupts, page faults, and other kernel actions. For more information on this command, refer to the following website:
2.6 vmstat
The vmstat command reports statistics about the process, virtual memory, trap, and CPU activity. For more information on this command, refer to the following website:
2.7 iostat
The iostat command iteratively reports I/O statistics for each active disk on the system. For more information on this command, refer to the following website:
2.8 swapinfo
The swapinfo command displays information about device and file system paging space. For more information on this command, refer to the following website:
2.9 top
The top command displays the top processes on the system, periodically updating the information; raw CPU percentage is used to rank the processes. For more information on this command, refer to the following website:
2.10 netstat
The netstat command displays statistics for network interfaces and protocols as well as the contents of various
2.11 Other Tools
The Developer and Solution Partner Program's (DSPP) technical information website contains links to debugging information. There are links from this page to other websites containing technical papers, tips, tutorials, and more. To review this information, refer to the following website:
52 Useful System Tools for Java Troubleshooting
3 Getting Help from
Sometimes you need help troubleshooting your Java application problems. Before opening a support call, search for information that may help you by referring to the Go Java! website:
This site contains much information about Java, including known issues, release notes, patches, downloads, documentation, and more. If you still need troubleshooting help after looking at this website and you have a support contract with
3.1 Problem Report Checklist
Use this checklist to collect information before you request support. Providing more information when you initiate your support call reduces the time it takes for support engineers to start working on your problem.
NOTE: More details about collecting problem data, system data, and Java environment data may be found in the sections following this checklist.
1.Problem Description
a.Did this Java application ever work?
b.What is the problem (abort, hang, performance, and so on)?
c.What messages are written to stdout or stderr relating to the problem?
d.Does the problem occur every time the application is run or intermittently?
e.What are the application details? Include the following:
???Name of the application.
???What the application does.
???The command line and options used to start the application.
???Description of the expected behavior.
???Description of the actual behavior.
???The application stack that you are
f.How do you reproduce the problem? If possible, provide source code and
g.Do you have a workaround for the problem? If so, describe it.
2.Problem Data (refer to Section 3.2 for details)
a.Core file, best collected with gdb's packcore command
b.Fatal error log (hs_err_pid<pid>.log)
c.Stack trace
3.System Information (refer to Section 3.3 for details)
a.What version of
b.What patches are installed on the system? This can be determined with HPjconfig or swlist.
c.What window manager is being used? For example, Reflections X or X Windows. Or is the application running inside a browser? If so, which one?
4.Java Environment (refer to Section 3.4 for details)
a.What is the version of the Java VM that is having the problem? Run the command java
b.What are the values of the environment variables used by Java?
c.What libraries are being loaded? This information is best collected with gdb's packcore command.
5.Contact Information
a.Contact name
b.Company name
c.Phone number
d.
The following subsections provide instructions for collecting the necessary problem, system, and Java environment information. The final subsection contains instructions for packaging the files you need to send to
3.2 Collecting Problem Data
Three pieces of information are essential for analyzing most
3.2.1 Collecting Core File Information
This section begins with a checklist to follow to make sure you can collect useful core files. It then reviews how you can generate a core file if one is not generated for you. Finally, there is a discussion about how to verify that your core file is valid.
3.2.1.1 Core File Checklist
Core files contain useful information, if they are complete. Sometimes you need to configure your system to make sure you can save complete core files. Consider the following items to ensure you can create complete core files.
1.Estimate the core file size.
2.Ensure your process can write large core files.
3.Verify you have enough free disk space.
4.Make sure the directory where the core file will reside supports a large file system. If not, write the core file to a directory that does.
5.Make sure you have the correct permissions to write core files.
Following are additional details on each of these steps.
3.2.1.1.1 Estimate Core File Size
The size of
???
???
???
3.2.1.1.2 Ensure Process Can Write Large Core Files
Check your coredump block size to make sure it is set to unlimited using the ulimit
$ ulimit
time(seconds) unlimited file(blocks) unlimited data(kbytes) 4292870144 stack(kbytes) 8192
54 Getting Help from
memory(kbytes) unlimited coredump(blocks) 4194303
If coredump is not set to unlimited, set it to unlimited using the ulimit
$ ulimit
3.2.1.1.3 Verify Amount of Disk Space
Check the amount of disk space available in the current working directory using the df
3.2.1.1.4 Check If Directory Supports Large File Systems
Use the fsadm command as root to check if your directory supports large file systems. If you do not execute this command as root, you may not retrieve meaningful results. Following is an example:
<root>$ /usr/sbin/fsadm <mount_point>
Following is example output when the file system is set up to support large files and when it is not set up to support large files:
<root>$ /usr/sbin/fsadm /extra
fsadm: /etc/default/fs is used for determining the file system type largefiles
<root>$ /usr/sbin/fsadm /stand
fsadm: /etc/default/fs is used for determining the file system type nolargefiles
You can use the /usr/sbin/fsadm command to set the directory to support large files. For example, to convert a hfs file system from nolargefiles to largefiles, issue the following command:
$ fsadm
Alternatively, if the directory does not support large file systems, you can write the core file to a different directory. Do this by setting the JAVA_CORE_DESTINATION environment variable (available starting with SDK 1.4.2) to the name of the directory and create the directory. For example:
$ export JAVA_CORE_DESTINATION=<alt_dir> $ mkdir $JAVA_CORE_DESTINATION
Java creates a directory named core under the JAVA_CORE_DESTINATION directory where the core and hs_err_pid<pid>.log files are written. For example:
$ cd $JAVA_CORE_DESTINATION $ ls
core.29757
3.2.1.1.5 Ensure Permissions Allow Core Files
Some Java processes run setuid; that is, a process where the effective uid or gid differs from the real uid or gid. On
$ echo "dump_all/W 1" | adb
This capability is turned on only for the current boot.
3.2.1.2 Generating a Core File
Analyzing the core file is essential for troubleshooting problems. Core files are automatically generated for application aborts. For hung processes and performance issues, you need to generate them using gdb's dumpcore command.
The gdb dumpcore command forces the generation of a core file without killing a running process. This command causes a core file named core.<pid> to be created. The current process state is not modified when this command is issued.
Following is an example for a Java application running on an Integrity system:
This generates a core file in the current directory with the name core.12290.
On
$ gcore 11050
3.2.1.3 Verifying a Core File
Once you have successfully collected your core file, you should verify that it is complete and valid with the following two steps.
First, open the core file in gdb and check the error and warning messages. If the message ???<corefilename> is not a core dump: File format not recognized??? is displayed when you open the file, your core file is invalid. Following is an example of verifying a core file produced by a
$ gdb /opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl core
HP gdb 5.5.7 for
..
Core was generated by `java'.
Program terminated with signal 6, Aborted.
#0 0xc0214db0 in kill+0x10 () from ./libc.2
Second, check to make sure that the core file was not trucated by issuing the ???what core??? command. If you do not see the dld.sl version at the bottom of the what output, then the core file is truncated and is not usable. In the following example, the dld.sl version exists at the bottom of the what output, so you know the core file is not truncated:
$ what core core:
some other library names and version information ...
3.2.2 Collecting Fatal Error Log Information
When a Java application aborts, the fatal error log file (hs_err_pid<pid>.log) is generated. The contents of this file vary depending on the architecture and the Java version (for example,
56 Getting Help from
early Java versions generate less information in the fatal error log). Following is a summary of the type of information contained in this file:
1.The error causing the Java VM to abort, including the pc, process id, and thread id at which the error occurred. For example:
#An unexpected error has been detected by HotSpot Virtual Machine:
#SIGSEGV (11) at pc=7541df20, pid=25675, tid=1
2.The Java version and problematic frame. For example:
#Java VM: Java HotSpot(TM) Server VM (1.4.2
#Problematic frame:
#j spin.main([Ljava/lang/String;)V+5
3.Information about the current thread, including:
a.the executing thread
b.siginfo at the point of failure
c.stack pointer and hex dump of the top of memory stack
d.hex dump at the location of the current pc
e.stack range and stack free space
4.Process information, including:
a.a dump of all active threads at the time of the abort (SDK 1.4.2.04+)
b.Java VM state (whether at safepoint or not) (SDK 1.4.2.10+)
c.mutex state (SDK 1.4.2.10+)
d.a summary of heap status; for example:
e.dynamic libraries loaded by the process (SDK 1.4.2.04+)
f.Java VM arguments (SDK 1.4.2.04+)
g.
5.System Information. This includes operating system name, version, CPU, memory, and system load. For example:
OS: HPUX
rlimit: STACK 98252k, CORE 2097151k, NOFILE 4096, AS infinity load average:0.12 0.19 0.22
Memory: 4k page, physical 16743644k
vm_info: Java HotSpot(TM) Server VM
3.2.3 Collecting Stack Trace Information
On
3.3 Collecting System Information
Along with
Following is an example using the swlist command to retrieve this list:
$ /usr/sbin/swlist
#Initializing...
#Contacting target "mutant"...
#Target: mutant:/
#
#
# Bundle(s):
#
Following is an example using HPjconfig to collect this information:
$ java
List of required patches:
More information about HPjconfig may be found in the HPjconfig section of this manual.
3.4 Collecting Java Environment Information
In order to perform core file analysis, you need to collect information about some environment variables and libraries used by the failed application. The following subsections describe how to do this.
3.4.1 Environment Variables
To facilitate troubleshooting, it is important to know the value of the environment variables that can affect the behavior of Java applications (for example, CLASSPATH). To collect these application
58 Getting Help from
runtime environment variable values, run the following command under the same environment (that is, the same user) that the Java application was executed:
(ksh)$ env > app_environment.txt (csh)$ getenv > app_environment.txt
Include the app_environment.txt file when you send in your collected data files to
3.4.2 Libraries
In order to perform core file analysis, you must have access to libraries used by the failed application. The method used for determining which libraries were used depends on whether or not gdb is available on the system.
If gdb is not available, then locate files by either examining the stdout of the failed application or the hs_err_pid<pid>.log file. Either of these should list all the libraries used. Using this list, manually copy the files.
If gdb is available on the system where the failure occurred, issue gdb's packcore command:
(gdb) packcore
This command creates a compressed tar file called packcore.tar.Z under the current directory. packcore.tar.Z contains the following:
???
???
???
In some situations, only a core file can be obtained. In this case limited troubleshooting can take place since some crucial pieces of information are missing
There is one additional library that should be collected: libjunwind. This library is used by gdb to unwind Java bytecode frames; its routines help make stack traces more readable and understandable. Since this library is only used during debugging, it is not included in the tar file generated by packcore.
The following table shows the location of the libjunwind library for
Table
On Integrity systems, beginning with SDK 1.4.0.10 and JDK 1.5.0.03, there are two libjunwind libraries for each Java VM, libjunwind64.so and libunwind.so. The following table shows the location of these libraries for both
Table
3.5 Packaging Files
The packcore command produced the packcore.tar.Z archive, which contains the core file, core, and the modules.tar file. You now need to package packcore.tar.Z with the other files needed for troubleshooting. One method for packaging is to use the Java archive tool, jar. This tool is included with all Java installations.
For example, to collect all files needed for core file analysis into file debug.jar, including packcore.tar.Z, hs_err_pid7145.log, app_environment.txt, and libjunwind.sl, issue the following command:
jar cvf debug.jar packcore.tar.Z hp_err_pid7145.log \ app_environment.txt libjunwind.sl
60 Getting Help from
4 Core File Analysis
The previous chapter described how to collect necessary information before opening a call to HP Support to get help troubleshooting Java applications. Sometimes it is possible to at least attempt the core file analysis on your own. This chapter walks through an example core file analysis step by step. By studying this example, you will learn some skills needed to analyze your own core file.
???Memory violations
???Illegal instructions
???
???Bus errors
???
???
Generally the core image file is called core and is written in the current working directory.
A core file is a dump of the process state at the time of the problem. The file contains sufficient information to determine what the process was doing when it failed. This information includes:
???Threads
???Register values
???Contents of attached data memory regions
???Kernel version
???Command name
4.1Sample Java Application
The sample application contains native code (C) and Java code. This particular application aborts in native code since this code contains a defect. The defect causes a runtime failure, which results in a core dump.
The example consists of three files for an application calledStackTrace:
???StackTraceJob ??? the script to create the core file
???StackTrace.java ??? the Java source code
???stacktrace.c ??? the C source code
These three files should be placed in a directory called StackTrace.
The following subsections contain listings of each of these three files. This example is run on a
4.1.1 StackTraceJob
#!/bin/ksh
# set JAVA_HOME
export JAVA_HOME=/opt/java1.4
# set PATH
export PATH=$JAVA_HOME/bin:$PATH
#Compile java bytecode /usr/bin/echo "Compile java code" javac StackTrace.java
#create header file
/usr/bin/echo "Create header file" javah
# compile jni code /usr/bin/echo "Compile c code"
/usr/bin/cc +z
# create shared library /usr/bin/echo "Create shared library"
/usr/bin/ld
/usr/bin/echo "Run StackTrace program" export SHLIB_PATH=.
export LD_LIBRARY_PATH=.
java StackTrace
NOTE: If this script is run on an Integrity system, change it from:
# create shared library
/usr/bin/echo "Create shared library"
/usr/bin/ld
to:
# create shared library
/usr/bin/echo "Create shared library"
/usr/bin/ld
4.1.2StackTrace.java
//File StackTrace.java
public class StackTrace {
native static String dumpCore(int i);
//************************************************** public static void method1(int ci) {
System.out.println("Calling method2()"); StackTrace.method2(ci);
}// end method1
//************************************************** public static void method2(int ci) {
System.out.println("Calling method3()"); StackTrace.method3(ci);
}// end method2
//************************************************** public static void method3(int ci) {
System.out.println("Calling methodMakeCall()"); StackTrace.methodMakeCall(ci);
}// end method3
//************************************************** public static void methodMakeCall(int ci) {
try { System.loadLibrary("stacktrace");
}
catch (UnsatisfiedLinkError Err) { System.out.println("error: " + Err); System.exit(1);
}
62 Core File Analysis
System.out.println("Call dumpCore to convert " + ci + " to a binary string!");
System.out.println("The binary String: " + StackTrace.dumpCore(ci));
}// end methodMakeCall
//************************************************** public static void main(String args[]) {
int convertInt;
System.out.println(); if(args.length == 1) {
convertInt = Integer.parseInt(args[0]);
}
else {
convertInt = 757;
}
System.out.println("Calling method1()"); StackTrace.method1(convertInt); System.out.println("Back in main, all done!");
}// end main
}// end StackTrace
4.1.3stacktrace.c
// File stacktrace.c
#include "StackTrace.h" #include <stdio.h>
JNIEXPORT jstring JNICALL
Java_StackTrace_dumpCore(JNIEnv *env, jclass class, jint intarg) {
jclass classid; jmethodID methodid;
printf("In dumpCore\n");
//Problem code. The Class: java.lang.IntegerX does not exist,
//but the exception is cleared and the program continues.
//Ultimately, failing and dumping core when getting the methodid.
//To fix, comment out the following 2 lines.
///*
classid =
//*/
//Working code. The Class: java.lang.Integer exists. The lines
//following the FindClass manage printing out of a stack trace,
//clearing an exception, and returning to the java main when
//FindClass fails.
//
// Uncomment the following 6 lines and rebuild to see the program work! /*
classid =
if(classid == NULL) {
return
}
*/
methodid =
"(I)Ljava/lang/String;");
if(methodid == NULL) {
return
}
return
}
4.2 Building the Application
The StackTraceJob script can be used on
This script can be used on an Integrity system if you make the changes described in the Note in the ???StackTraceJob??? section; that is, rename libstacktrace.sl to libstacktrace.so.
NOTE: There are instructions in stacktrace.c describing how to eliminate the errors so you can see how the corrected application will run.
Execute theStackTraceJob script. You will see output similar to the following:
$ StackTraceJob Compile java code Create header file
[Search path = /opt/java1.4/jre/lib/rt.jar:/opt/java1.4/jre/lib/i18n.jar: /opt/java1.4/jre/lib/sunrsasign.jar:/opt/java1.4/jre/lib/jsse.jar: /opt/java1.4/jre/lib/jce.jar:/opt/java1.4/jre/lib/charsets.jar: /opt/java1.4/jre/classes:.]
[Loaded ./StackTrace.class]
[Loaded /opt/java1.4/jre/lib/rt.jar(java/lang/Object.class)] [Creating file StackTrace.h]
Compile c code
Create shared library Run StackTrace program
Calling method1()
Calling method2()
Calling method3()
Calling methodMakeCall()
Call dumpCore to convert 757 to a binary string!
#An unexpected error has been detected by HotSpot Virtual Machine:
#SIGSEGV (11) at pc=c3ed2ec8, pid=28973, tid=1 # # Java VM: Java HotSpot(TM) Server VM
#(1.4.2
#Problematic frame:
#V [libjvm.sl+0x6d2ec8]
#
#An error report file with more information is saved as hs_err_pid28973.log
#Please report this error to HP customer support.
#
The following files are created as a result of running this script:
???
???
???
???
64 Core File Analysis
???
???
4.3Verify Core File
Before you proceed further, verify that the core file, core, is complete and valid. You can do this in two steps.
First, open the file in gdb and check the error and warning messages.
$ gdb /opt/java1.4/bin/PA_RISC2.0/java core
HP gdb 5.5.7 for
Copyright 1986 - 2001 Free Software Foundation, Inc.
GNU General Public License. Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by 'java'.
Program terminated with signal 6, Aborted.
#0 0xc0214db0 in kill+0x10() from ./libc.2
From the output above, you can tell that core file is valid.
Second, check that the core file was not truncated by issuing the ???what core??? command and searching for the existence of the dld.sl version at the bottom of the output:
$ what core core:
If the dld version is displayed at the bottom of the output, the core file is valid.
4.4 Debugging On Same System
If you are debugging the core file on the same system where it was created, you do not need to perform the steps outlined in the ???Packaging Files For Debugging On Different System??? and ???Unpacking Files On Debugging System??? sections. Instead, proceed to the ???Example gdb Session??? section.
4.5 Packaging Files For Debugging On Different System
If you will debug the core file on a different system than the one where it was created, you need to collect problem data essential for analyzing the core file.
Collect the following items:
???All source files (.java, .h, .c).
???All .class files.
???The command line used to run the application program. In this example, the StackTraceJob script.
Now, use gdb to pack the core file.
$ gdb /opt/java1.4/bin/PA_RISC2.0/java core
HP gdb 5.5.7 for
..
warning: core file may not match specified executable file. Core was generated by `java'.
Program terminated with signal 6, Aborted.
#0 0xc0214db0 in kill+0x10
warning: No debug info available - Trying to print as integer () from /usr/lib/libc.2
(gdb) packcore
The core file has been added to packcore.tar and the core file has been removed. (gdb) quit
The packcore command creates a compressed tar file named packcore.tar.Z. This file contains the core file and a tar file that has all the libraries used in the application. Refer to Section 3.4.2 for more details about the contents of the packcore.tar.Z file.
Now bundle all the files needed for debugging using jar. This also includes application files, if any. Before bundling the files, copy the correct libjunwind library (see Section 3.4.2) to the directory since this library needs to be included in the bundle.
$ cp /opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl libjunwind.sl
$ jar cvf bundleit.jar packcore.tar.Z hs_err_pid28973.log \ libjunwind.sl
added manifest
adding: packcore.tar.Z(in = 12323556) (out= 11532742)(deflated 6%)
adding: hs_err_pid28973.log(in = 5328) (out= 2195)(deflated 58%)
adding: libjunwind.sl(in = 217088) (out= 58863)(deflated 72%)
The following section shows the contents of the bundleit.jar file.
You can now delete packcore.tar.Z, hs_err_pid28973.log, and libjunwind.sl since all these files are included in the bundleit.jar file.
4.6 Unpacking Files On Debugging System
Skip this section if you are debugging the core file on the same system where it was created.
Move the bundleit.jar file to the system where you will be analyzing the core file, and use jar to extract the files.
66 Core File Analysis
Uncompress and extract the files from the packcore.tar.Z file:
$ uncompress packcore.tar.Z $ tar
Delete the packcore.tar file since you have extracted the files. Next, move the modules.tar and core files back to the current directory so all the files needed for debugging are together. Finally, unpack the modules.tar file.
$ rm packcore.tar
$ mv packcore/core .
$ mv packcore/modules.tar . $ tar
dld.sl
libjvm.sl
libpthread.1
libm.2
librt.2
libcl.2
libisamstub.1
libCsup.2
libc.2
libdld.2
libogltls.sl
libhpi.sl
libverify.sl
libjava.sl
libzip.sl
libstacktrace.sl
Remove the modules.tar file since you no longer need it.
$ rm modules.tar
Following is a listing of the files in the directory:
4.7 Example gdb Session
Before beginning core file analysis, examine the fatal error log file, hs_err_pid<pid>.log. This file contains useful information that will help you troubleshoot the problem. For more information about the contents of the hs_err_pid<pid>.log file, refer to Section 3.2.2., Collecting Fatal Error Log Information .
You are now ready to examine the core file.
This document assumes the reader understands
???
???Precision Architecture Runtime Architecture: http://devresource.hp.com/drc/STK/docs/archive/rad_11_0_32.pdf?jumpid=reg_R1002_USEN
???
???Assembler Reference:
???
Before you invoke gdb on the core file, you need to set some gdb environment variables to facilitate debugging. If you are debugging on a different system than the one where the core file was created, set GDB_SHLIB_PATH to your current directory; otherwise, it should not be set. You need to set GDB_JAVA_UNWINDLIB, and how you set it depends on whether you are debugging on the same system or a different one. If you are debugging on the same system, set it to the full path of the Java unwind library for the Java release (see Section 3.4.2). If you are debugging on a different system, set it to point to the libjunwind.sl file included in your bundle. The screen below illustrates the setting of these environment variables:
# Debugging on the same system
$ export GDB_JAVA_UNWINDLIB=/opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl
# Debugging on a different system
#
$ export GDB_SHLIB_PATH=.
$ export GDB_JAVA_UNWINDLIB=./libjunwind.sl
After setting the environment variable(s), you are ready to invoke gdb on the core file. For simplicity, you have placed all the files you need in the same directory. If you are debugging on a different system than the one where the core dump was created, invoke gdb using java as the program name since the java binary was included in the bundle you moved to the debugging system. However, if you are debugging on the same system where the core dump occurred, invoke gdb using the correct version of java for the executable. In this example, the executable is a
NOTE: Refer to the first set of examples in Section 1.5.2 to determine the complete java path when debugging on the same system where the core file was created.
68 Core File Analysis
Program terminated with signal 6, Aborted.
#0 0xc0214db0 in kill+0x10 () from ./libc.2
The first step is to look at the stack trace of the failing thread. Do this by issuing gdb's backtrace command. Following is the backtrace which has been annotated with the comment ???FAILED HERE??? at the point of failure:
(gdb) backtrace
#0 0xc0214db0 in kill+0x10 () from ./libc.2 #1 0xc01ab554 in raise+0x24 () from ./libc.2
#2 0xc01f1a78 in abort_C+0x160 () from ./libc.2 #3 0xc01f1ad4 in abort+0x1c () from ./libc.2
#4 0xc4042590 in os::abort+0x98 () from ./libjvm.sl
#5 0xc4145f24 in VMError::report_and_die+0xcc4 () from ./libjvm.sl
#6 0xc40463e4 in os::Hpux::JVM_handle_hpux_signal+0x2bc () from ./libjvm.sl #7 0xc40419dc in os::Hpux::signalHandler+0x4c () from ./libjvm.sl
#8 <signal handler called>
*** FAILED HERE ***
#9 0xc3ed2ec8 in get_method_id+0x128 () from ./libjvm.sl
#10 0xc3ed34d0 in jni_GetStaticMethodID+0xf0 () from ./libjvm.sl
#11 0xc00bf398 in Java_StackTrace_dumpCore+0xf8 () from ./libstacktrace.sl
#12 0x77c09e54 in interpreted frame: StackTrace::dumpCore (int)
#13 0x77c02e08 in interpreted frame: StackTrace::methodMakeCall (int)
#14 0x77c02ee4 in interpreted frame: StackTrace::method3 (int)
#15 0x77c02ee4 in interpreted frame: StackTrace::method2 (int)
#16 0x77c02ee4 in interpreted frame: StackTrace::method1 (int)
#17 0x77c02ee4 in interpreted frame: StackTrace::main (java.lang.String[])
#19 0xc3ec1f08 in JavaCalls::call_helper+0x1d8 () from ./libjvm.sl #20 0xc403d664 in os::os_exception_wrapper+0x34 () from ./libjvm.sl #21 0xc3ec1d04 in JavaCalls::call+0x8c () from ./libjvm.sl
#22 0xc3ed0ad0 in jni_invoke_static+0x1d8 () from ./libjvm.sl
#23 0xc3ee8214 in jni_CallStaticVoidMethod+0x15c () from ./libjvm.sl #24 0x581c in main+0xb14 ()
From the backtrace output, you see that the signal handler was called in frame 8. That means that the failure was in frame 9. Following is the address where the failure took place in frame 9:
#9 0xc3ed2ec8 in get_method_id+0x128 () from ./libjvm.sl
Print out the instruction at this address by using the following gdb command:
(gdb) x/i 0xc3ed2ec8
0xc3ed2ec8 <get_method_id()+0x128>: ldw 0(%r6),%r26
This instruction loads the value pointed to by R6+0 into R26. Print out the value of R6:
(gdb) frame 9
#9 0xc46d2ec8 in get_method_id+0x128 () from ./libjvm.sl (gdb) p/x $r6
$1 = 0x0
R6 contains zero, which is an invalid address.
To discover the root of the problem, you need to examine the instructions that lead up to the failure. Start by obtaining information about the get_method_id() function:
(gdb) info functions get_method_id
All functions matching regular expression "get_method_id":
0xc3ed2da0 get_method_id(JNIEnv_ *, _jclass *, char *, char *, bool, Thread *)
This output shows that the get_method_id() function has six parameters and it begins at address 0xc3ed2da0. You now want to list the instructions from the beginning of get_method_id() to the point of failure, which is get_method_id()+0x128 (see frame 9 in the backtrace output). Before you do this, determine how many instructions to display. Compute this by subtracting the address at the point of failure from the address of the start of get_method_id(), dividing by 4 (since there are 4 bytes per
(gdb) print
4.7 Example gdb Session 69
You want to display 75 instructions from the beginning of the get_method_id() function to the point of failure for frame 9. Since this is a substantial number of instructions, redirect the output to a file:
(gdb) set
Redirecting output to frame9instrs. (gdb) x /75i 0xc3ed2da0
(gdb) set redirect off
You would probably print out this file to examine it in detail.
Let's examine the listing of the redirect file, frame9instrs. The parameters to the get_method_id() function have been removed from the listing to improve readability. They are set to the following values for all calls to get_method_id() in this listing:
(JNIEnv_ *, _jclass *, char *, char *, bool, Thread *)
A quick recap of
Details about the assembly code follow the listing. The listing has been annotated with comments for purposes of discussion:
70 Core File Analysis
*** POINT OF FAILURE; R6 IS 0 ***
0xc3ed2ec8 <get_method_id()+0x128>: ldw 0(%r6),%r26
Now let's examine specific instructions that pertain to the failure. You know that at the point of the failure, R6 is equal to 0. You need to find out why. Do this by looking at the code to examine where R6 is set. The first place R6 is set is at instruction get_method_id()+0x34:
0xc3ed2dd4 <get_method_id()+0x34>: copy %r25,%r6
In this instruction, general register R25, which holds the value for _jclass (the second parameter passed to get_method_id()) is copied into general register R6.
Now trace the value passed into frame 9 by examining frame 10 and the contents of general register R25. Look at the backtrace again. It reveals that the address for frame 10 is:
#10 0xc3ed34d0 in jni_GetStaticMethodID+0xf0 () from ./libjvm.sl
This address is the instruction at offset 0xf0 in the jni_GetStaticMethodID() function. This is where the program returned from jni_GetStaticMethodID() to get_method_id(). Take this offset, which is a hexadecimal byte offset of the return point, divide it by 4, and add 1 in order to figure out the number of instructions from the beginning of the method to the return instruction. Do this in gdb as follows:
(gdb) p 0xf0/4+1 $1 = 61
You can get the address of the start of the jni_GetStaticMethodID() function from frame 10. The address of jni_GetStaticMethodID()+0xf0 is 0xc3ed34d0 , so the start of jni_GetStaticMethodID() is:
(gdb) p/x
Now display the instructions for frame 10. Since there are 61 instructions to display, redirect the output to a file:
(gdb) set
Redirecting output to frame10instrs.
4.7 Example gdb Session 71
(gdb) x /61i 0xc3ed33e0 (gdb) set redirect off
Following is the annotated listing of the redirected output file, frame10instrs. Note that the parameters to the jni_GetStaticMethodID() function have been removed to simplify the listing. The parameters to this function are:
(JNIEnv_ *, _jclass *, char const *, char const *)
72 Core File Analysis
Trace through the instructions to see where R25 was loaded with a value. The first place this happens is at offset 0xcc:
In this instruction, R25 is loaded with the value at
(gdb) x /2x
0x59a00: 0x00000000 0x00000000
The value of R25 is 0, and it is carried through to frame 9 leading to the abort.
R25 is loaded with
The R25 zero value was passed into jni_GetStaticMethodID() from frame 11, which is the native C routine Java_StackTrace_dumpCore().
Look at selected sections of the C source code (see Section 4.1.3) to find out where that parameter was set:
...
Java_StackTrace_dumpCore(JNIEnv *env, jclass class, jint intarg) {
jclass classid;
jmethodID methodid;
...
classid =
...
methodid =
...
}
You see the statement where GetStaticMethodID() is called. The second parameter to this function is classid, which was set previously by calling the FindClass() method, and the value of classid is zero since the call to FindClass() returned an invalid value. Examining the code, you see that a literal class name was passed to the FindClass() method. This literal contains a typographical error. It should read ???java/lang/Integer??? instead of ???java/lang/IntegerX???.
4.8 Summary
Java applications abort and generate core files for a variety of reasons. There are several useful tools on
There are some environmental issues to take into account to make sure that core files are created completely. Other files, such as the executable and the shared libraries used by the executable also should be collected before analyzing the core file.
In most cases, core file analysis is quite involved and the core file will need to be forwarded to HP Support for detailed analysis. However, there are times when users can attempt their own core file analysis. If they are successful, they will speed up the time it takes to resolve their programming problems.
Many times programs core dump because of bugs in user code. Occasionally, programs core dump because of bugs in the Java VM. If you suspect the problem is due to a bug in the Java VM, you may want to research whether the problem has been reported and if there is a workaround. Useful websites for finding information are the Go Java! website:
http://www.hp.com/products1/unix/java
and the Sun bug database:
4.8 Summary 73
74
Glossary
75
76
Index
Symbols
_JAVA_HEAPDUMP environment variable, 48
C
core file checklist, 54 crash analysis tools, 13
D
deadlocked process
tools and options for debugging, 13
Developer and Solution Partner Program (DSPP), 52 dumpcore, 56
F
fatal error handling options, 14
fatal error log, 17
information contained in, 56
G
gcore, 18 gdb
dumpcore, 56
invoking on a core file, 20 invoking on a hung process, 21 Java stack unwind features, 19 packcore, 59
subcommands for Java VM debugging, 19 support for Java, 18
GDB_JAVA_UNWINDLIB environment variable, 18 generating core files, 56
GlancePlus, 51
Go Java! website, 53
H
hat, 36 heap dump
monitoring memory usage, 49 options, 48
HP Caliper, 51
HPjconfig, 21
GUI mode, 22
HPjmeter, 24
analyzing garbage collection data, 28 analyzing profiling data , 26 connecting to node agent, 29 monitoring applications, 28 monitoring metrics, 32
sample programs, 32 session preferences, 30
hs_err_pid<pid>.log, 17 hung process
tools and options for debugging, 13
I
iostat, 51
J
jar, 60
Java archive tool, 60
java.security.debug system property, 37
JAVA_CORE_DESTINATION environment variable, 55 JAVA_LAUNCHER_OPTIONS environment variable, 38 JAVA_TOOL_OPTIONS environment variable, 37 jconsole, 38
jdb, 39 jhat, 39 jinfo, 16 jmap, 16 jps, 40
jstack, 16 jstat, 40 jstatd, 41 jvmstat tools, 41
L
libjunwind, 59
location on Integrity systems, 60 location on
M
memory monitoring tools and options, 14
miscellaneous troubleshooting tools and options, 15
N
netstat, 52
P
77