“hsperfdata_”-files in /tmp (or $TMPDIR) – jstat

The “hsperfdata_”-files are part of the java performance monitoring since version 1.4.2.
A general FAQ can be found here. A part of this performance monitoring aspect is “jstat”. It’s a Java SE bundled command-line JVM statistics monitoring tool.

To see what type of statistics you can choose from:
jstat -options

For example, to see a summary of garbage of collection statistics:
jstat -gcutil

[root@pingu:/tmp]# ls /tmp/hsperfdata_sandbox/
4828 4843 5024 5081
[root@pingu:/tmp]# jstat -gcutil 4828
S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 98.11 74.98 38.82 94.05 23 1.033 1 0.181 1.213

A syntax explanation on howto use “jstat” can be found here.

Debugging solaris : when nobody else can help, you better call dtrace

DTrace
DTrace is a comprehensive dynamic tracing framework for the Solaris Operating Environment. DTrace provides a powerful infrastructure to permit administrators, developers, and service personnel to concisely answer arbitrary questions about the behavior of the operating system and user programs.

I’ve been using the “execsnoop”-script (from brendangregg.com) today, and it’s been proving to be a very useful script.

execsnoop is a program to snoop process activity. As processes are executed on the server their details are printed out. Another user was logged in running a few commands which can be seen below. This is especially useful in troubleshooting short lived processes that are otherwise hard to spot. More examples.

# execsnoop
UID PID PPID CMD
100 3008 2656 ls
100 3009 2656 ls -l
100 3010 2656 cat /etc/passwd
100 3011 2656 vi /etc/hosts
100 3012 2656 date
100 3013 2656 ls -l
100 3014 2656 ls
100 3015 2656 finger
[…]

* Execsnoop has options to add the date and time (-v), or watch a particular command (-c command).
* Standalone execsnoop.d is the original DTrace only version..

Links
Guide
Howto
Scripts @ brendangregg.com (TIP)
Scripts @ Opensolaris.org
Scripts @ SolarisInternals.com

sed : slash before newline gets ignored

On some system, sed ignores the starting backslash from n (newline). So the following code:

$LAYOUT=”#n$INFOn$PARAMETER”;
sed “/$SECTION/a
$LAYOUT” $FILE

Had to be rewritten as follows:

sed “/$SECTION/ {G;s/$/#/;G;s/$/$INFO/;G;s/$/$PARAMETER/;}” $FILE

It doesn’t improve the readability, yet it gets the job done.

bash getopts

If you’re scripting with bash, and haven’t encountered the getopts function, then you should read the following
article @ linux.com.

The getopts function is an -easy to use- function when you’re working with script where parameters and flags have to be set.

Example

while getopts ":f:p:c:s:d r v" o ; do
        case $o in
                f ) FILE=$OPTARG;;
                p ) PARA=$OPTARG;;
                c ) VALU=$OPTARG;;
                s ) DELI=$OPTARG;;
                d ) DEBUG=y;;
                r ) RO=y;;
                v ) echo $VERSION
                    exit 0;;
        esac
done

The options with a colon (“:”) in front of them need to have an argument set. Yet if there is a space, then no argument is needed.

So -r -d & -v are the flags. Where the others are options/parameters that can be set.

Wireshark : prepare tcpdump / snoop output for it

When an certain communication flow isn’t working, it’s sometimes hard to know what’s causing it. So you’ll probably want to know what’s going over the wire. Thank god linux & unix offer good tools for this.

You can capture the data that is seen by a server with a simple tool. On linux this is called “tcpdump”, and on solaris it’s called “snoop”. Don’t be fooled by the name “tcpdump” as it handles more than only tcp packets.

Tcpdump & snoop will only output a truncated packetlength. Yet we want to see the full packets in wireshark (previously Ethereal) later on. So we’ll say that we want the packetlength to be the full 1500. And we want it to be exported to a nice file:

tcpdump -i [interface] -s 1500 -w [some -file]
snoop -d [interface] -s 1500 -o [some -file]

These commands will capture the full packets that pass on “interface”, and write it to “some-file”. Yet you can also add filters to this. Read the man page for more info, but here’s a small example:

tcpdump -s 1500 -i external port 162 -w /tmp/external-162.dmp

Good luck tracing 😉