Solaris – Measuring CPU : mpstat vs sar

Noticed something odd today;
The cpu idle showing when using “mpstat” differs from the one you have with “sar”, and would expect when reading “prstat”.

Check the following example:

a small script to generate CPU load
root@solaris> cat test

#!/bin/bash
for cpu in 1 ; do
( while true; do true; done ) &
done

an output of mpstat prior to starting the load
root@solaris> mpstat

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 724 66 581 720 602 832 40 139 38 0 1783 11 15 0 74
1 670 60 503 703 564 974 47 138 34 0 1598 9 14 0 77

we’ve got two processors, so we run the script twice so we’ll notice it more
root@solaris> ./test
root@solaris> ./test

sar shows we’re using 88% cpu, the system is using 12%, so 0% is idle
root@solaris> sar 1 1

SunOS solaris 5.10 Generic_125100-02 sun4u 04/26/2007
12:34:29 %usr %sys %wio %idle
12:34:30 88 12 0 0

mpstat says we’ve got around 74 to 77 % idle cpu time… how can that be?
root@solaris> mpstat

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 724 66 581 720 602 832 40 139 38 0 1783 11 15 0 74
1 670 60 503 703 564 974 47 138 34 0 1598 9 14 0 77

prstat shows that we’re using 62% at the moment
root@solaris> prstat

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
24715 root 2848K 1680K run 55 0 0:00:23 31% test/1
24709 root 2848K 1680K run 50 0 0:00:25 31% test/1
…
Total: 257 processes, 744 lwps, load averages: 1.40, 0.82, 0.70

sar shows us 80%, 20% for the system, so again 0% idle time
root@solaris> sar 1 1

SunOS solaris 5.10 Generic_125100-02 sun4u 04/26/2007
12:34:52 %usr %sys %wio %idle
12:34:53 80 20 0 0

When checking the man page of “mpstat”, we see the following explanation:

idl – percent idle time

So we’dd expect a value more towards 0% then towars 100%…

But further down we there is the following note:

The sum of CPU utilization might vary slightly from 100 due
to rounding errors in the production of a percentage figure.

The total time used for CPU processing is the sum of usr and
sys output values, reported for user and system operations.
The idl value reports the time that the CPU is idle for any
reason other than pending disk I/O operations.

So mpstat only shows the idle time in relation to the pending disk I/O operations. Useful to know!

The scheduling algorithm in regards to the nice value of a process

http://www.oreilly.com/catalog/linuxkernel/chapter/ch10.html

Summarized
The kernel does timeslicing of it’s cpu resource. Each process is given a certain slice. Yet the size of this size is determined by the priority (nice value). The priority is handled dynamically. So when a process hasn’t had enough cpu time, it’ll receive a higher priority. This also works the other way around, a process that had been running for a long time, will receive a low priority (cfr with a higher nice value).

The ‘Starvation’ protection

“In Linux, process priority is dynamic. The scheduler keeps track of what processes are doing and adjusts their priorities periodically; in this way, processes that have been denied the use of the CPU for a long time interval are boosted by dynamically increasing their priority. Correspondingly, processes running for a long time are penalized by decreasing their priority.”

Het scheduling concept

“Linux scheduling is based on the time-sharing technique already introduced in the section “CPU’s Time Sharing” in Chapter 5, Timing Measurements: several processes are allowed to run “concurrently,” which means that the CPU time is roughly divided into “slices,” one for each runnable process.[1] Of course, a single processor can run only one process at any given instant. If a currently running process is not terminated when its time slice or quantum expires, a process switch may take place. ”
…
“The Linux scheduling algorithm works by dividing the CPU time into epochs. In a single epoch, every process has a specified time quantum whose duration is computed when the epoch begins. In general, different processes have different time quantum durations. The time quantum value is the maximum CPU time portion assigned to the process in that epoch.”
…
“Each process has a base time quantum: it is the time-quantum value assigned by the scheduler to the process if it has exhausted its quantum in the previous epoch. The users can change the base time quantum of their processes by using the nice( ) and setpriority( ) system calls.”

Ubuntu 7.04 feisty & Vmware Server

Upgrade to Feisty, yet my vmware server wouldn’t start anymore. The update.pl wasn’t capable of solving it, yet after a bit of googling I found a patch for it…

Patching the software

Newer version of Kernel the VMware server needs to be patched, here is how.

* Get the code

wget http://ftp.cvut.cz/vmware/vmware-any-any-update109.tar.gz

* Unpack it

tar xvzf vwmare-any-any-update109.tar.gz

* Change to its directory

cd vwmare-any-any-update108

* Run the script that is going to patch the VMWare.

./runme.pl

Credits go to Nick @ tickus.com

UPDATE (10/08/2007)
* check within the index of http://ftp.cvut.cz/vmware/ first, as the vmware-any-any-updateXXX.tar.gz file gets updated a lot
* afterwards, run the “vmware-config.pl” tool again, and you should have no problems with your vmware

BackTrack 2 security live CD Review @ Linux.com

Check out the following article:
BackTrack 2 security live CD Review @ Linux.com

BackTrack is a live CD Linux distribution that focuses on penetration testing. A merger of two older security-related distros — Whax and Auditor Security Collection — BackTrack bundles more than 300 security tools.

Planning to test the tool later on… Seems like a nice thing to work with

Ubuntu 7.04 : smb issue after upgrade

After the upgrade, I encountered this error again when doing my samba mounts.
smbmnt must be installed suid root for direct user mounts (1000,1000)
smbmnt failed: 1


It resurfaced after the upgrade, but as I did it before, I recalled the fix:

$ sudo chmod u+s `which smbmnt`
$ sudo chmod u+s `which smbumount`

firefox plugins – the must haves!

Colorful Tabs
One of the things I got used to very fast when starting with “firebird” (the original name of FireFox) were the tabs. Yet when you have a lot them opened, then you can easily lose track of what was what. This addon gives each tab a nice pastel color, easily identifying your tabs, and nice on the eyes.

Firefox Showcase
I think this was stolen from the ideas from IE. Yet they’ve probably had stolen it from Mac… and they’ve stolen enough from Firefox in the past. So who cares? It’s a nice thing. You press F12, and you get a visual overlay of all the pages you have open.

Firebug
Building websites? Then you NEED this. I cannot believe this wasn’t developed ages ago.

FireFTP
A crossplatform ftp client in the shape of a firefox plugin.

Flashblock
Choose which flash things need to be shown (thus downloaded).

MeasureIt
Measure anything on a webpage. Handy when designing imo.

Ubuntu 7.04 Released

http://www.ubuntu.com/news/ubuntu704
Running the upgrade on my system as I type this… I’m curious! 🙂

solaris & md5 checksum

A helpful tip: Solaris (even 10) doesn’t ship with “md5sum” installed. So if you want to do a md5 checksum, you can try to see if digest is installed (man digest). Or download/install a md5 checker yourself. The file can be dowloaded from here

Java in real-time systems?

Today developerWorks published an interesting article on their site.

Some big clue lines:

Real-Time requirements

Real-time (RT) is a broad term used to describe applications that have real-world timing requirements. For example, a sluggish user interface doesn’t satisfy an average user’s generic RT requirements. This type of application is often described as a soft RT application. The same requirement might be more explicitly phrased as “the application should not take more than 0.1 seconds to respond to a mouse click.” If the requirement isn’t met, it’s a soft failure: the application can continue, and the user, though unhappy, can still use it. In contrast, applications that must strictly meet real-world timing requirements are typically called hard RT applications. An application controlling the rudder of an airplane, for example, must not be delayed for any reason because the result could be catastrophic. What it means to be an RT application depends in large part on how tolerant the application can be to faults in the form of missed timing requirements.

As you might guess, using Java for such RT (Real-Time) applications isn’t that obvious. A small summary of the challenges:
– Standard Java provides no guarantees for thread scheduling or thread priorities : It’s quite hard for an application to reacte within a given time if the trhead management doesn’t support this.
– Java-conformant JVM must delay loading a class until it’s first referenced by a program : This causes a delay in the application, and also in the Real-Time response of the applications. As the delay can cause an significant (& possibly unexpected) impact. The same goes for compiling to native code.
– Garbage collection: The GC can cause delays too, and same as the above… and thus cause an impact.

And then some promo on how IBM solves this matter… 😉 Yet it remains interesting reading material if you’re working with Java applications in an environment that depends on performance. It gives you some extra insight on how to tune your machines/code.

Nice

A small post how to tell processes to be nice to eachother. So yet again another addition to the entry about processes.

As the *nix systems do multitasking to handle their processes, there also has to be a system where it is possible to prioritize certain processes. This can be done thru the commands nice and renice.

The priority can be set within a range from -20 to 19. Where -20 is the most favorable scheduling. An easy way to remember is to think about gravity. The more weight you have, the lower you float, and the lower you are on the priority list.

So what’s the difference between the two commands;
– The nice command can be used to give a weight to the process when launching this process.
– The renice is to be used when an already active process needs to be re prioritized.

For example:
nice -n 10 /foo/bar
renice -p $PID -n 10

This small bit of knowledge can help you tune the process management of you system. Thus tune your server to be more effecient, as the important processes get more system resources.