AIX vs FreeBSD vs HP-UX vs RedHat vs Solaris vs Tru64

Working in lot of flavours of Unix/Linux? Not able to remember all the commands? 

Then here is the help for you:

http://www.unixguide.net/unixguide.shtml

Cheers,

Thiru

Glance - HP Way of optimizing the system's performance

I have found a webcast which explains how to use HP glance for monitoring and tuning HP Unix systems. Click here for more information. 

Few things to notice

1. We need to buy license from HP (or you can use 60 day trial version)

2. It even supports monitoring AIX, Linux, Solaris apart from HP Unix.

3. Supports drill down to find out what is going on system

To me Glance looks like more powerful nmon. For those of you who are sick of monitoring systems using vmstat, sar, iostat etc  glance (and nmon) are good tools.

Cheers,

Thiru

Be proud of being developer

Click the image to enlarge

Executing command on remote host, remote monitoring unix/Linux flavour

I was given some exercise on remote monitoring. Here is how I cracked the same. I have splited the task into 2 parts. 1) Executing a command on remote host 2) Run vmstat in remote host and collect performance statistics.

Steps for Executing the command on remote host

1. Create/update .netrc file in user home directory and add credentials of all the machines that we need to be accessed remotely. Format should be as follows: machine  login  password 
2. Execute “chmod 600 .netrc” so that only owner will have read/write access.
3. To execute the command on remote host use the following command:rexec   

Few examples for remote monitoring

HP Unix:

rexec remote_hp_ux_host vmstat 1 2 | tail -1 | awk '{print "Load Average : "$1"\nFree Mem :"$5"\nUser CPU : "$16"\nSystem CPU : "$17"\nIdle : "$18}'

 

AIX:

Rexec remote_aix_host vmstat 1 2 | awk -v  psize="`pagesize`" '{if (NR ==2) print "Total "$4"\nPage Size : " psize}{if (NR == 8) print "Free list sz: "$4"\nUser CPU   : "$14"\nSystem CPU : "$15"\nIdle       : "$16"\nIO Wait    : "$17}'

Rest of Flavours:

Give a try and send me a feed back.

Remote monitoring Linux/unix servers from linux/unix server

Steps for Executing the command on remote host

1. Create/update .netrc file in user home directory and add credentials of all the machines that we need to be accessed remotely. Format should be as follows

machine login password

 

2. Execute “chmod 600 .netrc” so that only owner will have read/write access.
3. To execute the command on remote host use the following command

rexec

To remote monitor

rexec remotehost vmstat 1 10

TCPIP State

Hi All,

     I have found a good article on TCPIP state and flow.

http://www.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf

Mapping the above article with http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/counters/counters2_lxyl.mspx?mfr=true will throw more light on perfmon counters.

Cheers,

Thiru

Quote from Thomas Kyte's book

About Thomas Kyte: Architect in Oracle Corp. Brain behind http://asktom.oracle.com/

if you want a 10 step guide to tuning a query, buy a piece of software. You are not needed in this process, anyone can put a query in, get a query out and run it to see if it is faster. There are tons of these tools on the market. They work using rules (heuristics) and can tune maybe 1% of the problem queries out there. They APPEAR to be able to tune a much larger percent but that is only because the people using these tools never look at the outcome -- hence they continue to make the same basic mistakes over and over and over.

If you want to really be aboe to tune the other 99% of the queries out there, knowledge of lots of stuff -- physical storage mechanisms, access paths, how the optimizer works -thats the only way.

..
..

Think about it for a moment. If there were a 10 step or even 1,000,000 step process by which any query can be tuned (or even X% of queries for that matter), we would write a program to do it. Oh don't get me wrong, there are many programs that actually try to do this - Oracle Enterprise Manager with its tuning pack, SQL Navigator and others. What they do is primarily recommend indexing schemes to tune a query, suggest materialized views, offer to add hints to the query to try other access plans. They show you different query plans for the same statement and allow you to pick one. They offer "rules of thumb" (what I generally call ROT since the acronym and the word is maps to are so appropriate for each other) SQL optimizations - which if they were universally applicable - the optimizer would do it as a matter of fact. In fact, the cost based optimizer does that already - it rewrites our queries all of the time. These tuning
tools use a very limited set of rules that sometimes can suggest that index or set of indexes you really should have thought of during your design.

Monitoring Windows Server Performance and Activity

Microsoft has given step by step guide on how to setup monitoring (using perfmon) on windows system. Click here to check the same.

I just don’t wanna reinvent wheel and also this saved my time.

For Linux/Unix monitoring, I am planning to write on my own(atleast 1 linux or unix version with detailed example).

Cheers,
Thiru

All I Ever Need to Know about Testing I Learned in Kindergarten - By Lee Copeland

While googling, I found an interesting blog (http://lazith.blogspot.com/2006/01/all-i-ever-need-to-know-about-testing.html). It is worth spending sometime reading this (to me, at least).

Cheers,
-Thiru

web_reg_save_param equallent for webload

I have found a site who claims that they have written a function equallent of web_reg_save_param. Click here to check.

Use at your own risk.

Thanks,
Thiru

OS Monitoring

Monitoring a server isn't something you should do chaotically. You need to have a clear plan—a set of goals that you hope to achieve. Troubleshooting server performance problems is a key reason for monitoring. Not just to plot good looking graphs and show it to superiors. Without monitoring, tuning is almost an impossible activity.

Basically monitoring is done for 2 purposes

1. Benchmark
2. Tuning

While monitoring OS, following are the basic things need to be considered regardless of OS platform.

1. CPU Utilization
2. Memory, paging
3. Throughput & retransmission statistics
4. TCP statistics
5. Disk statistics

Not all things need to be presented in report. Presenting 10-20 page report containing only relevant is always better than presenting a 100 page report (album). If you want to show your hard work, create annexure section and attached to it (I may not call it album here. After-all we are showing our hard work at relevant place. He he).

Linux Monitoring

Monitoring Linux based systems are not as complex as many people thing. All we need to know is just simple, basic shell scripting and few basic commands.

To know the version of Linux kernel
uname –a

vmstat command
vmstat reports information about processes, memory, paging, block IO, traps, and CPU activity.

Basic syntax
vmstat [[delay] count]

Example
vmstat 10 7

The first statistics that are printed are averaged over the system uptime.Don’t consider this unless it really make sense.

See man page for more information

iostat command
iostat displays kernel I/O statistics on terminal, device and cpu operations.

Basic syntax
iostat [[delay] count]

Example
Iostat 10 7

The first statistics that are printed are averaged over the system uptime.Don’t consider this unless it really make sense.

netstat command

netstat prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. There are a number of output formats, depending on the options for the information presented.

See man page for more information

Nmon utility

Quote from IBM
“This free tool gives you a huge amount of information all on one screen. Even though IBM doesn't officially support the tool and you must use it at your own risk, you can get a wealth of performance statistics. Why use five or six tools when one free tool can give you everything you need?”

The nmon tool runs on:

1. AIX® 4.1.5, 4.2.0 , 4.3.2, and 4.3.3 (nmon Version 9a: This version is functionally established and will not be developed further.)
2. AIX 5.1, 5.2, and 5.3 (nmon Version 10: This version now supports AIX 5.3 and POWER5™ processor-based machines, with SMT and shared CPU micro-partitions.
3. Linux® SUSE SLES 9, Red Hat EL 3 and 4, Debian on pSeries® p5, and OpenPower™
   Linux SUSE, Red Hat, and many recent distributions on x86 (Intel and AMD in 32-bit mode)
4. Linux SUSE and Red Hat on zSeries® or mainframe

Click here for more information.

Windows Monitoring
Windows provide huge set of monitoring counters to end users. Not all need to be monitored always, but all counters might be useful at one point or another. All windows OS related counters can be monitoring using perfmon (just goto run and type perfmon).

I am planning to write Step by step instruction on how to monitoring windows using perfmon later in as a separate blog.

Performance in Linux and Windows Servers

Performance of Windows Servers are quite unpredictable. The CPU mainly is totally not understandable. It does not show any trend. It could go to the peak load at very initial stages of the load test and would remain there for the remaining duration of the test run. Sometimes, it can also be observed that at some load, the CPU comes down and goes up again. Even the Memory and other parameters that are mainly considered for the performance tuning show sporadic nature. But the response times might show a constant trend. Due to these unpredictability, it is very difficult to predict the optimised Parameters of the performance.

Peformance of the same parameters in the Linux and AIX machines are very trendy. They show a particular trend with which it is very easy to predict and recommend various Performance parameters to optimise the performance of these Servers. It can even be observed that the response time statistics are very much in the trend with the UserLoad and CPU Utilisation, etc.

The observations are of my personal experience. Comments, suggestions and Views are welcome.

Understanding Architecture

Most important phase of performance tuning is to understand Architecture of the System under testing. Without understanding the end to end architecture, performance tuning is like “searching needle in hay”.
Few basic components and its definitions

1. Load balancer -> Technique to spread work between two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, throughput, or response time.
2. Web server -> A computer program that is responsible for accepting HTTP requests from web clients, which are known as web browsers, and serving them HTTP responses along with optional data contents, which usually are web pages such as HTML documents and linked objects such as images, css etc.,(usually static content)
3. LDAP server -> The Lightweight Directory Access Protocol, or LDAP, is an application protocol for querying and modifying directory services running over TCP/IP. The most common example is the telephone directory, which consists of a series of names (either of persons or organizations) organized alphabetically, with each name having an address and phone number attached. Due to this basic design (among other factors) LDAP is often used by other services for authentication(SSA).
4. Application Server -> An application server is a software engine that delivers applications to client computers or devices, typically through the Internet and using the HyperText Transfer Protocol. Application servers are distinguished from web servers by the extensive use of server-side dynamic content and frequent integration with database engines. An application server handles most, if not all, of the business logic and data access of the application.
5. Database server -> complex set of software programs that controls the organization, storage, management, and retrieval of data in a database.

During testing, monitoring and logging all the components + OS involved is mandatory. Without monitoring all components including network, improvement/tuning acheived is pure luck.

Next blog plan: monitoring

-Thiru

Windows XP getting weird ??

Windows XP will either get into weird state or stop working if we haven’t rebooted the system for long time. Either we have to logout and login or reboot to make it run smoother.
There are alternatives out there.
Create desktop shortcut for “%windir%\system32\rundll32.exe advapi32.dll,ProcessIdleTasks” and then invoke the created shortcut whenever you feel the system is slow/weird.
Don't want to do this?? Simple: Update to SP1. (Seems to be fixed in SP1).

Enjoy.

Performance - Go Green

Performance is not just making system to run faster. It effective use of whatever is available to us. Few questions/suggestions for us:

1. Have we switched off your monitor while leaving for break in the middle of work?
2. Have we switched off your computer while going home? Our systems are not server that is accessed 24x7. Nothing wrong is wasting 2 minutes next day morning to turn on.
3. Have we turned off your bike in signal?
4. Have we turned off the switch after charging the mobile?
5. Do we use printer effectively? (2 sides printing, print 2 pages on each side etc.,)
6. Use reusable plates rather than using plastic or paper plate.
7. Replace normal bulb with a compact fluorescent. Saves electricity. (google: ban the bulb).
8. Don’t keep your refrigerator door open for long time.
9. Please don’t use PLASTIC or try to minimize.

I am planning to follows this atleast from today.

These are very few things but can make big difference. Go Green.

Thanks

Thiru

Performance Testing Process/Phases

Performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. To become a successful Performance Engineer one should have

1. Basic concept of how things work (HTTP, Network, LB, Web Server, App server, DB server, LDAP, SSL Accelerator etc.,)
2. Eagerness to learn things
3. patience
4. In sting (list continues based on what you do)
   

Purpose

1. Check whether it met predefined SLA
2. Compare 2 systems and find out which is performing better
3. Find bottleneck and tune to make the system to perform better
   

Performance Testing/Engineering is not a science, it is rather art.
Phases in Performance Testing/Engineering

1. Requirement collection
2. Feasibility study
3. Test Plan
4. Scripting
5. Monitoring Design & Deployment
6. Smoke Test
7. Test Execution & Data Collection
8. Analysis and Reporting
9. Closure

Requirement Collection

• Understand want customer really wants
• Tell customer what performance team do and make sure they understand what it means.
• Ask customer what to do
• While selecting business flow for simulation, select flow(s) which obey the following rule. “20% of total business flow carried out by 80% of users”. (Traversal document)
• Understand the architecture (this has to done in detail if monitoring and tuning is involved)
• Stress to customer that

a. We are not here to do function testing.

b. We don’t/can’t performance test everything.

c. We don’t/can’t simulate 100% of production transactions.

Feasibility Study

• Can we do what customer wants? If not, ask customer for help. Better to say no rather than doing something wrong or giving false commitment. This will make sure we get repeated customer.
• How fast can we do this task? (this helps in planning)
• Does tool by native support this or do we need to build work around?
  Note: In the same service based company, marketing team can’t say no most of the times (May be due to aggressive targets set by management or earn commission and much more); in that case, always insist for employee training whenever possible.

Test Plan

• Form a team. Know their skills
• When to do what?
• Deadlines. Fix deadline based on team’s skill(not based on TL's skill).
• Get approval from client

Scripting

• Simulate as per traversal document
• Remember we simulate only core portion; so don’t make mistake in that.
• Build agenda/driver script based on mix.
• We are testing something to find issues. Make sure your script doesn’t have issues.

Monitoring design & Deployment

• Get the list of servers that needs to be monitored (including versions)
• If you haven’t worked on that specific version of servers, spend sometime in going though ‘new features’
• Design & deploy the monitoring for OS(perfmon, vmstat, netstat etc.,) and then corresponding servers specific monitors(perfservlet, server-status, statspack etc.,).
  Smoke Test
• Smoke Test is the test to make sure script and monitoring are working as required
• Make sure entire team is there during smoke (scripting team and monitoring design/deployment team)
• Have you detected something suspicious? Check and/or correct the same. Mistakes here will lead to failure of whole engagement.
• Redo smoke test if some changes were made
• Make sure enough time is allotted for smoke. Don’t do smoke just before 1 hour of test. Atleast give 1 or 2 days for smoke depends on complexity of engagement.
• Make sure you have check list

Test Execution & Data collection

• Should be only started if everything was gone fine in smoke.
• Make sure check list is correctly followed.
• After test execution, collect simulation tool logs, monitoring logs, configuration files, log files and whatever is required by analysis team.
• Most importantly timesync information of all servers.

Analysis and Reporting

• Nothing wrong in Newbie doing analysis. But?? (read the last point)
• Reporting should focus and answer for every question in requirement document. Nothing wrong in providing value add. Also nothing wrong in missing value add. But make sure customer gets what he wants. There is no point is giving value addition without addressing customers basic requirements.
• Recommendation should have substantiation or else say best practice.
• Review the report. After customer is expecting only this from us.

Closure

• Anything gone wrong? Why? Make sure that it is not getting repeated in future project
• Appreciate who ever deserves. After all PE guys doesn’t only work for money.
• Appreciation must be given for fresher who did excellent job in scripting inspite of this being his/her first project.
  

Whatever phase I listed below has to happen ideally; not necessarily always.

Comments are always welcome.

Welcome

Hello Everyone,

Welcome to Performance Hints Blog.
I will try to provide accurate information as much as possible in this blog. But whatever I write/say in the blog is my personal opinion. It has nothing to do with my employer.

Keep posting your ideas/queries and me this blog a interactive one. Anyone willing to join as author? Send me your personal email id.

Thanks,
Thiru