The Truth about performance testing tool
In modern day internet/web application, response time can be computed as follows
Response time = server side response time + network delay + client side processing time + rendering time
Where as our load testing tools, computes response time as follows.
Response time = server side response time + network delay.
Since client side logic is heavy in modern day web/internet application, most of the tools available today are really missing out the significant portion in response time.
Adding fuel to fire (Java Script Engine)
Because Java Script is so important to the web today, no one can even imagine a web application without java script. A JavaScript engine (also known as JavaScript interpreter or JavaScript implementation) is an interpreter that interprets JavaScript source code and executes the script accordingly in client’s browser. In real, java script execution time by Java Script Engine is also added to end user response time. To name few Spider monkey, V8 are couple of open source java script engine.
Even if some tools has ability (like Loadrunner click & script), it is very limited to a particular browser and particular java script engine.
More Fuel
Even if a tool is able to makeup the java script's response time, calculating full response time by tool has exception. PLUGINS. It is practically impossible for a tool vendor to work with all plugins available. Java and flash are the commonly used plugins in lot of web sites.
Just want to quote an example
In one of my previous client environment, single user test for a particular page click takes around 50 sec, whereas the loadrunner @ 1000 user load reported the max response time as 10 sec. After seeing the result, client has called in for a meeting and they demonstrated his portion and proved that in real, that particular page takes much longer time to view. To stick to our result, we have done a sniffing of that request using network sniffer and showed that client side JS doing some nasty calculation is the culprit not the server. Also we explained them that, whatever loadrunner gives are the server side response time not the end user response time.
So sad
Even if performance testing tool vendors wants to include client side response time, it is very difficult for them as whole lot of java script engine out in market with various market shares.
Even tough for some, it is hard to believe that server side response time and end user response time are very different, we have to accept(as it is fact).
Even if performance testing tool vendors wants to include client side response time, it is very difficult for them as whole lot of java script engine out in market with various market shares.
Even tough for some, it is hard to believe that server side response time and end user response time are very different, we have to accept(as it is fact).
8 comments:
"...whereas the loadrunner @ 1000 user load reported the max response time as < class="Apple-style-span" style="font-weight: bold;">"
Broken HTML in the post, or was the max response time really reported as a tag? :-)
Thanks Chuck for the comment. It was corrected now.
Thiru
Hi Thiru,
Nice to see a blog that talks specifically about performance engineering.
I liked the way you have put together this particular post w.r.t. issues in current tools.
Happy Testing,
Rahul Verma
:)
Hi,
This is really good and a very practical scenario and thanks for shedding some light on it. End user response time is actually what matters at the end of the day :)
BTW can you lightly elaborate the network sniffer you used and the way it has been used?
thanks
Tharinda
In reality the client side rendering time should not be more than fractions of a second. That is were the tools like firebug, plays a major role.
And more over that has to be test at the time of development itself. Because the rendering time is not depended (99%) on the current load on the server.
Client-side not only does rendering. It also does calculations sometimes (like my example or even stock ticker applications does this)
Even if the load testing tool vendors do come up with a way to identify client side response time, it would still be a concern. Client side performance depends on the machine it runs on. We would end up giving the details to client with far more disclaimers than how it is now :)... We will end up saying that it will take 10 ms for this js on this machine with so much CPU consumed :D
One thing to do is do the client side and server side performance testing separately.. Be done with the tool testing and tune the server. On the side run one iteration manually and using a client side tool like yslow, fiddler, pagedetailer etc find if there are any culprits eligible for tuning. Currently with the tool constraints, that is the only way I see working.
Post a Comment