Loading ...
December 4th, 2008 by 
Yaron Assa
So, what’s the fastest way for working with GUI objects? It seems that everyone has an opinion on the matter – some insist the Object-Repository yields the best results, while others are strong advocates for working with Descriptive Programming. While one will insert every object into a variable and use that variable, another would use the objects straight from the Object-Repository, thinking it was the most efficient way.
The Test
Well, I thought we’d better settle this once and for all, and test the matter (more or less) scientifically:
1. Test test objective would be to set changing values to Google’ main search page query box.
2. QTP was set to Fast run mode, with smart identification set to disabled.
3. The relevant page was the only one opened, in the only browser opened.
4. The test was run on IE7, in Windows XP with SP3 running on VMWare Fusion 2.1, with 1GB RAM. The test machine was a 2.4GHz MacBook pro (4GM RAM total).
5. The test was run 10 times – the figures are the average for these run sessions.
6. The test manipulated the GUI through these methods:
A. Regular use of the object repository.
B. Straightforward descriptive programming.
C. OR via fixed reference variable (settings a variable to point to an OR object, and then using that variable).
D. Descriptive Programming via fixed reference variable (similar to C).
E. Runtime through Object-Repository (using the object repository to access the WebEdit’s runtime object, and setting the value by changing a runtime property).
F. Runtime through Descriptive programming (similar to E).
G. Runtime through fixed object-repository reference (using the object-repository to get to the runtime object, inserting it into a variable, and using that variable)
H. Runtime through fixed descriptive-programming reference (similar to G).
7. Each method was tested for 50 consecutive inputs via a for loop.
I think this pretty much covers all of the mainstream possibilities for working with GUI objects. A timestamp was taken before each loop, and compared to a timestamp taken immediately afterwards.
Here’s the code for the test:
Dim i
Dim iTimer
Dim oEdit
'Regular OR
iTimer = Timer
For i = 0 to 50
Browser("Google").Page("Google").WebEdit("q").Set i
Next
Print "Regular OR =" & Timer-iTimer
'Descriptive Programming
iTimer = Timer
For i = 0 to 50
Browser("index:=0").Page("index:=0").WebEdit("name:=q").Set i
Next
Print "Descriptive Programming = " & Timer-iTimer
'OR via fixed reference
iTimer = Timer
Set oEdit = Browser("Google").Page("Google").WebEdit("q")
For i = 0 to 50
oEdit.Set i
Next
Print "OR via fixed reference = " & Timer-iTimer
'Descriptive programming via fixed reference
iTimer = Timer
Set oEdit = Browser("index:=0").Page("index:=").WebEdit("name:=q")
For i = 0 to 50
oEdit.Set i
Next
Print "Descriptive programming via fixed reference = " & Timer-iTimer
'Runtime through OR
iTimer = Timer
For i = 0 to 50
Browser("Google").Page("Google").WebEdit("q").Object.value = i
Next
Print "Runtime through OR = " & Timer-iTimer
'Runtime through Descriptive Programming
iTimer = Timer
For i = 0 to 50
Browser("index:=0").Page("index:=0").WebEdit("name:=q").Object.Value = i
Next
Print "Runtime through Descriptive Programming = " & Timer-iTimer
'Runtime through OR fixed reference
iTimer = Timer
Set oEdit = Browser("Google").Page("Google").WebEdit("q").Object
For i = 0 to 50
oEdit.value = i
Next
Print "Runtime through OR fixed reference = " & Timer-iTimer
'Runtime through DP fixed reference
iTimer = Timer
Set oEdit = Browser("index:=0").Page("index:=0").WebEdit("name:=q").Object
For i = 0 to 50
oEdit.value = i
Next
Print "Runtime through DP fixed reference = " & Timer-iTimer
Update:
Dani has suggested testing fetching the runtime object differently, through GetElement by ID:
iTimer = Timer
Set we = Browser("B").Page("P").Object.getElementById( "id" )
For I = 0 To 50
We.Value = i
Next
Print "runtime through the OR and GetElementByID = " & Timer-iTImer
iTImer = Timer
Set we = Browser("B").Page("P").Object.getElementByName( "name" )
For I = 0 To 50
We.Value = i
Next
Print "runtime through the OR and GetElementByName = " & Timer-iTImer
The Results
And here are the averaged results:
Regular OR = 5.59375
Descriptive Programming = 6.375OR via fixed reference = 3.31253
Descriptive programming via fixed reference = 3.25122
Runtime through OR = 2.90625
Runtime through Descriptive Programming = 3.921875
Runtime through OR fixed reference = 0.203125
Runtime through DP fixed reference = 0.265625
Update: Runtime through GetElementById = 0.20122; and through GetElementByName = 0.21019
Well, this turned up to pack some surprises, at least for me, but the bottom line remains the same – setting a variable to the GUI’s runtime objects is the most efficient method.
It seems that working directly through the object repository is faster than through descriptive programming (difference of almost a whole second – which quite shocked me, personally). This is reinforced by a more than a second gap when using OR and DP to access the WebEdit’s runtime object, however, the difference is wiped out by using a mediating variable.
It’s interesting to note that using the runtime object reduces the time by almost half, bringing it close to the performance of a fixed reference via a variable.
And the winner, by a full order of magnitude, is using the runtime object through a fixed reference variable. This is not surprising, as it avoids QTP’s GUI mapping mechanism completely (beside the first variable assignment).
I hope you enjoyed this article, and I’m looking forward to hearing more ideas for performance and other tests we can run on QTP (as opposed to with QTP)
Posted in General, QTP Techniques, Yaron Assa's Blog
December 4th, 2008 at 11:02 am
Fantastic work Yaron.
Now we only have the subjective argument of which approach is most appropriate for the team in question to support and debug.
December 4th, 2008 at 12:10 pm
Good article, I enjoyed to read it.
Thanks, Yaron :)
December 4th, 2008 at 12:25 pm
Yaron — Good article …
I am still puzzled at what it means to typical problems that people using QTP face …
In most of the Automation scenarios - fast execution of the automation testing is the least of concerns. Since most of the good automation is expected to run nearly unattended - it does not matter whether it takes 5 secs or 15 secs to complete a test.
Most of the cases the worry is about sychronization, free from false alarms, free from “Generic errors” or “object” not found errors.
How does this current study of yours and well explained ways of accessing GUI objects will help these QTP issues that are more painful and need help than speed issue.
Shrini Kulkarni
http://shrinik.blogspot.com
December 4th, 2008 at 1:36 pm
Interesting article!
To me test execution performance time matters a lot since “extra seconds” becomes minutes/hours when running a lot of tests in sequence. As a tester you want your test result ASAP for both manual and automated test execution.
/Stefan
http://abouttesting.blogspot.com/2008/05/qtp-test-execution-performance.html
December 4th, 2008 at 1:41 pm
Nice article Yaron.
Will and myself had a discussion on the same topic sometime back (in the comments)… http://www.learnqtp.com/2008/06/make-your-qtp-scripts-perform-better.html
December 4th, 2008 at 1:52 pm
Hi Yaron! Thanks for your systematical performance investigation and publishing the results. Good job.
However, I would recommend testing with a number of most frequently used page controls and operations with them. That might bring more reliable and proved results.
Thank you,
Alex.
December 4th, 2008 at 2:18 pm
This is very interesting. In the past I chose for DP approach instead of using QTP’s OR just for reason of performance.
However I’m very curious what the results would be if you do the same exercises on a more complex and extended script covering.
December 4th, 2008 at 5:12 pm
A very nice performance analysis Yaron! I was surprised to see that results indicated OR method consumes less time than DP method. I assumed DP is the best compared to OR method. But personally I would like to say here that in big QTP projects containing several hundred assertions DP method offers vb coding flexibility and saving of disk space - in this DP method better than using OR.
http://rameshnatesan.blogspot.com
December 4th, 2008 at 6:06 pm
Excellent article Yaron.
Some results are quite surprising.. didn’t know runtime OR would have such a huge difference in performance as compared to DP.
December 4th, 2008 at 7:18 pm
Yaron - Thanks. Great test
shrinik - You are right that this study doesn’t touch on other critical automation issues but even though performance should not and is not the main concern for good automated scripts but it does matter and this is how: if you are running 1000 scripts overnight, this study shows that the test would finish a lot sooner when using OR. This means that you don’t run into issues where your scripts are still running when morning batches kick in. How you handle sync points is a different subject and good automation should be independent of object interaction and object sync
Ramesh - I’m not sure what you mean when you say DP method gives flexibility and saves disk space. If you create objects for every thing starting from your window (that can be done in a global module or .vbs files) it doesn’t take that much more disk space and gives you the same flexibility as DP. In fact if you are creating your own objects it is less resourceful on the machines as all objects can and should be cleared from memory (when set to nothing). When you use DP, you do not clear memory and these objects are still sat in your memory. QTP does not (by default) clear memory
Saying that, down side of using objects is that in QTP, (self created) object approach only works on IE. It’s not cross browser compatible
December 5th, 2008 at 8:05 am
Fantastic work Yaron.
December 5th, 2008 at 7:47 pm
Great work Yaron.
It directly helped me to reduce my script performance drastically.
I had a script which have to run a look through 12000 records.
After applying Runtime through OR fixed reference method, it gave much better result.
Thanks a lot and looking for more such work!!
Raj
December 8th, 2008 at 6:25 am
Hi Yaron,
Good work yaron, In my case the fastest execution is “runtime through OR fixed reference”, below are the sequence of execution results.
Regular OR =3.796875
Descriptive Programming = 4.6875
OR via fixed reference = 2.359375
Descriptive programming via fixed reference = 2.3125
Runtime through OR = 1.875
Runtime through Descriptive Programming = 2.71875
Runtime through OR fixed reference = 0.078125
Runtime through DP fixed reference = 0.09375
But I try to use Dani method, but I am getting an error, that the method is not supported.
December 8th, 2008 at 8:45 am
Hi,I tried this for windows application.But “Object” method is not supporting.
Could you please suggest how to do for windows applications!
December 8th, 2008 at 8:50 am
Sorry, the windows environment doesn’t support access to runtime objects via .Object
December 8th, 2008 at 1:01 pm
By seeing this idea is excellent. Showing quantitative results given base for other thoughts.
Excellent.
December 9th, 2008 at 9:43 am
Excellent Stuff !!
December 9th, 2008 at 6:52 pm
A couple things I have noticed.
1. The speed in which each method works is dependent on the application. The OR is exponentially faster on AJAX / GWT applications.
2. The OR’s performance degrades as it increases in size.
December 11th, 2008 at 7:48 am
that’s awesome Yaron, thanks for that.
So basically, i call a ‘RegisterUserFunc’, replace my ‘WebEdit.Set’ with ‘object.value=’ and i’m done.
no code changes, and instanst performance improvement, great stuff.
December 15th, 2008 at 3:34 pm
Good article.
Thanks, Yaron
December 16th, 2008 at 10:41 am
Its really a useful info.thanks
December 17th, 2008 at 4:42 am
very good
many times my friends chat with me about this quetions
thanks a lot