Loading ...
December 4th, 2008 by 
Yaron Assa
So, what’s the fastest way for working with GUI objects? It seems that everyone has an opinion on the matter – some insist the Object-Repository yields the best results, while others are strong advocates for working with Descriptive Programming. While one will insert every object into a variable and use that variable, another would use the objects straight from the Object-Repository, thinking it was the most efficient way.
The Test
Well, I thought we’d better settle this once and for all, and test the matter (more or less) scientifically:
1. Test test objective would be to set changing values to Google’ main search page query box.
2. QTP was set to Fast run mode, with smart identification set to disabled.
3. The relevant page was the only one opened, in the only browser opened.
4. The test was run on IE7, in Windows XP with SP3 running on VMWare Fusion 2.1, with 1GB RAM. The test machine was a 2.4GHz MacBook pro (4GM RAM total).
5. The test was run 10 times – the figures are the average for these run sessions.
6. The test manipulated the GUI through these methods:
A. Regular use of the object repository.
B. Straightforward descriptive programming.
C. OR via fixed reference variable (settings a variable to point to an OR object, and then using that variable).
D. Descriptive Programming via fixed reference variable (similar to C).
E. Runtime through Object-Repository (using the object repository to access the WebEdit’s runtime object, and setting the value by changing a runtime property).
F. Runtime through Descriptive programming (similar to E).
G. Runtime through fixed object-repository reference (using the object-repository to get to the runtime object, inserting it into a variable, and using that variable)
H. Runtime through fixed descriptive-programming reference (similar to G).
7. Each method was tested for 50 consecutive inputs via a for loop.
I think this pretty much covers all of the mainstream possibilities for working with GUI objects. A timestamp was taken before each loop, and compared to a timestamp taken immediately afterwards.
Here’s the code for the test:
Dim i
Dim iTimer
Dim oEdit
'Regular OR
iTimer = Timer
For i = 0 to 50
Browser("Google").Page("Google").WebEdit("q").Set i
Next
Print "Regular OR =" & Timer-iTimer
'Descriptive Programming
iTimer = Timer
For i = 0 to 50
Browser("index:=0").Page("index:=0").WebEdit("name:=q").Set i
Next
Print "Descriptive Programming = " & Timer-iTimer
'OR via fixed reference
iTimer = Timer
Set oEdit = Browser("Google").Page("Google").WebEdit("q")
For i = 0 to 50
oEdit.Set i
Next
Print "OR via fixed reference = " & Timer-iTimer
'Descriptive programming via fixed reference
iTimer = Timer
Set oEdit = Browser("index:=0").Page("index:=").WebEdit("name:=q")
For i = 0 to 50
oEdit.Set i
Next
Print "Descriptive programming via fixed reference = " & Timer-iTimer
'Runtime through OR
iTimer = Timer
For i = 0 to 50
Browser("Google").Page("Google").WebEdit("q").Object.value = i
Next
Print "Runtime through OR = " & Timer-iTimer
'Runtime through Descriptive Programming
iTimer = Timer
For i = 0 to 50
Browser("index:=0").Page("index:=0").WebEdit("name:=q").Object.Value = i
Next
Print "Runtime through Descriptive Programming = " & Timer-iTimer
'Runtime through OR fixed reference
iTimer = Timer
Set oEdit = Browser("Google").Page("Google").WebEdit("q").Object
For i = 0 to 50
oEdit.value = i
Next
Print "Runtime through OR fixed reference = " & Timer-iTimer
'Runtime through DP fixed reference
iTimer = Timer
Set oEdit = Browser("index:=0").Page("index:=0").WebEdit("name:=q").Object
For i = 0 to 50
oEdit.value = i
Next
Print "Runtime through DP fixed reference = " & Timer-iTimer
Update:
Dani has suggested testing fetching the runtime object differently, through GetElement by ID:
iTimer = Timer
Set we = Browser("B").Page("P").Object.getElementById( "id" )
For I = 0 To 50
We.Value = i
Next
Print "runtime through the OR and GetElementByID = " & Timer-iTImer
iTImer = Timer
Set we = Browser("B").Page("P").Object.getElementByName( "name" )
For I = 0 To 50
We.Value = i
Next
Print "runtime through the OR and GetElementByName = " & Timer-iTImer
The Results
And here are the averaged results:
Regular OR = 5.59375
Descriptive Programming = 6.375OR via fixed reference = 3.31253
Descriptive programming via fixed reference = 3.25122
Runtime through OR = 2.90625
Runtime through Descriptive Programming = 3.921875
Runtime through OR fixed reference = 0.203125
Runtime through DP fixed reference = 0.265625
Update: Runtime through GetElementById = 0.20122; and through GetElementByName = 0.21019
Well, this turned up to pack some surprises, at least for me, but the bottom line remains the same – setting a variable to the GUI’s runtime objects is the most efficient method.
It seems that working directly through the object repository is faster than through descriptive programming (difference of almost a whole second – which quite shocked me, personally). This is reinforced by a more than a second gap when using OR and DP to access the WebEdit’s runtime object, however, the difference is wiped out by using a mediating variable.
It’s interesting to note that using the runtime object reduces the time by almost half, bringing it close to the performance of a fixed reference via a variable.
And the winner, by a full order of magnitude, is using the runtime object through a fixed reference variable. This is not surprising, as it avoids QTP’s GUI mapping mechanism completely (beside the first variable assignment).
I hope you enjoyed this article, and I’m looking forward to hearing more ideas for performance and other tests we can run on QTP (as opposed to with QTP)
Posted in General, QTP Techniques, Yaron Assa's Blog
December 4th, 2008 at 11:02 am
Fantastic work Yaron.
Now we only have the subjective argument of which approach is most appropriate for the team in question to support and debug.
December 4th, 2008 at 12:10 pm
Good article, I enjoyed to read it.
Thanks, Yaron :)
December 4th, 2008 at 12:25 pm
Yaron — Good article …
I am still puzzled at what it means to typical problems that people using QTP face …
In most of the Automation scenarios - fast execution of the automation testing is the least of concerns. Since most of the good automation is expected to run nearly unattended - it does not matter whether it takes 5 secs or 15 secs to complete a test.
Most of the cases the worry is about sychronization, free from false alarms, free from “Generic errors” or “object” not found errors.
How does this current study of yours and well explained ways of accessing GUI objects will help these QTP issues that are more painful and need help than speed issue.
Shrini Kulkarni
http://shrinik.blogspot.com
December 4th, 2008 at 1:36 pm
Interesting article!
To me test execution performance time matters a lot since “extra seconds” becomes minutes/hours when running a lot of tests in sequence. As a tester you want your test result ASAP for both manual and automated test execution.
/Stefan
http://abouttesting.blogspot.com/2008/05/qtp-test-execution-performance.html
December 4th, 2008 at 1:41 pm
Nice article Yaron.
Will and myself had a discussion on the same topic sometime back (in the comments)… http://www.learnqtp.com/2008/06/make-your-qtp-scripts-perform-better.html
December 4th, 2008 at 1:52 pm
Hi Yaron! Thanks for your systematical performance investigation and publishing the results. Good job.
However, I would recommend testing with a number of most frequently used page controls and operations with them. That might bring more reliable and proved results.
Thank you,
Alex.
December 4th, 2008 at 2:18 pm
This is very interesting. In the past I chose for DP approach instead of using QTP’s OR just for reason of performance.
However I’m very curious what the results would be if you do the same exercises on a more complex and extended script covering.
December 4th, 2008 at 5:12 pm
A very nice performance analysis Yaron! I was surprised to see that results indicated OR method consumes less time than DP method. I assumed DP is the best compared to OR method. But personally I would like to say here that in big QTP projects containing several hundred assertions DP method offers vb coding flexibility and saving of disk space - in this DP method better than using OR.
http://rameshnatesan.blogspot.com
December 4th, 2008 at 6:06 pm
Excellent article Yaron.
Some results are quite surprising.. didn’t know runtime OR would have such a huge difference in performance as compared to DP.
December 4th, 2008 at 7:18 pm
Yaron - Thanks. Great test
shrinik - You are right that this study doesn’t touch on other critical automation issues but even though performance should not and is not the main concern for good automated scripts but it does matter and this is how: if you are running 1000 scripts overnight, this study shows that the test would finish a lot sooner when using OR. This means that you don’t run into issues where your scripts are still running when morning batches kick in. How you handle sync points is a different subject and good automation should be independent of object interaction and object sync
Ramesh - I’m not sure what you mean when you say DP method gives flexibility and saves disk space. If you create objects for every thing starting from your window (that can be done in a global module or .vbs files) it doesn’t take that much more disk space and gives you the same flexibility as DP. In fact if you are creating your own objects it is less resourceful on the machines as all objects can and should be cleared from memory (when set to nothing). When you use DP, you do not clear memory and these objects are still sat in your memory. QTP does not (by default) clear memory
Saying that, down side of using objects is that in QTP, (self created) object approach only works on IE. It’s not cross browser compatible
December 5th, 2008 at 8:05 am
Fantastic work Yaron.
December 5th, 2008 at 7:47 pm
Great work Yaron.
It directly helped me to reduce my script performance drastically.
I had a script which have to run a look through 12000 records.
After applying Runtime through OR fixed reference method, it gave much better result.
Thanks a lot and looking for more such work!!
Raj
December 8th, 2008 at 6:25 am
Hi Yaron,
Good work yaron, In my case the fastest execution is “runtime through OR fixed reference”, below are the sequence of execution results.
Regular OR =3.796875
Descriptive Programming = 4.6875
OR via fixed reference = 2.359375
Descriptive programming via fixed reference = 2.3125
Runtime through OR = 1.875
Runtime through Descriptive Programming = 2.71875
Runtime through OR fixed reference = 0.078125
Runtime through DP fixed reference = 0.09375
But I try to use Dani method, but I am getting an error, that the method is not supported.
December 8th, 2008 at 8:45 am
Hi,I tried this for windows application.But “Object” method is not supporting.
Could you please suggest how to do for windows applications!
December 8th, 2008 at 8:50 am
Sorry, the windows environment doesn’t support access to runtime objects via .Object
December 8th, 2008 at 1:01 pm
By seeing this idea is excellent. Showing quantitative results given base for other thoughts.
Excellent.
December 9th, 2008 at 9:43 am
Excellent Stuff !!
December 9th, 2008 at 6:52 pm
A couple things I have noticed.
1. The speed in which each method works is dependent on the application. The OR is exponentially faster on AJAX / GWT applications.
2. The OR’s performance degrades as it increases in size.
December 11th, 2008 at 7:48 am
that’s awesome Yaron, thanks for that.
So basically, i call a ‘RegisterUserFunc’, replace my ‘WebEdit.Set’ with ‘object.value=’ and i’m done.
no code changes, and instanst performance improvement, great stuff.
December 15th, 2008 at 3:34 pm
Good article.
Thanks, Yaron
December 16th, 2008 at 10:41 am
Its really a useful info.thanks
December 17th, 2008 at 4:42 am
very good
many times my friends chat with me about this quetions
thanks a lot
January 7th, 2009 at 7:53 am
GREAT ARTICLE!!! Had a great time reading it and learning from it.
February 4th, 2009 at 3:42 am
Excellent article..really enjoyed…. :)
March 16th, 2009 at 7:50 pm
very interesting. Provides concrete evidence for efficient working with GUI objects.
Thank you
March 27th, 2009 at 9:51 pm
Really liked the article. Gave me a lot to consider in my next gig.
One of the biggest weaknesses to the typical use of descriptive programming I’ve seen is the maintenance hit. A typical developer will do this in-line and thus, when the test object changes, the developer has a lot of places he/she needs to update to fix the issue.
I’ve always went with the basic OR with a few Object ID tweaks, which seems to speed the OR a bit better.
But then I am less interested in test performance, that run automatically at 4AM, than I am with the maintenance cost to my clients.
AM
March 27th, 2009 at 9:59 pm
Runtime through OR fixed reference is what i do, tho I am interested in Dani’s strategy as well.
Question…
Why do you use the (.Object) at the end of the following?
Set oEdit = Browser(”Google”).Page(”Google”).WebEdit(”q”).Object
I use the following:
Set oEdit = Browser(”Google”).Page(”Google”).WebEdit(”q”)
I assume this is an implied modifier like the Value method for DataTables.
Am I right or am I missing something important???
AM
March 27th, 2009 at 10:09 pm
Okay one last thing to consider. If you use the prescribed method you’ll loose some of the auto-updating features in QTP. in other words, if you update an object in the OR, the reference in the code will not be updated in many cases.
For instance:
Set oPage = browser(”b”).Page(”P”)
oPage.WebEdit(”Edit1″)
If you change the name of the Edit1 control in the or, it will Not be updated automatically using fixed referencing. Personally i don’t care, but some of my buddies don’t like to loose this functionality.
I also do not generally fix ref my individual objects, only the pages. Perhaps i am loosing some performance doing so? Since I employ externalized/centralized code whenever possible (reusable code) it wouldn’t be a big deal to adopt fixed refs for all objects on a page for instance.
Have I lost performance by only going as far as the page objects?
As an aside, I originally adopted fixed refs as a way to make the code more readable and to shorten the lines of code in the days when we only had one small monitor and the code tended to run off the ‘page’ and not for speed at all (lol).
Now I use a multi monitor system with wide aspect monitors and that’s really not an issue so much anymore. I have my own lab at home and require my clients to at least have 2 monitor systems and two dev boxes before I start work (lol).
AM
April 6th, 2009 at 12:06 pm
Excellent comparison…..
April 8th, 2009 at 1:12 pm
By Runtime OR fixed reference can we use the Exist method
eg : Set oEdit = Browser(”Google”).Page”Google”).WebEdit(”q”).Object For i = 0 to 50
If oEdit.Exist then
oEdit.value = i
End if
Next
msgbox”Runtime through OR fixed reference = ” & Timer-iTimer
June 19th, 2009 at 3:16 pm
I repeated your tests for webextensibility. I used a webedit as the base class and identified the object using :
the code I used to run the test was :
iTimer = Timer
set oEdit = Browser(”index:=0″).Page(”index:=0″).myObject(”index:=0″).object
For i = 0 to 50
oEdit.value = i
Next
Print “Runtime through webext DP fixed reference = ” & Timer-iTimer
The results for DP were near enough identical to “Runtime through DP fixed reference” but interestingly when using OR with web extensibility the results were slightly slower (~0.01).
September 20th, 2009 at 7:31 am
excellent article,i like it.however, i wonder if from your sample(simple) case the conclusion can have meaning to more general and comlicated cases. vidbob test case somewhat hit my guess. thank you very much.
September 21st, 2009 at 8:07 pm
Thanks Yaron for such a nice analysis. But what about the scenario where we need to run multiple tests in batch and individual actions added to the batch, consists of local/shared OR. In that case it will take a bit more time to open the individual actions and there associated OR itself.
I guess, using DP will be a better option in this scenario. Will wait for your opinion on this. Thanks a lot.
January 13th, 2010 at 8:29 am
Simply Fantastic, fantabulous….great job. Honestly, didn`t know we can do the scrip0ting in so many ways.thouh i have used all :)
February 24th, 2010 at 7:59 am
I read it sometime back I remember but how could I didn’t left any Comment.
Yaron am using many of your approaches and getting praised for my automation approaches however I do mention your name in my Functions’ Header and in the starting of Function library that I took help from your and from AdvancedQTP.com.
It is better than Google for VbScript, Coding, OOPS, and QTP related queries.
Great Article and Great help !!!
I don’t know when will I think like that and write code like this, till then will learn.
-Subhash