Visual Studio Team System 2010 – The Testing Angle

I attended a briefing event on Visual Studio Team System 2010 today (22/7). The event was focused on the testing features of Visual Studio 2010, and while it only scratched the surface, it made quite an impression on me, and made me realize the Microsoft has finally pulled a potential “Quality Center Killer” from their labs.

Not only that, but the new Visual Studio 2010 has some killer-features which even compete with HP’s QTP; And while Microsoft tools are focused on .Net applications, in that specific arena, QTP doesn’t come out as a clear winner. In fact, even from the short brief I saw, QTP even lost on a couple of rounds.

I reported the event through twitter, and even though it doesn’t directly deal with QTP, I though it’s worth grouping my tweets and posting them here. I’d like to thank all of you who followed me live, and interacted with the reports, and especially @xerocube.

The tweets are shown in chronical order (oldest appear first). Feel free to follow me on twitter for future reports on events, and other automation news at http://twitter.com/yassa.

At Microsoft, in a convention on VS Team System 10.
VS 2010 focuses on the middle ground between testers and automation experts.
Focuses on MTLM – Microsoft Test Lab Manager. Not fully Visual Studio, but a central management framework for tests and test processes.
It seems that MTLM is MS contester against HP’s quality center.
The UI is slick enough http://twitpic.com/b7f0q.
The fact that many of the fields in the different forms come automatically from VS makes it simpler than QC for the average tester.
Very nice – very easy to configure automatic data collectors which makes it extremely easy to reproduce a bug.
You can even run a “profiler” which can add a log of the internal code structures which ran when the bug occurred – all automatically!
Automatic data collectors http://twitpic.com/b7f9f.
They keep hinting that the virtual environment settings is a killer feature – will be in the last lecture.
Did MS hire a former Apple designer for VS 2010? Big buttons, very clean UI, very consistent, very impressive.
Very easy to add parameters to tests – just write them with @ (e.g. @Param), and you got a parameter.
The manual test runner is quite nice. Has a “Start test and record”, which seems to be able to replay manual tests (semi QTP contester?)
Extremely efficient UI for the manual runner.
Manual runner http://twitpic.com/b7frg.
The automatic runner seems to work OK, but the demonstration is on the calculator. Let’s ask how it deals with changes in the UI.
The auto runner IDs  GUI controls by their backend name. If that changes, the DLL is considered 2 B “significantly difft”, & the test breaks.
However, you always know beforehand if the test broke or if it will work.
Creating a bug http://twitpic.com/b7g4b.
Even though a bug has many fields, most of them are automatically field, with very relevant information.
Very impressive – the logger records every keystroke, ON EVERY APPLICATION in the computer, not only the .Net AUT.
The quality dashboard is quite informative http://twitpic.com/b7glb
It seems that many of HP QC’s core features are now available from MS.
VS monitors significant changes to the assemblies, and recommends test suites that will provide the optimal regression coverage.
Shared steps – VS way for calling tests from other tests, with parameter support.
Shared steps are more flexible than QC’s “call to test”, since you can mark specific steps as shared, and not an entire tests.
Now they talked about what all this will support. NO IE6, NO flash / java, and unbelievably, NO Silverlight for the foreseeable future!
Surprisingly, Firefox 3 will be supported from the go-live.
Nice terminology, “no support, ever, never ever” = “An opportunity for partners”.
Endless and quite uninformative Q&A session
Very easy to create parameter iterations, either from VS, or by pointing to an external excel file.
Ha, very amusing, the Right To Left support pretty much sucks. At a Hebrew presentation. To an Israeli crowed. Pathetic.
Back from break, to automation session.   http://twitpic.com/b7itj
Starting with Unit Testing.
An amusing side note – the presentation computer has NOD32 installed on it, and it reported an error. FAIL.
This is weird, the presenter spends 10 minutes explaining what Unit Tests are to a room full of QA and automation project managers.
Very impressive debugging history in the Unit Test.
Automatic unit testing generation http://twitpic.com/b7jzc.
As usual, all unit tests relay on the Asset command, which reports the result back to the test manager.
Well, this is disappointing – this is actually almost identical to VS 2008. When I confronted the presenter, he kinda admitted it.
Amusing, after dissing me for bringing with a Mac in a MS building, the MS presentation has an image of a Macbook Pro on several slides
A Macbook Pro starring on a formal Microsoft presentation http://twitpic.com/b7kka
Now moving on to the new stuff – Coded UI testing
Well, this was quick – the second sentence was “This is up against QTP”.
The coded UI “Will not be coded with VBScript, but with real languages”.
UNBELIEVABLE – STILL NO SUPPORT FOR SILVERLIGHT. NOT EVEN FROM MICROSOFT.
The Coded UI testing will work very well even on third party custom controls (non standard wpf and windows forms).
Now demonstrating on a Demo WPF app.
Adding UI test http://twitpic.com/b7l5k.
The half transparent step recorder http://twitpic.com/b7l7x.
Very intuitive UI recorder.
It seems that the recorded steps have a lot of X,Y data in them (screen coordinates). Will ask about the robustness.
The UI test code http://twitpic.com/b7lg4.
The UI test can build UI maps for any windows application.
Very impressive data extraction from the application. However, only for .Net http://twitpic.com/b7ls2
What happens behind the scenes? You create controls in the code, and each control has a “search criteria” region which defines its ID props
The automatic comments added by VS are actually helpful. Quite the opposite of TestPartner, for example.
No built in support for regular expressions or wildcards in the ID properties of controls.
Now going through load testing.
Several load patterns we can use, constant, step pattern (increase users every X seconds), and goal (increase until response time changes).
You can define a very complex logic for which test would run after which.
This is actually a lot better than Load Runner – we can create very meaningful test mixes automatically.
Network emulator for client server apps! http://twitpic.com/b7n8x
The network emulator for client-server applications is integrated perfectly in defining the test, load and users mix.
Very detailed customizations of the different network types – packet losses, latency, TTL, and much more, all controlled via XML.
Very intuitive load wizards http://twitpic.com/b7nm1
Unlike Loadrunner and parallel load tools, VS allows you to run load with unlimited virtual users.
Amazing wizards for setting the user, script, and load mix http://twitpic.com/b7nsu
Adding monitors to remote computers http://twitpic.com/b7nz4
The monitors setup is quite nice, very intuitive, as these things go.
Counters are very neatly ordered, and available from the project tree http://twitpic.com/b7ob0
Oh, an innocent question reveals what I think can be a No-Go problem – load agents need the full VS suite, not just an agent component.
Nice analysis reports, quite similar to Load Runner http://twitpic.com/b7oku
The load agents report their counters during the test run. Nice feature, but it could really skew the application and network performance.
Off to a much needed break, the projector is giving me headaches.
Me and Vitaly at the convention http://twitpic.com/b7q10
Now Shai Mandle from Microsoft India will present on the VS Team-Test Lab Management
Lab management should make it much more easy to deploy machines and simulate installs
Lab management could automatically use Virtual Machine templates to simulate builds, installs and different environments.
Creating virtual environments automatically as part of the build release process. http://twitpic.com/b7qo1
The Virtual environments are automatically separated from the major network to prevent name and IP conflicts.
Adding virtual machines to a virtual environment is very easy http://twitpic.com/b7qwy.
Controlling multiple machines from the lab manager http://twitpic.com/b7r0e.
The virtual technology is currently only Windows Server 2008, but SP1 is planned to support VMWare as well.
The deeper implications of virtual deployments are quite substantial.
When reporting a bug, the information collector automatically gathers all the information on all the machines in the virtual environment.
OMG – the developer can click a link in the bug, and go back in time to how the environment was WHEN THE BUG WAS DETECTED.
The “go back to environment state” includes the memory snapshot of every server, so you could start remote debugging immediately!
Go back in time to the environment’s state when the bug was recorded.  http://twitpic.com/b7rla
I just think of all the problems the “go back in time” feature solves, and I’m actually at tears.
Bug traceability problems will be cut by some 99%. It’s absolutely groundbreaking.
Automation for the build-deploy-test cycle http://twitpic.com/b7s27
The virtual environment deployment opens the door to a very automated Build-Deploy-Test cycle.
Very detailed settings for the virtual build-deploy cycle http://twitpic.com/b7sjz
Managing the virtual build environments.  http://twitpic.com/b7sv6
And the million dollar question (literally) – how much would the hardware needed to run this cost.
The key is a HyperV enabled hardware (basically, any core due processor), with tons of memory.
A killer presentation. Almost too good to be true.
The event is over.I’m off to take care of my splitting headache.I’ll post these tweets as an article in AdvancedQTP.com . Thanks 4 following