Negative Tests

Background & Motivation

Negative tests are a crucial part of application testing – they ensure us the application contains “bad” user behavior and illegal inputs in a controlled fashion. Having said that, usually there’s never enough time, resources and personal to automate the “positive” tests; let alone the negative ones. This is not an inevitable situation – if we’ll write our positive tests in a generic and sophisticated manner, extending them into a set of negative tests will be fairly easy, and will require only minimal effort and resources.

For example, let’s take a banking account management application. One of the basic tests would involve creating a new account, by entering the costumer’s details, and assuring that the account was successfully created. If we’ll build the test correctly, we could immediately create a set of negative tests at almost no cost. We could validate mandatory fields, field value range, trying to create a double account etc.

Designing and building negative tests is a huge and complex subject, and implementing it can vary widely from project to project. I won’t try to explain all of it, but rather focus on a few important points and their implications. After the conceptual points, there’ll be a small example that better illustrates them (hopefully).

So, what does it take to derive a negative test from a positive one?

Design concept

When extending our positive test, we must abide by the basic code-design principles. Specifically, we must make sure that introducing the negative test will not break the black-box design and separation of concerns in the positive test’s “engine”.

On the one hand, our original test should remain “stupid” – a factory that receives execution instructions and an expected result, and works upon them. For the factory, there mustn’t be any difference between a negative test to a positive one, besides the content of the execution instructions, and the expected result.

On the other hand, the external code calling the test shouldn’t know or care how the test works. As far as it concerns, the test is a magical black box which receives inputs, and spits outputs.

So, how can we implement these design concepts?

Wrapping the test

The best way to extend a positive test to a negative one is to wrap the positive’s action/function with an additional layer, which will be used to run the negative test. For example, if we have a CreateAccount action, we should wrap it with a NegativeCreateAccount action. This layer will mediate between the negative inputs to the test-engine, and analyzing its outputs. If we’ll try to spare the extra layer, we’ll be forced to introduce negative-test logic into the positive test’s engine, thereby creating a maintenance hell.

Separation to cases

A single positive test might extend to multiple negative tests. For example, a positive CreateAccount test has many negative tests: not filling mandatory fields, inputting the wrong type of data (string instead of a number), trying to create two accounts for the same client etc. In this case, we’d like to make sure the test failed (more precisely succeeded to fail) for the right reason. If we didn’t enter the NAME mandatory field, and the account wasn’t created because there’s another account with a null name, then this in an Unexpected result, even though the test “successfully failed”.

It could be very tempting to code the different cases right into the original test-engine. You should avoid this at any cost! It will turn the code into an unreadable, tangled mess. Instead, we can make sure the original test reports enough data on the specific error that occurred, so the wrapper (NegativeCreateAccount) can analyze it and make sure it is the expected error. A good way for doing this is to report back the actual text of the error presented to the user, or a code-word describing the stage in which the error occurred. This way the wrapper can sort through the messages with a simple IF or Select-Case structure.

Creating negative data, focusing the test

Much like the separation to cases, generating data for the negative tests should also be done outside the original test-code. You might want to generate the data in the test wrapper, or have it sent-in from the external code calling the test. Either way, you’d want to focus and limit the possible variations of negative data. There are hundreds of thousands of combinations, and you sure can’t run through all of them.

For example, let’s say we got an account form with 10 fields from 4 different types, 5 of which are mandatory. All the possible combinations where some mandatory field is missing, or where there’s a mismatch between an input to a field type amount to 1953125 cases. Well, good luck with testing that.

However, focusing on checking each of the mandatory fields (separately) and a sample of non-valid input can lower that into a much saner total of 15 cases.

If you are considering generating negative values automatically, you can make the generating code much simpler by eliminating special treatment for mandatory fields.

You can simply define mandatory fields as a new type of field, which cannot accept null values (like date fields cannot accept strings). This way the code is more coherent and uniform.

Reporting errors

This turns out to be a tricky business. On the one hand, we’d like the positive test to report any error it encounters. On the other hand, we’d like the same code to carry out negative tests, where some of the errors are actually a success, and some are still reportable errors.

Here also, it’s imperative to keep the separation to different cases of negative tests outside. We should try to narrow the problem to two simple cases – either we report all the errors, or we don’t.

In order to differentiate the two possibilities, I recommend adding an ExpectedResult parameter to any relevant action / function. Whenever the test will encounter an error, it will check the parameter. If it’s true, then we got an unexpected error on our hands, and we should report it. If it’s false, then we expected some kind of error, but we can’t be sure if it’s the right one (because the case-separation logic is outside the test). So we report a Pass to the QTP log, and send the error details to the calling negative-test wrapper. The wrapper then analyzes the error, and if it’s not the one expected, reports a Fail to the QTP log.

Splitting the responsibility between the test-engine and the negative-wrapper, allows us to implement very complex reporting logic, without complicating and tangling our code, and enables us to integrate our code into a global framework of error analysis.

Resetting the application

Some applications are extremely sensitive to negative testing. We should always keep in mind that unlike manual negative tests, automatic negative tests are not meant to break the system. Make sure that you reset the application back to a state where it can undergo positive tests, and that no negative-data residues are left in the different screens the forms.

Summary

As usual, this article ends with the warning that the methods and ideas covered here are just the tip of the iceberg. As mentioned, designing and building negative tests is a huge and complex subject, and no finite amount of text can cover all the niches and possibilities.

I hope the conceptual ideas presented, coupled with the practical example will make it easier for you to build negative tests. The bottom line is that building your positive tests in a generic fashion can allow you to extend them into negative tests with little or no overhead.

Example

Below is an example for the conceptual ideas we’ve covered. Download the demo application, copy these code snippets into QTP, and run them. As you can see, it’s easy to add more types of negative tests, without touching the positive test “engine”. The example also demonstrates how strings can be used to pass complex data to functions.

First, we got the positive test:

'Actually tests the form. 

'sFieldsData = Field1>Value1|Field2>Value2|…

Function InputForm(sFieldsData, bExpectedResult)

    Dim arrFieldsData, arrInnerData

    Dim i

    Dim sFieldName, sValue

    Dim sSystemMessage

    arrFieldsData = Split(sFieldsData, "|") ‘Breaks the string into separate fields

    'Resets the form for the test. 

    VBWindow("vbname:=NegativeTests").VBButton("text:=Clear").Click

    'Input Fields

    For i = 0 to uBound(arrFieldsData)

      arrInnerData = Split(arrFieldsData(i), ">")

      sValue = arrInnerData(1)

      sFieldName = arrInnerData(0)

      VBWindow("vbname:=NegativeTests").VBEdit("vbname:=" & sFieldName).Set sValue

    Next

    'Submit Form

    VBWindow("vbname:=NegativeTests").VBButton("text:=Submit").Click

    'Get Message

   sSystemMessage = VBWindow("vbname:=NegativeTests").Dialog("micclass:=Dialog").Static("micclass:=Static").GetROProperty("text")

   wait 2

   VBWindow("vbname:=NegativeTests").Dialog("micclass:=Dialog").WinButton("text:=OK").Click

    If bExpectedResult = True Then

        InputForm = (sSystemMessage = "Form submitted")

        'Returns True if the message is "Form submitted", False otherwise

        'Notice that if we’re doing "positive" testing, the calling code doesn’t know

        'what is a "successful" system message. This information hides with-in our blackbox

    Else

      InputForm = sSystemMessage

      'Exectly the opposite - we want to keep the black-box "stupid", so if we’re

    'doing negative testing, the function doesn’t mess with the message, only

    'reports it back to where the negative-logic is

    End If

End Function

Then, we have the negative wrapper:

'Negativly tests the form

Function NegativeInputForm(sTestType)

    Dim sNegativeData

    Dim sExpectedMessage

    Dim sActualMessage

    Dim bResult

   Select Case sTestType

        Case "Mandatory Title"

             sNegativeData = "txtName>Name"

             sExpectedMessage = "Please enter all mandatory fields!"

        Case "Mandatory Name"

             sNegativeData = "txtTitle>Title"

             sExpectedMessage = "Please enter all mandatory fields!"

        Case "Wrong Type"

             sNegativeData = "txtAge>String|txtName>Name|txtTitle>Title"

             sExpectedMessage = "Age should only contain digits!"

        Case "Bad Example"

             sNegativeData = "txtAge>String"

             sExpectedMessage = "Please enter all mandatory fields!"

            'This will fail - the message will be about the age, though it’s

            'Supposed to be first about mandatory fields

    End Select

   sActualMessage = InputForm(sNegativeData, False)

   Select Case sActualMessage

        Case True

            Msgbox "Test was supposed to fail, but passed"

             bResult = False

        Case sExpectedMessage

            Msgbox "Negative test passed" & vbcrlf & "Press OK to continue"

             bResult = True

        Case else

            msgbox "Bad result!" & vbcrlf & _

                "Expected to get: " & sExpectedMessage & vbcrlf & _

                "But got: " & sActualMessage

             bResult = False

    End Select

   NegativeInputForm = bResult

End Function

And finally, the actual function calls for the demonstration:

'Performing "Positive" test that will pass: 

print InputForm("txtName>Name|txtTitle>Title|txtAge>24", True)

'Performing "Negative" test that will pass: 

print NegativeInputForm("Mandatory Title")

print NegativeInputForm("Mandatory Name")

print NegativeInputForm("Wrong Type")

'Performing "Negative" test that will fail: 

print NegativeInputForm("Bad Example")