Application Testing

Application TestingApplication Testing
Reggie Niccolo SantosReggie Niccolo Santos
UP ITDCUP ITDC

OutlineOutline
The Testing PhaseThe Testing Phase
Testing your Web AppsTesting your Web Apps
Testing Strategies and Tactics for Mobile ApplicationsTesting Strategies and Tactics for Mobile Applications
ReferencesReferences

In many software engineering methodologies, aIn many software engineering methodologies, a
separateseparate phase performed by aphase performed by a differentdifferent team afterteam after
the implementation is completedthe implementation is completed

Regression TestRegression Test
Internal TestingInternal Testing
Unit TestingUnit Testing
Stress TestingStress Testing

Form aForm a suitesuite of programs that test one or moreof programs that test one or more featuresfeatures ofof
the systemthe system
A valid regression test generatesA valid regression test generates verified resultsverified results
““gold standardgold standard””
Validity driven by the rValidity driven by the requirement documentequirement document
In practice, theIn practice, the implementation teamimplementation team is responsibleis responsible
for validity interpretationfor validity interpretation

As development continues, more tests are added, while oldAs development continues, more tests are added, while old
tests may remain validtests may remain valid
Because of new development, an old test may no longerBecause of new development, an old test may no longer
be validbe valid
If this is the case, the old test results areIf this is the case, the old test results are alteredaltered in thein the
“gold standard” to“gold standard” to matchmatch the current expectationsthe current expectations
The test suite is run generating new resultsThe test suite is run generating new results

Make sure allMake sure all internal, non-customer-visibleinternal, non-customer-visible
componentscomponents work wellwork well
Deals withDeals with low-levellow-level implementationimplementation
EachEach functionfunction oror componentcomponent is testedis tested
Accomplished by theAccomplished by the implementation teamsimplementation teams

Clear-box/white-box testingClear-box/white-box testing
All details areAll details are visiblevisible to the testto the test
Internal limitsInternal limits are tested hereare tested here

Make sure allMake sure all customer-visible componentscustomer-visible components workwork
wellwell
Deals with testing aDeals with testing a unit as a wholeunit as a whole
Test theTest the interaction of many functionsinteraction of many functions butbut confineconfine
the test withinthe test within one unitone unit
Supporting test code, sometimes calledSupporting test code, sometimes called scaffoldingscaffolding,,
may be necessary to support an individual testmay be necessary to support an individual test

Driven by theDriven by the architecture and implementationarchitecture and implementation
teamsteams
Black-box testingBlack-box testing because only thebecause only the details of thedetails of the
interface are visibleinterface are visible to the testto the test
Limits that areLimits that are globalglobal to a unit are tested hereto a unit are tested here

Make sure the application can completeMake sure the application can complete allall
scenariosscenarios
Driven by scenarios from theDriven by scenarios from the analysis teamanalysis team
Application limits and featuresApplication limits and features are tested hereare tested here

MustMust successfully execute all scenariossuccessfully execute all scenarios before it isbefore it is
ready for general customer availabilityready for general customer availability
Represents theRepresents the bulkbulk of the testing done by the industryof the testing done by the industry
Unlike the internal and unit testing, which areUnlike the internal and unit testing, which are
programmed, these tests are usuallyprogrammed, these tests are usually driven by scriptsdriven by scripts
that run the system with a collection of parameters andthat run the system with a collection of parameters and
collect resultscollect results

Run the application in an environment that isRun the application in an environment that is moremore
stressfulstressful than the target environmentthan the target environment
Deals with theDeals with the qualityquality of the application in theof the application in the
environmentenvironment
Requires a joint effort fromRequires a joint effort from all teamsall teams

Test environment established withTest environment established with many testingmany testing
stationsstations
At each station, aAt each station, a script is executingscript is executing the systemthe system
These scripts are usually based on the regression suiteThese scripts are usually based on the regression suite
More and more stations are added, all simultaneouslyMore and more stations are added, all simultaneously
hammering on the system, until the systemhammering on the system, until the system breaksbreaks

The system is repaired and the stress test is repeatedThe system is repaired and the stress test is repeated
until a level of stress is reached that isuntil a level of stress is reached that is higher thanhigher than
expectedexpected to be present at a customer siteto be present at a customer site
Race conditionsRace conditions andand memory leaksmemory leaks are oftenare often
found under stress testingfound under stress testing

Race ConditionRace Condition
Conflict betweenConflict between at least twoat least two teststests
Each test works correctly when done inEach test works correctly when done in isolationisolation
When the two tests are run in parallel, one or both ofWhen the two tests are run in parallel, one or both of
the teststhe tests failfail
Usually due to anUsually due to an incorrectly managed lockincorrectly managed lock

Memory LeakMemory Leak
Happens when a testHappens when a test leaves allocated memoryleaves allocated memory
behindbehind and does not correctly return the memory toand does not correctly return the memory to
the memory allocation schemethe memory allocation scheme
The test seems to run correctly, but after beingThe test seems to run correctly, but after being
exercised several times,exercised several times, available memory isavailable memory is
reducedreduced until the system failsuntil the system fails

1.1. ObjectivesObjectives
Establish your testing objectivesEstablish your testing objectives up frontup front and make sureand make sure
that they arethat they are measurablemeasurable
Make sure your objectives areMake sure your objectives are prioritizedprioritized
Ask yourself questions like:Ask yourself questions like:
1.1. ““What is most important:What is most important: minimal defectsminimal defects oror time-to-time-to-
marketmarket?”?”

2.2. Process and ReportingProcess and Reporting
Make sure that everyone on your testing teamMake sure that everyone on your testing team
knows his or herknows his or her rolerole
WhoWho should reportshould report whatwhat toto whomwhom andand whenwhen??
Define yourDefine your testing processtesting process

Guide questions:Guide questions:
HowHow will issues be reported?will issues be reported?
WhoWho will assign issues?will assign issues?
HowHow will issues be categorized?will issues be categorized?

WhoWho needsneeds whatwhat report andreport and whenwhen do they need it?do they need it?
2.2. Are team meetings scheduledAre team meetings scheduled in advancein advance or scheduledor scheduled asas
neededneeded??
Organize your team in a way thatOrganize your team in a way that supportssupports your testingyour testing
objectives and takes into account theobjectives and takes into account the individualindividual personalitiespersonalities
on your teamon your team

3.3. Tracking ResultsTracking Results
You will want a way to easilyYou will want a way to easily store, organize andstore, organize and
distribute informationdistribute information about bugs, issues, andabout bugs, issues, and
defects to the appropriate technical team membersdefects to the appropriate technical team members
You will also need a way toYou will also need a way to keep managementkeep management
informedinformed of the status of your testing effortsof the status of your testing efforts
e.g.e.g. AdminiTrack.comAdminiTrack.com

4.4. Test EnvironmentTest Environment
Separate from yourSeparate from your developmentdevelopment andand productionproduction
environmentenvironment
Includes aIncludes a separate web server, databaseseparate web server, database
server, and application serverserver, and application server if applicableif applicable

4.4. Test EnvironmentTest Environment
Create an explicitly defined procedure forCreate an explicitly defined procedure for movingmoving
codecode to and from your test environment and maketo and from your test environment and make
sure the procedure is followedsure the procedure is followed
Work with your development team to make sureWork with your development team to make sure
each new version of source code to be tested iseach new version of source code to be tested is
uniquely identifieduniquely identified

5.5. Usability TestingUsability Testing
Looking at aspects of your web application that affect theLooking at aspects of your web application that affect the
user’s experienceuser’s experience
HowHow easyeasy is it to navigate through your web application?is it to navigate through your web application?
Is itIs it obviousobvious to the user which actions are available to himto the user which actions are available to him
or her?or her?

5.5. Usability TestingUsability Testing
Is the look-and-feel of your web applicationIs the look-and-feel of your web application consistentconsistent
from page to page, including font sizes and colorsfrom page to page, including font sizes and colors
Design forDesign for accessibilityaccessibility especially if it is a legalespecially if it is a legal
requirementrequirement
http://www.w3.org/TR/AERThttp://www.w3.org/TR/AERT

6.6. Unit TestingUnit Testing
Focused on verifyingFocused on verifying smallsmall portions of functionalityportions of functionality
e.g. range checkinge.g. range checking

7.7. Verifying the HTMLVerifying the HTML
World Wide Web ConsortiumWorld Wide Web Consortium's free HTML's free HTML
Validation ServiceValidation Service
http://validator.w3.orghttp://validator.w3.org
Net MechanicNet Mechanic
http://www.netmechanic.comhttp://www.netmechanic.com

Make sure that your syntax is correctMake sure that your syntax is correct
Opening and closing tags matchOpening and closing tags match

Verify how your pagesVerify how your pages looklook inin different browsers, atdifferent browsers, at
different screen resolutionsdifferent screen resolutions, and on, and on differentdifferent
operating systemsoperating systems
Create aCreate a profileprofile of your target audience and makeof your target audience and make
some decision on what browsers you will support, onsome decision on what browsers you will support, on
which operating systems, and at what screen resolutionswhich operating systems, and at what screen resolutions

8.8. Load TestingLoad Testing
You want to simulate how users will use your webYou want to simulate how users will use your web
application in theapplication in the real worldreal world
TheThe earlierearlier you perform the load testing the betteryou perform the load testing the better
Performance tuningPerformance tuning should be tightly integratedshould be tightly integrated
with the design of your applicationwith the design of your application

8.8. Load TestingLoad Testing
Try to make sure that all of your pages load inTry to make sure that all of your pages load in 1515
seconds or lessseconds or less
Depends on yourDepends on your particular applicationparticular application and theand the
expectations of the people using itexpectations of the people using it

9.9. User Acceptance Testing (UAT)User Acceptance Testing (UAT)
You are making sure your web applicationYou are making sure your web application fitsfits thethe
use which it was intendeduse which it was intended
One way is to set up aOne way is to set up a beta testbeta test

10.10. Testing SecurityTesting Security
You need to test howYou need to test how securesecure your web applicationyour web application
is from bothis from both externalexternal andand internalinternal threatsthreats
Should be planned for and verified byShould be planned for and verified by qualityquality
security specialistssecurity specialists

Additional resources:Additional resources:
CERT Coordination CenterCERT Coordination Center
http://www.cert.org/http://www.cert.org/
Computer Security Resource CenterComputer Security Resource Center
http://csrc.nist.gov/http://csrc.nist.gov/

Make sure to also performMake sure to also perform ongoing securityongoing security
auditsaudits to ensure your web application remainsto ensure your web application remains
secure over time as people and technology changesecure over time as people and technology change

Testing Strategy and TacticsTesting Strategy and Tactics
for Mobile Applicationsfor Mobile Applications
1.1. Goal is not to find errors but to understand theGoal is not to find errors but to understand the
qualityquality of your offeringof your offering
Does itDoes it workwork??
Does it functionDoes it function as expectedas expected??
Will it meet theWill it meet the needsneeds of your users, so that theyof your users, so that they
come back again and again?come back again and again?

2.2. Mobile Testing for Native ApplicationsMobile Testing for Native Applications
The app download is often controlled by theThe app download is often controlled by the gate-gate-
keeping app storekeeping app store, with mechanisms in place to, with mechanisms in place to
charge potential consumerscharge potential consumers

Needs to ascertain that the app can beNeeds to ascertain that the app can be successfullysuccessfully
downloadeddownloaded to the device,to the device, executedexecuted on the device andon the device and
interactinteract with the supporting back content infrastructurewith the supporting back content infrastructure
When updates are made, you need to be sure that theWhen updates are made, you need to be sure that the
application can beapplication can be pushed out topushed out to andand acceptedaccepted by theby the
end userend user

Tied to the hip with theTied to the hip with the hardwarehardware andand operatingoperating
systemssystems for which they are writtenfor which they are written
Test on theTest on the physical devicesphysical devices supported by yoursupported by your
applicationapplication
EnsureEnsure backward compatibilitybackward compatibility with each olderwith each older
generation of the device you are expected to supportgeneration of the device you are expected to support

Ensure that when issues are found with your nativeEnsure that when issues are found with your native
apps, they can beapps, they can be quickly capturedquickly captured andand sharedshared
with otherswith others

3.3. Mobile Testing Challenges for Web ApplicationsMobile Testing Challenges for Web Applications
Understand theUnderstand the globalglobal dynamicdynamic

4.4. Devices: The Biggest Mobile Testing ChallengeDevices: The Biggest Mobile Testing Challenge
There areThere are potentially tens of thousands ofpotentially tens of thousands of
different client devicesdifferent client devices that could be used tothat could be used to
access your mobile app or website, and they mustaccess your mobile app or website, and they must
therefore all be considered when testing your mobiletherefore all be considered when testing your mobile
applicationsapplications

Can beCan be reducedreduced to an extent, but each time youto an extent, but each time you
reduce the number of device types that you testreduce the number of device types that you test
against, you are taking a chance that your applicationagainst, you are taking a chance that your application
might not workmight not work on a device, locking out a number ofon a device, locking out a number of
potential consumerspotential consumers

3 options:3 options:
Test exclusively usingTest exclusively using realreal devicesdevices
Test exclusively withTest exclusively with emulatedemulated devicesdevices
Use aUse a combinationcombination of eachof each

Have the advantage of having all theHave the advantage of having all the limitationslimitations
andand quirksquirks present in the actual client hardware andpresent in the actual client hardware and
firmware combination in the hands of your targetfirmware combination in the hands of your target
consumersconsumers
Can beCan be expensiveexpensive

Can beCan be disorganizeddisorganized andand labor-intensivelabor-intensive if theif the
testing environment is not conducive to creating,testing environment is not conducive to creating,
collecting and reproducing results in a consistentcollecting and reproducing results in a consistent
mannermanner

Test exclusively usingTest exclusively using emulatedemulated devicesdevices
RelativelyRelatively easiereasier to manageto manage
You can switch device types by simplyYou can switch device types by simply loading aloading a
new device profilenew device profile
4.4. Run on moreRun on more powerful PCspowerful PCs andand serversservers

Were designed forWere designed for testingtesting in mindin mind
TypicallyTypically fully instrumentedfully instrumented to capture detailedto capture detailed
diagnostics about the protocols that go back and forthdiagnostics about the protocols that go back and forth
between client and server at the various levels of thebetween client and server at the various levels of the
application stackapplication stack

Cost effectiveCost effective because a single platform with frequentbecause a single platform with frequent
updates of device profiles can be used to test every deviceupdates of device profiles can be used to test every device
on the market both today and tomorrowon the market both today and tomorrow
Big disadvantage is that they lack theBig disadvantage is that they lack the quirks, faultsquirks, faults andand
characteristicscharacteristics that only the real device can providethat only the real device can provide

May not give theMay not give the pixel-perfect accurate renderingpixel-perfect accurate rendering that youthat you
are assured to have with a real device solutionare assured to have with a real device solution
Processing power of a local PC canProcessing power of a local PC can hidehide any issues that youany issues that you
may have with themay have with the responsivenessresponsiveness of your web applicationof your web application
Not sensitiveNot sensitive to theto the ambient conditionsambient conditions that can impact thethat can impact the
behaviour of the devicebehaviour of the device

Use aUse a combinationcombination of eachof each
Start testing in an emulated environment to take advantage of theStart testing in an emulated environment to take advantage of the
speedspeed andand device diversitydevice diversity that an emulator can providethat an emulator can provide
Add real devicesAdd real devices into your test plan later in the developmentinto your test plan later in the development
cycle so you can validate the applications are functioning ascycle so you can validate the applications are functioning as
expected and certify that all development requirements andexpected and certify that all development requirements and
objectives have been metobjectives have been met

5.5. Network: A Regional ChallengeNetwork: A Regional Challenge
Each mobile operator mayEach mobile operator may support multiple networksupport multiple network
technologiestechnologies including LTE, CDMA, GSM, and some use lessincluding LTE, CDMA, GSM, and some use less
common or local networking standards such as iDEN, FOMA,common or local networking standards such as iDEN, FOMA,
and TD-SCDMAand TD-SCDMA
Each network has a unique combination of networkEach network has a unique combination of network
infrastructure thatinfrastructure that tunnels the packet-based protocolstunnels the packet-based protocols usedused
by mobile networks into TCP-IP protocols used by the mobileby mobile networks into TCP-IP protocols used by the mobile
WebWeb

When implemented, canWhen implemented, can restrictrestrict the flow ofthe flow of
information that travels between your server and theinformation that travels between your server and the
test clienttest client
Some limit the sites that can be accessed via a phoneSome limit the sites that can be accessed via a phone
to only those approved by the operator (“to only those approved by the operator (“walledwalled
gardengarden”)”)

Others might use “Others might use “transcodingtranscoding” in an attempt to scale” in an attempt to scale
down fixed Web content to better fit onto mobile phones,down fixed Web content to better fit onto mobile phones,
thus expanding the number of mobile sites that can bethus expanding the number of mobile sites that can be
seenseen
SomeSome strip vital informationstrip vital information from the HTTP headersfrom the HTTP headers
that your application might depend on to providethat your application might depend on to provide
functionality or to provide device adaptationfunctionality or to provide device adaptation

To fully test the full network stack on a particularTo fully test the full network stack on a particular
operator’s network infrastructure, you must beoperator’s network infrastructure, you must be
connected to the target networkconnected to the target network
Traveling to every network operator that you need toTraveling to every network operator that you need to
support can besupport can be very expensivevery expensive

We canWe can bypassbypass the lower layers of the network andthe lower layers of the network and
simply test over the Internet or LANsimply test over the Internet or LAN
You useYou use TCP/IPTCP/IP to connect directly to the serverto connect directly to the server
and youand you ignoreignore the GPRS tunnelling systems used bythe GPRS tunnelling systems used by
network operatorsnetwork operators

You will need to use aYou will need to use a device simulatordevice simulator to perform theto perform the
bypassbypass
Advantage is that you willAdvantage is that you will not need to usenot need to use andand thus pay forthus pay for
airtimeairtime
Disadvantages are that is that we oftenDisadvantages are that is that we often cannot emulatecannot emulate thethe
effects and timing of the network and the various networkeffects and timing of the network and the various network
elements such as proxieselements such as proxies

We can use theWe can use the real networkreal network by using either phones orby using either phones or
modemsmodems
Many device emulatorsMany device emulators support modemssupport modems that allow youthat allow you
to use your emulated devices on the local networkto use your emulated devices on the local network
But again, there is the cost ofBut again, there is the cost of travelingtraveling into range of theinto range of the
networknetwork

One piece of useful test equipment is aOne piece of useful test equipment is a real device in thereal device in the
cloudcloud
Consists of aConsists of a physical handsetphysical handset mounted in a remote box withmounted in a remote box with
aa remote control unitremote control unit and aand a remote antennaremote antenna
The remote control unit is physically connected to theThe remote control unit is physically connected to the device’sdevice’s
screen and keypad control circuitsscreen and keypad control circuits and is capable ofand is capable of
pressing keys and collecting screen imagespressing keys and collecting screen images

Often have the ability toOften have the ability to record a testrecord a test for subsequentfor subsequent
replayreplay
Reduce theReduce the cost of travelcost of travel to foreign networksto foreign networks
Most of the companies that make this type of equipmentMost of the companies that make this type of equipment
offer the ability to “offer the ability to “rentrent” testing time on a resource that is” testing time on a resource that is
shared with others and is managed for youshared with others and is managed for you

6.6. Scripting: The Repeatability ChallengeScripting: The Repeatability Challenge
The method that is actually used toThe method that is actually used to execute the test scriptexecute the test script
Script execution can either beScript execution can either be manualmanual oror automatedautomated
You either write down the scripts in a document or aYou either write down the scripts in a document or a
spreadsheet, which is then used by a test engineer whospreadsheet, which is then used by a test engineer who
manually enters keystrokesmanually enters keystrokes, or you, or you run automatedrun automated
scriptsscripts that in turn evoke the keystrokes and record thethat in turn evoke the keystrokes and record the
resultsresults

6.6. Scripting: The Repeatability ChallengeScripting: The Repeatability Challenge
Most emulators are capable of automating testMost emulators are capable of automating test
execution using aexecution using a higher-level, abstract scriptinghigher-level, abstract scripting
languagelanguage that is not device dependentthat is not device dependent
Many automated scripting tools have a special abilityMany automated scripting tools have a special ability
to “to “spiderspider” or “” or “crawlcrawl” a mobile Web site” a mobile Web site

7.7. SummarySummary
Invest in aInvest in a device emulatordevice emulator
Take advantage ofTake advantage of remote real devices in theremote real devices in the
cloudcloud
AutomateAutomate wherever possiblewherever possible

SummarySummary
7.7. SummarySummary
Invest in aInvest in a device emulatordevice emulator
Take advantage ofTake advantage of remote real devices in theremote real devices in the
cloudcloud
AutomateAutomate wherever possiblewherever possible

SummarySummary

SummarySummary
ObjectivesObjectives
Process and ReportingProcess and Reporting
Tracking ResultsTracking Results
Test EnvironmentTest Environment
Usability TestingUsability Testing

SummarySummary
Verifying the HTMLVerifying the HTML
Load TestingLoad Testing
User Acceptance Test (UAT)User Acceptance Test (UAT)
Testing SecurityTesting Security

SummarySummary

http://infolab.stanford.edu/~burback/watersluice/node19.htmlhttp://infolab.stanford.edu/~burback/watersluice/node19.html
10 Tips for Testing Apps for the Real World10 Tips for Testing Apps for the Real World
http://www.techrepublic.com/blog/10-things/10-tips-for-testing-apps-for-the-real-world/#http://www.techrepublic.com/blog/10-things/10-tips-for-testing-apps-for-the-real-world/#
http://www.adminitrack.com/articles/TestingYourWebApps.aspxhttp://www.adminitrack.com/articles/TestingYourWebApps.aspx
Web UsabilityWeb Usability
"Don't Make Me Think! A Common Sense Approach to Web Usability" by Steve Krug and Roger Black"Don't Make Me Think! A Common Sense Approach to Web Usability" by Steve Krug and Roger Black
http://www.keynote.com/mobile-app-testing.htmlhttp://www.keynote.com/mobile-app-testing.html

Application Testing

More Related Content

What's hot (20)

Similar to Application Testing (20)

More from Reggie Niccolo Santos (15)

Recently uploaded (20)

Application Testing