Magento Integration Tests

Editor's Notes

• #3: [Greetings]Who of you are Magento developers who customize Magento for their clients or implement Magento extensions? (hands)Good! *You* are the intended audience for this topic.The topic is – integration tests in Magento – intended for developers who would like to try automated tests in Magento and possibly start writing tests for their own Magento extensions TODAY.
• #4: 1) We’ll start from an overview of automated test types. What are the goals and how they are different.2) Then we’ll stop by on integration tests more in detail.3) I’m going to demonstrate integration testing framework in Magento and explain and how to use it.4) Then developers who are interested in creating tests, may find useful an explanation of the framework design.5) Finally, we’ll see where to get Magento integration tests and their documentation.So let’s proceed with overview of automated tests.
• #5: Automated tests is essential part of contemporary software, especially of large products. Development teams implement automated tests along with the product and put them on rails of continuous integration to ensure that several important concerns are addressed:1) Whether features of the product work as intended.2) Whether it is reliable.3) Does it perform good enough.4) And whether it is compatible with intended environments.
• #6: Functional tests is the most high-level type. Using functional tests, the product features are tested in the way as if user would interact with the system.The simplest implementation of functional tests is “manual”: a human (QA engineer) actually interacts with the system by a defined scenario or combination of multiple scenarios. Automation of these tests is usually implemented by recording QA engineer interaction with the system and then refining it with variables, that would allow to run these tests later on, automatically on different environment.
• #7: Performance testing is a special approach, when certain functional test (or we can call it “business scenario”) is executed multiple times in various environment conditions and various frequency to determine performance bottlenecks.There is a great variety of performance tests. For example:1) “Performance” tests execute the same scenario on regular basis and compare results over time2) “Load” tests assure that scenario runs successfully under normal and peak loads3) “Capacity” tests allow to determine the practical limit of algorithm load, while scenario executes successfully4) “Stress” tests inspect system behavior when scenario fails under enormous load
• #8: Unit tests are supposed to verify small portions of algorithm by executing its code with different combinations of arguments and asserting return values.Unlike functional tests, the unit tests are written by programmers and *for* programmers. They are to be used every day as a safety net when modifying code to preserve algorithm stability and backwards compatibility.Developing code along with unit tests cultivates better programming skills and results in better designed code.
• #9: Static code analysis tests perform syntax analysis of the source code without executing it to determine whether it is implemented according to coding standard and certain conventions.Like unit tests, this is also a “programmer-oriented” type, which helps team of developers to follow common standard, when working on source code together.There are tools for programming languages, that allow to perform static code analysis. For example Magento 2, as it is implemented on PHP, uses 3 tools:- PHP CodeSniffer – to enforce coding standard- PHP Mess Detector – to determine poor-designed code- PHP Copy-Paste detector – well, to detect copy-paste
• #10: Now let’s take a closer look at a different type of automated tests – integration tests.
• #11: The goal of integration tests is to execute programming code, deployed on close to real environment. In other words, in integration tests the product is *intended* to interact with environment.This type of tests concentrates on how the system works as a whole, rather than how its modules or *layers* would work by their own. It doesn’t aim to execute all combinations of algorithm – in integration tests it is enough to execute the code by “critical path”.A good example of interaction with environment is to run the same test suite, deployed on different DB platforms. Of course if the application has DB abstraction layer.* “Critical path” term in graph theory is the path of maximum length in an oriented acyclic graph. So if to consider an algorithm as such graph, in integration tests we want to execute the maximum amount of code in one pass.
• #12: Integration tests bring up value, a bit different from other types of tests.From business prospective, the integration tests can give measurement of “how stable the product is”. The user or product owner may not want to see detailed report of code coverage, they may prefer something simple, like one number. Percent of code coverage by integration tests can be this simple number, that gives some general idea about product stability.Integration tests have nature similar to unit tests, because they execute code in similar way. But they also allow to discover bugs, dependent on certain environment. Otherwise such bugs would emerge after product release, on customer environments, which is not a pleasant situation.
• #13: Unlike in unit tests, the less environment and objects are mocked during execution, the better it is for integration tests.Execution of unit tests has narrow scope of a module or class, while integration tests pierce through system architectural layers and touch as much modules as possible.
• #14: This is what kind of interaction with environment to expect when running integration tests:1) Before running test suite the product is to be deployed and installed2) It should interact with real database3) It is to utilize file system and all kind of caching
• #15: A use case from Magento 2 project.Magento is a product that supports multiple database platforms. Use of integration tests in continuous integration process allows to spare efforts of programmers from having to verify their code on all platforms all the time.A programmer can develop on his favorite DB platform, while continuous integration will execute integration tests on all platforms (and report errors).
• #16: So this kind of integration testing is implemented in Magento 2 by a test suite that works on “integration testing framework”.Let’s see how this framework can be used: we’ll review initial configuration, setup environment and then run the test suite.
• #17: Magento integration testing framework is built on top of PHPUnit framework, therefore the latter needs to be pre-installed. The PHPUnit version 3.5 or later is needed.Before running test suite, Magento application instance will be automatically installed into an empty database. And to operate, Magento requires a writable folder to store its cache and logs.
• #18: Integration tests and framework reside in “dev/tests/integration” folder, relative to Magento application root.Here we distinguish several major areas:1) The test suite – a folder where we can find a set of regular PHPUnit test cases, which are direct descendants of “PHPUnit_Framework_TestCase” class. One of key points in Magento integration tests is that we tend to stick with standard PHPUnit features as much as possible.2) Under “framework” folder there is “bootstrap.php”, a traditional thing to use with PHPUnit. And under “framework/Magento” there are classes, that implement Magento-specific PHPUnit customizations. The “framework/tests” folder contains unit tests for the integration testing framework, it has no relation with the integration test suite.3) Also there are configuration XML-files, which are used to customize Magento application framework. We’ll stop by on them in next slides.4) The “tmp” folder is that writable folder I mentioned before. Magento application instances are actually stored in “tmp” sub-folders.
• #19: PHPUnit features a fallback mechanism for configuration files. When you run “phpunit” from command line, it searches for “phpunit.xml” file or, if not found, for “phpunit.xml.dist” file. This allows to fiddle with configuration: out of the box Magento integration tests have some default presets in the “.xml.dist”, which can be locally overridden by copying it as “.xml” file.Integration testing framework replicates this behavior for “etc/local.xml” files.
• #20: Before running tests first time we start from setting up configuration in phpunit.xml. The PHPUnit framework allows to define constants in this file and use them in tests source code.There are a few such custom constants:1) “TESTS_LOCAL_CONFIG_FILE” specifies path to Magento local.xml configuration prototype, which contains database credentials2) The “TESTS_GLOBAL_CONFIG_FILES” and “TESTS_MODULE_CONFIG_FILES” allow to adjust the way how Magento reads configuration. Out of the box it says to load all XML-files from the specified folders3) “TESTS_SHUTDOWN_METHOD” is an optional constant, that allows to specify whether to uninstall or “restore database” after test suite execution. “Uninstalling” means a full cleanup of database and its temporary folder, “restoreDatabase” will result in pumping in the first database dump, that was created before executing tests first time.4) Also there are optional constants, that allow to collect profiling results in a CSV-file and use it, for example, to build some performance metrics.
• #21: Credentials for database, where application will be installed in, are specified in the “etc/local-<vendor>.xml” files, which are prototypes for Magento local.xml files.Note that besides section with DB credentials, this file also contains information about installation date and encryption key. It is important to not modify them, because valid installation date and encryption key values indicate that application is already installed and there is no need to run SQL-setup scripts.
• #22: Now we know enough to run integration tests.
• #23: Now that we know how to run integration tests in Magento, let's see from programmer's standpoint why there was need to build something on top of PHPUnit framework and how it is designed.
• #24: The first what integration testing framework does is bootstrapping Magento application. A fully-automatic installation, full cleanup or just database restoration. Entry point for this is the “framework/bootstrap.php” file, which has “Magento_Test_Bootstrap” under the hood. Besides of performing bootstrapping routines, this object can disclose customized paths to writable folders, where application stores cache and temporary files.Magento_Test_Db classes are database adapters, that are used by bootstrap for dealing with database cleanup and handling backups.
• #25: Executing a lot ofMagento code in one scoop without mocking anything is a quite challenging task:- there is application object with certain state,- some code may rely on certain configuration state,- and there is code that relies on objects, existing in database.To address these difficulties and simplify programmer’s task, the integration testing framework has a “Listener” – a standard way to customize PHPUnit framework behavior. As you may know, the PHPUnit framework utilizes PHPDoc annotations to execute some additional commands.So Magento_Test_Listener class aggregates custom PHPDoc annotations, that implement useful Magento-specific testing features.
• #26: @magentoAppIsolation annotation specifies to reset state of registry, re-instantiate application object and partially cleanup cache.Cleaning up global objects manually without this annotation would increase code complexity, especially that programmer would have to implement try-catch code blocks everywhere, because PHPUnit assertion can throw an exception. This example illustrates how compact can a test be in virtue of @magentoAppIsolation.This directive can be specified for tests (methods) and has “enabled” and “disabled” values, “disabled” by default. Isolation cleanup is performed automatically before executing test case classes.
• #27: To get into some conditional block, which is on critical path of algorithm, sometimes a system configuration value needs to be set up first. That’s a lot of code, especially considering the “try-catch” necessity.The @magentoConfigFixture annotation purpose is to tackle this problem: it allows to set system configuration value before test and reverts it back after test is complete (regardless of it failed or not).It can be specified on test case level or test level, or both (test level overrides test case level). A value can be set directly into global configuration, “current” store view configuration or other specified store view.
• #28: Some tests make sense only when there is an object existing in database, so that it can be loaded and asserted. Creating and persisting an entity would obfuscate and complicate the test, and again – the programmer would face with the “try-catch” problem when data is to be reverted.In this case the @magentoDataFixture directive comes for help. It works like an “include” script just wrapped into a database transaction:- Start transaction before including the script- Run the test- Revert transaction after test is executedOne important thing to know when composing such a script: use only the application API. No interaction with database directly. This will make tests more durable against database structure changes and independent from specific DB vendor.
• #29: The integration testing framework also contains a few utility classes, that would help to develop certain types of tests:- Abstract controller test case – allows to dispatch Magento controller actions, assert redirects and 404 result, as well contains HTTP-request and response stubs- Magento_Test_Entity is a helper to test CRUD operations with a typical model-entity, that is based on “Mage_Core_Model_Abstract” Other classes include special test suite that allows to exclude test groups using list of enabled/disabled Magento modules, a base class for code integrity tests and other smaller helpers.
• #30: An example: testing controller actions using “Magento_Test_TestCase_ControllerAbstract”. Among invoked methods, we can see a few custom ones, introduced by this class:- dispatch() – allows to dispatch a controller action by specifying just request URI- assert404NotFound() – analyse response (stub) object to determine whether it has 404 status- assertRedirect() – assert the response stub object for redirect- And there are getters and setters for request and response objects
• #31: Another example demonstrates how to use Magento_Test_Entity.- Constructor requires a Mage_Core_Model_Abstract instance (which represents an entity) and an array with data to change and assert during “update” part of the test- The “testCrud()” method starts test and assertions.It consecutively attempts to perform the following actions:1) Persist object in database and assert its new ID (the “create” operation)2) Read object from database by this ID3) Update object data from the passed array and persist it in database as “update” operation4) Delete the object from database.
• #32: Separately I’d like to mention code integrity tests. This is a sub-type of integration tests, that scan entire code base and verify compliance with certain framework feature.For example, the system has template engine and its components may introduce templates. A code integrity test would scan entire code base for templates declaration and verify whether they exist, execute them and check whether they don’t produce errors.Another example – scanning view files for references to static resources, such as images, CSS and JavaScript files and verifying that these files are accessible. Such test helps to find and eliminate broken references.The primary difference of code integrity tests from static code analysis is that code is executed in the process: the code works as an instrument to test itself.
• #33: A few last notes for programmers who would like to start implementing own integration tests.1) Always keep in mind the isolation problem: integration tests interact with environment, but still they should be isolated between each other.2) When dealing with fixtures or data providers, never perform direct operations with database. Otherwise tests will be more fragile.3) Keep performance of tests adequate to running on a local developer’s machine.
• #34: At this point I’ll just announce where to get the integration tests and framework, and documentation.
• #35: The code is publicly available in Magento 2 project. Here you can see Magento 2 SVN public repository URL.But soon it will be gone. We already internally have switched to Git and soon will replace this link with a URL to Git repository.Stay tuned.
• #36: Documentation for integration tests is available in product documentation wiki space, developer’s guide. There are several integration test guides, which are part of general-purpose “Magento Automated Testing Guide”.These documents have all the same information I presented today, just more detailed.The link will be announced, as soon as we deploy it on public server. Our sysadmins are working on it right now.