Why do I care about full stack functional testing?Because you really care about the quality of your product but you also care about the cost of QA. The most cost efficient option is probably full stack functional testing, which isn't exactly easy, but hey, you are a hacker, aren't you?
For the sake of conciseness, "full stack functional testing" will be mostly referred as "functional testing" in the remainder of this article.
Why unit tests, no matter how good they are, can't replace real QA?There are many reasons, here are several examples:
- It's common that when writing unit tests, developers use mocks to isolate the modules they are testing against and leave the integration points between modules untested. It is perfectly normal that you can have 100% unit tests coverage and your application is still completely broken.
- Unit tests, by its nature, do not cover integration points with external modules/APIs and thus do not protect your application from the behavior changes from them when you upgrade them. Traditionally developers are cautious about upgrading external libraries because often times it will trigger the need for a full regression tests.
- The logic unit tests are testing is sometimes too far away from the behavior logic end user is interacting with.
- Unit tests do not provide enough confidence during major refactoring especially when a significant number of unit tests themselves need to be changed due to the refactoring.
- For web applications, unit tests do not test cross browser compatibility.
Why full-stack functional tests? Does it count to have functional tests against a lower layer, e.g. the web service layer?Modern web applications have lots of UI logic happening in the browser (written in JS) and the integration between the UI and backend web service is too complex not to test. The point of full-stack functional tests testing from UI layer all the way to the persistence layer is that they perform the same interaction against the application exactly like real users (just faster). When they pass, you have 100% confidence your application works. Leaving out any layers will cost you that ultimate purpose.
Why functional tests are cheaper than manual QA?You are a hacker, you know the motto: automation, automation, automation. Automated regression tests is by several magnitudes faster than manual regression tests. Actually, since the scope of regression tests will keep increasing along with the growth of the application, the time manual regression tests take will soon start to hurt the application's time to market. So, if you care about both quality and time to market, automated functional tests is the only salable way to do regression tests.
Why full stack functional tests are often claimed to be too fragile to worth it?Functional testing is significantly different from unit testing which developers are more used to. The main factor that makes functional testing harder is the amount of uncertainties in functional testing:
- The simple fact that they test end to end means that there are a lot more internal and external factors can impact functional tests.
- It's harder for your to always test on a clean set of data - the amount of data needed for each functional test forbid you from setting up data on every test.
- UI interaction is asynchronous and the response time isn't exactly deterministic. Tests could fail randomly when the assertion is made too early, or, put in another way, the expected results show up too late.
How to improve the stability of functional tests over time?When a test fails due to the undeterministic nature of functional tests, instead of running it again and stops when the test passes, take the time to improve the stability. A very important tool to have here is the ability to automatically do screen captures when a functional test fails because you might not be able to repeat it easily.
Sometimes it takes some guessing to figure out what happened, but most of such "random" failings are caused by the fact that the responsiveness of UI isn't deterministic. To give a more specific example, suppose your test clicks a button on the UI and assert that some expected result. It could fail randomly when your assertion is made "faster" than the response. A common technique is to poll UI with certain frequency for a period of time, during which the test keeps reading the UI for some indication that the application has fully responded. Then the test can assert the expected result. This polling ability is a must for your functional test framework.
Why is it the developers' responsibility to write functional tests?There are at least 3 reasons:
- Only developers can make sure the UI is automatically testable and easily update functional tests according to UI change.
- Functional tests are code, lots of them. Writing readable functional tests requires at least basic OO design skills.
- Improving the stability of functional tests is a tough job that requires virtuosity in debugging, patience and the confidence to overcome technical challenges. Most people with such capabilities tend to developers, usually pretty good ones.
If functional tests cover everything, do I still write unit tests?The purpose of functional tests and unit tests in TDD is very different. Functional tests are mainly for code coverage to ensure functionality. In TDD, the main purpose of unit tests is to drive development, and the code coverage provided is more like a side effect. Thus functional tests cannot replace unit tests for that purpose. That being said, the code coverage provided by functional tests can help TDD because developers can now focus more on the drive-development purpose when writing unit tests without having to also keep code coverage in mind. Arguably, with functional tests you can chose other development methodologies without much immediate risk introduced to the quality of the software. So it opens up more options for the developers.
If the developers are writing functional tests that covers everything, do I still need dedicated QAs?You will have more edge case issues experienced by your end users without dedicated QAs or dedicated QA process. Developers are not trained, or interested, in testing the system against all special cases. Edge case issues will be discovered and fixed more in an on demand fashion. A good strategy would be to automatically monitor your application closely and alarm your developers (and/or automatically rollback changes) whenever any abnormal things happen, such as exceptions, sudden user activity changes and so on. Then your developers can fix them a.s.a.p and add functional tests accordingly. This approach is arguably more agile because you don't spend the cost in front to fix some edge issues that may never be met by the real end users.
What tools are needed for writing functional tests for a web application?Just a couple:
- Selenium - selenium 2.0 (webdriver) is much better and quite stable, although the documentation isn't that great (here are some tips)
- Chromedriver - faster than Firefox, but has some limitation comparing to Firefox whose selenium driver is the most mature one.
- Your favorite BDD framework.
Why BDD framework over Unit Testing framework when it comes to functional tests?Although using unit test structures to organize functional tests should work, it is more natural to organize functional tests into contexts and scenarios, which BDD frameworks usually support better.
Is there any good pattern in writing functional tests?The basic idea is to first model the application UI you are testing against and then write tests against that model. A popular pattern based on this idea is called page object pattern. In this pattern, you write classes to represent UI pages so that
- Detailed UI interaction implementation is hidden in these classes, e.g. the logic of how to locate a button.
- Only business meaningful methods are exposed, e.g. submit_order or set_shipping_address
class CheckoutPage def set_shipping_address(address) browser.find_element("input#street").send_keys(address[:street]) browser.find_element("input#city").send_keys(address[:city]) end def use_shipping_as_billing=(same) checkbox = browser.find_element("#same-shipping-billing") checkbox.click if checkbox.selected != same end def submit_order browser.find_element("input[type='submit']").click end endThen you can write tests like
checkout_page = CheckoutPage.new checkout_page.set_shipping_address(street: "20 Jane St", city: "London") checkout_page.set_credit_card_info(cc_num: "12345678", exp: "6/12") checkout_page.use_shipping_as_billing = true thank_you_page = checkout_page.submit_order thank_you_page.thank_you_message.should be_displayedAs you can see, by separating the modeling of the application from the tests, you can have much better readable tests as well as reusable UI manipulation code.
Of course, you don't have to stop at the page level, you can also create partial(or control) objects to model a smaller part of the page. In one sentence, use your OO skill to model the pages/ui components in the most sensible way.
Is it possible to let the business people read or even write functional tests?No. That's not going to work and don't waist your time or money on that. The goal for functional testing is to QA the product, it'll be wise not to get distracted by adding more responsibilities for functional testing.
Any comments are welcome. updated with one more Q&A about why only full-stack functional testing has to include every layer.