Full stack functional testing for hackers

Why do I care about full stack functional testing?

Because you really care about the quality of your product but you also care about the cost of QA. The most cost efficient option is probably full stack functional testing, which isn't exactly easy, but hey, you are a hacker, aren't you?

For the sake of conciseness, "full stack functional testing" will be mostly referred as "functional testing" in the remainder of this article.

Why unit tests, no matter how good they are, can't replace real QA?

There are many reasons, here are several examples:

• It's common that when writing unit tests, developers use mocks to isolate the modules they are testing against and leave the integration points between modules untested. It is perfectly normal that you can have 100% unit tests coverage and your application is still completely broken.
• Unit tests, by its nature, do not cover integration points with external modules/APIs and thus do not protect your application from the behavior changes from them when you upgrade them. Traditionally developers are cautious about upgrading external libraries because often times it will trigger the need for a full regression tests.
• The logic unit tests are testing is sometimes too far away from the behavior logic end user is interacting with.
• Unit tests do not provide enough confidence during major refactoring especially when a significant number of unit tests themselves need to be changed due to the refactoring.
• For web applications, unit tests do not test cross browser compatibility.

In one sentence, you can't be serious if you plan to live without QA because "my application is pretty well covered by unit tests."

Why full-stack functional tests? Does it count to have functional tests against a lower layer, e.g. the web service layer?

Modern web applications have lots of UI logic happening in the browser (written in JS) and the integration between the UI and backend web service is too complex not to test. The point of full-stack functional tests testing from UI layer all the way to the persistence layer is that they perform the same interaction against the application exactly like real users (just faster). When they pass, you have 100% confidence your application works. Leaving out any layers will cost you that ultimate purpose.

Why functional tests are cheaper than manual QA?

You are a hacker, you know the motto: automation, automation, automation. Automated regression tests is by several magnitudes faster than manual regression tests. Actually, since the scope of regression tests will keep increasing along with the growth of the application, the time manual regression tests take will soon start to hurt the application's time to market. So, if you care about both quality and time to market, automated functional tests is the only salable way to do regression tests.

Why full stack functional tests are often claimed to be too fragile to worth it?

Functional testing is significantly different from unit testing which developers are more used to. The main factor that makes functional testing harder is the amount of uncertainties in functional testing:

• The simple fact that they test end to end means that there are a lot more internal and external factors can impact functional tests.
• It's harder for your to always test on a clean set of data - the amount of data needed for each functional test forbid you from setting up data on every test.
• UI interaction is asynchronous and the response time isn't exactly deterministic. Tests could fail randomly when the assertion is made too early, or, put in another way, the expected results show up too late.

Thus debugging functional tests is more challenging than debugging the more deterministic unit test. It is very common that functional suites start fragile and take time to improve gradually until its robustness and stability reach a satisfactory level. Yes, there is a learning curve for developers how are used to writing unit tests, but it's not a "forget it, it's not worth it" one. All you need is the right developers.

How to improve the stability of functional tests over time?

When a test fails due to the undeterministic nature of functional tests, instead of running it again and stops when the test passes, take the time to improve the stability. A very important tool to have here is the ability to automatically do screen captures when a functional test fails because you might not be able to repeat it easily.

Sometimes it takes some guessing to figure out what happened, but most of such "random" failings are caused by the fact that the responsiveness of UI isn't deterministic. To give a more specific example, suppose your test clicks a button on the UI and assert that some expected result. It could fail randomly when your assertion is made "faster" than the response. A common technique is to poll UI with certain frequency for a period of time, during which the test keeps reading the UI for some indication that the application has fully responded. Then the test can assert the expected result. This polling ability is a must for your functional test framework.

Why is it the developers' responsibility to write functional tests?

There are at least 3 reasons:

• Only developers can make sure the UI is automatically testable and easily update functional tests according to UI change.
• Functional tests are code, lots of them. Writing readable functional tests requires at least basic OO design skills.
• Improving the stability of functional tests is a tough job that requires virtuosity in debugging, patience and the confidence to overcome technical challenges. Most people with such capabilities tend to developers, usually pretty good ones.

What does it take to successfully write and maintain functional tests?

As mentioned above, the biggest challenge in functional testing is the debugging bugs that appear to be random. Virtuosity in debugging, patience and confidence in tackling technical challenges are what you should look for when finding developers that can successively execute the use-functional-tests-for-QA strategy. They usually go together. If a developer easily gets frustrated or even annoyed by the undeterministic nature of functional tests, s/he won't be able to effectively debug it and improve the robustness of it, and in turn to make herself to believe that functional testing is always going to be fragile and thus won't work.

If functional tests cover everything, do I still write unit tests?

The purpose of functional tests and unit tests in TDD is very different. Functional tests are mainly for code coverage to ensure functionality. In TDD, the main purpose of unit tests is to drive development, and the code coverage provided is more like a side effect. Thus functional tests cannot replace unit tests for that purpose. That being said, the code coverage provided by functional tests can help TDD because developers can now focus more on the drive-development purpose when writing unit tests without having to also keep code coverage in mind. Arguably, with functional tests you can chose other development methodologies without much immediate risk introduced to the quality of the software. So it opens up more options for the developers.

If the developers are writing functional tests that covers everything, do I still need dedicated QAs?

You will have more edge case issues experienced by your end users without dedicated QAs or dedicated QA process. Developers are not trained, or interested, in testing the system against all special cases. Edge case issues will be discovered and fixed more in an on demand fashion. A good strategy would be to automatically monitor your application closely and alarm your developers (and/or automatically rollback changes) whenever any abnormal things happen, such as exceptions, sudden user activity changes and so on. Then your developers can fix them a.s.a.p and add functional tests accordingly. This approach is arguably more agile because you don't spend the cost in front to fix some edge issues that may never be met by the real end users.

What tools are needed for writing functional tests for a web application?

Just a couple:

• Selenium - selenium 2.0 (webdriver) is much better and quite stable, although the documentation isn't that great (here are some tips)
• Chromedriver - faster than Firefox, but has some limitation comparing to Firefox whose selenium driver is the most mature one.
• Your favorite BDD framework.

Given that the problem functional testing is trying to solve is already complex enough, it would be wise to keep the technology stack as simple as possible. The need for any extra layers on top of Selenium or BDD is very limited. You might need to write some simple helper methods to hide some boilerplate code needed for selenium, but not much more.

Why BDD framework over Unit Testing framework when it comes to functional tests?

Although using unit test structures to organize functional tests should work, it is more natural to organize functional tests into contexts and scenarios, which BDD frameworks usually support better.

Is there any good pattern in writing functional tests?

The basic idea is to first model the application UI you are testing against and then write tests against that model. A popular pattern based on this idea is called page object pattern. In this pattern, you write classes to represent UI pages so that

• Detailed UI interaction implementation is hidden in these classes, e.g. the logic of how to locate a button.
• Only business meaningful methods are exposed, e.g. submit_order or set_shipping_address

For a simplified ruby example, here is a page object for a checkout page

  class CheckoutPage
    def set_shipping_address(address)
      browser.find_element("input#street").send_keys(address[:street])
      browser.find_element("input#city").send_keys(address[:city])
    end

    def use_shipping_as_billing=(same)
      checkbox = browser.find_element("#same-shipping-billing")
      checkbox.click if checkbox.selected != same
    end

    def submit_order
      browser.find_element("input[type='submit']").click
    end
  end

Then you can write tests like

  checkout_page = CheckoutPage.new
  checkout_page.set_shipping_address(street: "20 Jane St", city: "London")
  checkout_page.set_credit_card_info(cc_num: "12345678", exp: "6/12")
  checkout_page.use_shipping_as_billing = true
  thank_you_page = checkout_page.submit_order
  thank_you_page.thank_you_message.should be_displayed

As you can see, by separating the modeling of the application from the tests, you can have much better readable tests as well as reusable UI manipulation code.

Of course, you don't have to stop at the page level, you can also create partial(or control) objects to model a smaller part of the page. In one sentence, use your OO skill to model the pages/ui components in the most sensible way.

Is it possible to let the business people read or even write functional tests?

No. That's not going to work and don't waist your time or money on that. The goal for functional testing is to QA the product, it'll be wise not to get distracted by adding more responsibilities for functional testing.



Any comments are welcome. updated with one more Q&A about why only full-stack functional testing has to include every layer.