Likewise, we should not narrow our view of testing as a means to eviscerate defects. Rather, testing is one aspect of a broader topic, called Quality Assurance(QA), which is the process to ensure that the delivered system conforms to requirements specified by target specifications (including functional, non-functional, regulatory, business objectives, coding standards, …). The process starts at project kickoff and continues through the system's operational lifecycle until decommissioning. Testing is a quality control tool to measure compliance to specifications, but in and of itself does not fix anything. A good testing process reveals developed system behavior, problem areas and specific defects: A quantitive view of system's conformance to specification and verification that system deliver required capability.
But, as an industry are we investing enough in QA to mitigate potential future failures? It appears not. The recent NASDAQ's Facebook IPO glitch, BATS own IPO failure , Knights Capital trading glitch, and BATS order mispricing do not share a root cause and are individually and uniquely serious failures. They have received extensive press coverage, incurred (or potentially incur) financial impairment, and caused reputation damage.
Potential causes for these failures:
* Stakeholder expectations leading to ambiguous or conflicting requirements,
* Insufficient test coverage and result analysis,
* Rushing system release (especially with known defects),
* Inadequate operational guidance to handle potential problem scenarios.
I can't definitely state that these reasons caused the above failures, but these are common QA deficiencies. In particular, disparate stakeholder (inclusive of business, regulators, operations, and users) expectations and the demand for rapid changes can lead to defects, which may manifest immediately, or more concerning, cause irrefutable damage over time. Components, such as trading platforms, matching engines, order routing systems, and market distribution systems will fail to properly operate, and when they do fail, we should have well defined vetted guidelines to handle these failures. Like FMECA, we should identify potential problems and identify actions and compensating provisions to remedy their impact, including at the market level.
Yet, even increased investment in QA will not guarantee absence of faults. As we started with, bug-free software is a fallacy; testing can prove the existence of defects, but can't prove non-existence. We can increase investment in the QA function to improve system resiliency, but ultimately, we need to test at the market level to ensure that component defects are controlled and limited. Many years ago, trading systems were simpler and we regularly conducted industry-wide tests; this would be very difficult to accomplish today. We should build to keep the market up, expect failures, but be ready to handle failures quickly and effectively. Achieving this goal requires a market-level QA process and matching investment to support this process.