Avoiding ‘Black Swans’ in the Software Testing Process

Technology firms that don’t follow and document strict, automated testing procedures are setting up themselves and their clients for failure. Here are 3 problems with the current testing and certification process.

Possibly one of the biggest technological advancements in financial technology, the introduction of the FIX protocol has allowed buy-side and sell-side firms to connect to each other in an infinitely quicker timeframe. For the most part, FIX engineering teams can get a certified connection in just a couple of days as opposed to spending weeks to months writing to a proprietary API.

One of the greatest strengths of the FIX protocol is that it’s a standardized communication link between software, but it does allow for flexibility. Most people think “FIX is FIX”; but most people don’t understand that flexibility in a standardized protocol leaves room for interpretation, which means firms can adjust and tweak their FIX engines to better suit their needs. There is nothing wrong with this, and we are constantly adjusting and tweaking our FIX engine to either conform to a counterparty’s needs or optimize on our end. The problem lies in the certification process we see in a majority of the exchanges and brokerages.

Let me walk you through a typical certification process:

  1. We reach out to counterparty with expressed interest in certifying with them.
  2. Counterparty responds with UAT credentials and its FIX spec documents.
  3. We spend up to a week tweaking and testing multiple use cases of our systems to replicate a live setting for the client, thus ensuring our system can facilitate all functionalities in the counterparty’s environment.
  4. We reach out to the counterparty to schedule the conformance test.
  5. The test usually consists of 10 to 15 manual tests verifying basic order submissions, rejection handling, cancels, replacements, partial fill handing, etc. The basic workflow messaging in any trading environment, from order submission to matching, is tested to confirm the two parties’ systems can communicate as they should.
  6. If you fail in any of the sub-tests, you must reschedule the test when you have the issue resolved.

Although it may only take our team a few days to successfully connect, the actual process could take a few weeks, depending on how prompt the connecting partner is in responding to issues or questions we have.

There are three major issues with the overall testing and certification process:

  1. The certification process is too vague and misses many use cases. Ten to 15 basic manual tests are just not enough.
  2. There is no standard test, no actual industry certification.
  3. Like a trendy NYC restaurant, we’ve seen some brokers booking certifications months out!

This content is for TabbForum Membership members only.
Log In Register
TabbFORUM is an open community that provides a platform for capital markets professionals to share their ideas and thought leadership with their peers. The views and opinions expressed are solely those of the author(s). They do not necessarily reflect the opinions of TABB Group, its analysts, TabbFORUM and its editors, or their employees, affiliates and partners.


Add a Comment