One of the most interesting problem areas has always been around market data. What are the prices? What were they? Who bought what? Who's selling and why? How does the back testing look?
As trading technology advances, the number of messages grow. As the messages grow, volumes follow. Volumes lead to more quotes. Market segmentation encourages more quoting venues. The world of trading has become the world of the best information in the fastest times … to receive it and act upon it.
Today’s markets throw off an ever-increasing amount of data that always contains a relative advantage to those who can fully absorb, analyze, understand and mine it. As volatility spikes become the norm, trading segments seek to analyze more events, compare venues with regions and assets to trade. As such, market data, the normalization of it and the storage and the analysis of it puts more and more stress on the enterprise. Not to mention the integrity of the analytics.
The current state of the art is that data is brought to the analytics. Data is kept somewhere deep in an archive, a database or storage unit. It is then removed and brought to the analytics in piecemeal to determine risk, build strategy, back test the strategy, view compliance metrics and on down to proper settlement.
Ideally, faster systems and/or larger capacity systems now available can help with growing data sets but moving data across the enterprise continues to put so much strain and latency on the process that most data is actually discarded and new data used, thus sacrificing information for performance.
Further, storage capacities have to be built out to allow mining of the growing “big data,” with average costs well above $100,000 per terabyte. Then there’s the movement of data across the LAN and the WAN that adds pressure to the bottom line in cost, time delay and data integrity.
Those who are able to archive and replay today’s market prices and run analysis on that info can easily be looking at tens of millions annually to buy, store, recall, replay, move, distribute and test with no guarantee of a profit. The tradeoff option is do it slower, take longer and analyze smaller sets at the risk of being last in the marketplace.
Analytics in and of themselves are by definition increasingly complex sets of patterns and models used in all aspects of finance. It’s arguably an art of designing interfaces and winning computations that make the difference to what is used successfully and what is not. Still, the size and depth make the opportunity of value moot.
Sometimes the best analytics can be held hostage to bottlenecks of technology that prohibit the data’s integrity, accuracy and timely arrival. However, if the analytics reside where the data is stored, a plethora of problematic implications dissipate. The accuracy and integrity of the solution is thus sharpened. The need to query data becomes almost obsolete as the data provides live results with constant computations running.
The bottom line is that putting powerful predictive analytics, which require no programming or quantitative teams, on the desktops of decision makers creates results heretofore only available to the largest of companies. The advent of in-database analytics offer this at a fraction of the cost of traditional solutions. In fact, we are currently seeing a 10:1 return on investment for our clients.
Why move the data to the analytics when you can move the analytics to the data? Analyzing large volume of data presents numerous challenges – it is time consuming, very expensive and requires management of complex technology infrastructure. In traditional approaches for analyzing data, end-users must move data into memory for processing. This activity accounts for up to 75 percent of the cycle time and imposes severe constraints on delivery of results. In addition, the client or server where the processing is done must have enough memory to store the data and intermediate results.
Fuzzy Logix, founded by investment bankers who have constructed highly functional and performance-driven analytics for a variety of asset classes and risk measurements, started the company for others with similar data-management and analytical bottlenecks.
In these efforts a series of algorithms, functions and computations were compiled as a library called DB Lytix.
The joint efforts by database, data warehousing and business intelligence technology are an example of the two worlds – business and technology – coming closer together. Some estimates indicate predictive analytics will account for 30 percent of all analytics by 2014. Of course these are, well, predictions, but as long as the data continues to compile together with the analytics, there are no limitations.