Vision for the Future of Data Storage Technology
We recently invited James Corcoran from Kx – a division of First Derivatives plc, to join us as a guest on the latest episode in our Artificial Intelligence for B2B Marketing podcast. Discussing the evolution of computer memory and data storage technology, we take an in-depth look into Kx’s unique role in the landscape of AI.
Data volumes today are growing faster than ever before and this continued growth in the amount of data being generated is true across all verticals, including those within our own clients’ organizations. The diversity of this volume of data is equally increasing. Historically the applications of data were usually quite specific, narrowly defined to solve specific problems. Today, as industries are fervently adopting data-driven cultures, the work of experts like James, has become more focused on building Big Data platforms which feed real-time analytics to multiple downstream applications.
To keep up with this rapid increase businesses can keep more data in-memory than ever before. This means bigger datasets representing real-time views of the world, and therefore more choices in how businesses scale systems both vertically and horizontally. With more performant storage technologies, businesses can also run simulations and backtest models against more and more historical data.
When Market Volatility Causes A Torrent of Data
This influx of data hasn’t come without its cost. Businesses are scrambling to build future-proof systems capable of handling data volumes greater than those which are observable today. Systems such as kdb+ can natively take advantage of future advances in computing power allows Kx to design its systems for massive scale, and for good reason. Recent financial crashes caused by rogue trading algorithms as well as unforeseen geopolitical events, usually produce a surge in volatility, leading to huge increases in the throughput of data flowing through capital markets systems. Regular stress tests are carried out by Kx to ensure their systems can continue to operate under these conditions, which in turn allows their clients to continue to make markets and provision services for their customers.
Kx has been the time-series and analytics engine of choice for a long time in front office electronic trading environments helping banks to automate their pricing, trading and hedging businesses. But they’ve also been providing surveillance and regulatory solutions to the capital markets industry for a number of years. Whilst the use-cases vary, ultimately these systems have similar requirements when it comes to the datasets, the need for high-speed analytics, scalability, resilience, and the integrity and security of data.
Kx’s surveillance solution contains a library of models designed to detect attempts to manipulate markets, whilst their MiFID II solution contains a comprehensive suite of pre and post-trade transparency reports. While also constructing solutions for the forthcoming Consolidated Audit Trail regulation in the US, Kx are working on building solutions for the Securities Financing Transactions Regulation in Europe, both of which are challenging the industry with complex data lineage requirements. Kx is a natural technology choice when it comes to implementing systems for the various regulatory regimes, since it handles a combination of streaming analytics, real-time, and batch-based processing in a single platform.
Future Proofing Software Solutions
It is important that businesses stay true to their commitments to maintaining a speed and performance for managing big fast data streams. Kx are investing heavily in machine learning, having recently set up a research team in London where they partner with leading academics in this space. While they plan to implement many machine learning algorithms natively in Kx, they’re also providing tight bindings to best-of-breed libraries such as TensorFlow for deep learning. As well as open sourcing their work under this initiative, Kx also plan to integrate machine learning techniques into their product portfolio this year. As software solutions have matured, the industry has recognized the need to build tools and platforms to automate many of the tasks associated with capturing, normalizing, cleansing and distributing large volumes of real-time and historical data. When coupled with their suite of data science and visualization products, Kx can create powerful solutions for our clients without reinventing the wheel each time.
MRP Prelytix is powered by Kx technology, the leading in-memory, time-series operational intelligence platform. Since the company’s inception, Kx’s singular goal has been to provide its customers with the fastest, most efficient, and most flexible tools for processing real-time and historical data. As a global technology provider with over 20 years of experience working with some of the world’s largest finance, technology, retail, pharma, manufacturing and energy institutions, Kx technology, incorporating the kdb+ times series database, is a leader in high-performance, in-memory computing, streaming analytics and operational intelligence.