Big data is not just having a lot of data
Big Data is more than just having a lot of data. Or at least that's what Dr Anthony Ledford, Chief Scientist at the Man AHL Research Lab, Oxford University, believes.
"[Big data] is having data from different sources, that are maybe in lots of formats; some might be numerical, some might be text, some might be pictures. Some might be sampled daily, some might be monthly. All these different sources of data - it's how you go about assimilating those in a coherent way and trying to make sense of them. That's what I understand of big data," says Dr Ledford.
Dr Ledford is no stranger to data and its applications. At Man AHL Research Lab, the day-to-day business is all about taking things that can be measured and observed today - our data that we use as input - and then using that information to forecast what's going to happen in financial markets. For the past 20 years, Man AHL Research Lab has used this data to build systems that can automatically make forecasts about where prices will evolve to, and capture that predictability for profit for clients.
However, in recent years the amount of data available has 'exploded'and the range of measurement and analysis tools and techniques have developed. It is in this development space where Dr Ledford spends the majority of his time.
It's a three stage process: research, test trading, client implementation. Man AHL Research Lab analyses trends and data and once they find something that may be viable for trading, they test it in real financial markets. Only once these first two stages have been completed will they roll out for client implementation.
But the industry hasn't always been willing to adopt these new trends and technologies. Dr Ledford says that as someone who works with big data on a daily basis he focusses more on the uncertainty of things than the opportunity. It's understanding the uncertainty whih can be a bit learning curve for clients.
"If you understand the uncertainty of it, it stops you taking large concentrated positions in any small group of markets or models. The rational way to run your portfolio is to spread your risk out across very many markets. That's why we trade hundreds of markets simultaneously across our portfolio, and we still need to do it for long time horizons. And that's because it's a very subtle thing that we are trying to capture."
"You need to actually scale across multiple markets and do it for a very long period of time - years - before this very elusive effect in any individual market can be distilled into something that looks like an investable product by the community."
During a panel at FundForum International 2016 in Berlin, Dr Ledford said that it's time for the industry to "open up the black box" of data and analytics.