Machine learning is the hottest thing in quantitative finance at the moment. Earlier this year, Marcos López de Prado, Adjunct Professor of Financial Machine Learning at Cornell University, presented numerous use cases, and there is certainly appetite for more. But as we move forward, how could we accelerate machine learning adoption? What are the challenges that still persist? And does quant finance have the necessary tools to overcome these obstacles?
The way mainstream media talks about artificial intelligence and machine learning, you’d think that we already live in a society where sophisticated AI & ML are around every corner, tricking us into thinking they are one of us, humans. But that’s not the case at all. Arguably, artificial intelligence, as we know it today, is not actually that intelligent.
“Moving from the current paradigm of “representation learning” (or learning by “imitation” from data) to a paradigm of “machine consciousness”, where the agent will be able to make decisions based on independent “thinking” by using an approach that is more akin to human behaviour, is still a distant milestone”, Cris Doloc, Founder and Principal of FintelligeX, said.
Where is finance in its machine learning adoption journey?
It is universally acknowledged that AI and ML provide huge competitive advantages to the businesses that are able to implement them correctly.
“I think the firms that haven't already adopted ML by now are probably unlikely to change because of some organisational reason”, Luca Lin, Partner at Domeyard, told us. “I think they're going to get competed out of their businesses very gradually and aren't likely to greatly increase their adoption if they haven't already done so by this point.”
But Lin reminded us that machine learning has been in finance long before it has become a hype.
“Clustering and regression techniques have been used in classical finance and time series prediction literature for a long time”, Lin explained. “Chances are that any quantitative long-short equity firm is already using these techniques in their factor models.”
Technology, however, is developing exponentially, and it seems that quant finance hasn’t been able to keep up.
“My impressions are that the industry still has a long way to go to adopt more advanced ML such as large deep learning models”, Niels Nygaard, Professor of Financial Mathematics at the University of Chicago told us. “This may be because of a lack of expertise and a reluctance to try new things when the current models seem to work fine.”
What are the obstacles you foresee during the adoption process?
Besides a reluctant industry, misinformation, high expectations, and ultimately disappointment could be the biggest challenge during the adoption process.
“It is hard work to get an ML model to function well and there are no out-of-the-box solutions”, Nygaard explained. “We are trying to close this gap by teaching ML to students in finance and economics courses.”
Building AI and ML solutions still require a lot of input from us.
“In our experience, it is the first step that takes the most time and effort, and it can lead to the greatest frustrations” Dr. Svetlana Borovkova, Head of Quantitative Modelling at Probability & Partners, said. “We see, both in academic research and in practical implementation, that a quant who is involved in ML, typically spends 50% (but often as much as 80%) of his or her time and effort on data cleaning and pre-processing and only 20 to 50% on actual training and testing of ML algorithms.”
Even Lin, whose start-up company isn’t restricted by legacy systems, struggles with the maintenance of machine learning models.
“These models tend to be less parsimonious and have a lot of configuration overhead and sensitivity to data provenance”, Lin explained. “The former means we have to keep careful versioning of our model configurations and parameters so our experiments are repeatable and cleanly organised. The latter means we need to make sure we track the history of our data carefully as it goes through iterations of cleaning, scrubbing, pre-processing, and so on. This would be very time-consuming for all firms, even firms with hundreds or thousands of employees, because it's not a problem that's linearly solvable with increasing number of people you throw at it. It requires careful design, planning, foresight and some good luck in making the right architectural choices.”
Does finance have the right talent and skills to put advanced machine learning into practice?
According to Lin, so far, there have been little issues with talent shortage, but the commoditisation of data science skillsets (and supporting software libraries) means finance will face some competition.
Nygaard told us: “Typically, the most talented ML students will seek to work at tech firms, both because of a higher pay, but mostly because the challenges and ability to have direct influence are seen as much better in the tech industry.”
“You'd be surprised how hard (and expensive) it is to keep a fresh graduate from the top 20 schools motivated and excited about data cleaning for more than 2 years”, Lin admitted.
Yet, that is exactly what finance needs.
“I think the most important skills are actually the least glamorous”, Lin explained. “An attention to detail, methodical and organised workflow, ability to communicate your ideas well, and a willingness to spend a disproportionate amount of time cleaning data and debugging, compared to the more interesting modelling work.”
No sweet without sweat. The machine learning developments are promising in finance, but that doesn’t mean that the threat of another AI winter has been eliminated. The hype and rumours of impossible possibilities can be misleading, and the gap between practitioners and academia needs to be bridged to foster in a new era of advanced machine learning in finance.