Very few can ignore the presence of Artificial Intelligence and Machine Learning in today's world, and even less so if you work in quantitative finance. Here, Michael Harris, quant systematic and discretionary trader and best selling author, discusses the impact these technologies are having on trading and investing.
Below are excerpts from a presentation I gave last year in Europe, as an invited speaker to a group of low profile but high net worth investors and traders. The subject was determined by the organizer to be about the impact of artificial intelligence and machine learning on trading and investing. The excerpts below are organized in four sections and cover about 50% of the original presentation.
General impact of artificial intelligence and machine learning on trading
Artificial Intelligence (AI) allows replacing humans with machines. In the 1980s, AI research focused primarily on expert systems and fuzzy logic. With computational power becoming cheaper, using machines to solve large-scale optimization problems became economically feasible. As a result of the advances in hardware and software, nowadays AI focuses on the use of neural networks and other learning methods for identifying and analyzing predictors, also known as features, or factors, that have economic value and can be used with classifiers to develop profitable models. This particular application of AI often goes by the name Machine Learning (ML).
Broad acceptance of this new technology is slow due to various factors, the most important being that AI requires investment in new tools and human talent.
The application of methods for developing trading strategies based on AI, both in short-term time frames and for longer-term investing, is gaining popularity and there are a few hedge funds that are very active in this field. However, broad acceptance of this new technology is slow due to various factors, the most important being that AI requires investment in new tools and human talent. The majority of funds use fundamental analysis because this is what managers learn in their MBA programs. There are not many hedge funds that rely solely on AI. Application of AI is growing at the retail level but the majority of traders still use methods that were proposed in mid twentieth century, including traditional technical analysis, because they are easy to learn and apply.
Note that AI and ML are not only used to develop trading strategies but also in other areas, for example in developing liquidity searching algos and suggesting portfolios to clients. Therefore, with AI applications gaining ground, the number of humans involved in trading and investment decisions decreases and this obviously affects markets and price action. It is early to speculate on the overall effects this new technology will have on the industry but it is possible that extensive use of AI will result in more efficient markets with lower volatility for extended periods of time followed by occasional volatility spikes due to regime changes. This is possible because the impact of subjective evaluation of information by humans will be minimized and with that the associated noise. But that remains to be seen in practice.
Impact of artificial intelligence and machine learning on alpha generation
During these initial phases of the adoption of AI technology there will be opportunities for those who understand it and know how to manage its risks. One problem with trading strategies based on AI is that they can yield models that are worse than random. I will try to explain what I mean by this: traditional technical analysis is an unprofitable method of trading because strategies based on chart patterns and indicators draw their returns from a distribution with zero mean before any transaction costs. Some traders will always be found at the right tail of the distribution and this gives the false impression that these methods have economic value. My research shows that especially in the futures and forex markets, longer-term profitability is hard to achieve no matter which method is used because these markets are designed to benefit market makers. However, in shorter periods of time some traders can realize large profits in leveraged markets due to luck. Then, these traders attribute their success to their strategies and skills, rather than to luck.
As the worse-than-random AI traders are being removed from the market and only those with robust models remain, the battle for profits will become intense.
With AI and ML, there are additional effects, such as the bias-variance trade-off. Data-mining bias can result in strategies that are over-fitted to past data but immediately fail on new data, or strategies that are too simple and do not capture important signals in the data that have economic value. The result of this trade-off is worse-than-random strategies and a negative skew in the distribution of returns of these traders even before transaction cost is added. This presents an opportunity for profit for large funds and investors in the post-quantitative easing era. However, as the worse-than-random AI traders are being removed from the market and only those with robust models remain, the battle for profits will become intense. It is too early to speculate whether AI traders or large investors will win this battle.
I would also like to mention a common misconception in this area: some people believe that the value is in the ML algos used. This is not true. The true value is in the predictors used, also known as features or factors. ML algorithms cannot find gold where there is none. One problem is that most ML professionals use the same predictors and try to develop models in an iterative fashion that will produce the best results. This process is plagued by data-mining bias and eventually fails. In a nutshell, data-mining bias results from the dangerous practice of using data multiple times with many models until results are acceptable in the training and testing samples. My research in this area indicates that if a simple classifier, such as Binary Logistic Regression, does not work satisfactorily with a given set of predictors, then it is highly likely that there is no economic value. Therefore, success depends on what is called feature engineering, and this is both a science and an art that requires knowledge, experience and imagination to come up with features that have economic value and only a small percentage of professionals can do that.
Impact of artificial intelligence and machine learning on technical analysis
We have to make a distinction between traditional and quantitative technical analysis because all methods that rely on the analysis of price and volume series fall under this subject. Traditional technical analysis, i.e., chart patterns, some simple indicators, certain theories of price action, etc., was not effective to start with. Other than a few incomplete efforts of limited scope and reach, publications that touted these methods never presented their longer-term statistical expectation but offered only promises that if this or that rule is used there would be profit potential. Since profits and losses in the markets follow some statistical distribution, there were always those who attributed their luck to these methods. At the same time, a whole industry developed around these methods because there were easy to learn. Unfortunately, many thought they could profit by being better at using methods known to everyone else and the result was massive wealth transfer from these naïve traders to market makers and other well-informed professionals.
Success depends on what is called feature engineering, and this is both a science and an art that requires knowledge, experience and imagination.
In the early 1990s, some market professionals realized that a large number of retail traders were trading using these naive methods. Some developed algos and AI expert systems to identify the formations in advance and then trade against them, causing in the process volatility that retail traders, also known as weak hands, could not cope with. In a more fundamental way, the failure of traditional technical analysis can be attributed to the disappearance of high serial correlation from the markets starting in the 1990s. It was basically the high serial correlation that offered the wrong impression that these methods worked. Nowadays, with few exceptions, markets are mean-reverting, not leaving room to simple technical analysis methods to work. However, some quantitative technical analysis methods often work well, such as mean-reversion and statistical arbitrage models, including ML algorithms that use features with economic value.
Note that this type of arbitrage is unlikely to be repeated in the case of AI and ML because of the great variety of models and the fact that most are being kept proprietary, but the main problem with this new technology is not confirmation bias, as in the case of traditional technical analysis, but data-mining bias.
In my opinion, observing the market and looking at charts is becoming an obsolete process. The future of trading is about processing information, developing and validating models in real-time. The hedge fund of the future will not rely on chart analysis. Some still do this because they are at the transition boundary where old ways meet with a new era. Many traders not familiar with AI will find it hard to compete in the future and will withdraw.
Winners and losers of the new trading technology
Application of AI will change trading in many ways and this is already happening. Investors may find out soon that medium-term returns will be much below expectations after the current trend caused by QE expires. If this scenario materializes, then investors will have to return to the old way of finding a good financial adviser that can suggest a portfolio mix and pick securities that will appreciate in value. In some cases, the adviser will be an AI program and this process will be executed online.
Traders need to get familiar with this new technology. Most traders are still struggling with old methods and just hope that “buy the dip” will work and provide profits for a few more years.
The future of trading is about processing information, developing and validating models in real-time.
One of the problems is the moral hazard cultivated by the central bank with direct support of the financial markets in the last eight years. Many traders and investors now believe that bear markets are not possible because the central bank will be there to redistribute their losses to everyone else although they can keep their profits. As a result, most market participants are unprepared for the next major market regime change and may face devastating losses.
There are excellent resources in the web about ML, AI and trading. The best way of learning is by trying to solve a few practical problems. But I believe the transition for most traders will not be possible. The combination of skills required for understanding and applying AI rules out 95% of traders used to drawing lines on charts and watching moving averages.
Investors should do their own research and consult a competent financial adviser who is familiar with these new developments. Every investor has different risk aversion profile and it is difficult to offer general guidelines. There will be a proliferation of robo-advisors soon and selecting one that suits particular needs and objectives may turn out to be a challenging task.
Anyone not familiar with ML and AI and their relation to trading and investing may find it more profitable to consult a professional who is up to speed in this area rather than embarking in a journey of reading books and articles, which is something that can be done after the basics are understood. I hope I have provided a general idea that can serve as a starting point in this interesting and potentially rewarding endeavor.
If you have any questions or comments, happy to connect on Twitter: @mikeharrisNY
About the author: Michael Harris is a trader and best selling author. He is also the developer of the first commercial software for identifying parameter-less patterns in price action 17 years ago. In the last seven years he has worked on the development of DLPAL, a software program that can be used to identify short-term anomalies in market data for use with fixed and machine learning models. Click here for more >>
This article was first published on towardsdatascience.com