It has been more than a decade since Mike Giles and Paul Glasserman introduced Algorithmic Differentiation (AD) and its adjoint mode in particular to Computational Finance. In their seminal paper titled “Smoking Adjoints: fast Monte Carlo Greeks” and published in Risk Magazine in January 2006 they use adjoint AD (AAD) to compute first- and second-order Greeks of a LIBOR market model. They show how AAD allows computation of potentially very large gradients at the cost of only a few function evaluations. This “cheap gradient” result had been an essential ingredient of numerical methods in computational science and engineering for many years. Contributions from data assimilation in meteorology/physical oceanography, shape/ topology optimization in automotive and aerospace engineering, or back-propagation in neural networks are collected in an extensive bibliography maintained by the AD community (autodiff.org).
The acceptance of AAD has been growing with the increasing number of sensitivities required, for example, by modern computational approaches to XVA and FRTB. Computational Finance has become one of the major target applications for AAD. Relatively simple proof-of-concept prototypes are increasingly replaced by production-grade AAD solutions targeting substantial parts of existing Quant libraries and running on heterogeneous hardware including large clusters and accelerators, for example, GPUs. Consequently, state of the art AAD solutions and software require a level of professionalism which goes beyond academic experiments. In the following I comment briefly on a collection of relevant issues to be considered in the context of what I would refer to as “production-grade AAD solutions.”
Memory Requirement
Due to the necessary reversal of the data flow AAD is known to suffer from potentially prohibitive memory requirement when applied naively to nontrivial numerical simulations. Checkpointing and preaccumulation techniques have been developed to facilitate construction of feasible AAD solutions for a given upper bound on the available memory. The combinatorial problem of minimizing the overall computational cost is hard. Powerful heuristics exist.
Second-Order Vector AAD
Second-order adjoints are computed as tangents (directional derivatives) of first-order adjoints. Vector tangent AD facilitates high-performance Hessian accumulation through parallelisation/acceleration.
Overloading/Source Transformation/Finite Differences
With C++ dominating the Quant library development landscape operator and function overloading has become the method of choice for AD in computational finance. Automatic source transformation is typically applied to simpler (domain-specific) scripting languages or to subsets of general-purpose programming languages. Hand-coding may prove useful in the context of isolated, run time critical, and stable parts of the target code. Near-optimal AD solutions typically combine local source transformation (for example, applied to scripted payoffs) and selective finite difference approximation (for example, applied to black-box routines) with an overall overloading approach. Template meta-programming in C++ and improved compiler optimisation result in a continuously decreasing performance gap between overloading and source transformation.
Symbolic Differentiation / Implicit Function Theorem
Basic AAD operates at the level of elemental functions consisting of arithmetic operators and intrinsic functions provided by the underlying programming language. For the sake of efficiency this low level of granularity should be avoided whenever possible. Higher-level elementals comprise symbolic derivatives of primal numerical kernels ranging from BLAS and (non-)linear system solvers via numerical integration methods to solvers of (partial) differential equations. Consistency of the resulting tangents and adjoints requires accurate primal solutions as well as careful numerical treatment of the symbolic adjoint formulations.
Non-smoothness / Noise
Conceptually, AD assumes its target code to be continuously differentiable up to the required order. It cannot be expected to “magically” deal with general non-smoothness. Smoothing of the target code is typically required as a preprocessing step for AD. Approximation by finite differences gives reasonable results in some cases. Similar issues arise in the context of noisy / chaotic functions.
Parallelism / GPU
AD solutions have been developed for parallel computers based on shared/distributed/hybrid memory models. Template meta programming in C++ allows for efficient and scalable implementations on massively parallel accelerators such as GPUs. The state of the art includes the efficient evaluation of second-order adjoints on GPUs.
Software Engineering / Modeling and Simulation
AD is “invasive.” First steps toward production-grade solutions restrict the use of AD to selected parts of the target code. Once the often substantial benefits are realised the use of AD is extended to larger code sections with nontrivial consequences for software design, coding guidelines, debugging, and testing. Tight integration of AD results in better code. Knowledge about parameter sensitivities contributes to the design of better models, methods and to the choice of suitable data.
Project Management
Production-grade AD solutions require appropriately trained personnel and a high degree of automation of the software development process to facilitate sustainability and robustness with respect to the ongoing evolution of the hard- and software infrastructure. Corresponding management decisions define the level of success of the adoption of AD as a fundamental ingredient of the computational “tool box” within financial institutions.
In conclusion, there is no viable alternative to adjoints as the method of choice for large-scale sensitivity analysis, error control, calibration, and deep learning. The development, maintenance, and evolution of robust, efficient, and sustainable adjoint-enabled Quant libraries remains a major challenge for modern computational finance. The ongoing collaboration between experts from AD and from computational finance contributes to more sophisticated AD software tools enabling improved security and predictivity of financial simulations.
