Hostname: page-component-745bb68f8f-g4j75 Total loading time: 0 Render date: 2025-02-07T22:46:18.917Z Has data issue: false hasContentIssue false
  • English
  • Français

Comparative analysis of machine learning methods for active flow control

Published online by Cambridge University Press:  10 March 2023

Fabio Pino Open the ORCID record for Fabio Pino [Opens in a new window] ,
Lorenzo Schena Open the ORCID record for Lorenzo Schena [Opens in a new window] ,
Jean Rabault Open the ORCID record for Jean Rabault [Opens in a new window]  and
Miguel A. Mendez Open the ORCID record for Miguel A. Mendez [Opens in a new window]
Fabio Pino*
Affiliation:
EA Department, von Kármán Institute for Fluid Dynamics, 1640 Sint Genesius Rode, Belgium Transfers, Interfaces and Processes (TIPs), Université libre de Bruxelles, 1050 Brussels, Belgium
Lorenzo Schena
Affiliation:
EA Department, von Kármán Institute for Fluid Dynamics, 1640 Sint Genesius Rode, Belgium Department of Mechanical Engineering, Vrije Universiteit Brussels, 1050 Brussels, Belgium
Jean Rabault
Affiliation:
Norwegian Meteorological Institute, 0313 Oslo, Norway
Miguel A. Mendez
Affiliation:
EA Department, von Kármán Institute for Fluid Dynamics, 1640 Sint Genesius Rode, Belgium
*
†Email address for correspondence: fabio.pino@vki.ac.be
Get access Rights & Permissions [Opens in a new window]

Abstract

Machine learning frameworks such as genetic programming and reinforcement learning (RL) are gaining popularity in flow control. This work presents a comparative analysis of the two, benchmarking some of their most representative algorithms against global optimization techniques such as Bayesian optimization and Lipschitz global optimization. First, we review the general framework of the model-free control problem, bringing together all methods as black-box optimization problems. Then, we test the control algorithms on three test cases. These are (1) the stabilization of a nonlinear dynamical system featuring frequency cross-talk, (2) the wave cancellation from a Burgers’ flow and (3) the drag reduction in a cylinder wake flow. We present a comprehensive comparison to illustrate their differences in exploration versus exploitation and their balance between ‘model capacity’ in the control law definition versus ‘required complexity’. Indeed, we discovered that previous RL control attempts of controlling the cylinder wake were performing linear control and that the wide observation space was limiting their performances. We believe that such a comparison paves the way towards the hybridization of the various methods, and we offer some perspective on their future development in the literature of flow control problems.

JFM classification

Flow Control: Flow Control
Type
JFM Papers
Information
Journal of Fluid Mechanics , Volume 958 , 10 March 2023 , A39
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Abu-Mostafa, Y.S., Magdon-Ismail, M. & Lin, H.-T. 2012 Learning from Data. AMLBook.Google Scholar
Ahmed, M.O., Vaswani, S. & Schmidt, M. 2020 Combining Bayesian optimization and Lipschitz optimization. Mach. Learn. 109 (1), 79–102.Google Scholar
Aleksic, K., Luchtenburg, M., King, R., Noack, B. & Pfeifer, J. 2010 Robust nonlinear control versus linear model predictive control of a bluff body wake. In 5th Flow Control Conference, p. 4833. American Institute of Aeronautics and Astronautics.Google Scholar
Alnæs, M., Blechta, J., Hake, J., Johansson, A., Kehlet, B., Logg, A., Richardson, C., Ring, J., Rognes, M.E. & Wells, G.N. 2015 The FEniCS project version 1.5. Arch. Numer. Softw. 3 (100).Google Scholar
Apata, O. & Oyedokun, D.T.O. 2020 An overview of control techniques for wind turbine systems. Sci. African 10, e00566.Google Scholar
Archetti, F. & Candelieri, A. 2019 Bayesian Optimization and Data Science. Springer.CrossRefGoogle Scholar
Bäck, T., Fogel, D.B. & Michalewicz, Z. 2018 Evolutionary Computation 1: Basic Algorithms and Operators. CRC.Google Scholar
Balabane, M., Mendez, M.A. & Najem, S. 2021 Koopman operator for Burgers's equation. Phys. Rev. Fluids 6 (6), 064401.Google Scholar
Banzhaf, W., Nordin, P., Keller, R.E. & Francone, F.D. 1997 Genetic Programming: An Introduction. Morgan Kaufmann.Google Scholar
Beintema, G., Corbetta, A., Biferale, L. & Toschi, F. 2020 Controlling Rayleigh–Bénard convection via reinforcement learning. J. Turbul. 21 (9–10), 585–605.Google Scholar
Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E. & Reglade, U. 2019 Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Adv. 9 (12), 125014.CrossRefGoogle Scholar
Benard, N., Pons-Prats, J., Periaux, J., Bugeda, G., Braud, P., Bonnet, J.P. & Moreau, E. 2016 Turbulent separated shear flow control by surface plasma actuator: experimental optimization by genetic algorithm approach. Exp. Fluids 57 (2), 1–17.CrossRefGoogle Scholar
Bergmann, M., Cordier, L. & Brancher, J.-P. 2005 Optimal rotary control of the cylinder wake using proper orthogonal decomposition reduced-order model. Phys. Fluids 17 (9), 097101.Google Scholar
Bersini, H. & Gorrini, V. 1996 Three connectionist implementations of dynamic programming for optimal control: a preliminary comparative analysis. In Proceedings of International Workshop on Neural Networks for Identification, Control, Robotics and Signal/Image Processing, pp. 428–437.Google Scholar
Bewley, T.R. 2001 Flow control: new challenges for a new renaissance. Prog. Aerosp. Sci. 37 (1), 21–58.CrossRefGoogle Scholar
Beyer, H.-G. & Schwefel, H.-P. 2002 Evolution strategies – a comprehensive introduction. Nat. Comput. 1, 3–52.Google Scholar
Blanchard, A.B., Cornejo Maceda, G.Y., Fan, D., Li, Y., Zhou, Y., Noack, B.R. & Sapsis, T.P. 2022 Bayesian optimization for active flow control. Acta Mech. Sin. 37, 1–13.Google Scholar
Brunton, S.L. & Noack, B.R. 2015 Closed-loop turbulence control: progress and challenges. Appl. Mech. Rev. 67 (5).Google Scholar
Brunton, S.L., Noack, B.R. & Koumoutsakos, P. 2020 Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52, 477–508.CrossRefGoogle Scholar
Bucci, M.A., Semeraro, O., Allauzen, A., Wisniewski, G., Cordier, L. & Mathelin, L. 2019 Control of chaotic systems by deep reinforcement learning. Proc. R. Soc. A 475 (2231), 20190351.CrossRefGoogle ScholarPubMed
Buşoniu, L., Babuška, R. & Schutter, B.D. 2010 Multi-agent reinforcement learning: An overview. In Innovations in Multi-Agent Systems and Applications – 1 (ed. D. Srinivasan & L.C. Jain), pp. 183–221. Springer.Google Scholar
Camarri, S. & Giannetti, F. 2010 Effect of confinement on three-dimensional stability in the wake of a circular cylinder. J. Fluid Mech. 642, 477–487.CrossRefGoogle Scholar
Castellanos, R., Cornejo Maceda, G.Y., de la Fuente, I., Noack, B.R., Ianiro, A. & Discetti, S. 2022 Machine-learning flow control with few sensor feedback and measurement noise. Phys. Fluids 34 (4), 047118.Google Scholar
Collis, S.S., Ghayour, K. & Heinkenschloss, M. 2002 Optimal control of aeroacoustic noise generated by cylinder vortex interaction. Intl J. Aeroacoust. 1 (2), 97–114.CrossRefGoogle Scholar
Cornejo Maceda, G.Y., Li, Y., Lusseyran, F., Morzyński, M. & Noack, B.R. 2021 Stabilization of the fluidic pinball with gradient-enriched machine learning control. J. Fluid Mech. 917, A42.CrossRefGoogle Scholar
Davidson, K.R. & Donsig, A.P. 2009 Real Analysis and Applications: Theory in Practice, p. 70. Springer Science & Business Media.Google Scholar
Debien, A., von Krbek, K.A.F.F., Mazellier, N., Duriez, T., Cordier, L., Noack, B.R., Abel, M.W. & Kourta, A. 2016 Closed-loop separation control over a sharp edge ramp using genetic programming. Exp. Fluids 57 (3), 1–19.Google Scholar
Dirk, M.L., Günther, B., Noack, B.R., King, R. & Tadmor, G. 2009 A generalized mean-field model of the natural and high-frequency actuated flow around a high-lift configuration. J. Fluid Mech. 623, 283–316.Google Scholar
Duriez, T., Brunton, S.L. & Noack, B.R. 2017 Machine Learning Control-Taming Nonlinear Dynamics and Turbulence, vol. 116. Springer.Google Scholar
Evans, L.C. 1983 An introduction to mathematical optimal control theory, lecture notes. Available at: https://math.berkeley.edu/~evans/control.course.pdf.Google Scholar
Fan, Y., Chen, L & Wang, Y. 2018 Efficient model-free reinforcement learning using Gaussian process. arXiv:1812.04359.Google Scholar
Fan, D., Yang, L., Wang, Z., Triantafyllou, M.S. & Karniadakis, G.E. 2020 Reinforcement learning for bluff body active flow control in experiments and simulations. Proc. Natl Acad. Sci. 117 (42), 26091–26098.CrossRefGoogle ScholarPubMed
Fasshauer, G.E. 2007 Meshfree Approximation Methods with MATLAB, vol. 6. World Scientific.CrossRefGoogle Scholar
Fleming, P.J. & Fonseca, C.M. 1993 Genetic algorithms in control systems engineering. IFAC Proc. Vols 26 (2), 605–612.Google Scholar
Forrester, A.I.J., Sóbester, A. & Keane, A.J. 2008 Engineering Design via Surrogate Modelling. Wiley.Google Scholar
Fortin, F.-A., De Rainville, F.-M., Gardner, M.-A., Parizeau, M. & Gagné, C. 2012 DEAP: evolutionary algorithms made easy. J. Mach. Learn. Res. 13, 2171–2175.Google Scholar
Frazier, P.I. 2018 A Tutorial on Bayesian Optimization. arXiv:1807.02811.Google Scholar
Fujimoto, S., van Hoof, H. & Meger, D. 2018 Addressing Function Approximation Error in Actor-Critic Methods. arXiv:1802.09477.Google Scholar
Gad-el Hak, M. 2000 Flow Control: Passive, Active, and Reactive Flow Management, pp. 1–9. Cambridge University Press.CrossRefGoogle Scholar
Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A. & Hachem, E. 2021 A review on deep reinforcement learning for fluid mechanics. Comput. Fluids 225, 104973.CrossRefGoogle Scholar
Gautier, N., Aider, J.-L., Duriez, T., Noack, B.R., Segond, M. & Abel, M. 2015 Closed-loop separation control using machine learning. J. Fluid Mech. 770, 442–457.Google Scholar
Gazzola, M., Hejazialhosseini, B. & Koumoutsakos, P. 2014 Reinforcement learning and wavelet adapted vortex methods for simulations of self-propelled swimmers. SIAM J. Sci. Comput. 36 (3), B622–B639.CrossRefGoogle Scholar
Goodfellow, I., Bengio, Y. & Courville, A. 2016 Deep Learning. MIT Press.Google Scholar
Goumiri, I.R., Priest, B.W. & Schneider, M.D. 2020 Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernels. arXiv:2004.05198.CrossRefGoogle Scholar
Griffith, M.D., Leontini, J., Thompson, M.C. & Hourigan, K. 2011 Vortex shedding and three-dimensional behaviour of flow past a cylinder confined in a channel. J. Fluids Struct. 27 (5–6), 855–860.CrossRefGoogle Scholar
Gunzburger, M.D. 2002 Perspectives in Flow Control and Optimization. Society for Industrial and Applied Mathematics.CrossRefGoogle Scholar
Guéniat, F., Mathelin, L. & Hussaini, M.Y. 2016 A statistical learning strategy for closed-loop control of fluid flows. Theor. Comput. Fluid Dyn. 30 (6), 497–510.CrossRefGoogle Scholar
van Hasselt, H.P., Guez, A., Hessel, M., Mnih, V. & Silver, D. 2016 Learning values across many orders of magnitude. Adv. Neural Inform. Proc. Syst. 29, 1–19.Google Scholar
Haupt, R.L. & Ellen Haupt, S. 2004 Practical Genetic Algorithms. Wiley Online Library.Google Scholar
Head, T., Kumar, M., Nahrstaedt, H., Louppe, G. & Shcherbatyi, I. 2020 scikit-optimize/ scikit-optimize. Available at: https://scholar.google.com/citations?view_op=view_citation&hl=hu&user=tQXS7LIAAAAJ&citation_for_view=tQXS7LIAAAAJ:ufrVoPGSRksC.Google Scholar
Jin, B., Illingworth, S.J. & Sandberg, R.D. 2020 Feedback control of vortex shedding using a resolvent-based modelling approach. J. Fluid Mech. 897, A26.CrossRefGoogle Scholar
Jones, D.R., Schonlau, M. & Welch, W.J. 1998 Efficient global optimization of expensive black-box functions. J. Global Optim. 13 (4), 455–492.CrossRefGoogle Scholar
Kanaris, N., Grigoriadis, D. & Kassinos, S. 2011 Three dimensional flow around a circular cylinder confined in a plane channel. Phys. Fluids 23 (6), 064106.Google Scholar
Kelley, H.J. 1960 Gradient theory of optimal flight paths. ARS J. 30 (10), 947–954.CrossRefGoogle Scholar
Kim, J., Bodony, D.J. & Freund, J.B. 2014 Adjoint-based control of loud events in a turbulent jet. J. Fluid Mech. 741, 28–59.Google Scholar
King, D.E. 2009 Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758.Google Scholar
Kirk, D.E. 2004 Optimal Control Theory: An Introduction. Courier Corporation.Google Scholar
Kober, J. & Peters, J. 2014 Reinforcement learning in robotics: a survey. In Springer Tracts in Advanced Robotics, pp. 9–67. Springer International Publishing.CrossRefGoogle Scholar
Koza, J.R. 1994 Genetic programming as a means for programming computers by natural selection. Stat. Comput. 4 (2), 87–112.CrossRefGoogle Scholar
Kubalik, J., Derner, E., Zegklitz, J. & Babuska, R. 2021 Symbolic regression methods for reinforcement learning. IEEE Access 9, 139697–139711.Google Scholar
Kumar, B. & Mittal, S. 2006 Effect of blockage on critical parameters for flow past a circular cylinder. Intl J. Numer. Meth. Fluids 50 (8), 987–1001.Google Scholar
Kuss, M. & Rasmussen, C. 2003 Gaussian processes in reinforcement learning. In Advances in Neural Information Processing Systems (ed. S. Thrun, L. Saul & B. Schölkopf), vol. 16. MIT Press.Google Scholar
Lang, W., Poinsot, T. & Candel, S. 1987 Active control of combustion instability. Combust. Flame 70 (3), 281–289.CrossRefGoogle Scholar
Lee, C., Kim, J., Babcock, D. & Goodman, R. 1997 Application of neural networks to turbulence control for drag reduction. Phys. Fluids 9 (6), 1740–1747.Google Scholar
Li, Y., Cui, W., Jia, Q., Li, Q., Yang, Z., Morzyński, M. & Noack, B.R. 2022 Explorative gradient method for active drag reduction of the fluidic pinball and slanted Ahmed body. J. Fluid Mech. 932, A7.CrossRefGoogle Scholar
Li, R., Noack, B.R., Cordier, L., Borée, J. & Harambat, F. 2017 Drag reduction of a car model by linear genetic programming control. Exp. Fluids 58 (8), 1–20.Google Scholar
Li, J. & Zhang, M. 2021 Reinforcement-learning-based control of confined cylinder wakes with stability analyses. J. Fluid Mech. 932, A44.CrossRefGoogle Scholar
Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. & Wierstra, D. 2015 Continuous control with deep reinforcement learning. arXiv:1509.02971.Google Scholar
Lin, J.C. 2002 Review of research on low-profile vortex generators to control boundary-layer separation. Prog. Aerosp. Sci. 38 (4–5), 389–420.CrossRefGoogle Scholar
Longuski, J.M., Guzmán, J.J. & Prussing, J.E. 2014 Optimal Control with Aerospace Applications. Springer.CrossRefGoogle Scholar
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P. & Mordatch, I. 2017 Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. arXiv:1706.02275.Google Scholar
Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas, J., Grefenstette, E., Whiteson, S. & Rocktäschel, T. 2019 A Survey of Reinforcement Learning Informed by Natural Language. arXiv:1906.03926.Google Scholar
Mahfoze, O.A., Moody, A., Wynn, A., Whalley, R.D. & Laizet, S. 2019 Reducing the skin-friction drag of a turbulent boundary-layer flow with low-amplitude wall-normal blowing within a Bayesian optimization framework. Phys. Rev. Fluids 4 (9), 094601.Google Scholar
Malherbe, C. & Vayatis, N. 2017 Global optimization of lipschitz functions. In International Conference on Machine Learning, pp. 2314–2323. PMLR.Google Scholar
Mathupriya, P., Chan, L., Hasini, H. & Ooi, A. 2018 Numerical investigations of flow over a confined circular cylinder. In 21st Australasian Fluid Mechanics Conference, AFMC 2018. Australasian Fluid Mechanics Society.Google Scholar
Mendez, F.J., Pasculli, A., Mendez, M.A. & Sciarra, N. 2021 Calibration of a hypoplastic model using genetic algorithms. Acta Geotech. 16 (7), 2031–2047.CrossRefGoogle Scholar
Mitchell, T. 1997 Machine Learning, vol. 1. McGraw-Hill.Google Scholar
Mnih, V., et al. 2015 Human-level control through deep reinforcement learning. Nature 518 (7540), 529–533.Google ScholarPubMed
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. & Riedmiller, M. 2013 Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602.Google Scholar
Munters, W. & Meyers, J. 2018 Dynamic strategies for yaw and induction control of wind farms based on large-eddy simulation and optimization. Energies 11 (1), 177.Google Scholar
Nian, R., Liu, J. & Huang, B. 2020 A review on reinforcement learning: introduction and applications in industrial process control. Comput. Chem. Engng 139, 106886.CrossRefGoogle Scholar
Noack, B.R. 2019 Closed-loop turbulence control-from human to machine learning (and retour). In Proceedings of the 4th Symposium on Fluid Structure-Sound Interactions and Control (FSSIC) (ed. Y. Zhou, M. Kimura, G. Peng, A.D. Lucey, & L. Huang), pp. 23–32. Springer.Google Scholar
Noack, B.R., Afanasiev, K., Morzyński, M., Tadmor, G. & Thiele, F. 2003 A hierarchy of low-dimensional models for the transient and post-transient cylinder wake. J. Fluid Mech. 497, 335–363.CrossRefGoogle Scholar
Noack, B.R., Cornejo Maceda, G.Y. & Lusseyran, F. 2023 Machine Learning for Turbulence Control. Cambridge University Press.Google Scholar
Novati, G. & Koumoutsakos, P. 2019 Remember and forget for experience replay. In Proceedings of the 36th International Conference on Machine Learning. PMLR.Google Scholar
Novati, G., Mahadevan, L. & Koumoutsakos, P. 2019 Controlled gliding and perching through deep-reinforcement-learning. Phys. Rev. Fluids 4 (9), 093902.CrossRefGoogle Scholar
Novati, G., Verma, S., Alexeev, D., Rossinelli, D., van Rees, W.M. & Koumoutsakos, P. 2017 Synchronisation through learning for two self-propelled swimmers. Bioinspir. Biomim. 12 (3), 036001.Google ScholarPubMed
Page, J. & Kerswell, R.R. 2018 Koopman analysis of Burgers equation. Phys. Rev. Fluids 3 (7), 071901.Google Scholar
Paris, R., Beneddine, S. & Dandois, J. 2021 Robust flow control and optimal sensor placement using deep reinforcement learning. J. Fluid Mech. 913, A25.Google Scholar
Park, D.S., Ladd, D.M. & Hendricks, E.W. 1994 Feedback control of von Kármán vortex shedding behind a circular cylinder at low Reynolds numbers. Phys. Fluids 6 (7), 2390–2405.Google Scholar
Pastoor, M., Henning, L., Noack, B.R., King, R. & Tadmor, G. 2008 Feedback shear layer control for bluff body drag reduction. J. Fluid Mech. 608, 161–196.CrossRefGoogle Scholar
Pedregosa, F., et al. 2011 Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830.Google Scholar
Pivot, C., Cordier, L. & Mathelin, L. 2017 A continuous reinforcement learning strategy for closed-loop control in fluid dynamics. In 35th AIAA Applied Aerodynamics Conference. American Institute of Aeronautics and Astronautics.Google Scholar
Powell, M.J.D. 2006 The newuoa software for unconstrained optimization without derivatives. In Large-Scale Nonlinear Optimization, pp. 255–297. Springer.Google Scholar
Rabault, J., Kuchta, M., Jensen, A., Réglade, U. & Cerardi, N. 2019 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech. 865, 281–302.Google Scholar
Rabault, J. & Kuhnle, A. 2019 Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Phys. Fluids 31 (9), 094105.Google Scholar
Rabault, J. & Kuhnle, A. 2022 Deep Reinforcement Learning applied to Active Flow Control. Cambridge University Press.Google Scholar
Rabault, J., Ren, F., Zhang, W., Tang, H. & Xu, H. 2020 Deep reinforcement learning in fluid mechanics: a promising method for both active flow control and shape optimization. J. Hydrodyn. 32 (2), 234–246.CrossRefGoogle Scholar
Rasmussen, C.E. & Williams, C.K.I. 2005 Gaussian Processes for Machine Learning. MIT Press.Google Scholar
Recht, B. 2019 A tour of reinforcement learning: the view from continuous control. Annu. Rev. Control Rob. Auton. Syst. 2 (1), 253–279.CrossRefGoogle Scholar
Rehimi, F., Aloui, F., Nasrallah, S.B., Doubliez, L. & Legrand, J. 2008 Experimental investigation of a confined flow downstream of a circular cylinder centred between two parallel walls. J. Fluids Struct. 24 (6), 855–882.Google Scholar
Ren, F., Rabault, J. & Tang, H. 2021 Applying deep reinforcement learning to active flow control in weakly turbulent conditions. Phys. Fluids 33 (3), 037121.CrossRefGoogle Scholar
Sahin, M. & Owens, R.G. 2004 A numerical investigation of wall effects up to high blockage ratios on two-dimensional flow past a confined circular cylinder. Phys. Fluids 16 (5), 1305–1320.Google Scholar
Schäfer, M., Turek, S., Durst, F., Krause, E. & Rannacher, R. 1996 Benchmark computations of laminar flow around a cylinder. In Flow Simulation with High-Performance Computers II, pp. 547–566. Springer.CrossRefGoogle Scholar
Schaul, T., Quan, J., Antonoglou, I. & Silver, D. 2015 Prioritized Experience Replay. arXiv:1511.05952.Google Scholar
Schlichting, H. & Kestin, J. 1961 Boundary Layer Theory, vol. 121. Springer.Google Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. 2017 Proximal Policy Optimization Algorithms. arXiv:1707.06347.Google Scholar
Seidel, J., Siegel, S., Fagley, C., Cohen, K. & McLaughlin, T. 2008 Feedback control of a circular cylinder wake. Proc. Inst. Mech. Engrs G 223 (4), 379–392.Google Scholar
Silver, D., et al. 2016 Mastering the game of go with deep neural networks and tree search. Nature 529 (7587), 484–489.Google ScholarPubMed
Silver, D., et al. 2018 A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362 (6419), 1140–1144.CrossRefGoogle ScholarPubMed
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D. & Riedmiller, M. 2014 Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on International Conference on Machine Learning – Volume 32, pp. 387–395. PMLR.Google Scholar
Singha, S. & Sinhamahapatra, K.P. 2010 Flow past a circular cylinder between parallel walls at low Reynolds numbers. Ocean Engng 37 (8–9), 757–769.CrossRefGoogle Scholar
Stengel, R.F. 1994 Optimal Control and Estimation. Courier Corporation.Google Scholar
Sun, S., Cao, Z., Zhu, H. & Zhao, J. 2019 A Survey of Optimization Methods from a Machine Learning Perspective. arXiv:1906.06821.Google Scholar
Sutton, R.S. & Barto, A.G. 2018 Reinforcement Learning: An Introduction. MIT Press.Google Scholar
Sutton, R.S., Barton, A.G. & Williams, R.J. 1992 Reinforcement learning is direct adaptive optimal control. IEEE Control Syst. Mag. 12 (2), 19–22.Google Scholar
Szita, I. 2012 Reinforcement learning in games. In Adaptation, Learning, and Optimization (ed. M. Wiering & M. van Otterlo), pp. 539–577. Springer.Google Scholar
Tang, H., Rabault, J., Kuhnle, A., Wang, Y. & Wang, T. 2020 Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning. Phys. Fluids 32 (5), 053605.CrossRefGoogle Scholar
Uhlenbeck, G.E. & Ornstein, L.S. 1930 On the theory of the Brownian motion. Phys. Rev. 36 (5), 823–841.Google Scholar
Vanneschi, L. & Poli, R. 2012 Genetic Programming – Introduction, Applications, Theory and Open Issues, pp. 709–739. Springer.Google Scholar
Verma, S., Novati, G. & Koumoutsakos, P. 2018 Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proc. Natl Acad. Sci. 115 (23), 5849–5854.Google ScholarPubMed
Vinuesa, R., Lehmkuhl, O., Lozano-Durán, A. & Rabault, J. 2022 Flow control in wings and discovery of novel approaches via deep reinforcement learning. Fluids 7 (2).CrossRefGoogle Scholar
Vladimir Cherkassky, F.M.M. 2008 Learning from Data. Wiley.Google Scholar
Wang, J. & Feng, L. 2018 Flow Control Techniques and Applications. Cambridge University Press.Google Scholar
Wiener, N. 1948 Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press.Google Scholar
Williamson, C.H.K. 1996 Vortex dynamics in the cylinder wake. Annu. Rev. Fluid Mech. 28 (1), 477–539.CrossRefGoogle Scholar
Zhang, H.-Q., Fey, U., Noack, B.R., König, M. & Eckelmann, H. 1995 On the transition of the cylinder wake. Phys. Fluids 7 (4), 779–794.CrossRefGoogle Scholar