Optimization and Machine Learning. Patrick Siarry
187
191 188
192 189
193 190
194 191
195 192
196 193
197 194
198 195
199 196
200 197
201 198
202 199
203 201
204 202
205 203
206 204
207 205
208 206
209 207
210 208
211 209
212 210
213 211
214 212
215 213
216 214
217 215
218 216
219 217
220 218
221 219
222 220
223 221
224 222
225 223
226 224
227 225
228 226
229 227
230 228
231 229
232 230
233 231
234 232
235 233
236 234
237 235
238 236
239 237
SCIENCES
Computer Science,
Field Directors – Valérie Berthé and Jean-Charles Pomerol
Operational Research and Decision, Subject Head – Patrick Siarry
Optimization and Machine Learning
Optimization for Machine Learning and Machine Learning for Optimization
Coordinated by
Rachid Chelouah
Patrick Siarry
First published 2022 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the under mentioned address:
ISTE Ltd
27-37 St George’s Road
London SW19 4EU
UK
John Wiley & Sons, Inc
111 River Street
Hoboken, NJ 07030
USA
© ISTE Ltd 2022
The rights of Rachid Chelouah and Patrick Siarry to be identified as the authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Control Number: 2021949293
British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISBN 978-1-78945-071-2
ERC code:
PE1 Mathematics
PE1_19 Control theory and optimization
PE6 Computer Science and Informatics
PE6_11 Machine learning, statistical data processing and applications using signal processing (e.g. speech, image, video)
Introduction
Rachid CHELOUAH
CY Cergy Paris University, France
Machine learning is revolutionizing our world. It is difficult to conceive of any other information technology that has developed so rapidly in recent years, in terms of real impact.
The fields of machine learning and optimization are highly interwoven. Optimization problems form the core of machine learning methods and modern optimization algorithms are using machine learning more and more to improve their efficiency.
Machine learning has applications in all areas of science. There are many learning methods, each of which uses a different algorithmic structure to optimize predictions, based on the data received. Hence, the first objective of this book is to shed light on key principles and methods that are common within both fields.
Machine learning and optimization share three components: representation, evaluation and iterative search. Yet while optimization solvers are generally designed to be fast and accurate on implicit models, machine learning methods need to be generic and trained offline on datasets. Machine learning problems present new challenges for optimization researchers, and machine learning practitioners seek simpler, generic optimization algorithms.
Quite recently, modern approaches to machine learning have also been applied to the design of optimization algorithms themselves, taking advantage of their ability to capture valuable information from complex structures in large spaces. Those capacities appear to be useful, especially for the representation and evaluation components. As large, complex structures are ubiquitous in optimization problems, and can be used as huge implicit datasets, the use of machine learning enabled the efficiency and genericity