The Big R-Book. Philippe J. S. De Brouwer
tries to understand what his army of quants, data scientists, and developers are up to, while having fun learning R. After reading this book you will be able to talk to all, challenge their work, and make most analysis yourself or be part of a bigger entity and specialize in one of the steps of modelling or data-manipulation.
In some way, this book can also be seen as a celebration of FOSS (Free and Open Source Software). We proudly mention that for this book no commercial software was used at all. The operating systemis Linux, the windows manager Fluxbox (sometimes LXDE or KDE),Kile and vi helped the editing process, Okular displayed the PDF-file, even the database servers and Hadoop/Spark are FOSS …and of course R and LATEX provided the icing on the cake. FOSS makes this world a more inclusive place as it makes technology more attainable in poorer places on this world.
FOSS
Hence, we extend a warm thanks to all people that spend somuch time to contributing to free software.
Acknowledgements
Writing a book that is so eclectic and holds so many information would not have been possible without tremendous support from so many people: mentors, family, colleagues, and ex-colleagues at work or at universities. This book is in the first place a condensation of a few decades of interesting work in asset management and banking and mixes things that I have learned in C-level jobs and more technical assignments.
I thank the colleagues of the faculties of applied mathematics at the AGH University of Science and Technology, the faculty of mathematics of the Jagiellonian University of Krakow, and the colleagues of HSBC for the many stimulating discussions and shared insights in mathematical modelling and machine learning.
To the MBA program of the Cracovian Business School, the University of Warsaw, and to the many leaders that marked my journey, I am indebted for the business insight, stakeholder management and commercial wit that make this book complete.
A special thanks goes to Piotr Kowalczyk, FRM and Dr. Grzegorz Goryl, PRM, for reading large chunks of this book and providing detailed suggestions. I am also grateful for the general remarks and suggestions from Dr. Jerzy Dzieża, faculty of applied mathematics at the AGH University of Science and Technology of Krakow and the fruitful discussions with Dr. Tadeusz Czernik, from the University of Economics of Katowice and also SeniorManager at HSBC, Independent Model Review, Krakow.
This book would not be what it is now without the many years of experience, the stimulating discussions with somany friends, and in particularmy wife, Joanna De Brouwerwho encouraged me to move from London in order to work for HSBC in Krakow, Poland. Somehow, I feel that I should thank the city council and all the people for the wonderful and dynamic environment that attracts so many new service centres and that makes the ones that already had selected forKrakow grow their successful investments. This dynamic environment has certainly been an important stimulating factor in writing this book.
However, nothing would have been possible without the devotion and support of my family: my wife Joanna, both children,Amelia and Maximilian, were wonderful and are a constant source of inspiration and support.
Finally, I would like to thank the thousands of people who contribute to free and open source software, people that spend thousands of hours to create and improve software that others can use for free. I profoundly believe that these selfless acts make this world a better and more inclusive place, because they make computers, software, and studying more accessible for the less fortunate.
A special honorary mentioning should go to the people that have built Linux, LATEX, R, and the ecosystems around each of them as well as the companies that contribute to those projects, such as Microsoft that has embraced R and RStudio that enhances R and never fails to share the fruits of their efforts with the larger community.
♣1♣ The Big Picture with Kondratiev and Kardashev
You have certainly heard the words: “data is the new oil,” and you probably wondered “are we indeed on the verge of a newera of innovation andwealth creation or …is this just hype andwill it blow over soon enough?”
Since our ancestors left the trees about 6 million years ago,we roamed theAfrican steppes and we evolved a more upright position and limbs better suited for walking than climbing. However, for about 4million years physiological changes did not include a larger brain. It is only in the last million years that we gradually evolved a more potent frontal lobe capable of abstract and logical thinking.
The first good evidence of abstract thinking is the Makapansgat pebble, a jasperite cobble – roughly 260 g and 5 by 8 cm – that by geological tear and wear shows a few holes and lines that vaguely resemble (to us) a human face. About 2.5 million years ago one of our australopithecine ancestors not only realized this resemblance but also deemed it interesting enough to pick up the pebble, keep it, and finally leave it in a cave miles from the river where it was found.
This development of abstract thinking that goes beyond vague resemblance was a major milestone. As history unfolded, it became clear that this was only the first of many steps that would lead us to the era of data and knowledge that we live in today. Many more steps towards more complex and abstract thinking, gene mutations and innovation would be needed.
abstract thinking:
Soon we developed language.With language we were able to transform learning from an individual level to a collective level. Now, experiences could be passed on to the next generation or peers much more efficiently, it became possible to prepare someone for something that he or she did not yet encounter and to accumulate more knowledge with every generation.
More than ever before this abstract thinking and accumulation of collective experiences lead to a “knowledge advantage” and smartness became an attractive trait in a mate. This allowed our brain to develop further and great innovations such as the wheel, scripture, bronze, agriculture, iron, specialisation of labour soon started to transform not only our societal coherence but also the world around us.
Without those innovations, we would not be where we are now. While it is discussable to classify these inventions as the fruit of scientific work, it is equally hard to deny that some kind of scientific approach was necessary. For example, realizing the patterns in themovements of the sun, we could predict seasons and weather changes to come and this allowed us to put the grains on the right moment in the ground. This was based on observations and experience.
Science and progress flourished, but the fall of the Western European empire made Europe sink in the dark medieval periodwhere thinkingwas dominated by religious fear and superstition and hence scientific progress came to grinding halt, and it is wake improvements in medical care, food production and technology.
The Arab world continued the legacy of Aristotle (384–322 BCE, Greece) and Alhazen (Ibn al-Haytham, 965–1039 Iraq), who by many is considered as the father of the modern scientific method.1 It was this modern scientific method that became a catalyst for scientific and technological development.
scientific method:
A class of people that accumulated wealth through smart choices emerged. This was made possible by private enterprise and an efficient way of sharing risks and investments. In 1602, the East Indies Company became the first common stock company and in 1601 the Amsterdam Stock Exchange created a platform where innovative, exploratory and trade ideas could find the necessary capital to flourish.
In 1775, James Watt's improvement of the steam engine allowed to leverage on the progress made around the joint stock company and the stock exchange. This combination powered the raise of a new societal organization, capitalism and fueled the first industrial wave based on automation (mainly in the textile industry).