Rank-Based Methods for Shrinkage and Selection. A. K. Md. Ehsanes Saleh
its Geometric Interpretation2.7.4 R-estimation with LASSO and aLASSO2.7.5 Oracle Properties2.8 Elastic Net (Enet)2.8.1 Naive Enet2.8.2 Standard Enet2.8.3 Enet in Machine Learning2.9 Example: Diabetes Data Set2.9.1 Model Building with R-aEnet2.9.2 MSE vs. MAE2.9.3 Model Building with LS-Enet2.10 Summary2.11 Problems
12 3 Location and Simple Linear Models3.1 Introduction3.2 Location Estimators and Testing3.2.1 Unrestricted R–estimator of θ3.2.2 Restricted R-estimator of θ3.3 Shrinkage R-estimators of Location3.3.1 Overview of Shrinkage R-estimators of θ3.3.2 Derivation of the Ridge-type R-estimator3.3.3 Derivation of the LASSO-type R-estimator3.3.4 General Shrinkage R-estimators of θ3.4 Ridge-type R-estimator of θ3.5 Preliminary Test R-estimator of θ3.5.1 Optimum Level of Significance of PTRE3.6 Saleh-type R-estimators3.6.1 Hard-Threshold R-estimator of θ3.6.2 Saleh-type R-estimator of θ3.6.3 Positive-rule Saleh-type (LASSO-type) R-estimator of θ3.6.4 Elastic Net-type R-estimator of θ3.7 Comparative Study of the R-estimators of Location3.8 Simple Linear Model3.8.1 Restricted R-estimator of Slope3.8.2 Shrinkage R-estimator of Slope3.8.3 Ridge-type R-estimation of Slope3.8.4 Hard-Threshold R-estimator of Slope3.8.5 Saleh-type R-estimator of Slope3.8.6 Positive-rule Saleh-type (LASSO-type) R-estimator of Slope3.8.7 The Adaptive LASSO (aLASSO-type) R-estimator3.8.8 nEnet-type R-estimator of Slope3.8.9 Comparative Study of R-estimators of Slope3.9 Summary3.10 Problems
13 4 Analysis of Variance (ANOVA)4.1 Introduction4.2 Model, Estimation and Tests4.3 Overview of Multiple Location Models4.3.1 Example: Corn Fertilizers4.3.2 One-way ANOVA4.3.3 Effect of Variance on Shrinkage Estimators4.3.4 Shrinkage Estimators for Multiple Location4.4 Unrestricted R-estimator4.5 Test of Significance4.6 Restricted R-estimator4.7 Shrinkage Estimators4.7.1 Preliminary Test R-estimator4.7.2 The Stein.Saleh-type R-estimator4.7.3 The Positive-rule Stein.Saleh-type R-estimator4.7.4 The Ridge-type R-estimator4.8 Subset Selection Penalty R-estimators4.8.1 Preliminary Test Subset Selector R-estimator4.8.2 Saleh-type R-estimator4.8.3 Positive-rule Saleh Subset Selector (PRSS)4.8.4 The Adaptive LASSO (aLASSO)4.8.5 Elastic-net-type R-estimator4.9 Comparison of the R-estimators4.9.1 Comparison of URE and RRE4.9.2 Comparison of URE and Stein.Saleh-type R-estimators4.9.3 Comparison of URE and Ridge-type R-estimators4.9.4 Comparison of URE and PTSSRE4.9.5 Comparison of LASSO-type and Ridge-type R-estimators4.9.6 Comparison of URE, RRE and LASSO4.9.7 Comparison of LASSO with PTRE4.9.8 Comparison of LASSO with SSRE4.9.9 Comparison of LASSO with PRSSRE4.9.10 Comparison of nEnetRE with URE4.9.11 Comparison of nEnetRE with RRE4.9.12 Comparison of nEnetRE with HTRE4.9.13 Comparison of nEnetRE with SSRE4.9.14 Comparison of Ridge-type vs. nEnetRE4.10 Summary4.11 Problems
14 5 Seemingly Unrelated Simple Linear Models5.1 Introduction5.1.1 Problem Formulation5.2 Signed and Signed Rank Estimators of Parameters5.2.1 General Shrinkage R-estimator of β5.2.2 Ridge-type R-estimator of β5.2.3 Preliminary Test R-estimator of β5.3 Stein.Saleh-type R-estimator of β5.3.1 Positive-rule Stein.Saleh R-estimators of β5.4 Saleh-type R-estimator of β5.4.1 LASSO-type R-estimator of the β5.5 Elastic-net-type R-estimators5.6 R-estimator of Intercept When Slope Has Sparse Subset5.6.1 General Shrinkage R-estimator of Intercept5.6.2 Ridge-type R-estimator of θ;5.6.3 Preliminary Test R-estimators of θ;5.7 Stein.Saleh-type R-estimator of θ;5.7.1 Positive-rule Stein.Saleh-type R-estimator of θ;5.7.2 LASSO-type R-estimator of θ;5.8 Summary5.8.1 Problems
15 6 Multiple Linear Regression Models6.1 Introduction6.2 Multiple Linear Model and R-estimation6.3 Model Sparsity and Detection6.4 General Shrinkage R-estimator of β6.4.1 Preliminary Test R-estimator6.4.2 Stein.Saleh-type R-estimator6.4.3 Positive-rule Stein.Saleh-type R-estimators6.5 Subset Selectors6.5.1 Preliminary Test Subset Selector R-estimator6.5.2 Stein.Saleh-type R-estimator6.5.3 Positive-rule Stein.Saleh-type R-estimator (LASSO-type)6.5.4 Ridge-type Subset Selector6.5.5 Elastic Net-type R-estimator6.6 Adaptive LASSO6.6.1 Introduction6.6.2 Asymptotics for LASSO-type R-estimator6.6.3 Oracle Property of aLASSO6.7 Summary6.8 Problems
16 7 Partially Linear Multiple Regression Model7.1 Introduction7.2 Rank Estimation in the PLM7.2.1 Penalty R-estimators7.2.2 Preliminary Test and Stein.Saleh-type R-estimator7.3 ADB and ADL2-risk7.4 ADL2-risk Comparisons7.5 Summary: L2-risk Efficiencies7.6 Problems
17 8 Liu Regression Models8.1 Introduction8.2 Linear Unified (Liu) Estimator8.2.1 Liu-type R-estimator8.3 Shrinkage Liu-type R-estimators8.4 Asymptotic Distributional Risk8.5 Asymptotic Distributional Risk Comparisons8.5.1 Comparison of SSLRE and PTLRE8.5.2 Comparison of PRSLRE and PTLRE8.5.3 Comparison of PRLRE and SSLRE8.5.4 Comparison of Liu-Type Rank Estimators With Counterparts8.6 Estimation of d8.7 Diabetes Data Analysis8.7.1 Penalty Estimators8.7.2 Performance Analysis8.8 Summary8.9 Problems
18 9 Autoregressive Models9.1 Introduction9.2 R-estimation of ρ for the AR(p)-Model9.3 LASSO, Ridge, Preliminary Test and Stein.Saleh-type R-estimators9.4 Asymptotic Distributional L2-risk9.5 Asymptotic Distributional L2-risk Analysis9.5.1 Comparison of Unrestricted vs. Restricted R-estimators9.5.2 Comparison of Unrestricted vs. Preliminary Test R-estimator9.5.3 Comparison of Unrestricted vs. Stein.Saleh-type R-estimators9.5.4 Comparison of the Preliminary Test vs. Stein.Saleh-type R-estimators9.6 Summary9.7 Problems
19 10 High-Dimensional Models10.1 Introduction10.2 Identifiability of β* and Projection10.3 Parsimonious Model Selection10.4 Some Notation and Separation10.4.1 Special Matrices10.4.2 Steps Towards Estimators10.4.3 Post-selection Ridge Estimation of βS1* and βS2*10.4.4 Post-selection Ridge R-estimators for βS1* and βS2*10.5 Post-selection Shrinkage R-estimators10.6 Asymptotic Properties of the Ridge R-estimators10.7 Asymptotic Distributional L2-Risk Properties10.8 Asymptotic Distributional Risk Efficiency10.9 Summary10.10 Problems
20 11 Rank-based Logistic Regression11.1 Introduction11.2 Data Science and Machine Learning11.2.1 What is Robust Data Science?11.2.2 What is Robust Machine Learning?11.3 Logistic Regression11.3.1 Log-likelihood Setup11.3.2 Motivation for Rank-based Logistic Methods11.3.3 Nonlinear Dispersion Function11.4 Application to Machine Learning11.4.1 Example: Motor Trend Cars11.5 Penalized Logistic Regression11.5.1 Log-likelihood Expressions11.5.2 Rank-based Expressions11.5.3 Support Vector Machines11.5.4 Example: Circular Data11.6 Example: Titanic Data Set11.6.1 Exploratory Data Analysis11.6.2 RLR vs. LLR vs. SVM11.6.3 Shrinkage and Selection11.7 Summary11.8 Problems
21 12 Rank-based Neural Networks12.1 Introduction12.2 Set-up for Neural Networks12.3 Implementing Neural Networks12.3.1 Basic Computational Unit12.3.2 Activation Functions12.3.3 Four-layer Neural Network12.4 Gradient Descent with Momentum12.4.1 Gradient Descent12.4.2 Momentum12.5 Back Propagation Example12.5.1 Forward Propagation12.5.2 Back Propagation12.5.3 Dispersion Function Gradients12.5.4 RNN Algorithm12.6 Accuracy Metrics12.7 Example: Circular Data Set12.8 Image Recognition: Cats vs. Dogs12.8.1 Binary Image Classification12.8.2 Image Preparation12.8.3 Over-fitting and Under-fitting12.8.4 Comparison of LNN vs. RNN12.9 Image Recognition: MNIST Data Set12.10 Summary12.11 Problems
22 Bibliography
23 Author Index
List