Gaël Varoquaux

This talk will cover how to build predictive models that handle well missing values, using scikit-learn. It will give on the one side the statistical considerations, both the classic statistical missing-values theory and the recent development in machine learning, and on the other side how to efficiently code solutions.

In many data-science applications, the data may come with missing values. There is a rich statistical literature on performing analysis with missing values. However, machine learning brings new tradeoffs: how to deal with missing-values at test time? Should we really care about recovering the model suitable for fully-observed data? I will cover both the classic theory and recent theoretical advances. I will show how scikit-learn can be used to implement various solutions, and how these illustrate the theory. Tentative outline: - The classic statistical view on missing values - Missing at Random Settings: why it is important - Imputation, and corresponding scikit-learn tools - Prediction for missing values - Simple predictors need very good predictors - Rich predictors work with simple imputers, even outside Missing at Random settings

This tutorial will guide towards good evaluation of machine-learning models, choosing metrics and procedures that match the intended usage, with code examples using the latest scikit-learn's features. We will discuss how good metrics should characterize all aspects of error, e.g. on the positive and negative class; the probability of a detection, or the probability of a true event given a detection; as they may need to catter for class imbalance. Metrics may also evaluate confidence scores, e.g. calibration. Model-evaluation procedures should gauge not only the expected generalization performance, but also its variations.

Model evaluation is a crucial aspect of machine-learning, to choose the best model, or to decide if a given model is good-enough for production. This tutorial will give didactic introductions to the various statistical aspects of model evaluation: what aspects of model prediction are important to capture, and how different metrics available in scikit-learn captures them. How to devise a model-evaluation procedure that is best suited to select the best model, or control that a model is suited for usage. This tutorial goes beyond mere application of scikit-learn and we expect even experts to learn useful considerations. The tutorial will be loosely based on the following preprint https://hal.archives-ouvertes.fr/hal-03682454, but with code examples for each important concept. A tentative outline is as follows: ### Performance metrics #### Metrics for classification - Binary classification - Confusion matrix - Simple summaries and their pitfalls - Probability of detection given true class, or vice versa? - Summary metrics for low prevalence - Metrics for shifts in prevalence - Multi-threshold metrics - Confidence scores and calibration - Multi-class classification - Adapting binary metrics to multi-class settings - Multilabel classification #### Metrics for regression - R2 score - Absolute error measures - Assessing the distribution of errors ### Evaluation strategies #### Evaluating a learning procedure - Cross-validation strategies - Driving model choices: nested cross-validation - Statistical testing - Sources of variance - Accounting for benchmarking variance #### Evaluating generalization to an external population - The notion of external validity - Confidence intervals for external validation

Other people: Arturo Amor