理论推导中常用到的一些概念

An unbiased risk estimator vs. The same minimizer

1. 矫正后的 loss 可以导致一个无偏的（consistent）估计，

[mathbb E_{p(x, ilde{y})}[ell_{correct}^1(h(x), ilde{y})] = mathbb E_{p(x,y)}[ell(h(x),y)]，forall\,h ]

2. 两者有相同的 minimizer:

[mathop{arg!min}_hmathbb E_{p(x, ilde{y})}[ell_{correct}^2(h(x), ilde{y})] = mathop{arg!min}_hmathbb E_{p(x,y)}[ell(h(x),y)] ]

上述条件 (2) 比 （1）要弱一些：

• (1) 可以推出（2）
• 在 (1) 不成立的情形下，（2）有可能成立。
• (2) 一般是在保留相同的 bayesian optimal classifier

Reference:

• NIPS-13. Learning with noisy label
• CVPR-17. Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
• ICML-20. Does Label Smoothing Mitigate Label Noise?
• ICML-20. Learning with Multiple Complementary Labels

Statistically consistent, classifier-consistency, risk-consistent

统计相容性，更多的考虑，在 (n oinfty) 的情况下，两者之间的差别

Reference:

• NIPS-19 Are anchor points really indispensable in label-noise learning?（提到 risk-consistent, classifier consistent）
• ICML-20 Does Label Smoothing Mitigate Label Noise? (提到 classification consistency）
• ICML-13 On the statistical Consistency of Algorithms for Binary Classification under Class Imbalance
• ICML-20. Progressive Identification of True Labels for Partial-Label Learning (classifier-consistency)
• ICML-20 Learning with multiple complementary labels (classifier-consistency, risk-consistent)
• NIPS-20 Provably consistent partial-label learning (risk consistent, classifier-consistent)

Excess risk bound vs. Generalization bound vs. learnability

(1). Excess risk 主要考虑的是当前 (ERM 算法所导出)分类器 与 最优的分类器的泛化误差的 gap
(2). Generalization bound 考虑的是经验误差与泛化误差的 uniform 的 gap，对假设空间中的所有假设同时成立，因此需要用 Rademacher complexity or VC dim 来刻画假设空间的复杂度。
(3). 有了 generalization bound, 就非常容易导出 excess risk bound, 几乎就是两倍的关系。（参见 Foundations of ML (2nd) Proposition 4.1 ）
(4). 可学习性考虑的是 ERM 算法输出的分类器的泛化误差 与 最优的分类器的泛化误差之间的 gap，其实就是 Excess risk。

参考文献：

• ICML20. Class-Weighted Classification: Trade-offs and Robust Approaches.
• ICML20. Learning with Bounded Instance- and Label-dependent Label Noise.

Plug-in classifiers

Reference

• NIPS09
• ICML20.
• ICML20. Class-Weighted Classification: Trade-offs and Robust Approaches
• 之前审稿的 rejection paper

Loss unbounded below 导致 overfit

不同于 0-1 error， 凸 loss 通常是无界的，会导致给与 outlier 过大的权重

Reference:

• NIPS-09
• ICML-20. Learning with Multiple Complementary Labels
• NIPS-19. Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

0-1 loss non-convex, non-smooth

Bayes classifier 其实是在优化 0-1 loss， 也就是在优化错误的概率。

Reference:

• NeuroComputing-15. Making Risk Minimization Tolerant to Label Noise