BlockSwap
2020-ICLR-BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
来源: ChenBong 博客园
- Institute:University of Edinburgh
- Author:Jack Turner,Michael O'Boyle
- GitHub:https://github.com/BayesWatch/pytorch-blockswap 【20+】
- Citation:1
Introduction
- backbone network (consist of standard block)
-
- cheap blocks pool ==> (swap backbone's block) candidate blcok-swap networks space
-
- under constraint sample ==> compute fisher score ==> ranking network
-
- Distillation (T: backbone network, S: block-swap network)
Contribution
- block-wise 的 替换,相对于 NAS(bottom-up method)来说降低了搜索空间,速度更快,相对于 改变网络的深度/宽度方法,如剪枝等(top-dowm method)来说搜索维度更高(剪枝只能修改每一层的filter个数,block swap 还可以修改层的类型)
- 基于 Fisher information 的候选网络快速评估算法
Method
Fisher Information
- 泰勒展开来计算filter重要性的方法 与 计算 Fisher information 的方法等价
- (Delta_{c}=frac{1}{2 N} sum_{n}^{N}left(sum_{i}^{W} sum_{j}^{H} a_{n i j} g_{n i j}
ight)^{2})
- feature map大小 W×H, (sum a_{ij}*g_{ij}) 衡量一个(filter输出的)channel 的重要性 (Delta_{c})
- (Delta_{b}=sum_{c}^{C} Delta_{c})
- C是一个 blcok 的总通道数;一个block的重要性表示为 (Delta_{b})
- (sum_B Delta_{b})
- B 是一个 swap-block network 的 blcok 数量,一个 swap-block network 的重要性表示为: (sum_B Delta_{b})
Substitute Blocks
-
Standard Block
- 参数量: (2N^2k^2)
-
Grouped+Pointwise Block – G(g)
- 参数量: (2((N^2k^2)/g+N^2))
-
Bottleneck Block – B(b)
- 参数量:((N/b)^2k^2+2N^2/b)
-
Bottleneck Grouped+Pointwise Block – BG(b, g)
- 参数量: ((N/bg)^2k^2+2N^2/b)
Distillation
(mathcal{L}_{A T}=mathcal{L}_{C E}+eta sum_{i=1}^{L}left|frac{mathbf{f}left(A_{i}^{t} ight)}{left|mathbf{f}left(A_{i}^{t} ight) ight|_{2}}-frac{mathbf{f}left(A_{i}^{s} ight)}{left|mathbf{f}left(A_{i}^{s} ight) ight|_{2}} ight|_{2} qquad (1))
(mathbf{f}left(A_{i} ight)=left(1 / N_{A_{i}} ight) sum_{j=1}^{N_{A_{i}}} mathbf{a}_{i j}^{2}) ,其中 (i=1,2,...,L) ,(N_{A_i}) is the number of channels at layer i.
- blocks pool ==> candidate blcok-swap networks space ==>
- under constraint sample ==> compute fisher score ==> ranking network
- Distillation (T: backbone network, S: block-swap network)
Experiments
CIFAR-10
Setup
- momentum:0.9
- lr:init 0.1,cosine
- minibatch size:128
- weight decay:5e-4
- β:1000
Teacher Network:
- 3 个 WRN-40-2(depth 40,width multiplier 2,18 blocks,2.2M params)
Student Network:
params constraint:200K, 400K, 600K, 800K
- WRN-16-2 / WRN-40-1 / WRN-16-1
- WRN-40-2 + mixed swap
- WRN-40-2 + Single swap (MBConv6 / DARTS / DenseNet)
- WRN-40-2 + SNIP pruning
- WRN-40-2 + (l1) pruning
ImageNet
Setup
- momentum:0.9
- lr:init 0.1,step:30,60,90
- minibatch size:256
- weight decay:1e-4
- β:750
Teacher Network:
- 1 个 ResNet34(16 blocks, 21.8M params)
Student Network:
params constraint:8M,3M
- ResNet18 / ResNet18-0.5 (the channel width in the last 3 sections has been halved)
- ResNet34 + mixed swap
- ResNet34 + Single swap (G(4) / G(N))
Ablation Study
mixed block VS. single blcok
mix swap 总是存在比 single swap 更好的结构
One minibatch VS. N minibatch && Ranking correlation
final err 与不同 batch 时下列指标的相关性:
- acc
- weight l2 norm
- grad l1 norm
- fisher score
Sample Num?
BlockSwap finds networks with final test errors of 4.85%. 4.54%, and 4.21% after 10, 100, and 1000 samples respectively.
We empirically found that 1000 samples.
Conclusion
Summary
To Read
Reference
https://blog.csdn.net/xbinworld/article/details/104591706