转载：DenseNet算法详解

原文连接：http://blog.csdn.net/u014380165/article/details/75142664

参考连接：http://blog.csdn.net/u012938704/article/details/53468483

 

本文这里仅当学习笔记使用，具体细节建议前往原文细度。

论文：Densely Connected Convolutional Networks 
论文链接：https://arxiv.org/pdf/1608.06993.pdf 
代码的github链接：https://github.com/liuzhuang13/DenseNet

文章详解： 
这篇文章是CVPR2017的best paper，文章提出的DenseNet（Dense Convolutional Network）主要还是和ResNet及Inception网络做对比，思想上有借鉴，但却是全新的结构，网络结构并不复杂，却非常有效！众所周知，最近一两年卷积神经网络提高效果的方向，要么深（比如ResNet，解决了网络深时候的梯度消失问题）要么宽（比如GoogleNet的Inception），而作者则是从feature入手，通过对feature的极致利用达到更好的效果和更少的参数。

先列下DenseNet的几个优点，感受下它的强大： 
1、减轻了vanishing-gradient（梯度消失）
2、加强了feature的传递 
3、更有效地利用了feature 
4、一定程度上较少了参数数量

在深度学习网络中，随着网络深度的加深，梯度消失问题会愈加明显，目前很多论文都针对这个问题提出了解决方案，比如ResNet，Highway Networks，Stochastic depth，FractalNets等，尽管这些算法的网络结构有差别，但是核心都在于：create short paths from early layers to later layers。那么作者是怎么做呢？延续这个思路，那就是在保证网络中层与层之间最大程度的信息传输的前提下，直接将所有层连接起来！

先放一个dense block的结构图。在传统的卷积神经网络中，如果你有L层，那么就会有L个连接，但是在DenseNet中，会有L(L+1)/2个连接。简单讲，就是每一层的输入来自前面所有层的输出。如下图：x0是input，H1的输入是x0（input），H2的输入是x0和x1（x1是H1的输出）……

DenseNet的一个优点是网络更窄，参数更少，很大一部分原因得益于这种dense block的设计，后面有提到在dense block中每个卷积层的输出feature map的数量都很小（小于100），而不是像其他网络一样动不动就几百上千的宽度。同时这种连接方式使得特征和梯度的传递更加有效，网络也就更加容易训练。原文的一句话非常喜欢：Each layer has direct access to the gradients from the loss function and the original input signal, leading to an implicit deep supervision.直接解释了为什么这个网络的效果会很好。前面提到过梯度消失问题在网络深度越深的时候越容易出现，原因就是输入信息和梯度信息在很多层之间传递导致的，而现在这种dense connection相当于每一层都直接连接input和loss，因此就可以减轻梯度消失现象，这样更深网络不是问题。另外作者还观察到这种dense connection有正则化的效果，因此对于过拟合有一定的抑制作用，博主认为是因为参数减少了（后面会介绍为什么参数会减少），所以过拟合现象减轻。

这篇文章的一个优点就是基本上没有公式，不像灌水文章一样堆复杂公式把人看得一愣一愣的。文章中只有两个公式，是用来阐述DenseNet和ResNet的关系，对于从原理上理解这两个网络还是非常重要的。

第一个公式是ResNet的。这里的l表示层，xl表示l层的输出，Hl表示一个非线性变换。所以对于ResNet而言，l层的输出是l-1层的输出加上对l-1层输出的非线性变换。

第二个公式是DenseNet的。[x0,x1,…,xl-1]表示将0到l-1层的输出feature map做concatenation。concatenation是做通道的合并，就像Inception那样。而前面resnet是做值的相加，通道数是不变的。Hl包括BN，ReLU和3*3的卷积。

这里Hl(.)是一个Composite function，是三个操作的组合Dense Block：BN−>(Scale)->ReLU−>Conv(3×3)->(Dropout)

所以从这两个公式就能看出DenseNet和ResNet在本质上的区别，太精辟。

前面的Figure 1表示的是dense block，而下面的Figure 2表示的则是一个DenseNet的结构图，在这个结构图中包含了3个dense block。作者将DenseNet分成多个dense block，原因是希望各个dense block内的feature map的size统一，这样在做concatenation就不会有size的问题。

这个Table1就是整个网络的结构图。这个表中的k=32，k=48中的k是growth rate，表示每个dense block中每层输出的feature map个数。为了避免网络变得很宽，作者都是采用较小的k，比如32这样，作者的实验也表明小的k可以有更好的效果。根据dense block的设计，后面几层可以得到前面所有层的输入，因此concat后的输入channel还是比较大的。另外这里每个dense block的3*3卷积前面都包含了一个1*1的卷积操作，就是所谓的bottleneck layer，目的是减少输入的feature map数量，既能降维减少计算量，又能融合各个通道的特征，何乐而不为。另外作者为了进一步压缩参数，在每两个dense block之间又增加了1*1的卷积操作。因此在后面的实验对比中，如果你看到DenseNet-C这个网络，每个dense block直接增加了这个Translation layer，该层的1*1卷积的输出channel默认是输入channel到一半, 具体：由BN−>（Scale）->(Relu)->Conv(1×1)−>(Dropout)->averagePooling(2×2)组成。如果你看到DenseNet-BC这个网络，表示既有bottleneck layer，又有Translation layer。

再详细说下bottleneck和transition layer操作。在每个Dense Block中都包含很多个子结构，以DenseNet-169的Dense Block（3）为例，包含32个1*1和3*3的卷积操作，也就是第32个子结构的输入是前面31层的输出结果，每层输出的channel是32（growth rate），那么如果不做bottleneck操作，第32层的3*3卷积操作的输入就是31*32+（上一个Dense Block的输出channel），近1000了。而加上1*1的卷积，代码中的1*1卷积的channel是growth rate*4，也就是128，然后再作为3*3卷积的输入。这就大大减少了计算量，这就是bottleneck。至于transition layer，放在两个Dense Block中间，是因为每个Dense Block结束后的输出channel个数很多，需要用1*1的卷积核来降维。还是以DenseNet-169的Dense Block（3）为例，虽然第32层的3*3卷积输出channel只有32个（growth rate），但是紧接着还会像前面几层一样有通道的concat操作，即将第32层的输出和第32层的输入做concat，前面说过第32层的输入是1000左右的channel，所以最后每个Dense Block的输出也是1000多的channel。因此这个transition layer有个参数reduction（范围是0到1），表示将这些输出缩小到原来的多少倍，默认是0.5，这样传给下一个Dense Block的时候channel数量就会减少一半，这就是transition layer的作用。文中还用到dropout操作来随机减少分支，避免过拟合，毕竟这篇文章的连接确实多。

实验结果： 
作者在不同数据集上采用的DenseNet网络会有一点不一样，比如在Imagenet数据集上，DenseNet-BC有4个dense block，但是在别的数据集上只用3个dense block。其他更多细节可以看论文3部分的Implementation Details。训练的细节和超参数的设置可以看论文4.2部分，在ImageNet数据集上测试的时候有做224*224的center crop。

Table2是在三个数据集（C10，C100，SVHN）上和其他算法的对比结果。ResNet[11]就是kaiming He的论文，对比结果一目了然。DenseNet-BC的网络参数和相同深度的DenseNet相比确实减少了很多！参数减少除了可以节省内存，还能减少过拟合。这里对于SVHN数据集，DenseNet-BC的结果并没有DenseNet(k=24)的效果好，作者认为原因主要是SVHN这个数据集相对简单，更深的模型容易过拟合。在表格的倒数第二个区域的三个不同深度L和k的DenseNet的对比可以看出随着L和k的增加，模型的效果是更好的。

Figure3是DenseNet-BC和ResNet在Imagenet数据集上的对比，左边那个图是参数复杂度和错误率的对比，你可以在相同错误率下看参数复杂度，也可以在相同参数复杂度下看错误率，提升还是很明显的！右边是flops（可以理解为计算复杂度）和错误率的对比，同样有效果。

Figure4也很重要。左边的图表示不同类型DenseNet的参数和error对比。中间的图表示DenseNet-BC和ResNet在参数和error的对比，相同error下，DenseNet-BC的参数复杂度要小很多。右边的图也是表达DenseNet-BC-100只需要很少的参数就能达到和ResNet-1001相同的结果。

另外提一下DenseNet和stochastic depth的关系，在stochastic depth中，residual中的layers在训练过程中会被随机drop掉，其实这就会使得相邻层之间直接连接，这和DenseNet是很像的。

总结： 
博主读完这篇文章真的有点相见恨晚的感觉，半年前就在arxiv上挂出来了，听说当时就引起了轰动，后来又被选为CVPR2017的oral，感觉要撼动ResNet的地位了，再加上现在很多分类检测的网络都是在ResNet上做的，这岂不是大地震了。惊讶之余来总结下这篇文章，该文章提出的DenseNet核心思想在于建立了不同层之间的连接关系，充分利用了feature，进一步减轻了梯度消失问题，加深网络不是问题，而且训练效果非常好。另外，利用bottleneck layer，Translation layer以及较小的growth rate使得网络变窄，参数减少，有效抑制了过拟合，同时计算量也减少了。DenseNet优点很多，而且在和ResNet的对比中优势还是非常明显的。

最后给出网络结构：

layer {
  name: "Data1"
  type: "Data"
  top: "Data1"
  top: "Data2"
  transform_param {
    mirror: true
    crop_size: 224
    mean_value: 83
    mean_value: 79
    mean_value: 80
    #mean_file: "./data/foodnet_mean.binaryproto"
  }
  data_param {
    source: "./data/densenettrain"
    #batch_size: 64
    batch_size: 1
    backend: LMDB
  }
}
layer {
  name: "Convolution1"
  type: "Convolution"
  bottom: "Data1"
  top: "Convolution1"
  convolution_param {
    num_output: 16
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "BatchNorm1"
  type: "BatchNorm"
  bottom: "Convolution1"
  top: "BatchNorm1"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale1"
  type: "Scale"
  bottom: "BatchNorm1"
  top: "BatchNorm1"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU1"
  type: "ReLU"
  bottom: "BatchNorm1"
  top: "BatchNorm1"
}
layer {
  name: "Convolution2"
  type: "Convolution"
  bottom: "BatchNorm1"
  top: "Convolution2"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout1"
  type: "Dropout"
  bottom: "Convolution2"
  top: "Dropout1"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat1"
  type: "Concat"
  bottom: "Convolution1"
  bottom: "Dropout1"
  top: "Concat1"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm2"
  type: "BatchNorm"
  bottom: "Concat1"
  top: "BatchNorm2"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale2"
  type: "Scale"
  bottom: "BatchNorm2"
  top: "BatchNorm2"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU2"
  type: "ReLU"
  bottom: "BatchNorm2"
  top: "BatchNorm2"
}
layer {
  name: "Convolution3"
  type: "Convolution"
  bottom: "BatchNorm2"
  top: "Convolution3"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout2"
  type: "Dropout"
  bottom: "Convolution3"
  top: "Dropout2"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat2"
  type: "Concat"
  bottom: "Concat1"
  bottom: "Dropout2"
  top: "Concat2"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm3"
  type: "BatchNorm"
  bottom: "Concat2"
  top: "BatchNorm3"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale3"
  type: "Scale"
  bottom: "BatchNorm3"
  top: "BatchNorm3"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU3"
  type: "ReLU"
  bottom: "BatchNorm3"
  top: "BatchNorm3"
}
layer {
  name: "Convolution4"
  type: "Convolution"
  bottom: "BatchNorm3"
  top: "Convolution4"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout3"
  type: "Dropout"
  bottom: "Convolution4"
  top: "Dropout3"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat3"
  type: "Concat"
  bottom: "Concat2"
  bottom: "Dropout3"
  top: "Concat3"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm4"
  type: "BatchNorm"
  bottom: "Concat3"
  top: "BatchNorm4"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale4"
  type: "Scale"
  bottom: "BatchNorm4"
  top: "BatchNorm4"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU4"
  type: "ReLU"
  bottom: "BatchNorm4"
  top: "BatchNorm4"
}
layer {
  name: "Convolution5"
  type: "Convolution"
  bottom: "BatchNorm4"
  top: "Convolution5"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout4"
  type: "Dropout"
  bottom: "Convolution5"
  top: "Dropout4"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat4"
  type: "Concat"
  bottom: "Concat3"
  bottom: "Dropout4"
  top: "Concat4"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm5"
  type: "BatchNorm"
  bottom: "Concat4"
  top: "BatchNorm5"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale5"
  type: "Scale"
  bottom: "BatchNorm5"
  top: "BatchNorm5"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU5"
  type: "ReLU"
  bottom: "BatchNorm5"
  top: "BatchNorm5"
}
layer {
  name: "Convolution6"
  type: "Convolution"
  bottom: "BatchNorm5"
  top: "Convolution6"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout5"
  type: "Dropout"
  bottom: "Convolution6"
  top: "Dropout5"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat5"
  type: "Concat"
  bottom: "Concat4"
  bottom: "Dropout5"
  top: "Concat5"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm6"
  type: "BatchNorm"
  bottom: "Concat5"
  top: "BatchNorm6"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale6"
  type: "Scale"
  bottom: "BatchNorm6"
  top: "BatchNorm6"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU6"
  type: "ReLU"
  bottom: "BatchNorm6"
  top: "BatchNorm6"
}
layer {
  name: "Convolution7"
  type: "Convolution"
  bottom: "BatchNorm6"
  top: "Convolution7"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout6"
  type: "Dropout"
  bottom: "Convolution7"
  top: "Dropout6"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat6"
  type: "Concat"
  bottom: "Concat5"
  bottom: "Dropout6"
  top: "Concat6"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm7"
  type: "BatchNorm"
  bottom: "Concat6"
  top: "BatchNorm7"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale7"
  type: "Scale"
  bottom: "BatchNorm7"
  top: "BatchNorm7"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU7"
  type: "ReLU"
  bottom: "BatchNorm7"
  top: "BatchNorm7"
}
layer {
  name: "Convolution8"
  type: "Convolution"
  bottom: "BatchNorm7"
  top: "Convolution8"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout7"
  type: "Dropout"
  bottom: "Convolution8"
  top: "Dropout7"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat7"
  type: "Concat"
  bottom: "Concat6"
  bottom: "Dropout7"
  top: "Concat7"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm8"
  type: "BatchNorm"
  bottom: "Concat7"
  top: "BatchNorm8"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale8"
  type: "Scale"
  bottom: "BatchNorm8"
  top: "BatchNorm8"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU8"
  type: "ReLU"
  bottom: "BatchNorm8"
  top: "BatchNorm8"
}
layer {
  name: "Convolution9"
  type: "Convolution"
  bottom: "BatchNorm8"
  top: "Convolution9"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout8"
  type: "Dropout"
  bottom: "Convolution9"
  top: "Dropout8"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat8"
  type: "Concat"
  bottom: "Concat7"
  bottom: "Dropout8"
  top: "Concat8"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm9"
  type: "BatchNorm"
  bottom: "Concat8"
  top: "BatchNorm9"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale9"
  type: "Scale"
  bottom: "BatchNorm9"
  top: "BatchNorm9"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU9"
  type: "ReLU"
  bottom: "BatchNorm9"
  top: "BatchNorm9"
}
layer {
  name: "Convolution10"
  type: "Convolution"
  bottom: "BatchNorm9"
  top: "Convolution10"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout9"
  type: "Dropout"
  bottom: "Convolution10"
  top: "Dropout9"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat9"
  type: "Concat"
  bottom: "Concat8"
  bottom: "Dropout9"
  top: "Concat9"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm10"
  type: "BatchNorm"
  bottom: "Concat9"
  top: "BatchNorm10"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale10"
  type: "Scale"
  bottom: "BatchNorm10"
  top: "BatchNorm10"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU10"
  type: "ReLU"
  bottom: "BatchNorm10"
  top: "BatchNorm10"
}
layer {
  name: "Convolution11"
  type: "Convolution"
  bottom: "BatchNorm10"
  top: "Convolution11"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout10"
  type: "Dropout"
  bottom: "Convolution11"
  top: "Dropout10"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat10"
  type: "Concat"
  bottom: "Concat9"
  bottom: "Dropout10"
  top: "Concat10"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm11"
  type: "BatchNorm"
  bottom: "Concat10"
  top: "BatchNorm11"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale11"
  type: "Scale"
  bottom: "BatchNorm11"
  top: "BatchNorm11"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU11"
  type: "ReLU"
  bottom: "BatchNorm11"
  top: "BatchNorm11"
}
layer {
  name: "Convolution12"
  type: "Convolution"
  bottom: "BatchNorm11"
  top: "Convolution12"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout11"
  type: "Dropout"
  bottom: "Convolution12"
  top: "Dropout11"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat11"
  type: "Concat"
  bottom: "Concat10"
  bottom: "Dropout11"
  top: "Concat11"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm12"
  type: "BatchNorm"
  bottom: "Concat11"
  top: "BatchNorm12"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale12"
  type: "Scale"
  bottom: "BatchNorm12"
  top: "BatchNorm12"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU12"
  type: "ReLU"
  bottom: "BatchNorm12"
  top: "BatchNorm12"
}
layer {
  name: "Convolution13"
  type: "Convolution"
  bottom: "BatchNorm12"
  top: "Convolution13"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout12"
  type: "Dropout"
  bottom: "Convolution13"
  top: "Dropout12"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat12"
  type: "Concat"
  bottom: "Concat11"
  bottom: "Dropout12"
  top: "Concat12"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm13"
  type: "BatchNorm"
  bottom: "Concat12"
  top: "BatchNorm13"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale13"
  type: "Scale"
  bottom: "BatchNorm13"
  top: "BatchNorm13"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU13"
  type: "ReLU"
  bottom: "BatchNorm13"
  top: "BatchNorm13"
}
layer {
  name: "Convolution14"
  type: "Convolution"
  bottom: "BatchNorm13"
  top: "Convolution14"
  convolution_param {
    num_output: 160
    bias_term: false
    pad: 0
    kernel_size: 1
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout13"
  type: "Dropout"
  bottom: "Convolution14"
  top: "Dropout13"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Pooling1"
  type: "Pooling"
  bottom: "Dropout13"
  top: "Pooling1"
  pooling_param {
    pool: AVE
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "BatchNorm14"
  type: "BatchNorm"
  bottom: "Pooling1"
  top: "BatchNorm14"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale14"
  type: "Scale"
  bottom: "BatchNorm14"
  top: "BatchNorm14"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU14"
  type: "ReLU"
  bottom: "BatchNorm14"
  top: "BatchNorm14"
}
layer {
  name: "Convolution15"
  type: "Convolution"
  bottom: "BatchNorm14"
  top: "Convolution15"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout14"
  type: "Dropout"
  bottom: "Convolution15"
  top: "Dropout14"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat13"
  type: "Concat"
  bottom: "Pooling1"
  bottom: "Dropout14"
  top: "Concat13"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm15"
  type: "BatchNorm"
  bottom: "Concat13"
  top: "BatchNorm15"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale15"
  type: "Scale"
  bottom: "BatchNorm15"
  top: "BatchNorm15"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU15"
  type: "ReLU"
  bottom: "BatchNorm15"
  top: "BatchNorm15"
}
layer {
  name: "Convolution16"
  type: "Convolution"
  bottom: "BatchNorm15"
  top: "Convolution16"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout15"
  type: "Dropout"
  bottom: "Convolution16"
  top: "Dropout15"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat14"
  type: "Concat"
  bottom: "Concat13"
  bottom: "Dropout15"
  top: "Concat14"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm16"
  type: "BatchNorm"
  bottom: "Concat14"
  top: "BatchNorm16"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale16"
  type: "Scale"
  bottom: "BatchNorm16"
  top: "BatchNorm16"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU16"
  type: "ReLU"
  bottom: "BatchNorm16"
  top: "BatchNorm16"
}
layer {
  name: "Convolution17"
  type: "Convolution"
  bottom: "BatchNorm16"
  top: "Convolution17"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout16"
  type: "Dropout"
  bottom: "Convolution17"
  top: "Dropout16"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat15"
  type: "Concat"
  bottom: "Concat14"
  bottom: "Dropout16"
  top: "Concat15"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm17"
  type: "BatchNorm"
  bottom: "Concat15"
  top: "BatchNorm17"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale17"
  type: "Scale"
  bottom: "BatchNorm17"
  top: "BatchNorm17"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU17"
  type: "ReLU"
  bottom: "BatchNorm17"
  top: "BatchNorm17"
}
layer {
  name: "Convolution18"
  type: "Convolution"
  bottom: "BatchNorm17"
  top: "Convolution18"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout17"
  type: "Dropout"
  bottom: "Convolution18"
  top: "Dropout17"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat16"
  type: "Concat"
  bottom: "Concat15"
  bottom: "Dropout17"
  top: "Concat16"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm18"
  type: "BatchNorm"
  bottom: "Concat16"
  top: "BatchNorm18"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale18"
  type: "Scale"
  bottom: "BatchNorm18"
  top: "BatchNorm18"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU18"
  type: "ReLU"
  bottom: "BatchNorm18"
  top: "BatchNorm18"
}
layer {
  name: "Convolution19"
  type: "Convolution"
  bottom: "BatchNorm18"
  top: "Convolution19"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout18"
  type: "Dropout"
  bottom: "Convolution19"
  top: "Dropout18"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat17"
  type: "Concat"
  bottom: "Concat16"
  bottom: "Dropout18"
  top: "Concat17"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm19"
  type: "BatchNorm"
  bottom: "Concat17"
  top: "BatchNorm19"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale19"
  type: "Scale"
  bottom: "BatchNorm19"
  top: "BatchNorm19"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU19"
  type: "ReLU"
  bottom: "BatchNorm19"
  top: "BatchNorm19"
}
layer {
  name: "Convolution20"
  type: "Convolution"
  bottom: "BatchNorm19"
  top: "Convolution20"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout19"
  type: "Dropout"
  bottom: "Convolution20"
  top: "Dropout19"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat18"
  type: "Concat"
  bottom: "Concat17"
  bottom: "Dropout19"
  top: "Concat18"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm20"
  type: "BatchNorm"
  bottom: "Concat18"
  top: "BatchNorm20"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale20"
  type: "Scale"
  bottom: "BatchNorm20"
  top: "BatchNorm20"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU20"
  type: "ReLU"
  bottom: "BatchNorm20"
  top: "BatchNorm20"
}
layer {
  name: "Convolution21"
  type: "Convolution"
  bottom: "BatchNorm20"
  top: "Convolution21"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout20"
  type: "Dropout"
  bottom: "Convolution21"
  top: "Dropout20"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat19"
  type: "Concat"
  bottom: "Concat18"
  bottom: "Dropout20"
  top: "Concat19"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm21"
  type: "BatchNorm"
  bottom: "Concat19"
  top: "BatchNorm21"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale21"
  type: "Scale"
  bottom: "BatchNorm21"
  top: "BatchNorm21"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU21"
  type: "ReLU"
  bottom: "BatchNorm21"
  top: "BatchNorm21"
}
layer {
  name: "Convolution22"
  type: "Convolution"
  bottom: "BatchNorm21"
  top: "Convolution22"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout21"
  type: "Dropout"
  bottom: "Convolution22"
  top: "Dropout21"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat20"
  type: "Concat"
  bottom: "Concat19"
  bottom: "Dropout21"
  top: "Concat20"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm22"
  type: "BatchNorm"
  bottom: "Concat20"
  top: "BatchNorm22"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale22"
  type: "Scale"
  bottom: "BatchNorm22"
  top: "BatchNorm22"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU22"
  type: "ReLU"
  bottom: "BatchNorm22"
  top: "BatchNorm22"
}
layer {
  name: "Convolution23"
  type: "Convolution"
  bottom: "BatchNorm22"
  top: "Convolution23"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout22"
  type: "Dropout"
  bottom: "Convolution23"
  top: "Dropout22"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat21"
  type: "Concat"
  bottom: "Concat20"
  bottom: "Dropout22"
  top: "Concat21"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm23"
  type: "BatchNorm"
  bottom: "Concat21"
  top: "BatchNorm23"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale23"
  type: "Scale"
  bottom: "BatchNorm23"
  top: "BatchNorm23"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU23"
  type: "ReLU"
  bottom: "BatchNorm23"
  top: "BatchNorm23"
}
layer {
  name: "Convolution24"
  type: "Convolution"
  bottom: "BatchNorm23"
  top: "Convolution24"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout23"
  type: "Dropout"
  bottom: "Convolution24"
  top: "Dropout23"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat22"
  type: "Concat"
  bottom: "Concat21"
  bottom: "Dropout23"
  top: "Concat22"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm24"
  type: "BatchNorm"
  bottom: "Concat22"
  top: "BatchNorm24"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale24"
  type: "Scale"
  bottom: "BatchNorm24"
  top: "BatchNorm24"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU24"
  type: "ReLU"
  bottom: "BatchNorm24"
  top: "BatchNorm24"
}
layer {
  name: "Convolution25"
  type: "Convolution"
  bottom: "BatchNorm24"
  top: "Convolution25"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout24"
  type: "Dropout"
  bottom: "Convolution25"
  top: "Dropout24"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat23"
  type: "Concat"
  bottom: "Concat22"
  bottom: "Dropout24"
  top: "Concat23"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm25"
  type: "BatchNorm"
  bottom: "Concat23"
  top: "BatchNorm25"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale25"
  type: "Scale"
  bottom: "BatchNorm25"
  top: "BatchNorm25"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU25"
  type: "ReLU"
  bottom: "BatchNorm25"
  top: "BatchNorm25"
}
layer {
  name: "Convolution26"
  type: "Convolution"
  bottom: "BatchNorm25"
  top: "Convolution26"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout25"
  type: "Dropout"
  bottom: "Convolution26"
  top: "Dropout25"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat24"
  type: "Concat"
  bottom: "Concat23"
  bottom: "Dropout25"
  top: "Concat24"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm26"
  type: "BatchNorm"
  bottom: "Concat24"
  top: "BatchNorm26"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale26"
  type: "Scale"
  bottom: "BatchNorm26"
  top: "BatchNorm26"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU26"
  type: "ReLU"
  bottom: "BatchNorm26"
  top: "BatchNorm26"
}
layer {
  name: "Convolution27"
  type: "Convolution"
  bottom: "BatchNorm26"
  top: "Convolution27"
  convolution_param {
    num_output: 304
    bias_term: false
    pad: 0
    kernel_size: 1
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout26"
  type: "Dropout"
  bottom: "Convolution27"
  top: "Dropout26"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Pooling2"
  type: "Pooling"
  bottom: "Dropout26"
  top: "Pooling2"
  pooling_param {
    pool: AVE
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "BatchNorm27"
  type: "BatchNorm"
  bottom: "Pooling2"
  top: "BatchNorm27"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale27"
  type: "Scale"
  bottom: "BatchNorm27"
  top: "BatchNorm27"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU27"
  type: "ReLU"
  bottom: "BatchNorm27"
  top: "BatchNorm27"
}
layer {
  name: "Convolution28"
  type: "Convolution"
  bottom: "BatchNorm27"
  top: "Convolution28"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout27"
  type: "Dropout"
  bottom: "Convolution28"
  top: "Dropout27"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat25"
  type: "Concat"
  bottom: "Pooling2"
  bottom: "Dropout27"
  top: "Concat25"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm28"
  type: "BatchNorm"
  bottom: "Concat25"
  top: "BatchNorm28"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale28"
  type: "Scale"
  bottom: "BatchNorm28"
  top: "BatchNorm28"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU28"
  type: "ReLU"
  bottom: "BatchNorm28"
  top: "BatchNorm28"
}
layer {
  name: "Convolution29"
  type: "Convolution"
  bottom: "BatchNorm28"
  top: "Convolution29"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout28"
  type: "Dropout"
  bottom: "Convolution29"
  top: "Dropout28"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat26"
  type: "Concat"
  bottom: "Concat25"
  bottom: "Dropout28"
  top: "Concat26"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm29"
  type: "BatchNorm"
  bottom: "Concat26"
  top: "BatchNorm29"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale29"
  type: "Scale"
  bottom: "BatchNorm29"
  top: "BatchNorm29"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU29"
  type: "ReLU"
  bottom: "BatchNorm29"
  top: "BatchNorm29"
}
layer {
  name: "Convolution30"
  type: "Convolution"
  bottom: "BatchNorm29"
  top: "Convolution30"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout29"
  type: "Dropout"
  bottom: "Convolution30"
  top: "Dropout29"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat27"
  type: "Concat"
  bottom: "Concat26"
  bottom: "Dropout29"
  top: "Concat27"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm30"
  type: "BatchNorm"
  bottom: "Concat27"
  top: "BatchNorm30"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale30"
  type: "Scale"
  bottom: "BatchNorm30"
  top: "BatchNorm30"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU30"
  type: "ReLU"
  bottom: "BatchNorm30"
  top: "BatchNorm30"
}
layer {
  name: "Convolution31"
  type: "Convolution"
  bottom: "BatchNorm30"
  top: "Convolution31"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout30"
  type: "Dropout"
  bottom: "Convolution31"
  top: "Dropout30"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat28"
  type: "Concat"
  bottom: "Concat27"
  bottom: "Dropout30"
  top: "Concat28"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm31"
  type: "BatchNorm"
  bottom: "Concat28"
  top: "BatchNorm31"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale31"
  type: "Scale"
  bottom: "BatchNorm31"
  top: "BatchNorm31"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU31"
  type: "ReLU"
  bottom: "BatchNorm31"
  top: "BatchNorm31"
}
layer {
  name: "Convolution32"
  type: "Convolution"
  bottom: "BatchNorm31"
  top: "Convolution32"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout31"
  type: "Dropout"
  bottom: "Convolution32"
  top: "Dropout31"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat29"
  type: "Concat"
  bottom: "Concat28"
  bottom: "Dropout31"
  top: "Concat29"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm32"
  type: "BatchNorm"
  bottom: "Concat29"
  top: "BatchNorm32"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale32"
  type: "Scale"
  bottom: "BatchNorm32"
  top: "BatchNorm32"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU32"
  type: "ReLU"
  bottom: "BatchNorm32"
  top: "BatchNorm32"
}
layer {
  name: "Convolution33"
  type: "Convolution"
  bottom: "BatchNorm32"
  top: "Convolution33"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout32"
  type: "Dropout"
  bottom: "Convolution33"
  top: "Dropout32"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat30"
  type: "Concat"
  bottom: "Concat29"
  bottom: "Dropout32"
  top: "Concat30"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm33"
  type: "BatchNorm"
  bottom: "Concat30"
  top: "BatchNorm33"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale33"
  type: "Scale"
  bottom: "BatchNorm33"
  top: "BatchNorm33"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU33"
  type: "ReLU"
  bottom: "BatchNorm33"
  top: "BatchNorm33"
}
layer {
  name: "Convolution34"
  type: "Convolution"
  bottom: "BatchNorm33"
  top: "Convolution34"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout33"
  type: "Dropout"
  bottom: "Convolution34"
  top: "Dropout33"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat31"
  type: "Concat"
  bottom: "Concat30"
  bottom: "Dropout33"
  top: "Concat31"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm34"
  type: "BatchNorm"
  bottom: "Concat31"
  top: "BatchNorm34"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale34"
  type: "Scale"
  bottom: "BatchNorm34"
  top: "BatchNorm34"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU34"
  type: "ReLU"
  bottom: "BatchNorm34"
  top: "BatchNorm34"
}
layer {
  name: "Convolution35"
  type: "Convolution"
  bottom: "BatchNorm34"
  top: "Convolution35"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout34"
  type: "Dropout"
  bottom: "Convolution35"
  top: "Dropout34"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat32"
  type: "Concat"
  bottom: "Concat31"
  bottom: "Dropout34"
  top: "Concat32"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm35"
  type: "BatchNorm"
  bottom: "Concat32"
  top: "BatchNorm35"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale35"
  type: "Scale"
  bottom: "BatchNorm35"
  top: "BatchNorm35"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU35"
  type: "ReLU"
  bottom: "BatchNorm35"
  top: "BatchNorm35"
}
layer {
  name: "Convolution36"
  type: "Convolution"
  bottom: "BatchNorm35"
  top: "Convolution36"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout35"
  type: "Dropout"
  bottom: "Convolution36"
  top: "Dropout35"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat33"
  type: "Concat"
  bottom: "Concat32"
  bottom: "Dropout35"
  top: "Concat33"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm36"
  type: "BatchNorm"
  bottom: "Concat33"
  top: "BatchNorm36"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale36"
  type: "Scale"
  bottom: "BatchNorm36"
  top: "BatchNorm36"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU36"
  type: "ReLU"
  bottom: "BatchNorm36"
  top: "BatchNorm36"
}
layer {
  name: "Convolution37"
  type: "Convolution"
  bottom: "BatchNorm36"
  top: "Convolution37"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout36"
  type: "Dropout"
  bottom: "Convolution37"
  top: "Dropout36"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat34"
  type: "Concat"
  bottom: "Concat33"
  bottom: "Dropout36"
  top: "Concat34"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm37"
  type: "BatchNorm"
  bottom: "Concat34"
  top: "BatchNorm37"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale37"
  type: "Scale"
  bottom: "BatchNorm37"
  top: "BatchNorm37"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU37"
  type: "ReLU"
  bottom: "BatchNorm37"
  top: "BatchNorm37"
}
layer {
  name: "Convolution38"
  type: "Convolution"
  bottom: "BatchNorm37"
  top: "Convolution38"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout37"
  type: "Dropout"
  bottom: "Convolution38"
  top: "Dropout37"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat35"
  type: "Concat"
  bottom: "Concat34"
  bottom: "Dropout37"
  top: "Concat35"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm38"
  type: "BatchNorm"
  bottom: "Concat35"
  top: "BatchNorm38"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale38"
  type: "Scale"
  bottom: "BatchNorm38"
  top: "BatchNorm38"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU38"
  type: "ReLU"
  bottom: "BatchNorm38"
  top: "BatchNorm38"
}
layer {
  name: "Convolution39"
  type: "Convolution"
  bottom: "BatchNorm38"
  top: "Convolution39"
  convolution_param {
    num_output: 12
    bias_term: false
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "Dropout38"
  type: "Dropout"
  bottom: "Convolution39"
  top: "Dropout38"
  dropout_param {
    dropout_ratio: 0.2
  }
}
layer {
  name: "Concat36"
  type: "Concat"
  bottom: "Concat35"
  bottom: "Dropout38"
  top: "Concat36"
  concat_param {
    axis: 1
  }
}
layer {
  name: "BatchNorm39"
  type: "BatchNorm"
  bottom: "Concat36"
  top: "BatchNorm39"
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
  param {
    lr_mult: 0
    decay_mult: 0
  }
}
layer {
  name: "Scale39"
  type: "Scale"
  bottom: "BatchNorm39"
  top: "BatchNorm39"
  scale_param {
    filler {
      value: 1
    }
    bias_term: true
    bias_filler {
      value: 0
    }
  }
}
layer {
  name: "ReLU39"
  type: "ReLU"
  bottom: "BatchNorm39"
  top: "BatchNorm39"
}
layer {
  name: "Pooling3"
  type: "Pooling"
  bottom: "BatchNorm39"
  top: "Pooling3"
  pooling_param {
    pool: AVE
    global_pooling: true
  }
}
layer {
  name: "InnerProduct1"
  type: "InnerProduct"
  bottom: "Pooling3"
  top: "InnerProduct1"
  inner_product_param {
    num_output: 1000
    bias_term: true
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "SoftmaxWithLoss1"
  type: "SoftmaxWithLoss"
  bottom: "InnerProduct1"
  bottom: "Data2"
  top: "SoftmaxWithLoss1"
}
layer {
  name: "Accuracy1"
  type: "Accuracy"
  bottom: "InnerProduct1"
  bottom: "Data2"
  top: "Accuracy1"
}