zoukankan      html  css  js  c++  java
  • 对Kaldi nnet3进行奇异值分解(SVD)以减小模型大小

    用处

    基于SVD实现模型压缩以适配低功耗平台

       

    根据nnet3bin/nnet3-copynnet3-copynnet3-am-copy的"--edits-config"参数中,新支持了以下选项:

    apply-svd name=<name-pattern> bottleneck-dim=<dim>

    查找所有名字与<name-pattern>匹配的组件,类型需要是AffineComponent或其子类。如果<dim>小于组件的输入或输出维数,则对组件参数进行奇异值分解,只保留最大<dim>奇异值,将这些组件替换为两个组件:LinearComponentNaturalGradientAffineComponent(的序列)。又见'reduce-rank'

       

    示例cd

    dir=`mktemp -d`

    nnet3-am-copy --edits='apply-svd name=*.affine bottleneck-dim=64' $dir/final.mdl $dir/final_svd.mdl

       

    vimdiff <(nnet3-info --print-args=false $dir/final.raw 2>&1|sort) <(nnet3-info --print-args=false $dir/final.raw 2>&1|sort)

       

    component-node name=tdnn1.affine component=tdnn1.affine input=lda input-dim=195 output-dim=1024

    component name=tdnn1.affine type=NaturalGradientAffineComponent, input-dim=195, output-dim=1024, learning-rate=0.00136, max-change=0.75, linear-params-rms=0.4864, linear-params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.92,1.0,1.1,1.2 1.3,1.6,6.4,8.8,10 11,13,14,17), mean=5.79, stddev=3.56], linear-params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(5.0,5.5,8.3,12 13,14,15,17,18 19,20,20,24), mean=15.4, stddev=2.46], bias-{mean,stddev}=-0.06099,0.2027, rank-in=20, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

    component-node name=tdnn1.affine_a component=tdnn1.affine_a input=lda input-dim=195 output-dim=64

    component-node name=tdnn1.affine_b component=tdnn1.affine_b input=tdnn1.affine_a input-dim=64 output-dim=1024

    component name=tdnn1.affine_a type=LinearComponent, input-dim=195, output-dim=64, learning-rate=0.00136, max-change=0.75, params-rms=0.3461, params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(4.0,4.0,4.0,4.1 4.2,4.3,4.8,5.2,5.5 5.6,5.8,5.8,6.0), mean=4.81, stddev=0.496], params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.36,0.45,0.75,1.9 2.0,2.3,2.7,3.2,3.4 3.6,3.7,3.9,4.7), mean=2.7, stddev=0.618], use-natural-gradient=true, rank-in=40, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

    component name=tdnn1.affine_b type=NaturalGradientAffineComponent, input-dim=64, output-dim=1024, learning-rate=0.00136, max-change=0.75, linear-params-rms=0.151, linear-params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.12,0.13,0.14,0.16 0.19,0.23,1.1,1.6,1.8 2.1,2.5,2.6,3.8), mean=1, stddev=0.676], linear-params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(4.0,4.0,4.0,4.1 4.2,4.3,4.8,5.2,5.5 5.6,5.8,5.8,6.0), mean=4.81, stddev=0.496], bias-{mean,stddev}=-0.06099,0.2027, rank-in=20, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

    结果

    经过解码测试,SVD后的模型识别率极差,完全无法使用。

    需要再对模型进行retrain

       

    使用SVD实现模型压缩后,再进行几轮迭代

    在已有训练样本的情况在,假设总iteration=2000,,将final.mdl进行SVD得到final_svd.mdl,再链接为0.mdl,运行一个epochlocal/chain/run_tdnn.sh --stage 16 --num_epochs 1

    在之前的epoch的基础上,再训几个epochs

    local/chain/run_tdnn.sh --stage 16 --num_epochs 2 --train_stage 1155

  • 相关阅读:
    5G网络逐渐普及TSINGSEE青犀视频云边端架构网页视频实时互动直播系统又将如何发展?
    【开发记录】TSINGSEE青犀视频云边端架构Visual Studio 2017自建WebRTC中peerconnection_client编译报无法解析错误
    安防视频云服务平台EasyCVR视频智能分析系统运行控制台报404错误如何排查?
    一对一或一对多音视频通话会议系统可以通过哪些方式实现?
    TSINGSEE青犀视频云边端视频智能分析平台开发VMware下安装Ubuntu系统后无法安装VMwaretools问题解决
    最简单的Windows套接字(Socket)例子(源码,实例)
    KJAVA虚拟机Hack笔记实现MIDP的SLAVE事件模型
    系统程序员成长计划你的数据放在哪里(下)
    使用new实现realloc操作
    KJava虚拟机hack笔记基于GTK的移植
  • 原文地址:https://www.cnblogs.com/JarvanWang/p/11063702.html
Copyright © 2011-2022 走看看