zoukankan      html  css  js  c++  java
  • What are the advantages of logistic regression over decision trees?FAQ

    What are the advantages of logistic regression over decision trees?FAQ

    The answer to "Should I ever use learning algorithm (a) over learning algorithm (b)" will pretty much always be yes. Different learning algorithms make different assumptions about the data and have different rates of convergence. The one which works best, i.e. minimizes some cost function of interest (cross validation for example) will be the one that makes assumptions that are consistent with the data and has sufficiently converged to its error rate.

    Put in the context of decision trees vs. logistic regression, what are the assumptions made?

    Decision trees assume that our decision boundaries are parallel to the axes, for example if we have two features (x1, x2) then it can only create rules such as x1>=4.5, x2>=6.5 etc. which we can visualize as lines parallel to the axis. We see this in practice in the diagram below.

    So decision trees chop up the feature space into rectangles (or in higher dimensions, hyper-rectangles). There can be many partitions made and so decision trees naturally scale up to creating more complex (say, higher VC) functions - which can be a problem with over-fitting.

    What assumptions does logistic regression make? Despite the probabilistic framework of logistic regression, all that logistic regression assumes is that there is one smooth linear decision boundary. It finds that linear decision boundary by making assumptions that the P(Y|X) of some form, like the inverse logit function applied to a weighted sum of our features. Then it finds the weights by a maximum likelihood approach. 

    However people get too caught up on that... The decision boundary it creates is a linear* decision boundary that can be of any direction. So if you have data where the decision boundary is not parallel to the axes,

    then logistic regression picks it out pretty well, whereas a decision tree will have problems.

    So in conclusion,

    • Both algorithms are really fast. There isn't much to distinguish them in terms of run-time.
    • Logistic regression will work better if there's a single decision boundary, not necessarily parallel to the axis.
    • Decision trees can be applied to situations where there's not just one underlying decision boundary, but many, and will work best if the class labels roughly lie in hyper-rectangular regions.
    • Logistic regression is intrinsically simple, it has low variance and so is less prone to over-fitting. Decision trees can be scaled up to be very complex, are are more liable to over-fit. Pruning is applied to avoid this.


    Maybe you'll be left thinking, "I wish decision trees didn't have to create rules that are parallel to the axis." This motivates support vector machines.

    Footnotes:
    * linear in your covariates. If you include non-linear transformations or interactions then it will be non-linear in the space of those original covariates.

  • 相关阅读:
    C#,asp.net,命名空间名,类名,方法名的获得
    asp.net引用用户控件
    SQL数据是否存在(是否有数据)判断,表,存储过程是否存在
    asp:Button 事件,点击事件 html Button runat="sever"
    CSS图片最大大小限制
    asp.net 路径
    Js实现网站的重定向,Js转向网址,Js跳转
    ViewState 页面状态保留
    vs 附加到进程
    sql XML处理,sp_xml_preparedocument,openxml
  • 原文地址:https://www.cnblogs.com/yymn/p/4662396.html
Copyright © 2011-2022 走看看