zoukankan      html  css  js  c++  java
  • What are the advantages of logistic regression over decision trees?FAQ

    What are the advantages of logistic regression over decision trees?FAQ

    The answer to "Should I ever use learning algorithm (a) over learning algorithm (b)" will pretty much always be yes. Different learning algorithms make different assumptions about the data and have different rates of convergence. The one which works best, i.e. minimizes some cost function of interest (cross validation for example) will be the one that makes assumptions that are consistent with the data and has sufficiently converged to its error rate.

    Put in the context of decision trees vs. logistic regression, what are the assumptions made?

    Decision trees assume that our decision boundaries are parallel to the axes, for example if we have two features (x1, x2) then it can only create rules such as x1>=4.5, x2>=6.5 etc. which we can visualize as lines parallel to the axis. We see this in practice in the diagram below.

    So decision trees chop up the feature space into rectangles (or in higher dimensions, hyper-rectangles). There can be many partitions made and so decision trees naturally scale up to creating more complex (say, higher VC) functions - which can be a problem with over-fitting.

    What assumptions does logistic regression make? Despite the probabilistic framework of logistic regression, all that logistic regression assumes is that there is one smooth linear decision boundary. It finds that linear decision boundary by making assumptions that the P(Y|X) of some form, like the inverse logit function applied to a weighted sum of our features. Then it finds the weights by a maximum likelihood approach. 

    However people get too caught up on that... The decision boundary it creates is a linear* decision boundary that can be of any direction. So if you have data where the decision boundary is not parallel to the axes,

    then logistic regression picks it out pretty well, whereas a decision tree will have problems.

    So in conclusion,

    • Both algorithms are really fast. There isn't much to distinguish them in terms of run-time.
    • Logistic regression will work better if there's a single decision boundary, not necessarily parallel to the axis.
    • Decision trees can be applied to situations where there's not just one underlying decision boundary, but many, and will work best if the class labels roughly lie in hyper-rectangular regions.
    • Logistic regression is intrinsically simple, it has low variance and so is less prone to over-fitting. Decision trees can be scaled up to be very complex, are are more liable to over-fit. Pruning is applied to avoid this.


    Maybe you'll be left thinking, "I wish decision trees didn't have to create rules that are parallel to the axis." This motivates support vector machines.

    Footnotes:
    * linear in your covariates. If you include non-linear transformations or interactions then it will be non-linear in the space of those original covariates.

  • 相关阅读:
    hive sql的常用日期处理函数总结
    超详细的六款主流ETL工具介绍及功能对比
    什么是ETL?
    什么是星型模型和雪花型模型,以及区别
    深入解读和应用RFM分析方法(模型)
    分析方法
    Kimball和Inmon方法论的适用场景比较
    数字化转型的本质(10个关键词)
    64个数据分析常用语
    @Transactional
  • 原文地址:https://www.cnblogs.com/yymn/p/4662396.html
Copyright © 2011-2022 走看看