https://research.aimultiple.com/automl-comparison/
Automated machine learning (AutoML) is a recent technology that is becoming popular for the last few years. While many businesses are looking to integrate autoML into their systems, it is a challenging task to choose the correct autoML solution provider for your company. We have compiled all relevant resources to compare different solutions and help your business select the most suitable one for its needs. We can group autoML providers into three main categories:
- Tech-giants
- Startups
- Open-source solutions
You can learn more about the types of autoML solution providers in our autoML tools article.
Besides our observations, if you want to compare different autoML solution providers by yourself, you can also visit our website for a sortable list of autoML software solutions.
Learn the top vendors according to each metric
Different companies lead different evaluation criteria. Below are our summarized insights:
- Funding: DataRobot is the most funded autoML company.
- Features: DataRobot, Google Cloud AutoML, Darwin, and H2O.ai offer a more extensive range of features than other providers.
- Performance: H2O.ai has greater performance measures in classification and regression tasks.
- Automation: Tazi.ai and DataRobot offer greater automation rates.
- Popularity: Along with the Google Cloud AutoML platform, H2O.ai is also the most searched autoML vendor.
DataRobot, H2O.ai, and Google Cloud AutoML are the leading vendors. However, you should be aware that there is no perfect autoML solution and that you need to test the leading solution with your own data science problems to understand better the performance of different solutions.
For more on how we evaluated each solution, please see the details below,
AutoML Market
The autoML market size is growing rapidly as the technology is getting more popular. A 2020 report by Research&Markets shares that the market generated has a revenue of $0.3 billion in 2019 and is expected to increase to $14.5 billion by 2030. According to the same report, the main drivers of this growths are:
- Increasing demand for more efficient fraud detection solutions
- The growing need for personalized product recommendation
- The rising importance of predictive lead scoring
The interest in autoML is also rapidly increasing, and we expect this increase to continue for at least a few more years.
Source: Google Trends
Funding
The total funding amount is a good indicator of success, as the investors would prefer to put their money on successful companies. Thus, the best-funded vendor can be considered as a successful autoML provider that stands to make the most profit in the long run.
Based on Crunchbase, the best-funded autoML solution is currently DataRobot by a total funding amount of $431 million. After DataRobot, H2O.ai comes second, and Dataiku is the third vendor with $151.1 million and $147 million, respectively. In the below table, you can find the other autoML vendors that are funded more than $1 million, including their founded year and company sizes.
However, we should also consider several shortcomings while evaluating autoML vendors by the total funding amount. Those shortcomings include:
- Older companies have greater potential to raise more money because they have more time to get traction and funding.
- Open-source autoML libraries like Caret and tech-giants like Google Cloud AutoML are not included in this evaluation because they are not funded by private organizations.
Features
In the evaluation of CapitalOne, Google Cloud AutoML, DataRobot, Darwin, and H2O.ai offer the most comprehensive features to their users. Most of the other tools lack the compatibility of different data types like time-series or hierarchical types and cannot provide hyperparameter optimization.
CapitalOne claims that they have been objective while completing this assessment however since they invested in H2O.ai around the time of this research, we need to consider their observation about the players not listed above with a grain of salt.
Automation Capability
The Targetbase has created a helpful chart for handling data science tasks and automation percentage. According to the figure, DataRobot and Tazi.ai stand out with the capability of a higher percentage of automation in all data science tasks evaluated, compared to others. NumberTheory.ai and NeuralStudio.ai are the other two autoML vendors that perform at least a medium level of automation in all data science tasks.
Source: Targetbase
Targetbase claims to have determined the level of automation based on the tools each autoML provider offers. If an autoML solution can handle more tasks automatically in one of the categories, then the automation rate is claimed to be rated as higher accordingly. The report does not share the sub categories under each category (e.g. data processing) so this picture is the most detailed side by side comparison of vendors from Targetbase.
Model performance
In both Targetbase and Capitalone reports, users have built models using these autoML tools with the same training datasets and predicted the labels of validation datasets. Model performance is one of the most important criteria for an autoML tool since this is the primary output of the model in most cases.
In this part, we have compiled resources from Targetbase and CapitalOne to compare the model performance of different autoML solutions, including open-source autoML libraries and startups. In both sources, the solution providers are evaluated in the following three cases:
- Binomial Classification
- Multi-class Classification
- Regression
H2O.ai seems to provide better performance compared to other autoML solutions overall. This is not a comprehensive assessment of all vendors in the space, there can be other autoML vendors that are not included in any of the resources and that can still provide competitive results. For a more detailed assessment, please see below:
Binomial Classification
When the task is binomial classification, CapitalOne indicates that H2O.ai slightly performs better than other providers. According to Targetbase, Neuralstudio.ai and Tazi.ai are better performance measures.
Multi-class Classification
While CapitalOne shares that the open-source library Auto-keras delivers better results than others, Targetbase’s analysis shows that DataRobot and H2O.ai perform better than other AutoML vendors.
Regression
H2O.ai and MLJar handle regression tasks more accurately than other providers, according to CapitalOne and Targetbase, respectively.
Popularity
Source: Google Trends
We observe that the interest in autoML companies starts around 2014, as commercial solutions were being launched. Currently, Google Cloud AutoML platform and H2O.ai are the most popular companies according to Google searches. We also see increasing trends in DataRobot and the open-source libraries Auto-sklearn and Auto-keras, which has emerged in the mid-2018.
To learn more about autoML, feel free to read our other autoML articles.
- AutoML: In-depth Guide to Automated Machine Learning
- Types of AutoML Software / Tools: In-depth Guide
- 22 AutoML Case Studies / Examples: In-Depth Guide
If you have questions about which automated machine learning (AutoML) vendor to choose for your business, don’t hesitate to contact us: