使用centos 测试安装madlib sql 机器学习类库
安装步骤
- 添加pg 10 repo
yum install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm- 安装pg python 基本包
yum -y install postgresql10-plpython supervisor- 安装madlib 依赖包
注意python 版本,我使用python 2.7 安装时候失败了，修改为了python34
yum update -y && yum install -y
git
gcc
wget
postgresql10-devel
openssl
m4
vim
flex
bison
graphviz
java
epel-release
python34-devel- 安装pip 包
默认一般是包含的
yum install -y python34-pip- pg_conf 配置(环境变量)
PATH="$PATH:/usr/pgsql-10/bin"- 安装python 依赖（通过pip）
pip3 install awscli pygresql paramiko --upgrade- 安装apache-madlib
下载rpm 包
wget https://dist.apache.org/repos/dist/release/madlib/1.15.1/apache-madlib-1.15.1-bin-Linux.rpm
安装
yum install -y apache-madlib-1.15.1-bin-Linux.rpm- 启动pg 数据库
/usr/pgsql-10/bin/postgresql-10-setup initdb
systemctl enable postgresql-10
systemctl start postgresql-10- 修改pg_hba.conf 添加访问支持
之后修改之后，需要重启服务，systemctl restart postgresql-10
/var/lib/pgsql/10/data/pg_hba.conf
修改如下：
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust- 初始化madlib 服务
/usr/local/madlib/bin/madpack -s madlib -p postgres -c postgres@localhost:5432/postgres install安装结果如下：
madpack.py: INFO : Detected PostgreSQL version 10.6.
madpack.py: INFO : *** Installing MADlib ***
madpack.py: INFO : MADlib tools version = 1.15.1 (/usr/local/madlib/Versions/1.15.1/bin/../madpack/madpack.py)
madpack.py: INFO : MADlib database version = None (host=localhost:5432, db=postgres, schema=madlib)
madpack.py: INFO : Testing PL/Python environment...
madpack.py: INFO : > Creating language PL/Python...
madpack.py: INFO : > PL/Python environment OK (version: 2.7.5)
madpack.py: INFO : > Preparing objects for the following modules:
madpack.py: INFO : > - array_ops
madpack.py: INFO : > - bayes
madpack.py: INFO : > - crf
madpack.py: INFO : > - elastic_net
madpack.py: INFO : > - linalg
madpack.py: INFO : > - pmml
madpack.py: INFO : > - prob
madpack.py: INFO : > - sketch
madpack.py: INFO : > - svec
madpack.py: INFO : > - svm
madpack.py: INFO : > - tsa
madpack.py: INFO : > - stemmer
madpack.py: INFO : > - conjugate_gradient
madpack.py: INFO : > - knn
madpack.py: INFO : > - lda
madpack.py: INFO : > - stats
madpack.py: INFO : > - svec_util
madpack.py: INFO : > - utilities
madpack.py: INFO : > - assoc_rules
madpack.py: INFO : > - convex
madpack.py: INFO : > - glm
madpack.py: INFO : > - graph
madpack.py: INFO : > - linear_systems
madpack.py: INFO : > - recursive_partitioning
madpack.py: INFO : > - regress
madpack.py: INFO : > - sample
madpack.py: INFO : > - summary
madpack.py: INFO : > - kmeans
madpack.py: INFO : > - pca
madpack.py: INFO : > - validation
madpack.py: INFO : Installing MADlib:
madpack.py: INFO : > Created madlib schema
madpack.py: INFO : > Created madlib.MigrationHistory table
madpack.py: INFO : > Wrote version info in MigrationHistory table
madpack.py: INFO : MADlib 1.15.1 installed successfully in madlib schema.- 检查madlib 服务的安装
/usr/local/madlib/bin/madpack -s madlib -p postgres -c postgres@localhost:5432/postgres install-check输出结果如下：
madpack.py: INFO : Detected PostgreSQL version 10.6.
TEST CASE RESULT|Module: bayes|bayes.ic.sql_in|PASS|Time: 117 milliseconds
TEST CASE RESULT|Module: crf|crf_train_small.ic.sql_in|PASS|Time: 112 milliseconds
TEST CASE RESULT|Module: crf|crf_test_small.ic.sql_in|PASS|Time: 131 milliseconds
TEST CASE RESULT|Module: elastic_net|elastic_net.ic.sql_in|PASS|Time: 123 milliseconds
TEST CASE RESULT|Module: linalg|linalg.ic.sql_in|PASS|Time: 43 milliseconds
TEST CASE RESULT|Module: linalg|svd.ic.sql_in|PASS|Time: 151 milliseconds
TEST CASE RESULT|Module: linalg|matrix_ops.ic.sql_in|PASS|Time: 231 milliseconds
TEST CASE RESULT|Module: prob|prob.ic.sql_in|PASS|Time: 21 milliseconds
TEST CASE RESULT|Module: svm|svm.ic.sql_in|PASS|Time: 151 milliseconds
TEST CASE RESULT|Module: tsa|arima.ic.sql_in|PASS|Time: 130 milliseconds
TEST CASE RESULT|Module: conjugate_gradient|conj_grad.ic.sql_in|PASS|Time: 35 milliseconds
TEST CASE RESULT|Module: knn|knn.ic.sql_in|PASS|Time: 107 milliseconds
TEST CASE RESULT|Module: lda|lda.ic.sql_in|PASS|Time: 109 milliseconds
TEST CASE RESULT|Module: stats|correlation.ic.sql_in|PASS|Time: 93 milliseconds
TEST CASE RESULT|Module: stats|f_test.ic.sql_in|PASS|Time: 25 milliseconds
TEST CASE RESULT|Module: stats|robust_and_clustered_variance_coxph.ic.sql_in|PASS|Time: 116 milliseconds
TEST CASE RESULT|Module: stats|pred_metrics.ic.sql_in|PASS|Time: 98 milliseconds
TEST CASE RESULT|Module: stats|wsr_test.ic.sql_in|PASS|Time: 29 milliseconds
TEST CASE RESULT|Module: stats|mw_test.ic.sql_in|PASS|Time: 26 milliseconds
TEST CASE RESULT|Module: stats|cox_prop_hazards.ic.sql_in|PASS|Time: 101 milliseconds
TEST CASE RESULT|Module: stats|ks_test.ic.sql_in|PASS|Time: 28 milliseconds
TEST CASE RESULT|Module: stats|chi2_test.ic.sql_in|PASS|Time: 28 milliseconds
TEST CASE RESULT|Module: stats|t_test.ic.sql_in|PASS|Time: 26 milliseconds
TEST CASE RESULT|Module: stats|anova_test.ic.sql_in|PASS|Time: 29 milliseconds
TEST CASE RESULT|Module: utilities|minibatch_preprocessing.ic.sql_in|PASS|Time: 104 milliseconds
TEST CASE RESULT|Module: utilities|pivot.ic.sql_in|PASS|Time: 74 milliseconds
TEST CASE RESULT|Module: utilities|path.ic.sql_in|PASS|Time: 88 milliseconds
TEST CASE RESULT|Module: utilities|sessionize.ic.sql_in|PASS|Time: 74 milliseconds
TEST CASE RESULT|Module: utilities|text_utilities.ic.sql_in|PASS|Time: 78 milliseconds
TEST CASE RESULT|Module: utilities|utilities.ic.sql_in|PASS|Time: 77 milliseconds
TEST CASE RESULT|Module: utilities|transform_vec_cols.ic.sql_in|PASS|Time: 93 milliseconds
TEST CASE RESULT|Module: utilities|encode_categorical.ic.sql_in|PASS|Time: 97 milliseconds
TEST CASE RESULT|Module: assoc_rules|assoc_rules.ic.sql_in|PASS|Time: 114 milliseconds
TEST CASE RESULT|Module: convex|mlp.ic.sql_in|PASS|Time: 226 milliseconds
TEST CASE RESULT|Module: convex|lmf.ic.sql_in|PASS|Time: 118 milliseconds
TEST CASE RESULT|Module: glm|glm.ic.sql_in|PASS|Time: 234 milliseconds
TEST CASE RESULT|Module: graph|graph.ic.sql_in|PASS|Time: 218 milliseconds
TEST CASE RESULT|Module: linear_systems|dense_linear_sytems.ic.sql_in|PASS|Time: 88 milliseconds
TEST CASE RESULT|Module: linear_systems|sparse_linear_sytems.ic.sql_in|PASS|Time: 91 milliseconds
TEST CASE RESULT|Module: recursive_partitioning|random_forest.ic.sql_in|PASS|Time: 155 milliseconds
TEST CASE RESULT|Module: recursive_partitioning|decision_tree.ic.sql_in|PASS|Time: 130 milliseconds
TEST CASE RESULT|Module: regress|clustered.ic.sql_in|PASS|Time: 115 milliseconds
TEST CASE RESULT|Module: regress|robust.ic.sql_in|PASS|Time: 99 milliseconds
TEST CASE RESULT|Module: regress|logistic.ic.sql_in|PASS|Time: 101 milliseconds
TEST CASE RESULT|Module: regress|multilogistic.ic.sql_in|PASS|Time: 98 milliseconds
TEST CASE RESULT|Module: regress|marginal.ic.sql_in|PASS|Time: 279 milliseconds
TEST CASE RESULT|Module: regress|linear.ic.sql_in|PASS|Time: 24 milliseconds
TEST CASE RESULT|Module: sample|train_test_split.ic.sql_in|PASS|Time: 76 milliseconds
TEST CASE RESULT|Module: sample|sample.ic.sql_in|PASS|Time: 21 milliseconds
TEST CASE RESULT|Module: sample|stratified_sample.ic.sql_in|PASS|Time: 76 milliseconds
TEST CASE RESULT|Module: sample|balance_sample.ic.sql_in|PASS|Time: 81 milliseconds
TEST CASE RESULT|Module: summary|summary.ic.sql_in|PASS|Time: 86 milliseconds
TEST CASE RESULT|Module: kmeans|kmeans.ic.sql_in|PASS|Time: 155 milliseconds
TEST CASE RESULT|Module: pca|pca_project.ic.sql_in|PASS|Time: 152 milliseconds
TEST CASE RESULT|Module: pca|pca.ic.sql_in|PASS|Time: 242 milliseconds
TEST CASE RESULT|Module: validation|cross_validation.ic.sql_in|PASS|Time: 110 milliseconds- 添加数据库测试函数
创建数据表&&添加数据：
CREATE TABLE array_tbl (
id integer,
array1 integer[],
array2 integer[]
);
INSERT INTO "public"."array_tbl"("id","array1","array2")
VALUES
(1,E'{1,2,3,4,5,6}',E'{6,5,4,3,2,1}'),
(2,E'{1,1,0,0,99,8}',E'{0,0,0,-5,2,1}');查看插入的结果
select * from array_tbl;
id | array1 | array2
----+----------------+----------------
1 | {1,2,3,4,5,6} | {6,5,4,3,2,1}
2 | {1,1,0,0,99,8} | {0,0,0,-5,2,1}
(2 行记录)使用madlib API:
说明： 主要是计算数组的最大、最小值
select id,madlib.array_min(array1) min, madlib.array_max(array1) max from array_tbl;
select id,madlib.array_min(array1) min, madlib.array_max(array1) max from array_tbl;
id | min | max
----+-----+-----
1 | 1 | 6
2 | 0 | 99
(2 行记录)说明
这个只是简单的安装，实际生产还需要好多东西需要调整，同时注意madlib schema 的安装，是对应到数据库的，当然
测试环境我们可以直接使用docker 版本的，尽管有点大，但是还是很方便的
参考资料
https://cwiki.apache.org/confluence/display/MADLIB/Installation+Guide#InstallationGuide-Prerequisites
https://www.postgresql.org/download/linux/redhat/