对于minio 我们可以使用基于sql 的对象内容查询，特别适合进行特定文件内容的获取，强大方便。
以下是一个简单的试用
环境准备
集成了prometheus
- docker-compose 文件
version: "3"
services:
minio:
image: minio/minio
ports:
- "9000:9000"
command: server /data
volumes:
- "./data:/data"
environment:
- "MINIO_PROMETHEUS_AUTH_TYPE=public"
- "MINIO_ACCESS_KEY=dalongapp"
- "MINIO_SECRET_KEY=dalongapp"
grafana:
image: grafana/grafana
ports:
- "3000:3000"
prometheus:
image: prom/prometheus
volumes:
- "./prometheus.yml:/etc/prometheus/prometheus.yml"
ports:
- "9090:9090"
- prometheus 配置
scrape_configs:
- job_name: minio
metrics_path: /minio/prometheus/metrics
scrape_interval: 10s
scrape_timeout: 10s
static_configs:
- targets: ['minio:9000']
- select api 调用代码
select.py2 文件使用python venv
import boto3
s3 = boto3.client('s3',
endpoint_url='http://localhost:9000',
aws_access_key_id='dalongapp',
aws_secret_access_key='dalongapp',
region_name='us-east-1')
r = s3.select_object_content(
Bucket='demoapp',
Key='demo.json',
ExpressionType='SQL',
Expression="SELECT d.dir_name, d.files FROM S3Object[*] d",
InputSerialization={
'CompressionType': 'NONE',
'JSON': {
'Type': 'DOCUMENT'
}
},
OutputSerialization={'JSON': {}},
)
for event in r['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
print(records)
elif 'Stats' in event:
statsDetails = event['Stats']['Details']
print("Stats details bytesScanned: ")
print(statsDetails['BytesScanned'])
print("Stats details bytesProcessed: ")
print(statsDetails['BytesProcessed'])
- 测试json 内容
demo.json
{
"created": "936864000",
"dir_name": "important_docs",
"files": [
{
"name": "."
},
{
"name": ".."
},
{
"name": ".aws"
},
{
"name": "downloads"
}
],
"owner": "AWS S3"
}
- minio s3 信息
运行&&效果
- 启动环境
docker-compose up -d
- 运行
python select.py2
- 效果
python select.py2
{"dir_name":"important_docs","files":[{"name":"."},{"name":".."},{"name":".aws"},{"name":"downloads"}]}
Stats details bytesScanned:
300
Stats details bytesProcessed:
300
- prometheus 监控效果
说明
基于s3 的select api 我们可以方便的进行一些基于sql的数据分析操作
参考资料
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/dev/s3-glacier-select-sql-reference-select.html
https://docs.min.io/docs/minio-select-api-quickstart-guide.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html
https://github.com/rongfengliang/minio-s3-select-learning