支持s3是cubestore 一个很不错的特性，可以提高系统的扩展性
参考修改
主要是添加endpoint 以及配置使用path格式请求
s3.rs
impl S3RemoteFs {
pub fn new(
dir: PathBuf,
region: String,
endpoint:String,
bucket_name: String,
sub_path: Option<String>,
) -> Result<Arc<Self>, CubeError> {
let credentials = Credentials::new(
env::var("CUBESTORE_AWS_ACCESS_KEY_ID").as_deref().ok(),
env::var("CUBESTORE_AWS_SECRET_ACCESS_KEY").as_deref().ok(),
None,
None,
None,
)?;
let bucket = Bucket::new_with_path_style(&bucket_name, Region::Custom {
endpoint:endpoint,
region:region,
}, credentials)?;
Ok(Arc::new(Self {
dir,
bucket,
sub_path,
delete_mut: Mutex::new(()),
}))
}
}
config/mod.rs
#[derive(Debug, Clone)]
pub enum FileStoreProvider {
Local,
Filesystem {
remote_dir: Option<PathBuf>,
},
S3 {
region: String,
endpoint:String,
bucket_name: String,
sub_path: Option<String>,
},
GCS {
bucket_name: String,
sub_path: Option<String>,
},
}
- 添加环境变量支持
摘取了部分，详细参考源码
store_provider: {
if let Ok(bucket_name) = env::var("CUBESTORE_S3_BUCKET") {
FileStoreProvider::S3 {
bucket_name,
region: env::var("CUBESTORE_S3_REGION").unwrap(),
endpoint: env::var("CUBESTORE_S3_ENDPOINT").unwrap(),
sub_path: env::var("CUBESTORE_S3_SUB_PATH").ok(),
}
} else if let Ok(bucket_name) = env::var("CUBESTORE_GCS_BUCKET") {
FileStoreProvider::GCS {
bucket_name,
sub_path: env::var("CUBESTORE_GCS_SUB_PATH").ok(),
}
} else if let Ok(remote_dir) = env::var("CUBESTORE_REMOTE_DIR") {
FileStoreProvider::Filesystem {
remote_dir: Some(PathBuf::from(remote_dir)),
}
} else {
FileStoreProvider::Filesystem { remote_dir: None }
}
},
运行试用
- 参考配置
export CUBESTORE_LOG_LEVEL=debug
export CUBESTORE_SERVER_NAME=localhost:9999
export CUBESTORE_META_PORT=9999
export CUBESTORE_S3_BUCKET=test
export CUBESTORE_S3_ENDPOINT=http://localhost:9000
export CUBESTORE_S3_REGION=us-east-1
export CUBESTORE_AWS_ACCESS_KEY_ID=dalongrong
export CUBESTORE_AWS_SECRET_ACCESS_KEY=dalongrong
./target/debug/cubestored
- 效果
说明
以上是一个简单的使用，关于集群以及docker镜像，后期构建好会发布到官方仓库
参考资料
https://github.com/rongfengliang/mycubejs
https://github.com/durch/rust-s3/blob/master/s3/bin/simple_crud.rs
https://cube.dev/docs/caching/using-pre-aggregations#running-in-production