我们部署在 kubernetes 集群上的每个 ASP.NET Core 应用的 appsettings.Production.json 都保存在各个应用的 ConfigMap 中,这些 appsettings.Production.json 中有些重复的配置,如果要修改这些配置,需要到各个应用中一个一个修改,很是麻烦。
针对这个麻烦,我们想到一个解决方法,将这些重复的配置放到一个公用的 ConfigMap 中(appsettings.shared.json),但是要到各个应用的 deployment 配置文件中通过 volumeMounts 一个一个 mount 这个 ConfigMap 也很是麻烦。
针对新的麻烦,我们又想到一个解决方法,在代码中直接读取 ConfigMap,选用的 Kubernetes C# 客户端是 KubernetesClient,实现方法如下。
安装 nuget 包
dotnet add package KubernetesClient
在 Program 中添加读取 ConfigMap 的方法实现
private static byte[] ReadK8sConfigMap()
{
var config = KubernetesClientConfiguration.InClusterConfig();
IKubernetes client = new Kubernetes(config);
var cm = client.ReadNamespacedConfigMap(name: "appsettings.shared.json", "production");
return System.Text.Encoding.UTF8.GetBytes(cm.Data["appsettings.shared.json"]);
}
将从 ConfigMap 读取到的数据以 json 的方式加载到 IConfiguration 中
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.ConfigureAppConfiguration((hostingContext, config) =>
{
var env = hostingContext.HostingEnvironment;
config.AddJsonFile("appsettings.json", optional: true)
.AddJsonStream(new MemoryStream(ReadK8sConfigMap()))
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);
config.AddEnvironmentVariables();
});
webBuilder.UseStartup<Startup>();
});
改进后的代码
namespace Microsoft.Extensions.Configuration
{
public static class ConfigMapExtensions
{
public static IConfigurationBuilder AddJsonKubeConfigMap(this IConfigurationBuilder builder, string name, string @namespace, string key)
{
var config = KubernetesClientConfiguration.IsInCluster() ?
KubernetesClientConfiguration.InClusterConfig() :
KubernetesClientConfiguration.BuildDefaultConfig();
IKubernetes client = new Kubernetes(config);
var json = client.ReadNamespacedConfigMap(name, @namespace)?.Data[key];
if (!string.IsNullOrEmpty(json))
{
builder.AddJsonStream(new MemoryStream(Encoding.UTF8.GetBytes(json)));
}
return builder;
}
}
}
在 pod 中读取 ConfigMap 报错
Unhandled exception. Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'Forbidden'
at k8s.Kubernetes.ReadNamespacedConfigMapWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 exact, Nullable`1 export, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at k8s.KubernetesExtensions.ReadNamespacedConfigMapAsync(IKubernetes operations, String name, String namespaceParameter, Nullable`1 exact, Nullable`1 export, String pretty, CancellationToken cancellationToken)
at k8s.KubernetesExtensions.ReadNamespacedConfigMap(IKubernetes operations, String name, String namespaceParameter, Nullable`1 exact, Nullable`1 export, String pretty)
at Microsoft.Extensions.Configuration.ConfigMapExtensions.AddJsonKubeConfigMap(IConfigurationBuilder builder, String name, String namespace, String key)
在 pod 中用 curl 命令进行请求
curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/api/v1/namespaces/production/configmaps/appsettings.shared.json
响应如下
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "configmaps "appsettings.shared.json" is forbidden: User "system:serviceaccount:production:default" cannot get resource "configmaps" in API group "" in the namespace "production"",
"reason": "Forbidden",
"details": {
"name": "appsettings.shared.json",
"kind": "configmaps"
},
"code": 403
}
原来是 pod 的默认 service account system:serviceaccount:production:default
没有权限请求 api。
通过添加 Role 与 RoleBinding 解决了这个问题,详见博问:k8s 中如何授权 pod 内可以访问指定的 ConfigMap
后来将 AddJsonKubeConfigMap 扩展方法放到了 github 仓库 https://github.com/cnblogs/KubernetesClient.Extensions