zoukankan      html  css  js  c++  java
  • Kubernetes容器集群部署Flannel网络(四)

    部署Flannel网络

    Overlay Network: 覆盖网络,在基础网络上叠加的一种虚拟网络技术模式,该网络中的主机通过虚拟链路连接起来.

    VXLAN: 将源数据包封装到UDP中,并使用基础网络的IP/MAC作为外层报文头进行封装,然后在以太网传输,到达目的后由隧道端点解封装并将数据发送给目标地址。

    Flannel:是Overlay网络的一种,也是将源数据包封装在另一种网络包里面进行路由转发和通信,目前已经支持UDP,VXLAN,AWS,VPC,GCE路由等级传输

    下载Flannel网络包:

    wget https://github.com/coreos/flannel/releases/download/v0.9.1/flannel-v0.9.1-linux-amd64.tar.gz

    解压flannel包:

    [root@master ~]# tar xf flannel-v0.9.1-linux-amd64.tar.gz 

    分别把二进制包flanneld和mk-docker-opts.sh脚本文件传到两台node节点的bin下

    [root@master ~]# scp flanneld mk-docker-opts.sh root@192.168.1.102:/opt/kubernetes/bin/
    [root@master ~]# scp flanneld mk-docker-opts.sh root@192.168.1.103:/opt/kubernetes/bin/

    node1节点操作:

    在cfg目录建立文件flanneld,添加以下信息

    FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 -etcd-cafile=/opt/kubernetes/ssl/ca.pem -etcd-certfile=/opt/kubernetes/ssl/server.pem -etcd-keyfile=/opt/kubernetes/ssl/server-key.pem"

    添加flanneld启动命令:

    cat <<EOF >/usr/lib/systemd/system/flanneld.service
    [Unit]
    Description=Flanneld overlay address etcd agent
    After=network-online.target network.target
    Before=docker.service
    
    [Service]
    Type=notify
    EnvironmentFile=/opt/kubernetes/cfg/flanneld
    ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
    ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    
    EOF

    添加docker启动命令配置参数:

    cat <<EOF >/usr/lib/systemd/system/docker.service
    
    [Unit]
    Description=Docker Application Container Engine
    Documentation=https://docs.docker.com
    After=network-online.target firewalld.service
    Wants=network-online.target
    
    [Service]
    Type=notify
    EnvironmentFile=/run/flannel/subnet.env
    ExecStart=/usr/bin/dockerd  $DOCKER_NETWORK_OPTIONS
    ExecReload=/bin/kill -s HUP $MAINPID
    LimitNOFILE=infinity
    LimitNPROC=infinity
    LimitCORE=infinity
    TimeoutStartSec=0
    Delegate=yes
    KillMode=process
    Restart=on-failure
    StartLimitBurst=3
    StartLimitInterval=60s
    
    [Install]
    WantedBy=multi-user.target
    
    EOF

    在master端添加flannel网络信息分配给节点,

    分配的网段为172.17.0.0网段,封装类型为vxlan,更高效,更稳定

    /opt/kubernetes/bin/etcdctl 
    --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem 
    --endpoints="https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379" 
    set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

    在master端获取是否分配成功:

    [root@master ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379" get /coreos.com/network/config
    2018-08-07 17:20:20.214528 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
    { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

    node1节点启动flanned

    [root@node1 cfg]# systemctl restart flanneld

    查看ip信息

    查看flannel生成的文件,分配了一个子网,

    [root@node1 cfg]# cat /run/flannel/subnet.env 
    DOCKER_OPT_BIP="--bip=172.17.45.1/24"
    DOCKER_OPT_IPMASQ="--ip-masq=false"
    DOCKER_OPT_MTU="--mtu=1450"
    DOCKER_NETWORK_OPTIONS=" --bip=172.17.45.1/24 --ip-masq=false --mtu=1450"

    查看docker启动文件变化,引用了flannel网络文件变量

     重启docker

    systemctl daemon-reload
    systemctl restart docker

    再次查看网络变化,两个网段一样

    node2节点布署同上node1部署步骤或如下:

    //node2上启动文件传至node3
    scp /usr/lib/systemd/system/flanneld.service 192.168.1.104:/usr/lib/systemd/system/
    
    // node2配置文件传至node3
    scp flanneld 192.168.1.104:/opt/kubernetes/cfg/
    
    //  node3上添加/usr/lib/systemd/system/docker.service文件
      EnvironmentFile=/run/flannel/subnet.env
      ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
    
    //启动
    systemctl daemon-reload
    systemctl restart docker

    分别在两个节点上查看 路由信息

    [root@node1 cfg]# route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 ens33
    172.17.45.0     0.0.0.0         255.255.255.0   U     0      0        0 docker0
    172.17.47.0     172.17.47.0     255.255.255.0   UG    0      0        0 flannel.1
    192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33
    node1路由节点
    root@node2 cfg]# route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 ens33
    172.17.45.0     172.17.45.0     255.255.255.0   UG    0      0        0 flannel.1
    172.17.47.0     0.0.0.0         255.255.255.0   U     0      0        0 docker0
    192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33
    node2路由节点

    在master节点查看两个节点分配的网络

    [root@master ssl]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.104:2379" ls /coreos.com/network/subnets

    查看配置网络配置文件

    [root@master ssl]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.104:2379" ls /coreos.com/network/

    为node节点创建kubeconfig文件

    kubeconfig用于node节点上的kube-proxy进行集群通信认证,

    master端操作:

    安装kubectl客户端工具:

    https://kubernetes.io/docs/tasks/tools/install-kubectl/
    chmod + x /opt/kubernetes/bin/kubectl
    source /etc/profile

    创建 TLS Bootstrapping Token

    export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
    cat > token.csv <<EOF
    ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
    EOF

    指定一个变量k8s api访问入口,以方便后面引用

    export KUBE_APISERVER="https://192.168.1.101:6443"

    设置集群参数

    [root@master ssl]# kubectl config set-cluster kubernetes 
    >   --certificate-authority=/opt/kubernetes/ssl/ca.pem 
    >   --embed-certs=true 
    >   --server=${KUBE_APISERVER} 
    >   --kubeconfig=bootstrap.kubeconfig
    Cluster "kubernetes" set.

    查看生成的集群配置参数:

    [root@master ssl]# cat bootstrap.kubeconfig 
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVVjBJNXllYisrV0plTlNHY2dPK1NFR0VVc3drd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURnd056QTJNekF3TUZvWERUSXpNRGd3TmpBMk16QXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeGs3WnhubXZzUThsWkIyelJCeTcKZk9MZGJRakYxU2FJMUJlRzA4YUVoeU5KQW5RSmdBekUyeTVNSHF0SG5zWVdJczdhV09aM0VQS0lGY3MxT0dPYwprNlBObDd5RkMrb0I5bG9JWDdhb1NQSGdQcUxrd1Z0TGJpYnpkUjBuUVczT0ZNcmsyRGdtUU9kQVVkUWdkenk1Ck5FVksyMVJUQUJ2UFpvUTFMVmxnL2xjTFZqcml6aVF1WHFJWXBxaHg1cFREMTY1Uk5TQWh0Z3pQajRpdDNJT2sKbnZ4Um9EeWRGbk9jZUVLaVVHSDVpWUJvRTI0U0VxL3VIbVg4dG01NDBrSVZ2VlBIQkxTTGtSZFNabStYSGoxZQpQRlJSQUFhdW40aW9SQ2M1Q0d5RkUwelFITnRMdGRSb1pQcEF4ZzRXS1loaXFaUHpjNStXNmFMTnR6Vk9tZFkxCjB3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVVTVuWnhISWJJSUp2VXV5UG5jd1U2VW1hNzhVd0h3WURWUjBqQkJnd0ZvQVVVNW5aeEhJYgpJSUp2VXV5UG5jd1U2VW1hNzhVd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGY0Z2NndYWjJLRGJMYUlia2dyClBTcVBVbFdNVUphWUtjWExIUzJsRGRzWUxYYmthK0JGK084M05ZZ2dDcGlSZ3VaMG01YjNOVE9uZTlXRzBRY24KcEl2TTdGNjB2Tmh4NWVrNGhUYkZ4U2xzWmdKSFM0bTdFNWhneENvNmZ0N1VvbWpRTTdxdGFDSnNHTnV6NHpDTQovTjdmNFVBZ21xSWMwYmdTNmtzcld0RjJmVVBVdEptSUpJN2lxWFlhNEk0cEhkMVNhN1FyNUs1YitYcjhiL3duCllSOXlhanZ1dTl0c0wyQ2ZHMmtyTFBKZG9jeTI2bWp5V3FORnhESFgwdEx6ZE10SnVqSnR6SnNqbm56cFRKSTIKR1I2UGdoNFNTZVloSERJcGVxZ2d3SEpNeVlXYm9IUnR4Zk1neUl1VmZHa0pTZFhJRVFEUDlvMGtLY1ppUit4bApiQ1k9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
        server: https://192.168.1.101:6443
      name: kubernetes
    contexts: []
    current-context: ""
    kind: Config
    preferences: {}
    users: []

    设置客户端认证参数:

    [root@master ssl]# kubectl config set-credentials kubelet-bootstrap 
    >   --token=${BOOTSTRAP_TOKEN} 
    >   --kubeconfig=bootstrap.kubeconfig

    查看客户端生成的证书参数:

    设置上下文参数

    kubectl config set-context default 
      --cluster=kubernetes 
      --user=kubelet-bootstrap 
      --kubeconfig=bootstrap.kubeconfig

    设置默认上下文

    kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

    创建kube-proxy kubeconfig文件

    kubectl config set-cluster kubernetes 
      --certificate-authority=./ca.pem 
      --embed-certs=true 
      --server=${KUBE_APISERVER} 
      --kubeconfig=kube-proxy.kubeconfig
    
    kubectl config set-credentials kube-proxy 
      --client-certificate=./kube-proxy.pem 
      --client-key=./kube-proxy-key.pem 
      --embed-certs=true 
      --kubeconfig=kube-proxy.kubeconfig
    
    kubectl config set-context default 
      --cluster=kubernetes 
      --user=kube-proxy 
      --kubeconfig=kube-proxy.kubeconfig
    
    kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

    查看生成的配置参数:

    [root@master ssl]# cat kube-proxy.kubeconfig 
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVVjBJNXllYisrV0plTlNHY2dPK1NFR0VVc3drd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURnd056QTJNekF3TUZvWERUSXpNRGd3TmpBMk16QXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeGs3WnhubXZzUThsWkIyelJCeTcKZk9MZGJRakYxU2FJMUJlRzA4YUVoeU5KQW5RSmdBekUyeTVNSHF0SG5zWVdJczdhV09aM0VQS0lGY3MxT0dPYwprNlBObDd5RkMrb0I5bG9JWDdhb1NQSGdQcUxrd1Z0TGJpYnpkUjBuUVczT0ZNcmsyRGdtUU9kQVVkUWdkenk1Ck5FVksyMVJUQUJ2UFpvUTFMVmxnL2xjTFZqcml6aVF1WHFJWXBxaHg1cFREMTY1Uk5TQWh0Z3pQajRpdDNJT2sKbnZ4Um9EeWRGbk9jZUVLaVVHSDVpWUJvRTI0U0VxL3VIbVg4dG01NDBrSVZ2VlBIQkxTTGtSZFNabStYSGoxZQpQRlJSQUFhdW40aW9SQ2M1Q0d5RkUwelFITnRMdGRSb1pQcEF4ZzRXS1loaXFaUHpjNStXNmFMTnR6Vk9tZFkxCjB3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVVTVuWnhISWJJSUp2VXV5UG5jd1U2VW1hNzhVd0h3WURWUjBqQkJnd0ZvQVVVNW5aeEhJYgpJSUp2VXV5UG5jd1U2VW1hNzhVd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGY0Z2NndYWjJLRGJMYUlia2dyClBTcVBVbFdNVUphWUtjWExIUzJsRGRzWUxYYmthK0JGK084M05ZZ2dDcGlSZ3VaMG01YjNOVE9uZTlXRzBRY24KcEl2TTdGNjB2Tmh4NWVrNGhUYkZ4U2xzWmdKSFM0bTdFNWhneENvNmZ0N1VvbWpRTTdxdGFDSnNHTnV6NHpDTQovTjdmNFVBZ21xSWMwYmdTNmtzcld0RjJmVVBVdEptSUpJN2lxWFlhNEk0cEhkMVNhN1FyNUs1YitYcjhiL3duCllSOXlhanZ1dTl0c0wyQ2ZHMmtyTFBKZG9jeTI2bWp5V3FORnhESFgwdEx6ZE10SnVqSnR6SnNqbm56cFRKSTIKR1I2UGdoNFNTZVloSERJcGVxZ2d3SEpNeVlXYm9IUnR4Zk1neUl1VmZHa0pTZFhJRVFEUDlvMGtLY1ppUit4bApiQ1k9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
        server: https://192.168.1.101:6443
      name: kubernetes
    contexts:
    - context:
        cluster: kubernetes
        user: kube-proxy
      name: default
    current-context: default
    kind: Config
    preferences: {}
    users:
    - name: kube-proxy
      user:
        client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzakNDQXNhZ0F3SUJBZ0lVY2paWkpqanIwT0tOcDBaQ2EzUGZ3TWdkUTJFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURnd056QTJOREV3TUZvWERUSTRNRGd3TkRBMk5ERXdNRm93YkRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUm93R0FZRFZRUURFeEZ6ZVhOMFpXMDZhM1ZpClpTMXdjbTk0ZVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT1hQcCs4SXNHcy8KMFluVjNHZERodDc2dDVmTnFxdk9kNm9qdjJaUUNvbXF0SnZIRzVQTzQrRFJMdDBjdk5STWFMbEFmbzdsTFlIRgpKYVNIcVVmNDZnWGdsNllaK3loK0h3TVlXNzVCZGlOc081cmJsSEs3bncvcHhFOTYzak90VTlzejBFRVQ3dnZ3CkNOTDhOUWlPSXY4OFI0MG5TK1JjdThaQXJxaE9mQVExUXZwU2tRUlIrOUlwbzl3VEpWQll2bldJVHRxTWhISzAKMDZVVk1RcDNwVHd2bXI2SGtwVWxUVk5IRklLU3RCU285ZmtOalE4MndXTW9vRElhM2RRYlJzeTNDbmQ3aWVZTQp6enNZQzY1SnA1VjR1Y1NVamRodXErc05hZWFGbnVROGFDcU0wZjV5bE90THFrZUpxSm56RmNtUVp0dlczcU5OCmFDM0lCM2V1djRNQ0F3RUFBYU4vTUgwd0RnWURWUjBQQVFIL0JBUURBZ1dnTUIwR0ExVWRKUVFXTUJRR0NDc0cKQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUWd1T1ZBdjd5ZgpublFZL3UyVitOT3hIWWV1eXpBZkJnTlZIU01FR0RBV2dCUlRtZG5FY2hzZ2dtOVM3SStkekJUcFNacnZ4VEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQU5maWk5Yk9oVHdhTEFmU0wwVE1PZFVrUGY2VVVuRFY3cXlLR3J4OG0KbXR1WENXTG1NTDNGQm9aaXljRHNheW9YYWFwbHMzb0NMdHMvOHhYTWFQeGo1bXNpclJFNW9pRFJvZFg4RE5FeApvR3A5VFovNnpRY05IZ01tV0llVVFLbWR2RWR1a1pSNk1tU2tveTJTRHpFQmVkNUVzMlAxWWg4QWF4aGRFTWw4ClZsZUFscmVNMUdDYU4wMzU1SzU1MVd6eFNRbm1xaFpMWXZTRzhFeGlvMmYxTXcyUGRHL2RMaUNjdFpaNkJxY04KUDI3Q2ZIWTN6MXRwN2U2TjV2K0VvV0RwZ2diYlBaVWdTMVdwUlZKUEdjQzM1YXlSbktwNjBYK1RhRlo2NzBxcApTNlg1b3dBYjVobjBkWXBOV1kwNFVCUVd6S2VualpCZS9vekw1UWd1QzR4NWNRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
        client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNWMrbjd3aXdhei9SaWRYY1owT0czdnEzbDgycXE4NTNxaU8vWmxBS2lhcTBtOGNiCms4N2o0TkV1M1J5ODFFeG91VUIranVVdGdjVWxwSWVwUi9qcUJlQ1hwaG43S0g0ZkF4aGJ2a0YySTJ3N210dVUKY3J1ZkQrbkVUM3JlTTYxVDJ6UFFRUlB1Ky9BSTB2dzFDSTRpL3p4SGpTZEw1Rnk3eGtDdXFFNThCRFZDK2xLUgpCRkg3MGltajNCTWxVRmkrZFloTzJveUVjclRUcFJVeENuZWxQQythdm9lU2xTVk5VMGNVZ3BLMEZLajErUTJOCkR6YkJZeWlnTWhyZDFCdEd6TGNLZDN1SjVnelBPeGdMcmttbmxYaTV4SlNOMkc2cjZ3MXA1b1dlNUR4b0tvelIKL25LVTYwdXFSNG1vbWZNVnlaQm0yOWJlbzAxb0xjZ0hkNjYvZ3dJREFRQUJBb0lCQUFwV1lQTjdySlBtZHdLQQpRRDNVNThnMzl2STVLaWc0VFdHMTRmSTBQNFVaeks0Rk10UmZSaGppei96YVFoUTZxNnRKbjJmR0gxbnJDRnB5ClQ4Y1JCMVhrV0czOFZUUHBHSFFITUhsS2NJbi9ZRjhDNlphUG9nUVJPK2RuVzM5dm1Jb1B0cHAyenhVK0l1ZDYKdU5mOXJNMVdaTkdRcHZWS3hwZVFhVXJnM090RGREcEJPL3B1akVlR2Foc3pMR2VaZjNzOGhyNE1TN3dSOG5vQQpWVnBuZGlzZkJ0QmRtRGY2Um9Zc2ZsTHkxY3FxaVovQ2JwaG1FY1VDTThEeWF1My9zT3BrZktOdjBuTURaM1Y2CmhHa0M4UmpBUStxZlp4ZkR1MHZJdU8zaVVSWUZieGp0bHdpNVlGeVNMZElocE9NV1FQZkFncjJ5TFJWRkd4VEYKRThLdzZpRUNnWUVBLzBXUStTWDVSc0JlK291UkgyYkxHQnZxa2hrcEQrTGFSdE5lWVJrdzZQR2tOK1J2dGhMVApETjE5ZXh0bFprZ2RZMVd1L25ZUHUrcVlqTUJDVUZPYUNocXNYcFJmUVVOdnVtc3JoU3V6ejhlWjlLQWRXV2w1Cjg5SVAxdHVoNFNLMWt1azJDMURTSytwenJPOGN4bXpYOU9EVHczMFFWNW1YdGdwT2dTQlNZajhDZ1lFQTVuZCsKdFJORnR2aytNRXVJcUR1MkJjOEZiS1B5YVJGRlVHSlJGL1JVN1dRbEdKSW55ZlozNVdvUUVhTFNPNXN4UEV0Nwo0S3prMWZhbm9UMDA5YXdMT3lBSCtneG4xOWlqMEU2SlhaQXdPNFdBbEVNWGlKREFXYWpMSldaUEtjRDE0Nk9VCm04Q3lCK3doRXRxNnhxdjIvN3grbDdkRmF2TkovRTg2UXFDWUNiMENnWUVBMmgvWGdRMnhzNE10cFdrYzJIZzkKa3B4SUZBN3J4MmRRbkxTQU1QZ1ZrdmJvRzlwV21uMzdXVUxwdGdmeTNUTkI2VitONkdGNWtGa2xnUi9IaWx2bApoRHpHalFxN1dra0ZiNFdqNm1jdGZwNVFMM2VobklWRWI5TlBOWWozQ2lsZWZlcEpBK0pMdExyaG5TTzBzdGpPCnI3QnRRdDNhMlArbFl5ZnJiT3J4ZW9FQ2dZQTdSM2N1NmpNZldrSzF0MmdDSmNHdkFzM1pZTjRrU2szeTNDY1IKcTBZQS9hWlJDSjcxWXRENkVHcnJybDYyK0RjVTFER2IyN3pNa0RJVDFpTitJVytPR0xJVWpnWlR6YVJ0R3haWQpHNU5jYS8xanBWSXpUVkE4ZjU4cEVvVEFMNkxQbDY3TTFTYTF4UFhJM0x2NENCbHpOQyszcEd0VmxCaHVybzVFCnZ2djhYUUtCZ1FDbGdFQjRlUzN0Zk1DeS9GaEpYNHo3TDl1SUJrTzNJNGQ4TjkreEo1cVBPMXE3S3ZOSytBMksKSlBGdDlzSm1kdkpGVkZmbU9Vd2ZncEtKcHFweUphZ2dRT0JlSTFmejEvY0tITit4LzlFK0dKekpnNDZ0NlN4Swo4eE9LOHRmNU53YXk5RGdmZW9sMXpUYU9oM0E1SnNBbWx0TVZEREFMWUE4dXFtWmJOU1B0cnc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
  • 相关阅读:
    Android之针对WebView的全屏播放
    Android之Android WebView常见问题及解决方案汇总
    android之针对fragment多次调用onCreateView的问题
    Android之在string.xml配置文字颜色粗体等效果
    ios成长之每日一遍(day 8)
    Android之TextView灵活使用
    ubuntu忘记root密码 的解决方法
    Mono Touch Table应用
    判断checkbox选中的个数
    C指针原理(14)
  • 原文地址:https://www.cnblogs.com/zhangzihong/p/9438446.html
Copyright © 2011-2022 走看看