k8s使用细节整理

1. k8s与harbor联合使用

使用私有仓库

• 命令行手动创建

[k8s@k8s-master ~]$ kubectl -n test-yapin create secret docker-registry  jinboharbor --docker-server=docker-hub.qhgctech.com  --docker-username=jinbo --docker-password=xxx --docker-email=1165773573@qq.com
[k8s@k8s-master ~]$ kubectl -n test-yapin  describe secrets jinboharbor 
Name:         jinboharbor
Namespace:    test-yapin
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/dockerconfigjson

Data
====
.dockerconfigjson:  145 bytes

注意：必须加上--docker-server，否则无法拉取镜像
通过kubectl describe观察新建的Secret的内容时会发现一个条目.dockercfg, 相当于用户主目录的
.dockercfg文件。该文件通常会在运行docker login命令时由Docker自动创建。

• 使用一台已经登录过harbor服务器的机器的认证信息
  使用cat ~/.docker/config.json, 确认是否有harbor服务器的认证信息

$ cat ~/.docker/config.json | base64 -w 0

将该认证信息BASE64编码

jinkins in k8s下的镜像拉取和推送

2. 手动删除pod

3. 使用指定条目初始化卷和挂载卷的指定条目

• 卷内暴露指定的ConfigMap条目（volumes.configMap.items）

通过卷的items属性能够指定哪些条目会被暴露作为configMap卷中的文件

      volumes:
        - name: config
          configMap:
            name: fortune-config
            items:
            - key: a.conf
              path: aa.conf

指定单个条目时需要同时设置条目的键名称和对应的文件名, a.conf和aa.conf

示例（如下configMap包含a.conf、b.conf两个条目）：

[k8s@k8s-master jinbo-test]$ cat a.conf 
aaaaaaaaaaaaa
aaaaaaaaaaaaa
[k8s@k8s-master jinbo-test]$ cat b.conf 
bbbbbbbbbb
bbbbbbbbbb
[k8s@k8s-master jinbo-test]$ cat test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: jinbo-test
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox
          image: busybox:glibc
          imagePullPolicy: "IfNotPresent"
          command: ['sleep','infinity']
          volumeMounts:
            - name: config
              mountPath: /tmp
          resources:
            limits:
              cpu: 40m
              memory: 100Mi
            requests:
              cpu: 40m
              memory: 100Mi
      volumes:
        - name: config
          configMap:
            name: fortune-config
            items:
            - key: a.conf
              path: aa.conf
[k8s@k8s-master jinbo-test]$ kubectl -n jinbo-test exec busybox-7d76c9f58f-mfxmg ls /tmp/
aa.conf
[k8s@k8s-master jinbo-test]$ kubectl -n jinbo-test exec busybox-7d76c9f58f-mfxmg cat /tmp/aa.conf
aaaaaaaaaaaaa
aaaaaaaaaaaaa
[k8s@k8s-master jinbo-test]$ 

/tmp文件夹下仅包含aa.conf文件

• ConfigMap独立条目作为文件被挂载且不隐藏文件夹中的其他文件
  volumeMounts额外的subPath字段可以被用于挂载卷中的某个独立文件或文件夹，无需挂载完整卷

    spec:
      containers:
          image: some/image
          volumeMounts:
            - name: myvolume
              mountPath: /etc/someconfig.conf    //挂载至某一文件，而不是文件夹
              subPath: myconfig.conf        //仅挂载指定条目myconfig.conf,并非完整的卷

示例：

[k8s@k8s-master jinbo-test]$ cat b.conf 
bbbbbbbbbb
bbbbbbbbbb
[k8s@k8s-master jinbo-test]$ 
[k8s@k8s-master jinbo-test]$ cat test.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: jinbo-test
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox
          image: busybox:glibc
          imagePullPolicy: "IfNotPresent"
          command: ['sleep','infinity']
          volumeMounts:
            - name: myvolume
              mountPath: /lib/bb.conf
              subPath: b.conf
          resources:
            limits:
              cpu: 40m
              memory: 100Mi
            requests:
              cpu: 40m
              memory: 100Mi
      volumes:
        - name: myvolume
          configMap:
            name: fortune-config
[k8s@k8s-master jinbo-test]$ kubectl -n jinbo-test exec busybox-6bd9c59c47-ws4gw ls /lib
bb.conf
ld-linux-x86-64.so.2
libc.so.6
libm.so.6
libnsl.so.1
libnss_compat.so.2
libnss_dns.so.2
libnss_files.so.2
libnss_hesiod.so.2
libnss_nis.so.2
libnss_nisplus.so.2
libpthread.so.0
libresolv.so.2
[k8s@k8s-master jinbo-test]$ kubectl -n jinbo-test exec busybox-6bd9c59c47-ws4gw cat /lib/bb.conf
bbbbbbbbbb
bbbbbbbbbb
[k8s@k8s-master jinbo-test]$ 

结论:
kubernetes key (pod.spec.volums[0].configMap.items[0].key)用于指定configMap中的哪些条目可用于挂载
kubernetes path (pod.spec.volums[0].configMap.items[0].path)用于将key重命名

kubernetes subPath (pod.spec.containers[0].volumeMounts.subPath)用于挂载卷中的指定目录或文件

4, k8s集群新增和删除节点

• 新增节点
  默认情况下加入集群的token是24小时过期，24小时后如果是想要新的node加入到集群，需要重新生成一个token，命令如下

# 显示获取token列表
$ kubeadm token list
# 生成新的token
$ kubeadm token create

除token外，join命令还需要一个sha256的值，通过以下方法计算

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

用上面输出的token和sha256的值或者是利用kubeadm token create --print-join-command拼接join命令即可

• 删除节点

[k8s@k8s-master ~]$ kubectl drain k8s-node2 --delete-local-data --force --ignore-daemonsets
node/k8s-node2 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-dzw84, kube-system/kube-proxy-86vsn
node/k8s-node2 drained
[k8s@k8s-master ~]$ kubectl delete nodes k8s-node2
node "k8s-node2" deleted

Options:
--delete-local-data=false: Continue even if there are pods using emptyDir (local data that will be deleted when the node is drained).
--dry-run=false: If true, only print the object that would be sent, without sending it.
--force=false: Continue even if there are pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet.
--ignore-daemonsets=false: Ignore DaemonSet-managed pods.

5. coredns 添加自定义DNS解析记录

参考文档: https://blog.csdn.net/kunyus/article/details/88841159

[k8s@k8s-master ~]$ kubectl -n kube-system get configmaps coredns -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        hosts {
           1.1.1.1  docker-hub.abcd.com   //自定义dns解析

           fallthrough                       //此处很关键
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
...
[k8s@k8s-master ~]$ kubectl -n kube-system scale deployment coredns --replicas=0
[k8s@k8s-master ~]$ kubectl -n kube-system scale deployment coredns --replicas=2

fallthrouth，如果没有配置其属性，你会发现虽然服务访问正常，并且自定义解析也正常，但是其他外网解析失败，我的理解是：当配置了fallthrouth后，当一个外部域名没有在自定义解析中找到，其会再通过forward . /etc/resolv.conf去查询。

6. nginx ingress配置支持低版本TLS

6.1 问题描述

如下图所示，通过curl命令(服务器版本redhat 6.5)或客户端程序访问ingress controler报TLS握手失败 image.png

6.2 解决方法

在没有配置任何nginx下，k8s的nginx默认只支持TLS1.2，不支持TLS1.0和TLS1.1

新建或修改nginx-configuration :

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"
  ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2"

更新nginx-configuration并重启pod

验证能正常响应:

$ curl -v --tlsv1.0 https://test.com
$ curl -v --tlsv1.1 https://test.com
$ curl -v --tlsv1.2 https://test.com

参考文档: https://www.cnblogs.com/lyc94620/p/11345124.html