k8s中的Docker容器内部无法apt-get update

k8s的pod有报解析不了域名的bug，为了debug，进入docker容器内部，希望使用curl，结果没有，想要安装一下结果遇到报错

# apt install curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package curl

于是apt-get update，结果继续报错

# apt-get update
0% [Connecting to deb.debian.org] [Connecting to security.debian.org]
Err:1 http://security.debian.org/debian-security buster/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian buster InRelease
  Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian buster-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists... Done
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.

因为环境是运行在虚拟机上的k8s集群里的docker容器，所以先具体定位了一下问题发生的情景

• 宿主机上直接启动的docker容器可以正常apt-get update
• 每台虚拟机上直接启动的docker容器均可以正常apt-get update
• 宿主机上使用kubectl create -f deploy.yaml创建的容器可以正常apt-get update`
• 虚拟机上使用kubectl create -f deploy.yaml创建的容器无法正常apt-get update`

宿主机和虚拟机上k8s集群中的docker容器内/etc/resolv.conf内容均如下

# cat /etc/resolv.conf
nameserver 10.96.0.10
search flask-demo.svc.cluster.local svc.cluster.local cluster.local

虚拟机上/etc/docker/daemon.json默认不配置dns，容器内/etc/resolv.conf内容如下

nameserver 192.168.1.1
search .

宿主机（Windows Docker Deskctop）上/etc/docker/daemon.json默认不配置dns，容器内/etc/resolv.conf内容如下

# DNS requests are forwarded to the host. DHCP DNS options are ignored.
nameserver 192.168.65.5

因此感觉是dns出了问题，先直接替换了一下/etc/resolv.conf，发现报错不一样了但仍然无法正确解析（通过和其他环境执行ping对比）

不过如果退出容器再重新进入，就可以正常解析域名了，由此可知修改后需要重启容器或dns服务生效，只是如果不apt-get update我们并做不到这一点

于是，对于自建服务，直观的方法就是在deployment中加入dns配置

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-demo-deploy
  namespace: flask-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: fd-tag
  template:
    metadata:
      labels:
        app: fd-tag
    spec:
      containers:
      - name: fd-ct
        image: lyudmilalala/flask-demo:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
      # dnsPolicy保持default可以不影响默认的dns配置
      dnsConfig:
        nameservers:
          - 8.8.8.8

对于更多想要部署在k8s上的第三方应用，很明显不能挨个儿去改他们的dns配置，还是需要修好自己网卡层的问题，要去flannel，calico，weaver的方向多调查

此处推荐K8s Troubleshooting — How to Debug CoreDNS Issues

我的情况是flannel，在搜索过程中找到一种情况，在使用vagrant的情景下，会默认创建两张网卡，然而flannel连向了不正确的那一张，这种Bug的表现如下：

kubectl logs --namespace kube-system kube-flannel-ds-5dxdm -c kube-flannel

I0622 17:53:13.690431       1 main.go:463] Found network config - Backend type: vxlan
I0622 17:53:13.690716       1 match.go:248] Using interface with name enp0s3 and address 10.0.2.15
I0622 17:53:13.690734       1 match.go:270] Defaulting external address to interface address (10.0.2.15)

然而事实上我们希望使用的网卡是ip为192.168.1.111的那一张

因此需要手动在flannel.yaml中设置

containers:
  - name: kube-flannel
    image: quay.io/coreos/flannel:v0.10.0-amd64
    command:
    - /opt/bin/flanneld
    args:
    - --ip-masq
    - --kube-subnet-mgr
    - --iface=enp0s8

Kubernetes: Pods Can't Resolve Hostnames

设置完后可以发现网卡和地址对了，但是仍然无法解析域名