环境：

• 操作系统： ubuntu 18.04
• k8s集群: v1.15.12 （kubeadm安装）

阿里云坑爹的文档，害我搞了3天才搞定
https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md

获取region_id.instance_id

echo `curl -s http://100.100.100.200/latest/meta-data/region-id`.`curl -s http://100.100.100.200/latest/meta-data/instance-id`

修改 kubelet启动参数

添加 --cloud-provider=external --provider-id=region_id.instance_id替换为上面获取到的id（provider-id是阿里云控制器用来识别ecs，添加路由表，添加负载均衡监听和虚拟服务器组）
不需要--hostname-override，会导致kubeadm join无法正常结束
（未验证）如果是已经加入集群的node，只修改kubectl并重启没用，估计可以直接修改node的spec，添加 providerID: cn-hongkong.i-xxxx

$ sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

...
Environment="KUBELET_CLOUD_PROVIDER_ARGS=--cloud-provider=external --provider-id=$region_id.$instance_id"
...
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CLOUD_PROVIDER_ARGS

创建access_key ConfigMap

获取阿里云账户access_key，需要负载均衡和专用网络VPC路由表权限
https://usercenter.console.aliyun.com/#/manage/ak

$ vi cloud-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system
data:
  special.keyid: __access_key__
  special.keysecret: __access_secret__

安装阿里云控制器

官方资源文档：https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/examples/cloud-controller-manager.yml
需要修改下面3项

• ${CLUSTER_CIDR}为集群pod子网地址
• 镜像版本（一定要修改！）
  推荐镜像版本：cloud-controller-manager-amd64:v1.9.3.81-gca19cd4-aliyun
• 配置文件的挂载方式，默认是挂载宿主机目录，建议改为ConfigMap

云控制器镜像版本目前是：cloud-controller-manager-amd64:v1.9.3.10-gfb99107-aliyun
有几个问题

• node name必须是region_id.instance_id，类似cn-hangzhou.i-xxxx，没有可读性
• 默认使用share规格的负载均衡，阿里云已经停用了，导致控制器无法创建新的负载均衡，需要通过service的annotation指定规格 service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
• 并且很多k8s service 的annotation不可用，比如指定使用已创建的负载均衡（推荐）service.beta.kubernetes.io/alicloud-loadbalancer-id：lb-xxxx
  详情见注解文档

注解文档：https://help.aliyun.com/document_detail/86531.html#title-6ou-i9x-2lr

注意！！！老版注解前缀为alicloud，很多网上的其他文档用的是新版前缀alibaba-cloud（坑爹）

配置文件

将云控制器配置文件调整为ConfigMap挂载方式（推荐）
也可以按照官方文档创建etc/kubernetes/cloud-controller-manager.conf，改为下面data的内容

$ vi cloud-controller-manager.yml

...
apiVersion: extensions/v1beta1
kind: DaemonSet
...
      volumes:
      - configMap:
          name: cloud-controller-manager
        name: k8s
...

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-controller-manager
  namespace: kube-system
data:
  cloud-controller-manager.conf: |-
    kind: Config
    contexts:
    - context:
        cluster: kubernetes
        user: system:cloud-controller-manager
      name: system:cloud-controller-manager@kubernetes
    current-context: system:cloud-controller-manager@kubernetes
    users:
    - name: system:cloud-controller-manager
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: ${ca_data}
        server: ${k8s_master_url}
      name: kubernetes

修改${ca_data}替换为下面的证书内容

cat /etc/kubernetes/pki/ca.crt|base64 -w 0

修改${k8s_master_url} 为集群地址
安装

kubectl apply -f cloud-controller-manager.yml

测试

$ vi nginx.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:latest
        name: nginx
        ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/alicloud-loadbalancer-id: "lb-xxxx"
    service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: "true"
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer


$ kubectl apply -f nginx.yaml