一、自建k8s集群（docker19.03.15+k8s1.22.3）-元一软件

未分类3个月前发布 元一软件
90 0

一、自建k8s集群（docker19.03.15+k8s1.22.3）

一、自建k8s集群（docker19.03.15+k8s1.22.3）-元一软件

0.101字数 537阅读 970

一、环境说明

先修改主机名

主机名 IP地址 角色 系统
master 172.16.79.130 k8s-master Centos7.6
node1 172.16.79.131 k8s-node1 Centos7.6
node2 172.16.79.132 k8s-node2 Centos7.6

注意：官方建议每台机器至少双核2G内存，同时需确保MAC和product_uuid唯一（参考下面的命令查看）

ip link
cat /sys/class/dmi/id/product_uuid

docker和k8s版本对应关系待整理，目前对照阿里云k8s服务版本

二、环境配置

以下命令在三台主机上均需运行

1、设置yum源

#阿里源需要另做整理,目前使用可能会有连接失败报错
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

rm -rf /var/cache/yum && yum makecache && yum -y update && yum -y autoremove
# 注意： 网络条件不好，可以不用 update,直接复制下面命令
rm -rf /var/cache/yum && yum makecache && yum -y autoremove

2、安装依赖包

yum install -y epel-release conntrack ipvsadm ipset jq sysstat curl iptables libseccomp

3、关闭防火墙

systemctl stop firewalld && systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT

4、关闭SELinux

setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

5、关闭swap分区

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

6、加载内核模块

modprobe br_netfilter
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
modprobe -- br_netfilter
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules

7、设置内核参数

cat << EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

sysctl -p /etc/sysctl.d/k8s.conf

8、安装 Docker

1、首先卸载旧版

yum -y remove docker \
           docker-client \
           docker-client-latest \
           docker-common \
           docker-latest \
           docker-latest-logrotate \
           docker-logrotate \
           docker-selinux \
           docker-engine-selinux \
           docker-engine \
           docker-ce-cli

2、安装依赖包

yum install -y yum-utils device-mapper-persistent-data lvm2

3、设置安装源（阿里云）

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

4、启用测试库（可选）

yum-config-manager --enable docker-ce-edge
yum-config-manager --enable docker-ce-test

5、安装docker

yum makecache fast
yum list docker-ce --showduplicates | sort -r
yum -y install docker*19.03.15

6、启动&自启动

systemctl start docker
systemctl enable docker

7、配置阿里云镜像加速

Docker建议配置阿里云镜像加速
安装完成后配置启动时的命令，否则 docker 会将 iptables FORWARD chain 的默认策略设置为DROP
另外Kubeadm建议将 systemd 设置为 cgroup 驱动，所以还要修改 daemon.json

sed -i "13i ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT" /usr/lib/systemd/system/docker.service

tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://bk6kzfqm.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

9、安装 kubeadm 和 kubelet

1、配置安装源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum makecache fast

2、安装

yum list kubelet --showduplicates | sort -r
yum install -y kubelet-1.22.3-0 kubeadm-1.22.3-0 kubectl-1.22.3-0
systemctl enable kubelet
注意：不需要 start

10、拉取所需镜像

1、拉取镜像

先从阿里云拉取所需的镜像，不然会从谷歌拉取，导致拉取失败

kubeadm config images list | sed -e 's/^/docker pull /g' -e 's#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g' | sh -x

docker images | grep registry.cn-hangzhou.aliyuncs.com/google_containers | awk '{print "docker tag",$1":"$2,$1":"$2}' | sed -e 's/registry.cn-hangzhou.aliyuncs.com\/google_containers/k8s.gcr.io/2' | sh -x
#coredns单独拉取
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4

docker images | grep registry.cn-hangzhou.aliyuncs.com/google_containers | awk '{print "docker rmi """$1""":"""$2}' | sh -x 



#可以在每台机器上执行以上三条命令，就不需要执行下面两条了~

2、Master 导出镜像

docker save $(docker images | grep -v REPOSITORY | awk 'BEGIN{OFS=":";ORS=" "}{print $1,$2}') -o k8s-images.tar

3、node 节点导入镜像

docker image load -i k8s-images.tar

三、初始化集群

~~~以下命令如无特殊说明，均在master上执行~~~

1、使用kubeadm init初始化集群（注意修 apiserver 地址为本机IP）

kubeadm init  --kubernetes-version=v1.22.11 --apiserver-advertise-address=172.16.79.130 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.1.0.0/16 

###注意###
此时我们安装的版本是1.22.3，kubelet --version 也是显示 Kubernetes v1.22.3
但是 在进行kubeadm config images list 时显示 镜像都是 v1.22.11
应该是1.22版本 最高的小版本是1.22.11，所以会用11的镜像（安装时请以kubeadm config images list 时显示 镜像版本为准）

--apiserver-advertise-address=172.16.79.130 master服务器的ip
--pod-network-cidr=10.244.0.0/16 ：（Pod 中间网络通讯我们用flannel，flannel要求是10.244.0.0/16，这个IP段就是Pod的IP段）
--service-cidr=10.1.0.0/16 ： Service（服务）网段（和微服务架构有关）


###清除k8s配置
kubeadm reset ---y
systemctl stop kubelet && systemctl stop etcd
docker rm -f -v  $(docker ps -aq)   #小心谨慎操作，该指令清除机器上所有在run的容器
find /var/lib/kubelet | xargs -n 1 findmnt -n -t tmpfs -o TARGET -T | uniq | xargs -r umount -v
rm -r -f /etc/kubernetes /var/lib/kubelet  /etc/etcd.env  /var/lib/etcd  /var/etcd/

初始化成功后会输出类似下面的加入命令，暂时无需运行，先记录。

kubeadm join 172.16.79.130:6443 --token 49r3gq.8mq7ettvlw32nz8r \
    --discovery-token-ca-cert-hash sha256:56c0961c9679bea85d538ca4cfa9bc7475ba281f354d66386e4f47c85dee497e

2、为需要使用kubectl的用户进行配置

#把密钥配置加载到自己的环境变量里
export KUBECONFIG=/etc/kubernetes/admin.conf

#每次启动自动加载$HOME/.kube/config下的密钥配置文件（K8S自动行为）
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

3、 集群网络配置（2选1）

1、安装 flannel 网络

#每台机器执行
docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
docker pull rancher/mirrored-flannelcni-flannel:v0.18.1
docker pull rancher/mirrored-flannelcni-flannel:v0.18.1

#master执行
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

kube-flannel.yml可复制下面

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
       #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
       #image: flannelcni/flannel:v0.18.1 for ppc64le and mips64le (dockerhub limitations may apply)
        image: rancher/mirrored-flannelcni-flannel:v0.18.1
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
       #image: flannelcni/flannel:v0.18.1 for ppc64le and mips64le (dockerhub limitations may apply)
        image: rancher/mirrored-flannelcni-flannel:v0.18.1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: xtables-lock
          mountPath: /run/xtables.lock
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate

2、安装 calico 网络 #v3.24.5

#每台服务器执行
docker pull  docker.io/calico/kube-controllers:v3.24.5
docker pull  docker.io/calico/node:v3.24.5
docker pull  docker.io/calico/pod2daemon-flexvol:v3.24.5
docker pull  docker.io/calico/cni:v3.24.5
docker pull docker.io/calico/typha:v3.24.5
docker pull docker.io/calico/csi:v3.24.5
docker pull docker.io/calico/node-driver-registrar:v3.24.5
#上面是需要用到的镜像，先pull完再下载文档安装
#按照官方文档步骤进行安装：
https://docs.tigera.io/calico/3.24/getting-started/kubernetes/quickstart

#calico版本和k8s对应版本可以到官网选择版本后，查看安装要求：
https://docs.tigera.io/calico/3.24/getting-started/kubernetes/requirements

4、向Kubernetes集群中添加Node节点

在node1和node2上运行之前在master输出的命令

kubeadm join 172.16.79.130:6443 --token 49r3gq.8mq7ettvlw32nz8r \
    --discovery-token-ca-cert-hash sha256:56c0961c9679bea85d538ca4cfa9bc7475ba281f354d66386e4f47c85dee497e

注意：没有记录集群 join 命令的可以通过以下方式重新获取

kubeadm token create --print-join-command --ttl=0

5、为需要使用kubectl的用户进行配置

kubectl get nodes
kubectl get pods -A -o wide

6、kube-proxy 开启 ipvs

kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-configmap.yaml
sed -i 's/mode: ""/mode: "ipvs"/' kube-proxy-configmap.yaml
kubectl apply -f kube-proxy-configmap.yaml
rm -f kube-proxy-configmap.yaml
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

最后编辑于

：

©

著作权归作者所有,转载或内容合作请联系作者
【社区内容提示】社区部分内容疑似由AI辅助生成，浏览时请结合常识与多方信息审慎甄别。
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

更多精彩内容，就在简书APP

“小礼物走一走，来简书关注我”

还没有人赞赏，支持一下

一、自建k8s集群（docker19.03.15+k8s1.22.3）-元一软件

总资产0.102共写了537字获得2个赞共1个粉丝

相关阅读更多精彩内容

  • 环境说明： 主机名操作系统版本ipdocker versionkubelet versionkubeadm ver…

  • Kubernetes(K8s)-k8s服务集群安装 一、环境准备 1、机器环境 节点CPU核数必须是 ：>= 2核…

  • 1. k8s简介与本文目的 1.1. 概述 k8s 本身涉及到大量的技术知识，包括操作系统、网络、存储、调度、分布…

  • kubeadm部署Kubernetes 1.15 资源准备 环境准备 注意：在所有节点进行如下操作 设置hostn…

  • 环境：centOS 7.6 2C 核4G 内存1 master 2 slave 腾讯云防火墙端口设置30080 ；…

© 版权声明
文章版权归作者所有，未经允许请勿转载。
添加小工具
点此为“正文侧边栏”添加小工具