k8s StorageClass Storage class

I. Overview

A cluster-level resource, StorageClass, is a resource object in Kubernetes that defines policies and methods for creating Persistent Volumes (PVs).StorageClass is primarily used to enable dynamic provisioning of PVs, which means that when a user creates a Persistent Volume Claim (PVC), Kubernetes automatically creates a conforming PV based on the specified StorageClass and binds it to the PVC.

StorageClass As an abstract definition of a storage resource, a user-setPVC Application blocking details of the back-end storage, on the one hand, reduces the user's attention to the details of the storage resources, on the other hand, alleviates the administrator manual managementPV The work is done automatically by the systemPV creation and binding to enable dynamic resource provisioning. Based on theStorageClass The dynamic resource provisioning model will gradually become the standard storage management model for cloud platforms.

1、StorageClass object definition

The definition of the StorageClass resource object includes the name, the provider of the back-end storage, the configuration of the parameters and reclaim policy of the back-end storage, and the volume binding mode (volumeBindingMode).

• The name of the StorageClass is important and will be referenced when the PVC is created, and administrators should accurately name StorageClasses with different storage characteristics.

• Once a StorageClass has been created, it cannot be modified; if it needs to be changed, the original StorageClass resource object can only be deleted and recreated.

2、StorageClass YAML example

apiVersion: storage./v1
kind: StorageClass
metadata.
  # Name of the storage class that the user references in the PVC
  name: example-storage-class
  /is-default-class: "true" # Set as the default StorageClass
# Name of the dynamic provisioner, needs to match the installed provisioner
provisioner: /aws-ebs # The example uses the AWS EBS provisioner.
# reclaimPolicy defines what happens to the PV when the PVC is deleted.
reclaimPolicy: Delete # Optional values are Retain or Delete
# allowVolumeExpansion
allowVolumeExpansion: true # Optional values are true or false
# Defines the mode in which the volume is bound to the Pod
volumeBindingMode: Immediate # Selectable values are Immediate or WaitForFirstConsumer
# Defines the parameters required by the storage system, which are passed to the preparer
parameters.
  # Storage type, set according to the requirements of the preparer and storage system
  type: gp2 # AWS EBS example, may have different values for other systems
  # Storage IOPS performance, some storage systems may require this parameter
  iopsPerGB: "10" # Example, set according to requirements and preparer support
  # Minimum IOPS value for storage, some storage systems may require this parameter
  minimumIOPS: "1000" # Example, values based on requirements and preparer support
  # Maximum IOPS value for storage, may be required for some storage systems
  maximumIOPS: "20000" # Example, value to be set based on requirements and preparer support
  # encryption options for storage, some storage systems may support encryption
  encrypted: "true" # Example, value depends on requirements and preparer support.
  # region of storage, may be required for cross-region storage systems
  availabilityZone: "us-east-1a" # Example, set according to the requirements of the preparer and storage system.
  # Performance level of the storage, some storage systems may offer different performance levels
  performance: "high" # Example, set according to the requirements of the Preparer and Storage System
# Defines mount options that will be used when the PV is mounted to the node
mountOptions.
  - debug # Example, values are based on requirements and may include "debug", "defaults", "ro", etc. # - other-option # Can be used when the PV is mounted to a node.
  # - other-option # Additional mount options can be added.
# Allowed topology constraints that define which nodes the storage can be accessed by
allowedTopologies.
  - matchLabelExpressions.
    - key: /region
      values: /region
        - us-east-1
    - key: /zone
      values: /zone
        - us-east-1a

II. StorageClass field

1. Provisioner (storage preparer)

provisioner specifies the Preparer (Provisioner) used to dynamically create PersistentVolume (PV). A provisioner is a plug-in that is responsible for creating storage resources in the back-end storage system in response to PersistentVolumeClaim (PVC) requests, and different storage plug-ins support different storage back-ends and service providers. When a PVC is created and the StorageClass associated with it specifies a preparer, Kubernetes invokes this preparer to automatically create the corresponding PV.

Volume Inserts	Built-in Preparer	Configuration example
AWSElasticBlockStore	√	AWS EBS
AzureFile	√	Azure files(Abandoned)
AzureDisk	√	Azure Disk
CephFS	-	-
Cinder	√	Open Stack Cinder
FC	-	-
FlexVolume	-	-
GCEPersistentDisk	√	gcePD
Glusterfs	√	GlusterFS
iSCSI	-	-
Local	-	Local
NFS	-	NFS
PortworxVolume	√	portworx-volume
RBD	√	ceph-rbd
VsphereVolume	√	Vsphere

1.1、Built-in Preparer

Provisioners with built-in Kubernetes support are named with a "/" at the beginning.

• /aws-ebs: for creating Elastic Block Store (EBS) volumes on AWS.
• /azure-disk: for creating disks on Azure.
• /gce-pd: Used to create persistent disks on Google Cloud Platform (GCP).
• /cinder: Used to create Cinder volumes on OpenStack.

1.2. Third-party preparers

In order to conform to StorageClass usage, a custom Provisioner needs to conform to the Storage Volumes development specification, where the author of the external storage provider has full and free control over the code, how it is provisioned, how it runs, the storage plug-ins (including Flex), and so on.

code repositorykubernetes-sigs/sig-storage-lib-external-provisioner Contains a class library for writing functional implementations for external preparers. You can access the code repositorykubernetes-sigs/sig-storage-lib-external-provisioner Understand the list of external drivers.

For example, for NFS types, Kubernetes does not provide an internal Provisioner, but an external Provisioner can be used. there are also many third-party storage providers that provide their own external Provisioners.

2、reclaimPolicy(Recycling Strategy)

PVs created through the Dynamic Resource Provisioning model will inherit the reclaim policy set on the StorageClass resource object. The configuration field is called "reclaimPolicy" and the options that can be set include Delete and Retain.

• If StorageClass does not specify a reclaimPolicy, the default value is Delete.
• For PVs created manually by the administrator that are still managed by StorageClass, the resource reclamation policy set at the time of PV creation will be used.

3、allowVolumeExpansion(allowVolumeExpansion)

PVs can be configured to allow expansion, and when the allowVolumeExpansion field of the StorageClass resource object is set to true, it will allow the user to automatically complete the expansion of the PV by editing the storage space of the PVC.

The following table describes the types of Volumes that support storage expansion and the required minimum version of Kubernetes:

Volume types that support storage expansion	Kubernetes Minimum Version
gcePersistentDisk	1.11
awsElasticBlock Store	1.11
Cinder	1.11
glusterfs	1.11
RBD	1.11
Azure File	1.11
Azure Disk	1.11
Portworx	1.13
FlexVolume	1.14(Alpha)
CSI	1.16(Beta)

This function can only be used to expand volumes, not to reduce them.

4. mountOptions (mount options)

With the mountOptions field of the StorageClass resource object, the system will set mount options for dynamically created PVs.

Not all PV types support the mount option, if a PV does not support it but StorageClass sets the field, the PV will fail to be created. In addition, the system does not validate mount options, so if the wrong option is set, the container will simply fail when mounting the storage.

5, volumeBindingMode (volume binding mode)

The volumeBindingMode field of the StorageClass resource object is set to control when the PVC is bound to a dynamically created PV.

Currently supported binding modes include: Immediate and WaitForFirstConsumer.

5.1、Immediate

The default value of the storage binding mode is Immediate, which indicates that when a PersistentVolumeClaim (PVC) is created, the PV is dynamically created and the binding operation of the PVC to the PV is performed.

Note that for topology-limited or back-end storage that cannot be accessed from all Nodes, the binding of PVs will be done without knowledge of the Pod scheduling requirements, which may result in some Pods not being able to complete the scheduling.

5.2、WaitForFirstConsumer

The WaitForFirstConsumer binding pattern indicates that the binding operation of a PVC to a PV is delayed until the first Pod using the PVC is created.

The system will create PVs on the Node where the Pod is located based on the scheduling requirements of the Pod, which can be set by the following conditions (not limited to).

• Pod's need for resources

• Node Selector

• Pod affinity and anti-affinity settings

• Taint and Toleration Settings

Storage volumes that currently support the WaitForFirstConsumer binding mode include:

• AWSElasticBlockStore
• AzureDisk
• GCEPersistentDisk.

In addition, some storage plugins support WaitForFirstConsumer mode through pre-created PV bindings, such as.

• AWSElasticBlockStore
• AzureDisk
• GCEPersistentDisk
• Local

6. allowedTopologies (allowed topologies)

In environments using the WaitForFirstConsumer pattern, if it is still desirable to perform PV binding operations based on specific topology information (Topology), this can also be set in the StorageClass definition via the allowedTopologies field.

6.1. Examples

The following example sets the label selection criteria for the target Node by matchLabelExpressions (zone=us-central1-a or us-central1-b) PVs will be allowed to be created on Nodes that satisfy these conditions

apiVersion: storage./v1
kind: StorageClass
metadata:
  name: standard
provisioner: /example
parameters:
  type: pd-standard
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: /zone
    values:
    - us-central-1a
    - us-central-1b

7. parameters (storage parameters)

Parameter settings for the back-end storage resource provider, different Provisioners may offer different parameter settings. Some parameters can be set without displaying the settings, and the Provisioner will use their default values.

Currently, the StorageClass resource object supports a maximum of 512 storage parameters, and the space occupied by all keys and values cannot exceed 256KiB.

7.1. Examples

The following are examples of StorageClass storage parameters provided by common storage providers (Provisioners) (AWSElasticBlockStore storage volumes are used as an example):

apiVersion: storage./v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: 
volumeBindingMode: WaitForFirstConsumer
parameters:
  ./fstype: xfs
  type: io1
  iopsPerGB: "50"
  encrypted: "true"
allowedTopologies:
- matchLabelExpressions:
  - key: /zone
    values:
    - us-east-2c

7.2. storage parameters (AWSElasticBlockStore)

AWS EBS Storage Parameters

• type (required)

  • Defines the storage type of the EBS volume (default value gp3). Example:
    • gp2: general purpose SSD
    • io1: SSDs Delivering High IOPS
    • io2: A New Generation of SSDs for Applications Requiring High IOPS
    • st1: Volumes optimized by HDD storage
    • sc1: Volumes optimized by cold HDD storage

• iopsPerGB (optional, valid only when type is io1 or io2)

  • Defines the number of IOPS provided per GiB.
  • For example, if type is io1 and iopsPerGB is set to 10, a 100 GiB volume will provide 1000 IOPS.

• iops (optional, valid only if type is io1)

  • Directly defines the total number of IOPS for the volume. For example, iops: "1000" means that the volume will provide 1000 IOPS.

• throughput (optional, valid only if type is st1)

  • Defines the volume throughput in MiB/s. For example, throughput: "500" means that the volume throughput is 500 MiB/s.

• encrypted (optional)

  • Boolean value indicating whether the EBS volume should be encrypted. For example, encrypted: "true".

• kmsKeyId (optional)

  • Specify the KMS key ID used to encrypt the EBS volume. for example, kmsKeyId: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab".

• fsType (optional)

  • Defines the file system type. For example, fsType: "ext4".

• volumeSize (optional)

  • Defines the requested volume size in GiB. For example, volumeSize: "100" means that a volume of 100 GiB is requested.

• availabilityZone (optional)

  • Defines in which availability zone the EBS volume should be created. For example, availabilityZone: "us-west-2a".

• multiAttachEnabled (Optional)

  • Boolean value indicating whether multi-attach is enabled. For example, multiAttachEnabled: "true" allows volumes to be attached to multiple instances simultaneously.

• snapshotId (optional)

  • Define the snapshot ID used to create the EBS volume. for example, snapshotId: "snap-0123456789abcdef0".

• tags (optional)

  • Defines a set of key-value pairs that are used to label EBS volumes. Example:

  • tags:
      - key: "project"
        value: "myproject"
      - key: "owner"
        value: "myteam"
    

8. Set the default StorageClass (/is-default-class)

When creating a YAML file for a SC, you need to add a comment in the metadata section to mark that SC as the default.

8.1. Examples

apiVersion: storage./v1
kind: StorageClass
metadata:
  name: low-latency
  annotations:
    /is-default-class: "false"
provisioner:
reclaimPolicy: Retain # The default value is Delete
allowVolumeExpansion: true
mountOptions:
  - discard # This may be enabled at the block storage level UNMAP/TRIM
volumeBindingMode: WaitForFirstConsumer
parameters:
  guaranteedReadWriteLatency: "true" # This is a service provider specific

8.2 Modifying an Existing StorageClass

8.2.1 Setting the Existing SC as the Default

kubectl patch storageclass <sc-name> -p '{"metadata": {"annotations":{"/is-default-class":"true"}}}'

8.2.2 Setting an existing SC as non-default

kubectl patch storageclass <sc-name> -p '{"metadata": {"annotations":{"/is-default-class":"false"}}}'

If you set the /is-default-class annotation to true on multiple StorageClasses in the cluster, and then create a PersistentVolumeClaim (PVC) with the storageClassName unset, Kubernetes will use the most recently created default Kubernetes will use the most recently created default StorageClass.

Example -- Dynamically creating PVs using an NFS-type StorageClass

character	hostname (of a networked computer)	ip address
nfs server + master node	k8s-master1	192.168.112.10
nfs client + node node	k8s-node1	192.168.112.20
nfs client + node node	k8s-node2	192.168.112.30

1. Configure the NFS server

1.1 Installing the nfs-utils package on the master node

yum install -y nfs-utils

1.2 Creating a shared directory on the master node

mkdir -pv /data/nfs

1.3 Editing the /etc/exports file

echo "/data/nfs 192.168.112.0/24(rw,sync,no_root_squash)" > /etc/exports

1.4. Apply new export settings

exportfs -arv

1.5. Reboot and set NFS to boot itself

systemctl restart nfs && systemctl enable nfs

1.6. Checking NFS Shares

showmount -e localhost

2. Configure the NFS client

All node nodes

2.1 Install nfs-utils package

yum install -y nfs-utils

2.2 Creating a directory to mount an NFS share

mkdir -pv /mnt/nfs

mount -t nfs 192.168.112.10:/data/nfs /mnt/nfs

2.2.1. Unmounting the wrong mount point

If you execute themount -t nfs command, such as using the wrong arguments or paths, you can fix the error by unmounting the current mount point and remounting it again.

• Confirm the current mount status

  • utilizationmount command to view all current mounts and find entries that are mounted incorrectly.

  • mount
    

• Unmounting the wrong mount point

  • # Know the wrong mount point
    sudo umount /mnt/nfs
    

  • # Know the IP address of the nfs server and the path to the shared directory
    sudo umount 192.168.112.10:/data/nfs
    

• Encountered device/file in use while uninstalling

  • umount -f /mnt/nfs
    

  • Forced unloading may result in data loss, so only do this if you are certain that no data is being written.

2.3. Reboot and set NFS to boot itself

systemctl restart nfs && systemctl enable nfs

2.4. System boot automatically mounts NFS shares (not required)

echo '192.168.112.10:/data/nfs /mnt/nfs nfs defaults 0 0' >> /etc/fstab

2.5 Check the mount status

mount | grep 192.168.112.10

3、Create storage class

cat >>  << EOF
apiVersion: storage./v1
kind: StorageClass
metadata:
  name: nfs-storage
  namespace: default
  labels:
    environment: test
provisioner: /ifs
reclaimPolicy: Retain
volumeBindingMode: Immediate
EOF

kubectl apply -f 

4. Creating RBAC Privileges

cat >>  << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: default
---
kind: ClusterRole
apiVersion: ./v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage."]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: ./v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default                                      
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: .
---
kind: Role
apiVersion: ./v1
metadata:
  name: leader-locking-nfs-client-provisioner
  namespace: default                                        
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: ./v1
metadata:
  name: leader-locking-nfs-client-provisioner
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: .
EOF

kubectl apply -f 

5. Create the provisioner

Deploy NFS Client Provisioner, a Kubernetes external storage plugin for dynamically creating NFS PVs.

cat >> << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: /mydlq/nfs-subdir-external-provisioner:v4.0.0
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: /ifs # This must be filled in herestorageclasshit the nail on the headPROVISIONERConsistent name information
            - name: NFS_SERVER
              value: 192.168.112.10 # indicate clearly and with certaintyNFSserver-basedIPaddress
            - name: NFS_PATH
              value: /data/nfs # indicate clearly and with certaintyNFS服务器hit the nail on the head共享挂载目录
      volumes:
        - name: nfs-client-root # Define the name of the persistent volume，It has to be up there.volumeMountsThe name of the mount is the same
          nfs:
            server: 192.168.112.10 # indicate clearly and with certaintyNFSlocatedIPaddress
            path: /data/nfs # indicate clearly and with certaintyNFS服务器hit the nail on the head共享挂载目录
EOF

kubectl apply -f 

6. Create PVC

cat >>  << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
  namespace: default
  labels:
    environment: test
    app: nginx
spec:
  storageClassName: nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Mi
EOF

kubectl apply -f 

7. Observe whether the PV is dynamically created

kubectl get pvc,pv,sc

Discovered that using NFS Client Provisioner you can dynamically create pv's and bind them to PVC's in a Bound state.

Create multiple Pods that use the same PVC

cat >>  << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nfs-pod1
spec:
  containers:
    - name: container1
      image: nginx:1.16.0
      volumeMounts:
        - name: shared-data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: shared-data
      persistentVolumeClaim:
        claimName: nginx-pvc
EOF

cat >>  << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nfs-pod2
spec:
  containers:
    - name: container2
      image: nginx:1.16.0
      volumeMounts:
        - name: shared-data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: shared-data
      persistentVolumeClaim:
        claimName: nginx-pvc
EOF

kubectl apply -f  -f 

9. Write data to shared storage

kubectl exec -it nfs-pod1 -- /bin/bash
echo "hello from nfs-pod1" > /usr/share/nginx/html/
exit

10. Read data from another Pod

kubectl exec -it nfs-pod2 -- /bin/bash
cat /usr/share/nginx/html/
exit

11, view nfs server and client shared directory

tree /data/nfs
tree /mnt/nfs

This emerging directory (PV directory) default-nginx-pvc-pvc-c8a8b825-1577-4f76-ba1f-a1302941b333 is used to map PersistentVolume (PV) to specific paths on NFS servers

11.1 Flow of PV Catalog Creation

• Create the PVC:
  • When you create a PVC, Kubernetes automatically creates PVs based on StorageClass.
• Provisioner creates the PV:
  • Provisioner creates a new directory under the shared directory on the NFS server with a directory name that contains the name and UUID of the PVC.
• Mount to Pod:
  • The created PV is mounted to the path specified in the Pod.

11.2 Structure of the PV Catalog

/<shared directory>/<namespace>-<PVC name>-<PV name>

• Shared directory:
  • This is the directory you create and share out on the NFS server, such as /data/nfs for the nfs server and /mnt/nfs for the nfs client.
• Namespace:
  • This is a logical grouping in Kubernetes that is used to isolate different applications and services. the name of the namespace to which the PVC belongs.
• PVC Name:
  • This is the name of the PersistentVolumeClaim created in Kubernetes.
• PV Name:
  • This is the name of the PersistentVolume created dynamically from the PVC, usually a string with a UUID.

kubectl get pvc nginx-pvc -o custom-columns='PVC-NAMESPACE:.,PVC-NAME:.'

kubectl get pv pvc-c8a8b825-1577-4f76-ba1f-a1302941b333 -o custom-columns='PV-NAME:.'

12, modify the contents of the NFS server shared directory to view changes within the container

You can find synchronized updates as well

cd /data/nfs/default-nginx-pvc-pvc-c8a8b825-1577-4f76-ba1f-a1302941b333

echo "hello from nfs-server" > 

kubectl get pods -o wide
 
curl 10.244.1.3

curl 10.244.2.8

IV. Example - Dynamic Creation of PVCs and PVs

1. Create Headless Service

cat >>  << EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
EOF

kubectl apply -f 

kubectl get svc -l app=nginx -o wide

2、Create StatefulSet

cat >> << EOF
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.16.0
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates: # Binding by templating
  - metadata:
      name: www # indicate clearly and with certaintypvcyour name
      annotations:
        /storage-class: "nfs-storage" # 只indicate clearly and with certainty了storageClass
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Mi
EOF

kubectl apply -f 

3. Observe Pod status

Since it's a statefulset controller, pods are created sequentially

kubectl get pods -l app=nginx -w

4. Observe that the PVC and PV have been dynamically created and bound to each other.

kubectl get pvc -l 'app=nginx,environment!=test'

kubectl get pv | grep -v "default/nginx-pvc"

5、Each Pod will get a piece of independent PV by binding PVC.

kubectl get pod web-0 -o custom-columns='PVC-NAME:.[*].'

kubectl get pod web-1 -o custom-columns='PVC-NAME:.[*].'

kubectl get pod web-2 -o custom-columns='PVC-NAME:.[*].'

kubectl get pod web-3 -o custom-columns='PVC-NAME:.[*].'

kubectl get pod web-4 -o custom-columns='PVC-NAME:.[*].'