-DirectPV Distributed Multi-Tenant Storage Deployment

MinIO Deployment Preparation

Deployment overview

This experiment combines MinIO deployment with Kubernetes to realize the integration of MinIO in Kubernetes.

minio is officially supported through simple rapid deployment for basic testing:

curl /minio/docs/master/source/extra/examples/ -O
kubectl apply -f 

Refer to the official:Rapid Deployment of MinIO 。

This experiment is based on a production environment and deploys a set of multi-tenant Minio that can be used in a production environment.

MinIO officially recommends using minio-operator to deploy multi-tenant minio systems.
Each MinIO Tenant represents a separate MinIO Object Store in a Kubernetes cluster.

Deployment Architecture

The official best practice recommended architecture is as follows:

preliminary planning

Kubernetes Installation

This experiment does not involve Kubernetes deployment, the Kubernetes deployment reference isKubernetes_v1.30.3 Highly Available Deployment Architecture II 。

The plans and related information for the completed deployed Kubernetes clusters are listed below:

Cluster VIP: 172.24.10.100
The relevant domain names can be resolved normally, you can resolve the relevant domain names to VPI via hosts.

helm installation

The official recommendation for minio in production is to deploy it using helm, so you need to deploy the helm tool in Kubernetes in advance.
Reference:Kubernetes Cluster Management - Helm Deployment and Usage 。

MinIO Operator Components

Introduction to Minio Operator

Before deploying minio, you need to complete the deployment of minio operator, this experiment uses the latest stable Operator version 6.0.2.

MinIO Operator installs a Custom Resource Definition (CRD) to support describing MinIO tenants as Kubernetes objects.

The MinIO Operator exists in its own separate namespace and creates Kubernetes resources in it, which include pods, svc, replicasets, and deployments.

By default, Operator pods use the MinIO CRD to monitor all namespaces of an object and automatically manage these resources.

When a tenant is created using Operator, the tenant must have its own namespace in which Operator generates the pods required for tenant configuration.

Each Tenant pod runs three containers.

• MinIO Container A container that runs all standard MinIO functionality, equivalent to a basic MinIO installation on bare metal. This container stores and retrieves objects in the provided mount point (pv).
• InitContainer, which exists only during the startup of the pod, is used to manage the configuration secret during the startup, and this container is terminated when the startup is complete.
• The Sidecar container is used to initialize the MinIO tenants. sidecar retrieves and validates the configuration of each tenant and creates the necessary local resources in the pod.

Starting with Operator 6.0.0, Sidecar has its own image and release cycle, separate from the rest of MinIO Operator.
MinIO Operator stores the tenant's environment variables in the sidecar, allowing Operator to update the variables without requiring a rolling reboot.

The tenant enables pvc to communicate with the persistent volume in which the object is stored.

Tip:
1: kubectl -k and kubectl --kustomize deployment are also officially supported:Deploy the MinIO Operator . The limited use of helm installation for this experiment is also available:Deploy Operator With Helm 。
2: However, if you use Helm chart to install Operator, you must use Helm to manage that installation process. Do not use kubectl krew, Kustomize, or similar methods to update or manage the MinIO Operator installation.

Kubernetes TLS Certificates

MinIO Operator automatically generates TLS Certificate Signing Requests (CSRs) and creates signed TLS certificates using the Kubernetes certificates. the TLS Certificate Management API.

MinIO Operator therefore requires that the Kubernetes kube-controller-manager configuration include the following configuration settings:

• --cluster-signing-key-file : Specifies the PEM-encoded RSA or ECDSA private key used to sign cluster-wide certificates.
• --cluster-signing-cert-file : Specifies the x.509 Certificate Authority certificate for the PEM encoding used to issue cluster-wide certificates.

Operator cannot complete initialization if the Kubernetes cluster is not configured with the corresponding generated CSR.
By default, Kubernetes is configured for autoblocking as above, and you can check whether the current Kubernetes cluster is configured for TLS certificates in the following way.

kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane -n kube-system -o yaml

Tip: MinIO Operator automatically generates TLS certificates for all MinIO tenant pods using a specified Certificate Authority (CA). clients external to the Kubernetes cluster must trust the Kubernetes cluster CA in order to connect to MinIO Operator or a MinIO tenant.

It is also possible to generate x.509 TLS certificates signed by a known and trusted CA and pass these certificates to MinIO tenants.

Add repo

Add the minio chart repo using helm.

[root@master01 ~]# mkdir ten-minio
[root@master01 ~]# cd ten-minio
[root@master01 ten-minio]# helm repo add minio-operator 
[root@master01 ten-minio]# helm repo update
[root@master01 ten-minio]# helm search repo minio-operator
NAME                         	CHART VERSION	APP VERSION	DESCRIPTION                    
minio-operator/minio-operator	4.3.7        	v4.3.7     	A Helm chart for MinIO Operator
minio-operator/operator      	6.0.3        	v6.0.3     	A Helm chart for MinIO Operator
minio-operator/tenant        	6.0.3        	v6.0.3     	A Helm chart for MinIO Operator

Tip: Available domesticallyhelm repo add minio-operator Make additions.
minio-operator/minio-operator is an older version of operator and does not need to be installed.

Installation operator

Run the helm install command to install operator.
The following command specifies and creates a dedicated minio-operator namespace for installation.

[root@master01 ten-minio]# helm show values minio-operator/operator > #selectable，Export Defaultschart values
[root@master01 ten-minio]# cat

[root@master01 ten-minio]# helm install \
  --namespace minio-operator \
  --create-namespace \
  operator minio-operator/operator

Verify operator

Verify the operator installation.

[root@master01 ten-minio]# kubectl get all -n minio-operator #View installed resources
NAME READY STATUS RESTARTS AGE
pod/minio-operator-95fd896dc-cdt84 1/1 Running 0 2m6s
pod/minio-operator-95fd896dc-rfjsr 1/1 Running 0 2m6s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/operator ClusterIP 10.20.194.160 <none> 4221/TCP 2m7s
service/sts ClusterIP 10.20.24.158 <none> 4223/TCP 2m7s

NAME READY UP-TO-DATE AVAILABLE AGE
/minio-operator 2/2 2 2 2m7s

NAME DESIRED CURRENT READY AGE
/minio-operator-95fd896dc 2 2 2 2m7s

Permanent Volume

MinIO requires dedicated access to the drive or volume that gives access to the object storage, this experiment uses the local disk /dev/nvme0n2 , which is used independently to deploy minio.

Tip: This proprietary disk is used for deploying minio distributed storage, and no other processes, software, scripts, etc. are allowed to perform any operations or perform operations on objects or files placed on it by MinIO, otherwise it may bring the risk of data corruption.

MinIO can use any Kubernetes PV that supports the ReadWriteOnce access mode.
MinIO's consistency guarantees require exclusive storage access, and ReadWriteOnce provides that support.

Additionally, MinIO recommends setting a Retain recycling policy for PVC StorageClass, and also recommends formatting volumes to XFS when configuring storage class, CSI, or PV-based storage to ensure optimal performance.

MinIO can use any Kubernetes Persistent Volume (PV) that supports ReadWriteOnce access mode.
MinIO's consistency guarantees require the exclusive storage access provided by ReadWriteOnce, and the persistent volume must already exist before the tenant is deployed.

Additionally, MinIO recommends setting up Retain's recycling policy for PVC StorageClass, and under best practices, configuring StorageClass, CSI, or other providers at the bottom of the PV, it is recommended that volumes be formatted as XFS to ensure optimal performance.

For Kubernetes clusters where nodes have direct attached storage, MinIO strongly recommends using the DirectPV CSI driver.

DirectPV provides a distributed persistent volume manager that discovers, formats, mounts, schedules, and monitors drives on Kubernetes nodes, DirectPV addresses the limitations of manually configuring and monitoring local persistent volumes.

For details on DirectPV usage, see:Introduction and Installation cap (a poem)Storage Management 。

This experiment has completed the installation of directpv with the following information:

[root@master01 ten-minio]# kubectl -n directpv get all
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-79f6b96bf8-t9c4l   3/3     Running   0          25m
pod/controller-79f6b96bf8-vrb2q   3/3     Running   0          25m
pod/controller-79f6b96bf8-x76gj   3/3     Running   0          25m
pod/node-server-4tj7p             4/4     Running   0          25m
pod/node-server-67c7b             4/4     Running   0          25m
pod/node-server-cjhd5             4/4     Running   0          25m
pod/node-server-k6ks2             4/4     Running   0          25m

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
/node-server   4         4         4       4            4           <none>          25m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
/controller   3/3     3            3           25m

NAME                                    DESIRED   CURRENT   READY   AGE
/controller-79f6b96bf8   3         3         3       25m
[root@master01 ten-minio]# kubectl -n directpv get sc
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io      directpv-min-io      Delete          WaitForFirstConsumer   true                   26m

Creating a tenant namespace

Create a namespace that simulates multi-tenancy.

[root@master01 ten-minio]# vim 
apiVersion: v1
kind: Namespace
metadata:
  name: minio-ns01
  labels:
    name: minio-ns01

[root@master01 ten-minio]# kubectl apply -f 

MinIO Helm deployment

helm overview

Helm is a tool that automates the deployment of applications to Kubernetes clusters ref:Cluster Management - Helm Deployment and Usage 。

Helm charts are a set of YAML files, templates and other files needed to define deployment details.
The following procedure uses Helm Chart to deploy tenants managed by MinIO Operator.

This process requires that the Kubernetes cluster has a valid Operator deployment and cannot use the MinIO Operator tenant chart to deploy tenants independent of Operator.

Tip: The MinIO Operator Tenant Chart is different from the community-managed MinIO chart.

Community Helm Chart is built, maintained and supported by the community.
MinIO does not guarantee support for any given bug, feature request, or issue arising from an update that references this chart.

The Operator Tenant Chart is officially maintained and supported by MinIO, and MinIO strongly recommends using the official Helm Chart for Operator and tenant in production environments.

precondition

To install a MinIO tenant with Helm, the following requirements need to be met:

• Available Kubernetes clusters;
• The kubectl CLI tool that matches the Kubernetes cluster;
• Hlem 3.8 or later;
• yq 4.18.1 or later;
• MinIO Operator is installed.

This installation must have Kubernetes cluster access administrative privileges.

Helm Charts deployed MinIO Tenant

The following uses the official MinIO Tenant chart for deploying MinIO tenants, which supports a simplified installation path compared to a local chart installation.

The following steps use Helm to deploy MinIO tenants based on the official MinIO Tenant chart.

More chart configuration references:MinIO Server Config Guide 。

Verify MinIO Operator

Add repo reference:Add repo

Configuring helm

Configure the related helm vaule .

[root@master01 ten-minio]# curl -sLo  /minio/operator/master/helm/tenant/

Tip: If you use Helm to deploy MinIO tenant, you must use Helm to manage or upgrade that deployment. Do not use kubectl krew, Kustomize, or similar methods to manage or upgrade MinIO tenant.
2: You can also use the helm command to export the default of minio chart.

[root@master01 ten-minio]# helm show values minio-operator/tenant > 

Configure tenant topology

The pool[0] prefix controls the number of servers, the number of volumes per server, and the storage class for all pods deployed in the tenant.

Configuration Notes:

[root@master01 ten-minio]# vim
tenant: name: tenant-01
  name: tenant-01
  pools.
    - servers: 4
      volumesPerServer: 4
      name: pool-0
      size: 2Gi #Limited by experimental disk space, adjusted to 2Gi
      storageClassName: directpv-min-io #Specify the sc created with directpv.

Configuration affinity/ or anti-affinity

Tenant Chart supports the following Kubernetes Selector, affinity, and anti-affinity configurations:

• Node Selector ()
• Node/Pod affinity or anti-affinity ([n].affinity)

MinIO recommends configuring Pod anti-affinity for tenants to ensure that the Kubernetes scheduler does not deploy multiple Pods to the same worker node.

If you need to deploy Pods to specific worker nodes, you can pass the labels or filters for those nodes to the nodeSelector or affinity field to restrict the scheduler from deploying Pods to those nodes.

[root@master01 ten-minio]# kubectl label nodes worker0{1,2,3,4} tenant-minio=enabled #Tagging nodes based on deployment planning
tenant:
  pools:
    - servers: 4
      name: pool-0
      size: 2Gi #Limited by experimental disk space，turn2Gi
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: tenant-minio
                operator: In
                values:
                - enabled

Configuring TLS Certificates

The MinIO Tenant CRD provides the following fields; use the appropriate fields to configure tenant TLS network encryption:

Create certificates in advance.

[root@master01 ten-minio]# ll *com*
-rw-r--r-- 1 root root 3.9K Aug 24 06:19 
-rw-r--r-- 1 root root 1.7K Aug 24 06:19 
-rw-r--r-- 1 root root 3.9K Aug 24 05:36 
-rw-r--r-- 1 root root 1.7K Aug 24 05:36 


[root@master01 ten-minio]# kubectl -n minio-ns01 create secret tls minio-webui-tls --cert= --key=
[root@master01 ten-minio]# kubectl -n minio-ns01 create secret tls minio-api-tls --cert= --key=

[root@master01 ten-minio]# kubectl -n minio-ns01 describe secrets minio-webui-tls
[root@master01 ten-minio]# kubectl -n minio-ns01 describe secrets minio-api-tls

Turn off cert-manager automatic certificates.

  certificate:
    externalCaCertSecret: [ ]
    externalCertSecret: [ ]
    requestAutoCert: false
    certConfig: { }

Configuring Environment Variables

You can use Set MinIO Server environment variables.

This setting allows you to set minio related users and passwords, this experiment keeps the default minio/minio123 .

Configuring the ingress

Configure ingress rules to expose minio within the cluster to the outside world.

ingress:
  api:
    enabled: true
    ingressClassName: "nginx"
    labels: { }
    annotations: 
      /rewrite-target: /
    tls: 
      - hosts:
          - ''
        secretName: minio-api-tls
    host: 
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: "nginx"
    labels: { }
    annotations: 
      /rewrite-target: /
    tls: 
      - hosts:
          - ''
        secretName: minio-webui-tls
    host: 
    path: /
    pathType: Prefix

Complete below:

tenant:
  name: tenant-01
  image:
    repository: /minio/minio
    tag: RELEASE.2024-08-17T01-24-54Z
    pullPolicy: IfNotPresent
  imagePullSecret: { }
  scheduler: { }
  configuration:
    name: myminio-env-configuration
  configSecret:
    name: myminio-env-configuration
    accessKey: minio
    secretKey: minio123

  pools:
    - servers: 4
      name: pool-0
      volumesPerServer: 4
      size: 2Gi
      storageClassName: directpv-min-io
      storageAnnotations: { }
      annotations: { }
      labels: { }
      tolerations: [ ]
      nodeSelector: { }
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: tenant-minio
                operator: In
                values:
                - enabled
      resources: { }
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        fsGroupChangePolicy: "OnRootMismatch"
        runAsNonRoot: true
      containerSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints: [ ]
  mountPath: /export
  subPath: /data
  metrics:
    enabled: false
    port: 9000
    protocol: http
  certificate:
    externalCaCertSecret: [ ]
    externalCertSecret: [ ]
    requestAutoCert: false
    certConfig: { }
  features:
    bucketDNS: false
    domains: { }
    enableSFTP: false
  buckets: [ ]
  users: [ ]
  podManagementPolicy: Parallel
  liveness: { }
  readiness: { }
  startup: { }
  lifecycle: { }
  exposeServices: { }
  serviceAccountName: ""
  prometheusOperator: false
  logging: { }
  serviceMetadata: { }
  env: [ ]
  priorityClassName: ""
  additionalVolumes: [ ]
  additionalVolumeMounts: [ ]

ingress:
  api:
    enabled: true
    ingressClassName: "nginx"
    labels: { }
    annotations: 
      /rewrite-target: /
    tls: 
      - hosts:
          - ''
        secretName: minio-api-tls
    host: 
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: "nginx"
    labels: { }
    annotations: 
      /rewrite-target: /
    tls: 
      - hosts:
          - ''
        secretName: minio-webui-tls
    host: 
    path: /
    pathType: Prefix

Official deployment

Use the configured one for deployment.

[root@master01 ten-minio]# helm install \
--namespace minio-ns01 \
--create-namespace \
--values  tenant-01 minio-operator/tenant

validate

Confirm that the related deployment was successful.

• Deployment validation
  View minio.

[root@master01 ten-minio]# kubectl get all -n minio-ns01 -o wide
NAME                     READY   STATUS    RESTARTS   AGE    IP             NODE       NOMINATED NODE   READINESS GATES
pod/tenant-01-pool-0-0   2/2     Running   0          6m6s   10.10.5.9      worker01   <none>           <none>
pod/tenant-01-pool-0-1   2/2     Running   0          6m6s   10.10.30.65    worker02   <none>           <none>
pod/tenant-01-pool-0-2   2/2     Running   0          6m6s   10.10.196.79   worker04   <none>           <none>
pod/tenant-01-pool-0-3   2/2     Running   0          6m5s   10.10.19.79    worker03   <none>           <none>

NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE    SELECTOR
service/minio               ClusterIP   10.20.208.139   <none>        80/TCP     6m7s   /tenant=tenant-01
service/tenant-01-console   ClusterIP   10.20.114.138   <none>        9090/TCP   6m7s   /tenant=tenant-01
service/tenant-01-hl        ClusterIP   None            <none>        9000/TCP   6m7s   /tenant=tenant-01

NAME                                READY   AGE    CONTAINERS      IMAGES
/tenant-01-pool-0   4/4     6m6s   minio,sidecar   /minio/minio:RELEASE.2024-08-17T01-24-54Z,/minio/operator-sidecar:v6.0.2

[root@master01 ten-minio]# kubectl -n minio-ns01 get ingress -o wide
NAME                CLASS   HOSTS               ADDRESS        PORTS     AGE
tenant-01           nginx        172.24.10.11   80, 443   6m31s
tenant-01-console   nginx      172.24.10.11   80, 443   6m31s

View directpv.

[root@master01 ten-minio]# kubectl -n minio-ns01 get pvc
[root@master01 ten-minio]# kubectl directpv list drives

• browser authentication
  Accessed using a browser The default minio account password: minio / minio123.

• MC Validation
  The MinIO Client mc command line tool provides commands such as ls, cat, cp, mirror, and diff to support file systems and Amazon s3-compatible cloud storage services.

The mc command line tool is built for compatibility with the AWS S3 API and has been tested on MinIO and AWS S3 for expected functionality and behavior.

Install mc:

[root@master01 minio]# curl /client/mc/release/linux-amd64/mc \
  --create-dirs \
  -o /usr/local/bin/mc

[root@master01 minio]# chmod +x /usr/local/bin/mc
[root@master01 minio]# mc --help

Connect minio:
Use the mc alias set command to add Amazon s3-compatible services to the mc configuration, replacing the alias with the name to be associated with the S3 service.
The mc command usually requires alias as an argument to identify which S3 service it is to be executed against. If you omit ACCESS_KEY and SECRET_KEY, you will be prompted to enter these values in the CLI when executing the command.

[root@master01 minio]# mc alias set myminio  minio minio123
Added `myminio` successfully.
[root@master01 minio]# mc admin info myminio

Deployment Reference:Installing the MinIO Book with helm in K8S 。