Version: 3.4

Installation on a Bare Metal Kubernetes Cluster using Custom Container Registry

This topic provides instructions for installing Portworx on a bare metal Kubernetes cluster using a custom container registry.

note

The steps in this document use the air-gapped-install bootstrap script to create a custom registry for internet connected clusters. For informationon how to install Portworx on an air-gapped bare metal cluster, see Installation on Air-Gapped Bare Metal Kubernetes Cluster.

The following collection of tasks describe how to install Portworx on a bare metal Kubernetes cluster using the Portworx Operator:

Configure Custom Registry
Generate Portworx Specification
Deploy Portworx Operator
Deploy StorageCluster
Verify Portworx Pod Status
Verify pxctl Cluster Provision Status

Complete all the tasks to install Portworx.

Configure Custom Registry

Set an environment variable for your Kubernetes version:

KBVER=$(kubectl version --short | awk -F'[v+_-]' '/Server Version: / {print $3}')

Set an environment variable to specify the latest major version of Portworx:
```
PXVER=<portworx-version>
```
On an internet-connected host that matches the architecture and OS version of the Kubernetes cluster nodes intended for Portworx installation, download the air-gapped installation bootstrap script for the Kubernetes and Portworx versions:
```
curl -o px-ag-install.sh -L "https://install.portworx.com/$PXVER/air-gapped?kbver=$KBVER"
```
Pull the container images for the Kubernetes and Portworx versions:
```
sh px-ag-install.sh pull
```
Log in to docker:
```
docker login <your-custom-registry>
```
Push the container images to a custom registry that is accessible to your nodes. Do not include http:// in your custom registry path:
```
sh px-ag-install.sh push <your-registry-path>
```
For example:
```
sh px-ag-install.sh push myregistry.net:5443
```
Example for pushing image to a specific repo:
```
sh px-ag-install.sh push myregistry.net:5443/px-images
```

Create a version manifest configmap for the Portworx Operator

Download the Portworx version manifest:

curl -o versions.yaml "https://install.portworx.com/$PXVER/version?kbver=$KBVER"

(Optional) If your installation uses images from multiple custom registries, update the version manifest with the custom registry location details. You can use a DNS hostname and domain, or IP addresses (IPv4 or IPv6), to specify the container registry server in the following format:
```
<dns-host.domain or IPv4 or IPv6>[:<port>]/repository/image:tag
```
The following example demonstrates registries using a custom DNS hostname + domain, IPv4, and IPv6:
version-config.yaml
```
version: 2.13.3
components:
  stork: custom-registry.acme.org/portworx/backup/stork:23.2.1
  autopilot: 192.168.1.2:5433/tools/autopilot:1.3.7
  nodeWiper: [2001:db8:3333:4444:5555:6666:7777:8888]:5443/portworx/px-node-wiper:2.13.2
```
note
- Ensure that the Custom Container Registry location field is empty for any specs you generate in the spec generator.
- kubeScheduler, kubeControllerManager, and pause may not appear in the version manifest, but you can include them in the px-version configmap:
... kubeScheduler: custom-registry.acme.org/k8s/kube-scheduler-amd64:v$KBVER kubeControllerManager: custom-registry.acme.org/k8s/kube-controller-manager-amd64:v$KBVER pause: custom-registry.acme.org/k8s/pause:3.1

Create a configmap from the downloaded or updated version manifest:

kubectl -n <px-namespace> create configmap px-versions --from-file=versions.yaml

Generate Portworx Specification

To install Portworx, you must first generate Kubernetes manifests that you will deploy in your bare metal Kubernetes cluster by following these steps.

Sign in to the Portworx Central console.
In the Welcome to Portworx! section, select Get Started.
On the Product Line page, in the Portworx Enterprise section, select Continue.
From the Portworx Version dropdown menu, select the Portworx version to install.
From the Platform dropdown menu, select DAS/SAN.
From the Distribution Name dropdown menu, select None.
Click Customize.
On the Basic tab:
1. Select the Use the Portworx Operator and Built-in ETCD checkboxes.
2. From the Portworx version dropdown, select the same value that you have set as your Portworx version in the Configure Custom Registry section.
3. Click Next.
On the Storage tab, retain the recommended default values and click Next.
On the Network tab:
1. Enter the Data Network Interface to be used for data traffic.
2. Enter the Management Network Interface to be used for management traffic.
3. Enter the Starting port for Portworx services.
4. Click Next.
On the Customize tab:
1. In the Customize section, under Are you running on either of these?, select None.
2. In the Registry and Image Settings section:
  - If you use a single private registry, enter the internal registry path and the details for how to connect to your private registry in the Custom Container Registry Location field.
  - If you use multiple private registries, leave the Custom Container Registry Location field blank.
Click Finish to generate the specs.
Log in to the custom container registry using your JFrog credentials. The registry automatically creates the following sample config.json file:
```
cat /root/.docker/config.json
{
        "auths": {
                <custom registry name>: {
            "auth": <Base 64 version of Jfrog username: Jfrog token>
                }
        }
}
```
Replace custom registry name with the name of the custom registry and Base 64 version of Jfrog username: Jfrog token with the base 64 version of your jfrog username and token.
Use the config.json file, and create a registry secret to pull container images from the custom container registry:
```
kubectl create secret generic regcred -n portworx \
--from-file=.dockerconfigjson=/root/.docker/config.json \
--type=kubernetes.io/dockerconfigjson
```
After creating the regcred registry secret, add the registry secret in the Kubernetes Docker Registry Secret field.
Configure the STC configuration file to pull OCI monitor component images from the custom container registry:
```
env:
- name: REGISTRY_USER
    value: <repository user credentials>
- name: REGISTRY_PASS
    value: <repository token>
spec:
image: registry.portworx.io/portworx/oci-monitor:3.3.0.1
```
Replace repository user credentials with the actual username used to authenticate with the registry.

Deploy Portworx Operator

Use the Operator specifications you generated in the Generate Portworx Specification section, and deploy Portworx Operator by running the following command.

kubectl apply -f 'https://install.portworx.com/<PX-version-number>?comp=pxoperator'

serviceaccount/portworx-operator created
podsecuritypolicy.policy/px-operator created
clusterrole.rbac.authorization.k8s.io/portworx-operator created
clusterrolebinding.rbac.authorization.k8s.io/portworx-operator created
deployment.apps/portworx-operator created

Deploy StorageCluster

Use the StorageCluster specifications you generated in the Generate Portworx Specification section, and deploy StorageCluster by running the following command.

kubectl apply -f 'https://install.portworx.com/<PX-version-number>?operator=true&mc=false&kbver=&b=true&c=px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-8dfd338e915b&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true'

storagecluster.core.libopenstorage.org/px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-8dfd338e915b created

(Optional) If you have a disaggregated setup, after you generate the StorageCluster spec, you must create two separate node sections in the spec to define the device settings for the storage and storageless (compute) nodes. Here is a sample StorageCluster spec that uses node-specific overrides:

Verify Portworx Pod Status

Enter the following command to list and filter the results for Portworx pods and specify the namespace where you have deployed Portworx:

kubectl get pods -n <px-namespace> -o wide | grep -e portworx -e px

NAME                                                    READY   STATUS    RESTARTS         AGE     IP                NODE                   NOMINATED NODE   READINESS GATES
portworx-api-774c2                                      1/1     Running   0                2m55s   192.168.121.196   username-k8s1-node0    <none>           <none>
portworx-api-t4lf9                                      1/1     Running   0                2m55s   192.168.121.99    username-k8s1-node1    <none>           <none>
portworx-api-dvw64                                      1/1     Running   0                2m55s   192.168.121.99    username-k8s1-node2    <none>           <none>
portworx-kvdb-94bpk                                     1/1     Running   0                4s      192.168.121.196   username-k8s1-node0    <none>           <none>
portworx-kvdb-8b67l                                     1/1     Running   0                10s     192.168.121.196   username-k8s1-node1    <none>           <none>
portworx-kvdb-fj72p                                     1/1     Running   0                30s     192.168.121.196   username-k8s1-node2    <none>           <none>
portworx-operator-58967ddd6d-kmz6c                      1/1     Running   0                4m1s    10.244.1.99       username-k8s1-node0    <none>           <none>
prometheus-px-prometheus-0                              2/2     Running   0                2m41s   10.244.1.105      username-k8s1-node0    <none>           <none>
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d-9gs79   2/2     Running   0                2m55s   192.168.121.196   username-k8s1-node0    <none>           <none>
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d-vpptx   2/2     Running   0                2m55s   192.168.121.99    username-k8s1-node1    <none>           <none>
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d-bxmpn   2/2     Running   0                2m55s   192.168.121.191   username-k8s1-node2    <none>           <none>
px-csi-ext-868fcb9fc6-54bmc                             4/4     Running   0                3m5s    10.244.1.103      username-k8s1-node0    <none>           <none>
px-csi-ext-868fcb9fc6-8tk79                             4/4     Running   0                3m5s    10.244.1.102      username-k8s1-node2    <none>           <none>
px-csi-ext-868fcb9fc6-vbqzk                             4/4     Running   0                3m5s    10.244.3.107      username-k8s1-node1    <none>           <none>
px-prometheus-operator-59b98b5897-9nwfv                 1/1     Running   0                3m3s    10.244.1.104      username-k8s1-node0    <none>           <none>

Note the name of a px-cluster pod. You will run pxctl commands from these pods in Verify pxctl Cluster Provision Status.

Verify pxctl Cluster Provision Status

Access the Portworx CLI.

Run the following command to find the storage cluster:

kubectl -n <px-namespace> get storagecluster

NAME                                              CLUSTER UUID                           STATUS   VERSION   AGE
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d   xxxxxxxx-xxxx-xxxx-xxxx-6f3fd5522eae   Online   2.11.0    10m

The status must display the cluster is Online.

Run the following command to find the storage nodes:

kubectl -n <px-namespace> get storagenodes

NAME                  ID                                     STATUS   VERSION          AGE
username-k8s1-node0   xxxxxxxx-xxxx-xxxx-xxxx-fad8c65b8edc   Online   2.11.0-81faacc   11m
username-k8s1-node1   xxxxxxxx-xxxx-xxxx-xxxx-70c31d0f478e   Online   2.11.0-81faacc   11m
username-k8s1-node2   xxxxxxxx-xxxx-xxxx-xxxx-19d45b4c541a   Online   2.11.0-81faacc   11m  

The status must display the nodes are Online.

Verify the Portworx cluster provision status by running the following command.
Specify the pod name you retrieved in Verify Portworx Pod Status.

kubectl exec <px-pod> -n <px-namespace> -- /opt/pwx/bin/pxctl cluster provision-status

NODE					                NODE STATUS	 POOL						              POOL STATUS  IO_PRIORITY	SIZE	AVAILABLE	USED   PROVISIONED ZONE REGION	RACK
0c99e1f2-9d49-xxxx-xxxx-xxxxxxxxxxxx	Up		    0 ( 8ec9e6aa-7726-xxxx-xxxx-xxxxxxxxxxxx )	Online		HIGH		32 GiB	32 GiB		33 MiB	0 B		default	default	default
1e89102f-0510-xxxx-xxxx-xxxxxxxxxxxx	Up		    0 ( 06fcc73a-7e2f-xxxx-xxxx-xxxxxxxxxxxx )	Online		HIGH		32 GiB	32 GiB		33 MiB	0 B		default	default	default
24508311-e2fe-xxxx-xxxx-xxxxxxxxxxxx	Up		    0 ( 58ab2e3f-a22e-xxxx-xxxx-xxxxxxxxxxxx )	Online		HIGH		32 GiB	32 GiB		33 MiB	0 B		default	default	default

What to do next

Create a PVC. For more information, see Create your first PVC.

Configure Custom Registry​

Create a version manifest configmap for the Portworx Operator​

Generate Portworx Specification​

Deploy Portworx Operator​

Deploy StorageCluster​

Verify Portworx Pod Status​

Verify pxctl Cluster Provision Status​

What to do next​