Kubernetes: Harbor Private Registry

by

We will see how to deploy Harbor Private registry in a highly available kubernetes cluster

Prerequisites for this is a Highly available Kubernetes cluster, a working Nginx Ingress, Some load balancer (We are using MetalLB here.) See how to setup metallb here.

First Lets do the OS Level Configuration

OS Level Configurations

Create working directory for Harbor and Navigate into it – [On All Nodes]

#mkdir /harbor

Create certs and the persistent volume directory – [On All Nodes]

#mkdir /harbor/certs && mkdir /harbor/data

Create sub directories for persistence of various components – [On All Nodes]

#mkdir /harbor/data/harbor-data /harbor/data/harbor-data-cm /harbor/data/harbor-data-js /harbor/data/harbor-data-db /harbor/data/harbor-data-redis /harbor/data/harbor-data-trivy

Kubernetes Configuration

Add harbor helm repo

#helm repo add harbor https://helm.goharbor.io

Fetch and Untar the repo so that we can modify the values

#helm fetch harbor/harbor --untar

Navigate in certs Directory to generate the certificates

#cd /harbor/certs && openssl genrsa -out ca.key 4096

Creating the Certificates

Create the cert file

#openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=IN/ST=MH/L=MUM/O=demolabs/OU=labs/CN=platform.demolabs.com" -key ca.key -out ca.crt

Generate the V3 extension file for adding additional DNS names

#cat <<'EOF' >> v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=platform.demolabs.com
DNS.2=demolabs
DNS.3=harbor
DNS.4=notary.platform.demolabs.com
EOF

Use the v3.ext created above generate the domain CRT

#openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in platform.demolabs.com.csr -out platform.demolabs.com.crt

Deploying Harbor

Create Namespace harbor

#kubectl create ns harbor

Create secret in harbor namespace with above generated certificates

#kubectl create secret tls harbor-cert --key platform.demolabs.com.key --cert platform.demolabs.com.crt -n harbor

Create the Persistent volumes

#cat <<'EOF' >> pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-data-pv
  namespace: harbor
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /root/harbor/data/harbor-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - platform
          - platform-n2
          - platform-n3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-data-cm-pv
  namespace: harbor
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /root/harbor/data/harbor-data-cm
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - platform
          - platform-n2
          - platform-n3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-data-js-pv
  namespace: harbor
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /root/harbor/data/harbor-data-js
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - platform
          - platform-n2
          - platform-n3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-data-db-pv
  namespace: harbor
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /root/harbor/data/harbor-data-db
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - platform
          - platform-n2
          - platform-n3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-data-redis-pv
  namespace: harbor
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /root/harbor/data/harbor-data-redis
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - platform
          - platform-n2
          - platform-n3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-data-trivy-pv
  namespace: harbor
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /root/harbor/data/harbor-data-trivy
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - platform
          - platform-n2
          - platform-n3
EOF

Create Persistent Volume Claims

#cat <<'EOF' >> pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-data
  namespace: harbor
spec:
  storageClassName: local-storage
  volumeName: harbor-data-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-data-cm
  namespace: harbor
spec:
  storageClassName: local-storage
  volumeName: harbor-data-cm-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-data-js
  namespace: harbor
spec:
  storageClassName: local-storage
  volumeName: harbor-data-js-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-data-db
  namespace: harbor
spec:
  storageClassName: local-storage
  volumeName: harbor-data-db-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-data-redis
  namespace: harbor
spec:
  storageClassName: local-storage
  volumeName: harbor-data-redis-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-data-trivy
  namespace: harbor
spec:
  storageClassName: local-storage
  volumeName: harbor-data-trivy-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi
EOF

Create the PV and PVC

#kubectl apply -f pv.yaml
#kubectl apply -f pvc.yaml

Modify the values.yaml as required

Line 4: 
	[type: ingress] - [if you need any other service edit it here]

Line 19: 
	[certSource: secret]

Line 23:
	[commonName: "platform.demolabs.com"]

Line 28:
	[secretName: "harbor-cert"]

Line 33: 
	[notarySecretName: "harbor-cert"]

Line 36:
	[core: platform.demolabs.com]

Line 37:
	[notary: notary.platform.demolabs.com]Line 4: 
	[type: ingress] - [if you need any other service edit it here]

Line 47:
	[className: "nginx"]

Line 127:
	[externalURL: https://platform.demolabs.com]

Line 205 - 251
	Add existingClaim: "" and storageClass: "" for each component
	For ex: for harbor-data we have created pvc as "harbor-data" and storage class is "local-storage" so
	 existingClaim: "harbor-data"
	 storageClass: "local-storage"
	Similarly add for [chartmuseum, jobservice, database, redis and trivy] 
	Make sure that the existing claim name matches to the pvc name created above for the components
	Storage Class will remain the same for ALL

---> Example:
persistence:
enabled: true
resourcePolicy: "keep"
persistentVolumeClaim:
 registry:
   existingClaim: "harbor-data"
   storageClass: "local-storage"
   subPath: ""
   accessMode: ReadWriteOnce
   size: 5Gi
   annotations: {}
 chartmuseum:
   existingClaim: "harbor-data-cm"
   storageClass: "local-storage"
   subPath: ""
   accessMode: ReadWriteOnce
   size: 5Gi
   annotations: {}
 jobservice:
   jobLog:
     existingClaim: "harbor-data-js"
     storageClass: "local-storage"
     subPath: ""
     accessMode: ReadWriteOnce
     size: 1Gi
     annotations: {}
 database:
   existingClaim: "harbor-data-db"
   storageClass: "local-storage"
   subPath: ""
   accessMode: ReadWriteOnce
   size: 1Gi
   annotations: {}
 redis:
   existingClaim: "harbor-data-redis"
   storageClass: "local-storage"
   subPath: ""
   accessMode: ReadWriteOnce
   size: 1Gi
   annotations: {}
 trivy:
   existingClaim: "harbor-data-trivy"
   storageClass: "local-storage"
   subPath: ""
   accessMode: ReadWriteOnce
   size: 5Gi
   annotations: {}
--------

Line 363:
	harborAdminPassword: "Harbor12345"

Deploy harbor with the values file and make sure you are in the directory where values.yaml is present

#helm install harbor . -n harbor

Check and wait for pods to come in running state

#kubectl get pods -n harbor -o wide

Check for Service

#kubectl get svc -n harbor

Check for ingress

#kubectl get ingress -n harbor

Finally Access the URL

Access the URL https://platform.demolabs.com to check id harbor is up

Conclusion

We have successfully deployed Harbor in high Availability in Kubernetes cluster using Local Storage as storage class, nginx ingress and MetalLB as Load Balancer

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

error: Content is protected !!