Longhorn-backed CNPG PostgreSQL PVCs

And why you shouldn't do it.

January 4, 2025

This tutorial explains how to recover data from a CNPG PostgreSQL cluster in the unfortunate event where you have the PV but not the PVC annotations. It might sound far-fetched, but I've been in this situation a few times before establishing properly supported backups (a.k.a. object store).

Background

First time I had to do this was when I was doing backups through Longhorn, I had a policy in place to run a backup once per day and send to a NFS-mounted storage. My first big mistake was not testing my backups and trusting that it would work because all others do, but after a disk failure, the database refused to start on another node.

After recovering the databases, I switched to using CSI volume snapshots, following the official CNPG documentation. This worked fine until I lost my Kubernetes server node due to a misconfiguration while tinkering with K3s certificates and tokens. Recreating the K3s cluster with FluxCD revealed that restoring volume snapshots required volumesnapshots.snapshot.storage.k8s.io resources, which weren't versioned in my GitOps workflow. This recovery procedure saved my databases once again.

My Current Setup

Today, I use a MinIO server outside my K3s cluster to store backups. This approach allows for easy restores from a fresh cluster without relying on auto-generated custom resources. While tools like Velero could back up these dynamic resources, its lack of native NFS support makes an object store necessary either way, so I'd rather cut the middleman.

Recovery Process

This process involves initializing a new cluster and replacing its PVC while the cluster is hibernating.

⚠️ Note: You perform this at your own risk, and I'm only responsible for my data loss! ⚠️

Step 1: Create a New CNPG Cluster

Start by creating a new CNPG PostgreSQL cluster as if it were a fresh install, complete with initdb and all. Keep the deployments using this database scaled to 0. Wait for the cluster status to show "Cluster in healthy state".

For example, let's use a cluster for an Authentik instance:

postgresql.yaml

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: authentik-postgres
  labels:
    app.kubernetes.io/component: database
spec:
  description: "Authentik Cluster"
  imageName: ghcr.io/cloudnative-pg/postgresql:16.3-7
  instances: 2
  startDelay: 300
  stopDelay: 300
  primaryUpdateStrategy: unsupervised

  postgresql:
    parameters:
      log_connections: "false"
      log_disconnections: "false"
      client_min_messages: "error"
      log_hostname: "false"
      max_connections: "100"
      archive_timeout: "1800"

  bootstrap:
    initdb:
      database: authentik
      owner: authentik
      secret:
        name: authentik-postgres-user

  storage:
    storageClass: longhorn-cnpg
    size: 1Gi
  
  affinity:
    enablePodAntiAffinity: true

  nodeMaintenanceWindow:
    inProgress: false
    reusePVC: false

  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "1Gi"

Run the following command to check the cluster status: kubectl get cluster -n authentik When the output shows "Cluster in healthy state", you're ready to proceed:

❯ kubectl get cluster -n authentik
NAME                 AGE     INSTANCES   READY   STATUS                     PRIMARY
authentik-postgres    3m49s   2           2       Cluster in healthy state   authentik-postgres-1

Step 2: Hibernate the Cluster

Hibernate the cluster to safely replace its data:

kubectl cnpg hibernate on -n authentik authentik-postgres

Let's have a look at the annotations we need to save by getting the YAML from our authentik-postgres cluster with kubectl get -n authentik pvc authentik-postgres-1 -o yaml.

Copy the labels and annotations to a notepad for later.

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: authentik-postgres-1
  namespace: authentik
  uid: 8aedb33c-0b96-43f7-86aa-bde018fd45d4
  resourceVersion: '183891682'
  creationTimestamp: '2025-01-05T00:47:21Z'
  labels:
    cnpg.io/cluster: authentik-postgres
    cnpg.io/instanceName: authentik-postgres-1
    cnpg.io/instanceRole: primary
    cnpg.io/pvcRole: PG_DATA
    role: primary
  annotations:
    cnpg.io/clusterManifest: >-
      {"metadata":{"name":"authentik-postgres","namespace":"authentik","uid":"028d9f29-933c-4da7-88e6-07eee2071f20","resourceVersion":"183888366","generation":1,"creationTimestamp":"2025-01-05T00:47:20Z","labels":{"app.kubernetes.io/component":"database","kustomize.toolkit.fluxcd.io/name":"apps-authentik","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"managedFields":[{"manager":"kustomize-controller","operation":"Apply","apiVersion":"postgresql.cnpg.io/v1","time":"2025-01-05T00:47:20Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{"f:app.kubernetes.io/component":{},"f:kustomize.toolkit.fluxcd.io/name":{},"f:kustomize.toolkit.fluxcd.io/namespace":{}}},"f:spec":{"f:affinity":{"f:enablePodAntiAffinity":{}},"f:bootstrap":{"f:initdb":{"f:database":{},"f:owner":{},"f:secret":{"f:name":{}}}},"f:description":{},"f:imageName":{},"f:instances":{},"f:nodeMaintenanceWindow":{"f:inProgress":{},"f:reusePVC":{}},"f:postgresql":{"f:parameters":{"f:archive_timeout":{},"f:client_min_messages":{},"f:log_connections":{},"f:log_disconnections":{},"f:log_hostname":{},"f:max_connections":{}}},"f:primaryUpdateStrategy":{},"f:resources":{"f:limits":{"f:memory":{}},"f:requests":{"f:cpu":{},"f:memory":{}}},"f:startDelay":{},"f:stopDelay":{},"f:storage":{"f:size":{},"f:storageClass":{}}}}},{"manager":"manager","operation":"Update","apiVersion":"postgresql.cnpg.io/v1","time":"2025-01-05T00:48:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:availableArchitectures":{},"f:certificates":{".":{},"f:clientCASecret":{},"f:expirations":{".":{},"f:authentik-postgres-ca":{},"f:authentik-postgres-replication":{},"f:authentik-postgres-server":{}},"f:replicationTLSSecret":{},"f:serverAltDNSNames":{},"f:serverCASecret":{},"f:serverTLSSecret":{}},"f:cloudNativePGCommitHash":{},"f:cloudNativePGOperatorHash":{},"f:conditions":{},"f:configMapResourceVersion":{".":{},"f:metrics":{".":{},"f:cnpg-default-monitoring":{}}},"f:currentPrimary":{},"f:currentPrimaryTimestamp":{},"f:healthyPVC":{},"f:image":{},"f:instanceNames":{},"f:instances":{},"f:instancesReportedState":{".":{},"f:authentik-postgres-1":{".":{},"f:isPrimary":{},"f:timeLineID":{}},"f:authentik-postgres-2":{".":{},"f:isPrimary":{},"f:timeLineID":{}}},"f:instancesStatus":{".":{},"f:healthy":{}},"f:latestGeneratedNode":{},"f:managedRolesStatus":{},"f:phase":{},"f:poolerIntegrations":{".":{},"f:pgBouncerIntegration":{}},"f:pvcCount":{},"f:readService":{},"f:readyInstances":{},"f:secretsResourceVersion":{".":{},"f:applicationSecretVersion":{},"f:clientCaSecretVersion":{},"f:replicationSecretVersion":{},"f:serverCaSecretVersion":{},"f:serverSecretVersion":{}},"f:switchReplicaClusterStatus":{},"f:targetPrimary":{},"f:targetPrimaryTimestamp":{},"f:timelineID":{},"f:topology":{".":{},"f:instances":{".":{},"f:authentik-postgres-1":{},"f:authentik-postgres-2":{}},"f:nodesUsed":{},"f:successfullyExtracted":{}},"f:writeService":{}}},"subresource":"status"}]},"spec":{"description":"Recovery
      Cluster","imageName":"ghcr.io/cloudnative-pg/postgresql:16.3-7","postgresUID":26,"postgresGID":26,"instances":2,"postgresql":{"parameters":{"archive_mode":"on","archive_timeout":"1800","client_min_messages":"error","dynamic_shared_memory_type":"posix","log_connections":"false","log_destination":"csvlog","log_directory":"/controller/log","log_disconnections":"false","log_filename":"postgres","log_hostname":"false","log_rotation_age":"0","log_rotation_size":"0","log_truncate_on_rotation":"false","logging_collector":"on","max_connections":"100","max_parallel_workers":"32","max_replication_slots":"32","max_worker_processes":"32","shared_memory_type":"mmap","shared_preload_libraries":"","ssl_max_protocol_version":"TLSv1.3","ssl_min_protocol_version":"TLSv1.3","wal_keep_size":"512MB","wal_level":"logical","wal_log_hints":"on","wal_receiver_timeout":"5s","wal_sender_timeout":"5s"},"syncReplicaElectionConstraint":{"enabled":false}},"replicationSlots":{"highAvailability":{"enabled":true,"slotPrefix":"_cnpg_"},"updateInterval":30},"bootstrap":{"initdb":{"database":"authentik","owner":"authentik","secret":{"name":"authentik-postgres-user"},"encoding":"UTF8","localeCollate":"C","localeCType":"C"}},"enableSuperuserAccess":false,"storage":{"storageClass":"longhorn-cnpg","size":"1Gi","resizeInUseVolumes":true},"startDelay":300,"stopDelay":300,"smartShutdownTimeout":180,"switchoverDelay":3600,"affinity":{"enablePodAntiAffinity":true,"podAntiAffinityType":"preferred"},"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}},"primaryUpdateStrategy":"unsupervised","primaryUpdateMethod":"restart","nodeMaintenanceWindow":{"reusePVC":false},"monitoring":{"disableDefaultQueries":false,"customQueriesConfigMap":[{"name":"cnpg-default-monitoring","key":"queries"}]},"logLevel":"info"},"status":{"instances":2,"readyInstances":2,"instancesStatus":{"healthy":["authentik-postgres-1","authentik-postgres-2"]},"instancesReportedState":{"authentik-postgres-1":{"isPrimary":true,"timeLineID":1},"authentik-postgres-2":{"isPrimary":false,"timeLineID":1}},"managedRolesStatus":{},"timelineID":1,"topology":{"instances":{"authentik-postgres-1":{},"authentik-postgres-2":{}},"nodesUsed":2,"successfullyExtracted":true},"latestGeneratedNode":2,"currentPrimary":"authentik-postgres-1","targetPrimary":"authentik-postgres-1","pvcCount":2,"healthyPVC":["authentik-postgres-1","authentik-postgres-2"],"writeService":"authentik-postgres-rw","readService":"authentik-postgres-r","phase":"Cluster
      in healthy
      state","secretsResourceVersion":{"replicationSecretVersion":"183887343","applicationSecretVersion":"12113939","clientCaSecretVersion":"183887338","serverCaSecretVersion":"183887338","serverSecretVersion":"183887341"},"configMapResourceVersion":{"metrics":{"cnpg-default-monitoring":"166723476"}},"certificates":{"serverCASecret":"authentik-postgres-ca","serverTLSSecret":"authentik-postgres-server","replicationTLSSecret":"authentik-postgres-replication","clientCASecret":"authentik-postgres-ca","serverAltDNSNames":["authentik-postgres-rw","authentik-postgres-rw.authentik","authentik-postgres-rw.authentik.svc","authentik-postgres-r","authentik-postgres-r.authentik","authentik-postgres-r.authentik.svc","authentik-postgres-ro","authentik-postgres-ro.authentik","authentik-postgres-ro.authentik.svc"],"expirations":{"authentik-postgres-ca":"2025-04-05
      00:42:20 +0000 UTC","authentik-postgres-replication":"2025-04-05 00:42:20
      +0000 UTC","authentik-postgres-server":"2025-04-05 00:42:20 +0000
      UTC"}},"cloudNativePGCommitHash":"2b489ad6","currentPrimaryTimestamp":"2025-01-05T00:47:55.749766Z","targetPrimaryTimestamp":"2025-01-05T00:47:21.653581Z","poolerIntegrations":{"pgBouncerIntegration":{}},"cloudNativePGOperatorHash":"144e71b00bdcfc5edafa10055fb0cc4a6efa9f467a8e66826d5e7bb2b254b706","conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2025-01-05T00:48:41Z","reason":"ClusterIsReady","message":"Cluster
      is
      Ready"},{"type":"ContinuousArchiving","status":"True","lastTransitionTime":"2025-01-05T00:47:55Z","reason":"ContinuousArchivingSuccess","message":"Continuous
      archiving is
      working"}],"instanceNames":["authentik-postgres-1","authentik-postgres-2"]}}
    cnpg.io/hibernateClusterManifest: >-
      {"metadata":{"name":"authentik-postgres","namespace":"authentik","uid":"028d9f29-933c-4da7-88e6-07eee2071f20","resourceVersion":"183888366","generation":1,"creationTimestamp":"2025-01-05T00:47:20Z","labels":{"app.kubernetes.io/component":"database","kustomize.toolkit.fluxcd.io/name":"apps-authentik","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"managedFields":[{"manager":"kustomize-controller","operation":"Apply","apiVersion":"postgresql.cnpg.io/v1","time":"2025-01-05T00:47:20Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{"f:app.kubernetes.io/component":{},"f:kustomize.toolkit.fluxcd.io/name":{},"f:kustomize.toolkit.fluxcd.io/namespace":{}}},"f:spec":{"f:affinity":{"f:enablePodAntiAffinity":{}},"f:bootstrap":{"f:initdb":{"f:database":{},"f:owner":{},"f:secret":{"f:name":{}}}},"f:description":{},"f:imageName":{},"f:instances":{},"f:nodeMaintenanceWindow":{"f:inProgress":{},"f:reusePVC":{}},"f:postgresql":{"f:parameters":{"f:archive_timeout":{},"f:client_min_messages":{},"f:log_connections":{},"f:log_disconnections":{},"f:log_hostname":{},"f:max_connections":{}}},"f:primaryUpdateStrategy":{},"f:resources":{"f:limits":{"f:memory":{}},"f:requests":{"f:cpu":{},"f:memory":{}}},"f:startDelay":{},"f:stopDelay":{},"f:storage":{"f:size":{},"f:storageClass":{}}}}},{"manager":"manager","operation":"Update","apiVersion":"postgresql.cnpg.io/v1","time":"2025-01-05T00:48:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:availableArchitectures":{},"f:certificates":{".":{},"f:clientCASecret":{},"f:expirations":{".":{},"f:authentik-postgres-ca":{},"f:authentik-postgres-replication":{},"f:authentik-postgres-server":{}},"f:replicationTLSSecret":{},"f:serverAltDNSNames":{},"f:serverCASecret":{},"f:serverTLSSecret":{}},"f:cloudNativePGCommitHash":{},"f:cloudNativePGOperatorHash":{},"f:conditions":{},"f:configMapResourceVersion":{".":{},"f:metrics":{".":{},"f:cnpg-default-monitoring":{}}},"f:currentPrimary":{},"f:currentPrimaryTimestamp":{},"f:healthyPVC":{},"f:image":{},"f:instanceNames":{},"f:instances":{},"f:instancesReportedState":{".":{},"f:authentik-postgres-1":{".":{},"f:isPrimary":{},"f:timeLineID":{}},"f:authentik-postgres-2":{".":{},"f:isPrimary":{},"f:timeLineID":{}}},"f:instancesStatus":{".":{},"f:healthy":{}},"f:latestGeneratedNode":{},"f:managedRolesStatus":{},"f:phase":{},"f:poolerIntegrations":{".":{},"f:pgBouncerIntegration":{}},"f:pvcCount":{},"f:readService":{},"f:readyInstances":{},"f:secretsResourceVersion":{".":{},"f:applicationSecretVersion":{},"f:clientCaSecretVersion":{},"f:replicationSecretVersion":{},"f:serverCaSecretVersion":{},"f:serverSecretVersion":{}},"f:switchReplicaClusterStatus":{},"f:targetPrimary":{},"f:targetPrimaryTimestamp":{},"f:timelineID":{},"f:topology":{".":{},"f:instances":{".":{},"f:authentik-postgres-1":{},"f:authentik-postgres-2":{}},"f:nodesUsed":{},"f:successfullyExtracted":{}},"f:writeService":{}}},"subresource":"status"}]},"spec":{"description":"Recovery
      Cluster","imageName":"ghcr.io/cloudnative-pg/postgresql:16.3-7","postgresUID":26,"postgresGID":26,"instances":2,"postgresql":{"parameters":{"archive_mode":"on","archive_timeout":"1800","client_min_messages":"error","dynamic_shared_memory_type":"posix","log_connections":"false","log_destination":"csvlog","log_directory":"/controller/log","log_disconnections":"false","log_filename":"postgres","log_hostname":"false","log_rotation_age":"0","log_rotation_size":"0","log_truncate_on_rotation":"false","logging_collector":"on","max_connections":"100","max_parallel_workers":"32","max_replication_slots":"32","max_worker_processes":"32","shared_memory_type":"mmap","shared_preload_libraries":"","ssl_max_protocol_version":"TLSv1.3","ssl_min_protocol_version":"TLSv1.3","wal_keep_size":"512MB","wal_level":"logical","wal_log_hints":"on","wal_receiver_timeout":"5s","wal_sender_timeout":"5s"},"syncReplicaElectionConstraint":{"enabled":false}},"replicationSlots":{"highAvailability":{"enabled":true,"slotPrefix":"_cnpg_"},"updateInterval":30},"bootstrap":{"initdb":{"database":"authentik","owner":"authentik","secret":{"name":"authentik-postgres-user"},"encoding":"UTF8","localeCollate":"C","localeCType":"C"}},"enableSuperuserAccess":false,"storage":{"storageClass":"longhorn-cnpg","size":"1Gi","resizeInUseVolumes":true},"startDelay":300,"stopDelay":300,"smartShutdownTimeout":180,"switchoverDelay":3600,"affinity":{"enablePodAntiAffinity":true,"podAntiAffinityType":"preferred"},"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}},"primaryUpdateStrategy":"unsupervised","primaryUpdateMethod":"restart","nodeMaintenanceWindow":{"reusePVC":false},"monitoring":{"disableDefaultQueries":false,"customQueriesConfigMap":[{"name":"cnpg-default-monitoring","key":"queries"}]},"logLevel":"info"},"status":{"instances":2,"readyInstances":2,"instancesStatus":{"healthy":["authentik-postgres-1","authentik-postgres-2"]},"instancesReportedState":{"authentik-postgres-1":{"isPrimary":true,"timeLineID":1},"authentik-postgres-2":{"isPrimary":false,"timeLineID":1}},"managedRolesStatus":{},"timelineID":1,"topology":{"instances":{"authentik-postgres-1":{},"authentik-postgres-2":{}},"nodesUsed":2,"successfullyExtracted":true},"latestGeneratedNode":2,"currentPrimary":"authentik-postgres-1","targetPrimary":"authentik-postgres-1","pvcCount":2,"healthyPVC":["authentik-postgres-1","authentik-postgres-2"],"writeService":"authentik-postgres-rw","readService":"authentik-postgres-r","phase":"Cluster
      in healthy
      state","secretsResourceVersion":{"replicationSecretVersion":"183887343","applicationSecretVersion":"12113939","clientCaSecretVersion":"183887338","serverCaSecretVersion":"183887338","serverSecretVersion":"183887341"},"configMapResourceVersion":{"metrics":{"cnpg-default-monitoring":"166723476"}},"certificates":{"serverCASecret":"authentik-postgres-ca","serverTLSSecret":"authentik-postgres-server","replicationTLSSecret":"authentik-postgres-replication","clientCASecret":"authentik-postgres-ca","serverAltDNSNames":["authentik-postgres-rw","authentik-postgres-rw.authentik","authentik-postgres-rw.authentik.svc","authentik-postgres-r","authentik-postgres-r.authentik","authentik-postgres-r.authentik.svc","authentik-postgres-ro","authentik-postgres-ro.authentik","authentik-postgres-ro.authentik.svc"],"expirations":{"authentik-postgres-ca":"2025-04-05
      00:42:20 +0000 UTC","authentik-postgres-replication":"2025-04-05 00:42:20
      +0000 UTC","authentik-postgres-server":"2025-04-05 00:42:20 +0000
      UTC"}},"cloudNativePGCommitHash":"2b489ad6","currentPrimaryTimestamp":"2025-01-05T00:47:55.749766Z","targetPrimaryTimestamp":"2025-01-05T00:47:21.653581Z","poolerIntegrations":{"pgBouncerIntegration":{}},"cloudNativePGOperatorHash":"144e71b00bdcfc5edafa10055fb0cc4a6efa9f467a8e66826d5e7bb2b254b706","conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2025-01-05T00:48:41Z","reason":"ClusterIsReady","message":"Cluster
      is
      Ready"},{"type":"ContinuousArchiving","status":"True","lastTransitionTime":"2025-01-05T00:47:55Z","reason":"ContinuousArchivingSuccess","message":"Continuous
      archiving is
      working"}],"instanceNames":["authentik-postgres-1","authentik-postgres-2"]}}
    cnpg.io/hibernatePgControlData: >
      pg_control version number:            1300

      Catalog version number:               202307071

      Database system identifier:           7456226693945323540

      Database cluster state:               shut down

      pg_control last modified:             Sun 05 Jan 2025 12:55:46 AM UTC

      Latest checkpoint location:           0/5000028

      Latest checkpoint's REDO location:    0/5000028

      Latest checkpoint's REDO WAL file:    000000010000000000000005

      Latest checkpoint's TimeLineID:       1

      Latest checkpoint's PrevTimeLineID:   1

      Latest checkpoint's full_page_writes: on

      Latest checkpoint's NextXID:          0:745

      Latest checkpoint's NextOID:          16391

      Latest checkpoint's NextMultiXactId:  1

      Latest checkpoint's NextMultiOffset:  0

      Latest checkpoint's oldestXID:        722

      Latest checkpoint's oldestXID's DB:   1

      Latest checkpoint's oldestActiveXID:  0

      Latest checkpoint's oldestMultiXid:   1

      Latest checkpoint's oldestMulti's DB: 1

      Latest checkpoint's oldestCommitTsXid:0

      Latest checkpoint's newestCommitTsXid:0

      Time of latest checkpoint:            Sun 05 Jan 2025 12:55:46 AM UTC

      Fake LSN counter for unlogged rels:   0/3E8

      Minimum authentik ending location:     0/0

      Min authentik ending loc's timeline:   0

      Backup start location:                0/0

      Backup end location:                  0/0

      End-of-backup record required:        no

      wal_level setting:                    logical

      wal_log_hints setting:                on

      max_connections setting:              100

      max_worker_processes setting:         32

      max_wal_senders setting:              10

      max_prepared_xacts setting:           0

      max_locks_per_xact setting:           64

      track_commit_timestamp setting:       off

      Maximum data alignment:               8

      Database block size:                  8192

      Blocks per segment of large relation: 131072

      WAL block size:                       8192

      Bytes per WAL segment:                16777216

      Maximum length of identifiers:        64

      Maximum columns in an index:          32

      Maximum size of a TOAST chunk:        1996

      Size of a large-object chunk:         2048

      Date/time type storage:               64-bit integers

      Float8 argument passing:              by value

      Data page checksum version:           0

      Mock authentication nonce:           
      b71a23a4ebf4c034c67a9a357fd68bdc07f2c1337cc900204eac4abe1cbccb1b
    cnpg.io/nodeSerial: '1'
    cnpg.io/operatorVersion: 1.23.3
    cnpg.io/pgControldata: >
      pg_control version number:            1300

      Catalog version number:               202307071

      Database system identifier:           7456226693945323540

      Database cluster state:               shut down

      pg_control last modified:             Sun 05 Jan 2025 12:55:46 AM UTC

      Latest checkpoint location:           0/5000028

      Latest checkpoint's REDO location:    0/5000028

      Latest checkpoint's REDO WAL file:    000000010000000000000005

      Latest checkpoint's TimeLineID:       1

      Latest checkpoint's PrevTimeLineID:   1

      Latest checkpoint's full_page_writes: on

      Latest checkpoint's NextXID:          0:745

      Latest checkpoint's NextOID:          16391

      Latest checkpoint's NextMultiXactId:  1

      Latest checkpoint's NextMultiOffset:  0

      Latest checkpoint's oldestXID:        722

      Latest checkpoint's oldestXID's DB:   1

      Latest checkpoint's oldestActiveXID:  0

      Latest checkpoint's oldestMultiXid:   1

      Latest checkpoint's oldestMulti's DB: 1

      Latest checkpoint's oldestCommitTsXid:0

      Latest checkpoint's newestCommitTsXid:0

      Time of latest checkpoint:            Sun 05 Jan 2025 12:55:46 AM UTC

      Fake LSN counter for unlogged rels:   0/3E8

      Minimum authentik ending location:     0/0

      Min authentik ending loc's timeline:   0

      Backup start location:                0/0

      Backup end location:                  0/0

      End-of-backup record required:        no

      wal_level setting:                    logical

      wal_log_hints setting:                on

      max_connections setting:              100

      max_worker_processes setting:         32

      max_wal_senders setting:              10

      max_prepared_xacts setting:           0

      max_locks_per_xact setting:           64

      track_commit_timestamp setting:       off

      Maximum data alignment:               8

      Database block size:                  8192

      Blocks per segment of large relation: 131072

      WAL block size:                       8192

      Bytes per WAL segment:                16777216

      Maximum length of identifiers:        64

      Maximum columns in an index:          32

      Maximum size of a TOAST chunk:        1996

      Size of a large-object chunk:         2048

      Date/time type storage:               64-bit integers

      Float8 argument passing:              by value

      Data page checksum version:           0

      Mock authentication nonce:           
      b71a23a4ebf4c034c67a9a357fd68bdc07f2c1337cc900204eac4abe1cbccb1b
    cnpg.io/pvcStatus: detached
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
    volume.kubernetes.io/storage-provisioner: driver.longhorn.io
  finalizers:
    - kubernetes.io/pvc-protection
  managedFields:
    - manager: k3s
      operation: Update
      apiVersion: v1
      time: '2025-01-05T00:47:24Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:pv.kubernetes.io/bind-completed: {}
            f:pv.kubernetes.io/bound-by-controller: {}
            f:volume.beta.kubernetes.io/storage-provisioner: {}
            f:volume.kubernetes.io/storage-provisioner: {}
        f:spec:
          f:volumeName: {}
    - manager: k3s
      operation: Update
      apiVersion: v1
      time: '2025-01-05T00:47:24Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:accessModes: {}
          f:capacity:
            .: {}
            f:storage: {}
          f:phase: {}
      subresource: status
    - manager: manager
      operation: Update
      apiVersion: v1
      time: '2025-01-05T00:47:56Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:cnpg.io/nodeSerial: {}
            f:cnpg.io/operatorVersion: {}
          f:labels:
            .: {}
            f:cnpg.io/cluster: {}
            f:cnpg.io/instanceName: {}
            f:cnpg.io/instanceRole: {}
            f:cnpg.io/pvcRole: {}
            f:role: {}
        f:spec:
          f:accessModes: {}
          f:resources:
            f:requests:
              .: {}
              f:storage: {}
          f:storageClassName: {}
          f:volumeMode: {}
    - manager: kubectl-cnpg
      operation: Update
      apiVersion: v1
      time: '2025-01-05T00:55:57Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:cnpg.io/clusterManifest: {}
            f:cnpg.io/hibernateClusterManifest: {}
            f:cnpg.io/hibernatePgControlData: {}
            f:cnpg.io/pgControldata: {}
            f:cnpg.io/pvcStatus: {}
  selfLink: /api/v1/namespaces/authentik/persistentvolumeclaims/authentik-postgres-1
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-8aedb33c-0b96-43f7-86aa-bde018fd45d4
  storageClassName: longhorn-cnpg
  volumeMode: Filesystem

Step 3: Replace the PVC

  1. Scale down the CNPG operator: kubectl scale --replicas 0 -n cnpg-system deployment cnpg-cloudnative-pg
  2. Delete the existing PVC.
  3. In the Longhorn UI, locate the backup of your old PVC and restore it. Ensure the new PVC has the same name and namespace as the deleted one.
  4. Add the saved labels and annotations to the restored PVC's manifest.
  5. Scale the CNPG operator back up: kubectl scale --replicas 1 -n cnpg-system deployment cnpg-cloudnative-pg
  6. Wake up the cluster: kubectl cnpg hibernate off -n authentik authentik-postgres

Step 4: Verify and Extract your Data

Once the cluster is awake, CNPG will initialize it with the restored PVC. You can now scale your workload back up.

Final Notes: Backup Your Data

While this process can get your PostgreSQL database up and running, I strongly recommend extracting your data and creating a fresh cluster with a proper backup setup. CNPG's preferred approach involves backing up to an object store, which allows for reliable recovery even in a fresh cluster.

Remember: Always test your backups before relying on them. This will save you from headaches down the road.

👋 Found this post helpful? Have suggestions or just want to chat about it? Throw me a message.

Hacking a Cheap Smart Plug

October 17, 2024