Back up your database in CI before every migration

Google’s SRE team found that roughly 70% of outages are due to changes in a live system. A deploy that touches your database is exactly that kind of change: a migration that drops a column, a data fix that updates more rows than it should, a one-off script that runs against the wrong server. When it goes wrong, the backup you reach for is often hours old, and it has to actually restore.

That last part is not a given. In January 2017, a deletion meant for a GitLab.com replica ran on the primary database. The nightly pg_dump backups had been failing without anyone noticing (“The S3 bucket was empty”), restores had never been tested, and about six hours of production data was lost. GitLab’s public postmortem is worth reading for anyone who runs a database.

This guide puts two steps into your pipeline. The deploy job takes a fresh backup and waits for it before the migration runs. A scheduled job restore-tests the newest backup and fails loudly when it doesn’t come back. Both run with the Norcube CLI and a CI token that can start backups and restore tests, and nothing else.

Why back up in the pipeline

The restore point is minutes old. If the migration goes wrong, you lose the minutes since the deploy started, not everything since last night’s run.

The deploy waits for it. nrc backup jobs run --wait exits only when the backup has finished. If the backup fails, the job fails and the migration never starts. You never migrate without a restore point.

“Backup succeeded” is not “it restores”. A green backup job proves that the dump ran. A restore test proves that it comes back: Norcube restores the backup into a throwaway database, checks the data against what the backup recorded (user tables for PostgreSQL, collections and documents for MongoDB) and deletes the copy. Run that on a schedule and a broken backup shows up as a failed pipeline, not as a surprise during an incident.

Who this is for

• Teams that ship schema migrations from CI. Rails, Django, Laravel, Prisma, Flyway, golang-migrate: if a merge to main changes the database, this is for you.
• Small teams without a DBA. Nobody has time to check backups by hand. A pipeline that fails when a backup doesn’t restore does it for you.
• Anyone who has to show tested backups. ISO 27001’s backup control (Annex A 8.13) asks for backup copies that are regularly tested, and customer security questionnaires often ask how you test restores. A scheduled restore test with a history of results is a straight answer.

You don’t need this if your database never changes shape during deploys and a day-old restore point is fine. Scheduled backups with automated restore tests in the dashboard already cover that.

How it works

Two jobs, each with one job to do:

1. In the deploy pipeline, before migrations: nrc backup jobs run backs up the database now and waits. A failure stops the deploy.
2. On a schedule, nightly or weekly: nrc backup restore-tests run restore-tests the newest successful backup and waits for the verdict. It exits 1 unless the test passed, so the scheduled pipeline fails and your CI notifies the team.

You can also put the restore test in the deploy pipeline, right after the backup, before especially risky migrations. It proves the very backup you’re about to rely on, at the cost of a few extra minutes per deploy.

Before you start

1. A data source with a policy. Connect the database in Norcube Backup and attach an enabled backup policy to it. A backup started from CI uses the attached policy (the highest-priority enabled one, or the one you name with --policy), so without one the command stops with an error.

2. A CI token. In the dashboard, open IAM › CI tokens and create a token with the scopes backup.write (start backups) and backup.verify (run restore tests). Or from your terminal:

nrc auth ci-token create --name deploy-backups \
  --scopes backup.write,backup.verify --expires 90d -o text

With -o text it prints only the token, so you can pipe it straight into your CI’s secret store, for example | gh secret set NORCUBE_TOKEN on GitHub. A CI token belongs to the organization, not to a person, always expires (90 days by default, up to a year) and can be revoked at any time.

3. The data source id. CI commands take an id, not a name. List your data sources and copy the value from the ID column:

nrc backup datasources list --search prod

4. Two values in your CI settings:

The CLI reads NORCUBE_TOKEN on its own. There’s no --org to set: the token already knows its organization.

Back up before you migrate

Pick your CI. The backup step sits before your migration and deploy steps, which stay whatever they are today (make migrate and make deploy below are placeholders).

name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install the Norcube CLI
        uses: norcubeplatform/setup-norcube@v1
        with:
          version: v0.21.4
          token: ${{ secrets.NORCUBE_TOKEN }}

      - name: Back up the database before migrating
        run: nrc backup jobs run -d "$NORCUBE_DATASOURCE_ID" --wait --wait-timeout 30m
        env:
          NORCUBE_DATASOURCE_ID: ${{ vars.NORCUBE_DATASOURCE_ID }}

      - name: Run migrations
        run: make migrate

      - name: Deploy
        run: make deploy

stages:
  - backup
  - migrate
  - deploy

.norcube:
  image: debian:bookworm-slim
  variables:
    NORCUBE_NO_UPDATE_CHECK: '1'
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
    - curl -fsSL https://github.com/norcubeplatform/cli/raw/main/install.sh | VERSION=v0.21.4 sh
    - export PATH="$HOME/.norcube/bin:$PATH"

backup:
  extends: .norcube
  stage: backup
  script:
    - nrc backup jobs run -d "$NORCUBE_DATASOURCE_ID" --wait --wait-timeout 30m
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"

migrate:
  stage: migrate
  script:
    - make migrate
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"

deploy:
  stage: deploy
  script:
    - make deploy
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"

image: debian:bookworm-slim

pipelines:
  branches:
    main:
      - step:
          name: Back up the database before migrating
          script:
            - apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
            - curl -fsSL https://github.com/norcubeplatform/cli/raw/main/install.sh | VERSION=v0.21.4 sh
            - export PATH="$HOME/.norcube/bin:$PATH"
            - export NORCUBE_NO_UPDATE_CHECK=1
            - nrc backup jobs run -d "$NORCUBE_DATASOURCE_ID" --wait --wait-timeout 30m
      - step:
          name: Migrate and deploy
          script:
            - make migrate
            - make deploy

Where the values go. GitHub: NORCUBE_TOKEN under Settings › Secrets and variables › Actions › Secrets, NORCUBE_DATASOURCE_ID under Variables. GitLab: both under Settings › CI/CD › Variables, the token masked and protected. Bitbucket: both under Repository settings › Repository variables, the token marked Secured.

What the install does. On GitHub, setup-norcube downloads the CLI, checks it against the release checksums, puts it on the PATH and exports the token. Elsewhere, the install script does the same download and checksum check, and VERSION pins the release so a new CLI version never changes your pipeline by surprise.

Prove it restores on a schedule

A backup you’ve never restored is a hope, not a plan. Run a restore test every night (or every week, for big databases) in a pipeline of its own. It picks the newest successful backup, restores it into a throwaway database, checks the data and reports the verdict. Your production database is never touched.

name: Restore test

on:
  schedule:
    - cron: '0 3 * * *' # every night at 03:00 UTC
  workflow_dispatch:

jobs:
  restore-test:
    runs-on: ubuntu-latest
    steps:
      - name: Install the Norcube CLI
        uses: norcubeplatform/setup-norcube@v1
        with:
          version: v0.21.4
          token: ${{ secrets.NORCUBE_TOKEN }}

      - name: Prove the newest backup restores
        run: nrc backup restore-tests run -d "$NORCUBE_DATASOURCE_ID" --wait --wait-timeout 1h
        env:
          NORCUBE_DATASOURCE_ID: ${{ vars.NORCUBE_DATASOURCE_ID }}

# Add this job to the .gitlab-ci.yml above, then create the schedule under
# Build › Pipeline schedules (for example 0 3 * * * on the default branch).
restore-test:
  extends: .norcube
  stage: backup
  script:
    - nrc backup restore-tests run -d "$NORCUBE_DATASOURCE_ID" --wait --wait-timeout 1h
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

# Add the custom pipeline next to "branches:" in bitbucket-pipelines.yml, then
# schedule it on the repository's Pipelines page under Schedules: pick the
# branch and the custom pipeline restore-test.
pipelines:
  custom:
    restore-test:
      - step:
          name: Prove the newest backup restores
          script:
            - apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
            - curl -fsSL https://github.com/norcubeplatform/cli/raw/main/install.sh | VERSION=v0.21.4 sh
            - export PATH="$HOME/.norcube/bin:$PATH"
            - export NORCUBE_NO_UPDATE_CHECK=1
            - nrc backup restore-tests run -d "$NORCUBE_DATASOURCE_ID" --wait --wait-timeout 1h

When the test fails, the pipeline fails, and your CI tells the team the way it tells you about any failed build. Norcube also emails your organization about failed restore tests on its own, and the result shows up in the dashboard’s restore-health score, so you have a history to show an auditor.

What the exit codes mean

The CLI’s exit code says what went wrong, so a failed step is easy to act on:

With -o json, errors come as JSON on stderr with a code and a message, if you want your own handling.

Good to know

A CI token can’t touch your data. It can start backups and restore tests and read their status. It can’t download a backup, run a real restore, read credentials or change policies: those stay with people in the dashboard and the CLI. A leaked deploy token can’t be used to copy your database.

Pin the versions. setup-norcube@v1 and VERSION=v0.21.4 keep the pipeline the same until you choose to upgrade.

A timeout doesn’t cancel the job. When --wait-timeout runs out, the CLI stops waiting and exits 7, but the backup or restore test carries on and finishes on the server. Size the timeout to your database: the default is 2 hours.

nrc backup health is a report, not a gate. It shows the restore-test score of every data source and always exits 0. Use restore-tests run --wait when you want the pipeline to fail.

Rotate before the token expires. Tokens always expire. Create the new one, update the secret, then revoke the old one with nrc auth ci-token revoke <name>.

Costs. A backup started from CI is an ordinary backup and counts as usage on your Backup plan, like one you start from the dashboard. Restore tests are included on every plan.

Try it

You need a free Norcube account, the Norcube CLI and five minutes. Connect a database, attach a policy, create a token with backup.write and backup.verify, and paste the workflow for your CI. Everything the CLI can do for Backup is on the Backup CLI page, and the reference lives in the docs: Backup from the CLI and CI tokens.