Domino V12 Veeam Backup and Replication Integration
Introduction
The Domino Backup integration for Veeam Backup and Replication provides integrated end to end backup and restore experiences.
Veeam Backup and Replication uses application aware processing integrating with the new Domino V12 Backup leveraging pre-freeze and post-thaw scripts. This Domino V12 Backup integration provides full Domino application support bringing all databases into an consistent state (Domino backup API) in combination with Veeam snapshot technology.
-
Backup operations are scheduled and initiated by Veeam Backup and Replication.
-
Domino restore operations are initiated by a Domino administrator leveraging the feature rich Domino based restore UI in the Domino Backup database
dominobackup.nsf
.
This integration is supported on Windows64 and Linux64.
High Level Architecture
Domino Backup
Backups are initiated by Veeam Backup and Replication leveraging pre-freeze
and post-thaw
scripts to allow application aware processing. Those scripts are executed on the Domino server before and after a Veeam snapshot and run Domino scripts installed on the Domino server to allow integrated Domino Backup operations.
-
The
pre-freeze
script triggers a Domino Backup snapshot operation to bring all Domino databases into backup mode. -
The
post-thaw
script triggers Domino Backup to bring all databases back from backup mode into normal operation mode and backup delta data accumulated while databases have been in backup mode.
This integration allows full and incremental backups with Veeam Backup and Replication. The inventory of backups is listed in the Veeam repository (“restore points
”) and also in the Domino backup database (dominobackup.nsf
). The restore operation is always initiated by the Domino administrator in the Domino native UI for restore leveraging the database inventory on the Domino side.
Domino Restore
Restore operations are fully integrated and initiated in the Domino Backup dominobackup.nsf
database. A Domino admin creates a restore request for databases.
Request are performed by the restore server task on the Domino server The restore task leverages scripts (batch/shell script) to
- Find the right restore point
- Mount a restore point to the Domino server
- Copy databases (nsf/ntf/box) and backup delta (*.DELTA files) from a Veeam backup into the Domino data directory.
The Domino restore server task takes care of bringing the database on-line and performs post processing like disable replication, change replia-id, restore documents and folders into the original database.
Backup Retention
In this scenario Veeam ensures the retention of backup data as defined in the backup job and Domino maintains the inventory, log database retention and delta file retention via “prune” operations.
Mount and Restore Operation Implementation
The mount operation requires direct communication with the Veeam Backup and Restore server and leverages the Veeam PowerShell command line interface to control the Veeam Backup and Replication server.
-
A Powershell script on the Veeam server identifies and mounts the right restore point to perform mount operations on the Domino server to allow Domino to restore databases.
-
The mount request is initiated by a Domino restore script triggered by the restore server task on the Domino server requesting the restore.
-
The mount operation on the traget Domino machine uses Admin credentials already defined for application aware scripts (
pre-freeze
&post-thaw
). -
The restore script on the Domino server machine finds the desired database in the mounted restore point and copies the database to the requested restore location
-
And finally, Domino sends an unmount request for the restore mount to the Veeam server.
Windows OpenSSH Server Configuration
The communication between the Domino servers and the Veeam Backup and Replication server leverages the Secure Shell protocol (ssh) authenticated via private/public key security. User/Password authentication is disabled by design.
This integration provides a safe and reliable cross platform communication channel and doesn’t require servers to belong to the same Windows domain or PowerShell remote execution. All Domino servers can use the same SSH key or individual keys for authentication.
Public/Private Key Authentication
This integration leverages the Microsoft OpenSSH with a restricted configuration only allowing restore mount and unmount operations for each Domino server.
A SSH key is configured via .ssh/authorized_keys
for a Windows user account with Veeam restore operator permissions.
The command execution is restricted to a single mount/unmount PowerShell script via authorized_keys
configuration to allow tight control of requested operations.
OpenSSH Server Configuration
Install Microsoft OpenSSL Server with the following type of configuration
The open OpenSSL server has been tested with Windows 10 and Windows 2019 Server on the Veeam Backup and Replication server. The installation instruction contains step by step instructions including references to the original Microsoft documentation for the OpenSSH server.
Configurating User and Access
A user with with Veeam restore administrator permissions is required to invoke the PowerShell operations.
The PowerShell script can distinguish between different servers by requesting IP address (environment variable SSH_CONNECTION
) and will only map the configured VM to the requesting host.
In case separate keys are required by platform or server, multiple entries can be added to the authorized_key file as shown in the example below. The command request is passed to the PowerShell script via SSH_ORIGINAL_COMMAND
environment variable.
Example for a user “domino”
C:\Users\domino\.ssh\authorized_keys
command="powershell.exe c:/domino/veeam/DominoRestore.ps1" ssh-ed25519 AAAAC3NzaC1lZD...
command="powershell.exe c:/domino/veeam/DominoRestore.ps1" ssh-rsa AAAAB3NzaC1yc2EAAA...
Note:
OpenSSH requires strict permission on the authorizied_keys. Make sure the file is only readable by admins and the user
Configuration File
An agent-less backup leveraging backup via a virtualization back-end like VMware
or Hyper-V
does not provide a direct mapping between the guest operation systems and their IP addresses. The restore operation triggered from the Domino server only provides the IP address of the requesting server and the requested restore time.
Therefore for mapping and verifying restore requests this integration leverages a central configuration file in JSON format on the Backup and Replication server.
The configuration defines which Domino server can perform mount/unmount operations (in addition to the SSH public/private key authentication) and also ensures each server can only restore from it’s own restore points.
For access control and mapping of Domino servers to the corresponding VMs a configuration file is used.
The file is located by default in the following location and is read by the PowerShell script.
c:/dominobackup/dominobackup.cfg
The configuration contains the following information:
- IP address
- Veeam admin credential description to find the right credential for mounting
-
Operating system (Linux windows) - Name of the operating system VM/host (the name used by Veeam to identify the virtual machine)
Example: dominobackup.cfg
[
{
"VmHost" : "Domino01-Linux",
"IpAddress" : "192.168.96.236",
"AccountName" : "Domin-root",
"OS" : "Linux"
},
{
"VmHost" : "Domino02-Win2019",
"IpAddress" : "192.168.96.220",
"AccountName" : "Domino-WinAdmin",
"OS" : "Windows"
}
]
Pre-Freeze/Post Thaw Scripts for Windows and Linux
A Veeam backup application copies and executes the pre-freeze
and post-thaw
to a temporary location on the guest machine. Those scripts are configured to call the corresponding Domino Backup integration scripts by the Veeam server.
Add following scripts to the backup job configuration. The scripts point to the default script location on the Domino server. To custom install directories adjust the scripts accordingly.
Windows
c:/scripts/domino/windows/pre-freeze.sh
c:/scripts/domino/windows/post-thaw.sh
Linux
c:/scripts/domino/linux/pre-freeze.sh
c:/scripts/domino/linux/post-thaw.sh
Domino Server Backup Configuration
Domino Backup integrates with Veeam for backup and restore operation leveraging batch (Windows) and shell scripts (Linux).
The scripts should be copied to the following directories. Ensure the files are executable by the Domino user (usually: notes) and root.
-
Windows:
c:/Program Files/HCL/Domino/backup/veeam
-
Linux:
/opt/hcl/domino/backup/veeam
The DXL configuration file provided in this repository contains the corresponding configuration and can be imported directly into the dominobackup.nsf
database.
For custom installation directories adjust the script directory in the configuraiton accordingly.
The following scripts are used for integration:
-
backup_domino_snapshot.cmd/sh
Veeam snapshot script invoked by the pre-freeze script to start Domino backup in snapshot mode
(brings databases into consistent/freeze state before a Veeam snapshot is started) -
backup_snapshot_start.cmd/sh
Helper script started by Domino Backup to indicate databases are in consistent state
(communicates the status back tobackup_domino_snapshot.cmd/sh
) -
backup_domino_snapshot_done.cmd/sh
Veeam snapshot script invoked by thepost-thaw
script to signal the snapshot has been created -
backup_snapshot.cmd/sh
Helper script started by Domino Backup to capture post snapshot operations.
The script waits until Veeam has performed a snapshot and communicated back viapost-thwa
script to Domino. -
backup_post.cmd/sh
Script executed when the backup is finished on the Domino server side to allow post processing.
Future integration point for scheduling backup of delta files created during backup. -
prune_backup.cmd/sh
Script to prune backup delta files and logs -
restore_db.cmd/sh
Restore script to mount Veem backups, find/copy databases and delta files.
Invokes a restore mount/unmount request via SSH connection to the Veeam Backup and Replication server. -
backup_translog.cmd/sh
Script to backup a translog extend – Not implemented yet -
prune_translog.cmd/sh
Script to prune transaction log files and logs – Not implemented yet -
restore_translog.cmd/sh
Restore script for translog extends – Not implemented yet
Special Consideration and Settings
The BackupStartDT
is always stored in UTC time and will be converted by the PowerShell script.
Domino and Veeam times need to be in sync. To ensure poper restore operations the clock skew time parameter RestoreClockSkewMinutes
can be set in the PowerShell script.
Technical Background
Sequence of Backup Operations
A Veeam backup job is scheduled for backup operations.
The integration into Domino Backup is implemented via pre-freeze
and post-thaw
scripts as described below.
-
Veeam Backup Job –>
pre-freeze.cmd
–>backup_domino_snapshot.cmd/sh
–>load backup -s
(Domino Backup) -
Domino Backup –>
backup_snapshot_start.cmd/sh
–> Brings all databases into backup mode and writes status file to confirm Domino is in snapshot backup mode -
Domino Backup –>
backup_snapshot.cmd/sh
–> Waits until snapshot status is confirmed -
Veeam Backup Job “Snapshot Created” –>
post-thaw.cmd
–>backup_domino_snapshot_done.cmd/sh
–> sets snapshot status to “DONE” to terminate thebackup_snapshot.cmd/sh
and return control to Domino Backup -
Domino Backup –> Gets all databases back from backup mode and stores delta files if needed –>
backup_post.cmd/sh
(currently no special operations)
Sequence of Restore Operations
Domino databases can be restored directly from the Domino Backup database (dominobackup.nsf). The integration used, leverages the Veeam Powershell integration on the Veeam Backup and replication server.
The restore operation leverages SSH with public/private key authentication with the Veeam server.
-
Domino Restore server task (restore) starts
restore_db.cmd/sh
on the Domino server. -
The
restore_db.cmd/sh
invokes a restore operation requesting a backup for a defined backup time (BackupStartDT
) -
The script sends a restore mount request with the desired backup date via
BackupStartDT
variable via SSH to the Veeam Backup and Replication server. -
On the Veeam server a PowerShell script is started to find the matching restore point and mounts it to the Domino server
-
restore_db.cmd/sh
leverages the mount and searches for the right database to restore -
The restore operation looks for a backup timestamp tag file (e.g.
/local/notesdata/dominobackup_20210514112233.tag
) added by Domino Backup to the Notes data directory before the snapshot was taken to identify matching backup location and to ensure the right backup was mounted. -
Once located the script copies the files to the target location (usually the restore directory with a .DAD extension)
-
Finally the restore script sends an unmount request for the restore point to the Veeam server