DbBackup (Oracle - Berkeley DB Java Edition API)
Date: 2014-12-16

com.sleepycat.je.util

## Class DbBackup

- java.lang.Object
- - com.sleepycat.je.util.DbBackup

-     public class DbBackup
      extends Object

  DbBackup is a helper class for stopping and restarting JE background activity in an open environment in order to simplify backup operations. It also lets the application create a backup which can support restoring the environment to a specific point in time.

  Backing up without DbBackup

  Because JE has an append only log file architecture, it is always possible to do a hot backup without the use of DbBackup by copying all log files (.jdb files) to your archival location. As long as the log files are copied in alphabetical order, (numerical in effect) and all log files are copied, the environment can be successfully backed up without any need to stop database operations or background activity. This means that your backup operation must do a loop to check for the creation of new log files before deciding that the backup is finished. For example:

       time    files in                    activity
               environment

        t0     000000001.jdb     Backup starts copying file 1
               000000003.jdb
               000000004.jdb

        t1     000000001.jdb     JE log cleaner migrates portion of file 3 to newly
               000000004.jdb     created file 5 and deletes file 3. Backup finishes
               000000005.jdb     file 1, starts copying file 4. Backup MUST include
                                 file 5 for a consistent backup!

        t2     000000001.jdb     Backup finishes copying file 4, starts and
               000000004.jdb     finishes file 5, has caught up. Backup ends.
               000000005.jdb

  In the example above, the backup operation must be sure to copy file 5, which came into existence after the backup had started. If the backup stopped operations at file 4, the backup set would include only file 1 and 4, omitting file 3, which would be an inconsistent set.

  Also note that log file 5 may not have filled up before it was copied to archival storage. On the next backup, there might be a newer, larger version of file 5, and that newer version should replace the older file 5 in archive storage.

  Backing up with DbBackup

  DbBackup helps simplify application backup by defining the set of files that must be copied for each backup operation. If the environment directory has read/write protection, the application must pass DbBackup an open, read/write environment handle.

  When entering backup mode, JE determines the set of log files needed for a consistent backup, and freezes all changes to those files. The application can copy that defined set of files and finish operation without checking for the ongoing creation of new files. Also, there will be no need to check for a newer version of the last file on the next backup.

  In the example above, if DbBackup was used at t0, the application would only have to copy files 1, 3 and 4 to back up. On a subsequent backup, the application could start its copying at file 5. There would be no need to check for a newer version of file 4.

  When it is important to minimize the time that it takes to recover using a backup, a checkpoint should be performed immediately before calling `startBackup()`. This will reduce recovery time when opening the environment with the restored log files. A checkpoint is performed explicitly by calling `Environment.checkpoint(com.sleepycat.je.CheckpointConfig)` using a config object for which `setForce(true)` has been called.

  Performing simple/full backups

  The following examples shows how to perform a full backup. A checkpoint is performed to minimize recovery time.

       void myBackup(Environment env, File destDir) {
           DbBackup backupHelper = new DbBackup(env);

           // Optional: Do a checkpoint to reduce recovery time after a restore.
           env.checkpoint(new CheckpointConfig().setForce(true));

           // Start backup, find out what needs to be copied.
           backupHelper.startBackup();
           try {
               // Copy the necessary files to archival storage.
               String[] filesToCopy = backupHelper.getLogFilesInBackupSet();
               myCopyFiles(env, filesToCopy, destDir);
           } finally {
               // Remember to exit backup mode, or the JE cleaner cannot delete
               // log files and disk usage will grow without bounds.
              backupHelper.endBackup();
           }
       }

       void myCopyFiles(Environment env, String[] filesToCopy, File destDir) {
           // See LogVerificationInputStream
       }

  When copying files to the backup directory, it is critical that each file is verified before or during the copy. If a file is copied that is corrupt (due to an earlier disk failure that went unnoticed, for example), the backup will be invalid and provide a false sense of security.

  The `example here` shows how to implement the `myCopyFiles` method using `LogVerificationInputStream`. A filter input stream is used to verify the file efficiently as it is being read. If you choose to use a script for copying files, the `DbVerifyLog` command line tool can be used instead.

  Assuming that the full backup copied files into an empty directory, to restore you can simply copy these files back into another empty directory.

  Always start with an empty directory as the destination for a full backup or a restore, to ensure that no unused files are present. Unused files -- perhaps the residual of an earlier environment or an earlier backup -- will take up space, and they will never be deleted by the JE log cleaner. Also note that such files will not be used by JE for calculating utilization and will not appear in the `DbSpace` output.

  Performing incremental backups

  Incremental backups are used to reduce the number of files copied during each backup. Compared to a full backup, there are two additional pieces of information needed for an incremental backup: the number of the last file in the previous backup, and a list of the files in the environment directory at the time of the current backup, i.e., the current snapshot. Their purpose is explained below.

  The number of the last file in the previous backup is used to avoid copying files that are already present in the backup set. This file number must be obtained before beginning the backup, either by checking the backup archive, or getting this value from a stored location. For example, the last file number could be written to a special file in the backup set at the time of a backup, and then read from the special file before starting the next backup.

  The list of files in the current snapshot, which should be obtained by calling `getLogFilesInSnapshot()` (after calling `startBackup()`), is used to avoid unused files after a restore, and may also be used to reduce the size of the backup set. How to use this list is described below.

  Some applications need the ability to restore to the point in time of any of the incremental backups that were made in the past, and other applications only need to restore to the point in time of the most recent backup. Accordingly, the list of current files (that is made at the time of the backup), should be used in one of two ways.

  1. If you only need to restore to the point in time of the most recent backup, then the list should be used to delete unused files from the backup set. After copying all files during the backup, any file that is not present in the list may then be deleted from the backup set. This both reduces the size of the backup set, and ensures that unused files will not be present in the backup set and therefore will not be restored.
  2. If you need to keep all log files from each backup so you can restore to more than one point in time, then the list for each backup should be saved with the backup file set so it can be used during a restore. During the restore, only the files in the list should be copied, starting with an empty destination directory. This ensures that unused files will not be restored.

  The following two examples shows how to perform an incremental backup. In the first example, the list of current files is used to delete files from the backup set that are no longer needed.

       void myBackup(Environment env, File destDir) {

           // Get the file number of the last file in the previous backup.
           long lastFileInPrevBackup =  ...

           DbBackup backupHelper = new DbBackup(env, lastFileInPrevBackup);

           // Optional: Do a checkpoint to reduce recovery time after a restore.
           env.checkpoint(new CheckpointConfig().setForce(true));

           // Start backup, find out what needs to be copied.
           backupHelper.startBackup();
           try {
               // Copy the necessary files to archival storage.
               String[] filesToCopy = backupHelper.getLogFilesInBackupSet();
               myCopyFiles(env, filesToCopy, destDir);

               // Delete files that are no longer needed.
               // WARNING: This should only be done after copying all new files.
               String[] filesInSnapshot = backupHelper.getLogFilesInSnapshot();
               myDeleteUnusedFiles(destDir, filesInSnapshot);

               // Update knowledge of last file saved in the backup set.
               lastFileInPrevBackup = backupHelper.getLastFileInBackupSet();
               // Save lastFileInPrevBackup persistently here ...
           } finally {
               // Remember to exit backup mode, or the JE cleaner cannot delete
               // log files and disk usage will grow without bounds.
              backupHelper.endBackup();
           }
       }

       void myDeleteUnusedFiles(File destDir, String[] filesInSnapshot) {
           // For each file in destDir that is NOT in filesInSnapshot, it should
           // be deleted from destDir to save disk space in the backup set, and to
           // ensure that unused files will not be restored.
       }

       void myCopyFiles(Environment env, String[] filesToCopy, File destDir) {
           // See LogVerificationInputStream
       }

  When performing backups as shown in the first example above, to restore you can simply copy all files from the backup set into an empty directory.

  In the second example below, the list of current files is saved with the backup set so it can be used during a restore. The backup set will effectively hold multiple backups that can be used to restore to different points in time.

       void myBackup(Environment env, File destDir) {

           // Get the file number of the last file in the previous backup.
           long lastFileInPrevBackup =  ...

           DbBackup backupHelper = new DbBackup(env, lastFileInPrevBackup);

           // Optional: Do a checkpoint to reduce recovery time after a restore.
           env.checkpoint(new CheckpointConfig().setForce(true));

           // Start backup, find out what needs to be copied.
           backupHelper.startBackup();
           try {
               // Copy the necessary files to archival storage.
               String[] filesToCopy = backupHelper.getLogFilesInBackupSet();
               myCopyFiles(env, filesToCopy, destDir);

               // Save current list of files with backup data set.
               String[] filesInSnapshot = backupHelper.getLogFilesInSnapshot();
               // Save filesInSnapshot persistently here ...

               // Update knowledge of last file saved in the backup set.
               lastFileInPrevBackup = backupHelper.getLastFileInBackupSet();
               // Save lastFileInPrevBackup persistently here ...
           } finally {
               // Remember to exit backup mode, or the JE cleaner cannot delete
               // log files and disk usage will grow without bounds.
              backupHelper.endBackup();
           }
       }

       void myCopyFiles(Environment env, String[] filesToCopy, File destDir) {
           // See LogVerificationInputStream
       }

  When performing backups as shown in the second example above, to restore you must choose one of the file lists that was saved. You may choose the list written by the most recent backup, or a list written by an earlier backup. To restore, the files in the list should be copied into an empty destination directory.

  Restoring from a backup

  As described in the sections above, the restore procedure is to copy the files from a backup set into an empty directory. Depending on the type of backup that was performed (see above), either all files from the backup set are copied, or only the files on a list that was created during the backup.

  After copying the files, an additional step is needed if the backup that was restored is to be used as the basis for future incremental backups. To enable the creation of future incremental backups, the `EnvironmentConfig.ENV_RECOVERY_FORCE_NEW_FILE` parameter should be set to true when opening the JE Environment for the first time after the restore. When this parameter is set to true, the last .jdb file restored will not be modified when opening the Environment, and the next .jdb file will be created and will become the end-of-log file.

  WARNING: If this property is not set to true when opening the environment for the first time after a restore, then the backup set that was restored may not be used as the basis for future incremental backups. If a future incremental backup were performed based on this backup set, it would be incomplete and data would be lost if that incremental backup were restored.

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `DbBackup(Environment env)` Creates a DbBackup helper for a full backup.
    `DbBackup(Environment env, long lastFileInPrevBackup)` Creates a DbBackup helper for an incremental backup.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `endBackup()` End backup mode, thereby re-enabling normal deletion of log files by the JE log cleaner.
    `long` | `getLastFileInBackupSet()` Can only be called in backup mode, after startBackup() has been called.
    `String[]` | `getLogFilesInBackupSet()` Get the minimum list of files that must be copied for this backup.
    `String[]` | `getLogFilesInBackupSet(long lastFileInPrevBackup)` Deprecated. replaced by `getLogFilesInBackupSet()`; pass lastFileInPrevBackup to the `DbBackup(Environment,long)` constructor.
    `String[]` | `getLogFilesInSnapshot()` Get the list of all files that are needed for the environment at the point of time when backup mode started, i.e., the current snapshot.
    `void` | `startBackup()` Start backup mode in order to determine the definitive backup set needed at this point in time.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### DbBackup

          public DbBackup(Environment env)
                   throws DatabaseException

      Creates a DbBackup helper for a full backup.

      This is equivalent to using `DbBackup(Environment,long)` and passing `-1` for the `lastFileInPrevBackup` parameter.

      Parameters:
        `env` - with an open, valid environment handle. If the environment directory has read/write permissions, the environment handle must be configured for read/write.
      Throws:
        `IllegalArgumentException` - if the environment directory has read/write permissions, but the environment handle is not configured for read/write.
        `DatabaseException`

    - #### DbBackup

          public DbBackup(Environment env,
                  long lastFileInPrevBackup)

      Creates a DbBackup helper for an incremental backup.
      Parameters:
        `env` - with an open, valid environment handle. If the environment directory has read/write permissions, the environment handle must be configured for read/write.
        `lastFileInPrevBackup` - the last file in the previous backup set when performing an incremental backup, or `-1` to perform a full backup. The first file in this backup set will be the file following `lastFileInPrevBackup`.
      Throws:
        `EnvironmentFailureException` - if an unexpected, internal or environment-wide failure occurs.
        `IllegalArgumentException` - if the environment directory has read/write permissions, but the environment handle is not configured for read/write.

  - ### Method Detail

    - #### startBackup

          public void startBackup()
                           throws DatabaseException

      Start backup mode in order to determine the definitive backup set needed at this point in time.

      This method determines the last file in the backup set, which is the last log file in the environment at this point in time. Following this method call, all new data will be written to other, new log files. In other words, the last file in the backup set will not be modified after this method returns.

      WARNING: After calling this method, deletion of log files in the backup set by the JE log cleaner will be disabled until `endBackup()` is called. To prevent unbounded growth of disk usage, be sure to call `endBackup()` to re-enable log file deletion. Additionally, the Environment can't be closed until endBackup() is called.

      Throws:
        `LogOverwriteException` - if a replication operation is overwriting log files. The backup can not proceed because files may be invalid. The backup may be attempted at a later time.
        `EnvironmentFailureException` - if an unexpected, internal or environment-wide failure occurs.
        `IllegalStateException` - if a backup is already in progress
        `DatabaseException`

    - #### endBackup

          public void endBackup()

      End backup mode, thereby re-enabling normal deletion of log files by the JE log cleaner.
      Throws:
        `LogOverwriteException` - if a replication operation has overwritten log files. Any copied files should be considered invalid and discarded. The backup may be attempted at a later time.
        `EnvironmentFailureException` - if an unexpected, internal or environment-wide failure occurs.
        `IllegalStateException` - if a backup has not been started.

    - #### getLastFileInBackupSet

          public long getLastFileInBackupSet()

      Can only be called in backup mode, after startBackup() has been called.
      Returns:
        the file number of the last file in the current backup set. Save this value to reduce the number of files that must be copied at the next backup session.
      Throws:
        `IllegalStateException` - if a backup has not been started.

    - #### getLogFilesInBackupSet

          public String[] getLogFilesInBackupSet()

      Get the minimum list of files that must be copied for this backup. When performing an incremental backup, this consists of the set of backup files that are greater than the last file copied in the previous backup session. When performing a full backup, this consists of the set of all files. Can only be called in backup mode, after startBackup() has been called.

      The file numbers returned are in the range from the constructor parameter `lastFileInPrevBackup + 1` to the last log file at the time that `startBackup()` was called.

      Returns:
        the names of all files to be copied, sorted in alphabetical order. The return values are generally simple file names, not full paths. However, if multiple data directories are being used (i.e. the je.log.nDataDirectories parameter is non-0), then the file names are prepended with the associated "dataNNN/" prefix, where "dataNNN/" is the data directory name within the environment home directory and "/" is the relevant file separator for the platform.
      Throws:
        `EnvironmentFailureException` - if an unexpected, internal or environment-wide failure occurs.
        `IllegalStateException` - if a backup has not been started.

    - #### getLogFilesInBackupSet

          @Deprecated
          public String[] getLogFilesInBackupSet(long lastFileInPrevBackup)

      Deprecated. replaced by `getLogFilesInBackupSet()`; pass lastFileInPrevBackup to the `DbBackup(Environment,long)` constructor.
      Get the minimum list of files that must be copied for this backup. This consists of the set of backup files that are greater than the last file copied in the previous backup session. Can only be called in backup mode, after startBackup() has been called.
      Parameters:
        `lastFileInPrevBackup` - file number of last file copied in the last backup session, obtained from getLastFileInBackupSet().
      Returns:
        the names of all the files to be copied that come after lastFileInPrevBackup.
      Throws:
        `EnvironmentFailureException` - if an unexpected, internal or environment-wide failure occurs.
        `IllegalStateException` - if a backup has not been started.

    - #### getLogFilesInSnapshot

          public String[] getLogFilesInSnapshot()

      Get the list of all files that are needed for the environment at the point of time when backup mode started, i.e., the current snapshot. Can only be called in backup mode, after startBackup() has been called.

      When performing an incremental backup, this method is called to determine the files that would needed for a restore. As described in the examples at the top of this class, this list can be used to avoid unused files after a restore, and may also be used to reduce the size of the backup set.

      When performing a full backup this method is normally not needed, since in that case it returns the same set of files that is returned by `getLogFilesInBackupSet()`.

      Returns:
        the names of all files in the snapshot, sorted in alphabetical order. The return values are generally simple file names, not full paths. However, if multiple data directories are being used (i.e. the je.log.nDataDirectories parameter is non-0), then the file names are prepended with the associated "dataNNN/" prefix, where "dataNNN/" is the data directory name within the environment home directory and "/" is the relevant file separator for the platform.
      Throws:
        `EnvironmentFailureException` - if an unexpected, internal or environment-wide failure occurs.
        `IllegalStateException` - if a backup has not been started.

Copyright (c) 2004, 2014 Oracle and/or its affiliates. All rights reserved.