• S3CLOUD Resource security protocols require very accurate time. You MUST either enable the NTP time daemon or use some other method of ensuring accurate time before attempting to use this service.
• Quota (maximum capacity) must be entered in the S3CLOUD Resource definition. The default for this Resource type is 100GB. This may be changed as needed. There is no limit. Multiple archives per medium utilizes archive expiration times and lazy reclamation to maximize the number of archives stored for maximum safety.
• Compression and optional encryption are supported. The connection and all data transported between BackupEDGE and S3CLOUD Endpoints is always encrypted.
• Full file checksumming, for maximum data integrity, is supported.
• Instant File Restore is available from any archive. It is not necessary to read through an entire archive to restore individual files and directories.
• MySQL™ hot backups are supported.
11.8 - Theory of Operation
BackupEDGE S3CLOUD Resources are very similar to the URL Resources used for FTP Backups. However, there are a few points you should be aware of before using them.
Segments
In a tape, DVD, or similar backup, BackupEDGE streams the data directly on to the media as a single complete archive. In S3CLOUD backups, as in FTP backups, BackupEDGE streams the data into archive files. One of the restrictions in S3CLOUD storage is that, before sending an archive file you must tell the cloud site length of the file you are sending.
To work with this restriction, BackupEDGE automatically segments archives; that is, it divides one logically long archive into many short archive files (called segments) that are small enough to keep from filling the hard drive while keeping the backup streaming at maximum network bandwidth. By default, these segments are 50 megabytes in length. This is configurable but usually not necessary.
BackupEDGE can write multiple archives to S3CLOUD servers, and each archive may contain
multiple segments. BackupEDGE handles segments automatically, and provides tools for managing the segments. To maintain consistent archives, the individual segments should never be manipulated by operating system commands or Bucket Manager tools. This is why segments do not have names that make sense to humans.
Quotas
Each S3CLOUD Resource is assigned a storage quota. BackupEDGE will not attempt to use more storage in the S3CLOUD storage cloud than that assigned by the quota.
Retention Times
By default, all archives created to an S3CLOUD Resource using the Scheduler have a Retention
Time (or expiration time) of one week. They will not be automatically erased until the Retention Time is up, but will not necessarily be erased just because its Retention Time is up. An archive
past its Retention Time is called an Expired Archive.
Space Reclamation
Archives are retained on S3CLOUD servers at least until their expiration time has passed. After that, they are deleted in one of two ways...
Lazy Reclamation Enabled (Default)
If Lazy Reclamation is enabled, archives will remain on the S3CLOUD server as long as possible, just in case they may be needed even after their retention time is up. This allows maximum space utilization on the server. For an archive to be deleted...
• The retention time must be up, i.e. it must be an Expired Archive.
• Adding a segment to a new archive would cause the defined quota to be exceeded.
If both conditions are true, the oldest Expired Archive will be deleted in its entirety. This process ensures that a maximum number of older archives are available on the URL Resource.
If the quota is reached and none of the archives has expired, the backup will prompt for additional media.
By default, each backup in a Scheduled Job has a Retention Time of 1 week. This is may be changed on a per-schedule basis in the default simple Scheduler, and on a per-backup basis in the advanced Scheduler.
Lazy Reclamation Disabled
Disabling Lazy Reclamation (un-checking the Lazy Reclamation field in the Resource
Definition) configures BackupEDGE to check for and immediately erase all expired archives any
time a new backup is started to the s3cloud Resource. Only unexpired archives will be retained. This allows only the minimum required amount of space to be used, while still retaining as many archives as are needed.
Note that usually, if you are backing up multiple machines and/ or schedules to the same cloud site, you will create multiple Resources, one per machine/schedule combination. Each Resource would use a different directory on the cloud server and have a different quota. A typical schedule would look like this:
Sample S3CLOUD Backup Schedule
This Schedule will perform Monday through Friday backups. In the example, a five backup rotation will be created. Because of the one week default retention, Expired Archives (those older than one week old) will be retained on the cloud server at least one week, and possibly longer based on the Lazy Reclamation flag in the Resource definition. If the quota is reached and none of the archives has expired, the backup will fail.
+ Edit Backup Schedule ---+ | Schedule Name: simple_job | | Time: [23:00 ] (15:20:21) Enabled: [X] | | Sequence: acme.microlite.com:esequence/onsite | | Backup Domain: system | | Primary Resource: [Change] acme.microlite.com:s3cloud!s3cloud0 | | | | +---+ January 2015 | | | Every Sunday of the week (None) | Su Mo Tu We Th Fr Sa | | | Every Monday of the week Master | M M 3 | | | Every Tuesday of the week Master | 4 M M M M M 10 | | | Every Wednesday of the week Master | 11 M M M M M 17 | | | Every Thursday of the week Master | 18 M M M M M 24 | | | Every Friday of the week Master | 25 M M M M M 31 | | | Every Saturday of the week (None) | | | +---+ | | | | Notify / Advanced: [Change] [Reset Dates] | | Mail Summary To: root Print Summary To: NONE | | Mail Failures To: NONE Print Failures To: NONE | |[Save] [Cancel]| +---+
Changing the retention time in the Schedule to 2 weeks, three weeks, etc. allows easy creation of multiple minimal storage rotations.
Create a BackupEDGE S3CLOUD Resource
Use edgemenu:Admin->Define Resources to do this. Select ‘[NEW]’, and use the down-arrow
keys to change the resource type to ‘S3 Cloud Storage (s3cloud)’. Press [Enter], give the Resource a name (or leave the default) and press [Enter], then [Next].
Directory
The default backup directory is “/backups”, which is in reality a directory stored in the Bucket
within your Amazon account. Bucket
This is the name of the Bucket you created in the appropriate Region of your Account. Access Key
This is the Access Key ID of the User you created in your Amazon Account. Secret Access Key
This is the Secret Access Key ID for the Access Key ID of the User you created in your Amazon
Account.
S3 Cloud Endpoint
This is the Access Address for the Region where you created the Bucket in your Account. For Amazon S3, you will find a list of available Endoints on page page 119.
For dinCloud D3, you will find a list of available Endpoints on page page 127.
For other S3-compatible providers, please see their own on-line documentation for Region and
Endpoint information.
Lazy Reclamation
This controls the behavior of space reclamation (deleting archives) on the s3cloud Resource. See “Space Reclamation” on page 113 for additional information. The default behaviour is Enabled.
+ BackupEDGE Resource Information ---+ |- General Resource Information ---| |Resource Type Amazon S3 and Compatible S3 Cloud Resource | |Resource Name [s3cloud0 ] Change as appropriate | |Description [S3CLOUD ] | |Changer Assoc [Standalone Device] | |Interface [Other ] | | | |-S3 INFO ---| |Directory [/backups ] [Test URL] | |Bucket [YOURS3BUCKET ] [X] Lazy Reclamation | |Access Key [ ] | |Secret Access Key [ ] | |S3 Cloud Endpoint [s3.amazonaws.com ] | | | |- Default Backup Properties ---| |Quota [100G ] [S] Compression Level [5] | |Edge Block Size [64 ] [Y] Double Buffering | ¦[Next] [Prev] [Cancel]¦ +---+
As with URL Resources, we recommend that you set the directory to reflect your system name and schedule name. Here is an example of an s3cloud Resource set for use with the default backup scheduled (simple_job) for system acme.
Testing the S3CLOUD Resource
Test the s3cloud server connection from the machine with BackupEDGE installed. The
[Test URL] button uses the information on the S3CLOUD Resource screen to create a
connection with the S3 ENdpoint and tests transferring files back and forth to the appropriate directory. If a failure occurs, the reasons will be displayed on the screen to help with debugging. For reference, a copy of the most recent test failure log (if any) will be saved in the file
/usr/lib/edge/tmp/testurl.log.
Initialize the S3CLOUD Resource
When you press [Next] to save the resource, you will be asked if you want to ‘Initialize’ it. You
must let BackupEDGE initialize the resource. This tests the connection again and creates a control file named CTL in the destination directory. To initialize at a later time, use
edgemenu:Admin->Initialize Medium. Note that initializing the resource will not erase any
existing backups. If existing backups exits, the CTL file, which contains information about the
individual archive segments, will be re-calculated.
Selecting the S3CLOUD Resource
Select the S3CLOUD Resource as you would any other resource in EDGEMENU or in the
Scheduler.
When you set up a new schedule, it's a good idea to use
edgemenu:Verify->Show Archive Label to see how many archives are actually present after a
few days and check the amount of space used.