Configuration

Installation

HybridStore can be fully configured using its WCF-API, most basic configurations can be done via command line.

Setup

After finishing the setup, you get a service "Inpoint.HybridStore", which is installed in C:\Program Files (x86)\HS Europe\inPoint.HybridStore. Next you need to ensure that a valid license is placed in the directory of C:\Program Files (x86)\HS Europe. After that the service will run and use default connections. That means there is no tenant yet, but it can already be accessed by WCF. The url for the interface are written to the DebugOutput and they can also be seen in the config-file (it is created during first start).

Configure using Commandline

After installing open a command line (with admin rights) in the folder of HybridStore.

For the help of all commands execute

"HybridStoreSv.exe CONFIG"

The basic settings are explained in this documentation.

IMPORTANT In case that existing objects are changed, a restart of the service is required! The command line tool will not do that!

Tenant

This creates a tenant with the given name and database credentials, the id is randomly created. This id is needed for all calls to HybridStore!

TIP: Use "^" at the end of a line, allows to continue the command in the next line to make it more readable!

HybridStoreSv.exe CONFIG /CreateTenant ^
  /Name HS-Test             ^
  /DbType SQLSERVER         ^
  /DbScheme hsUser          ^
  /DbConnectionString "User Id=hsUser;Password=hsPass;Initial Catalog=HybridStore;Data Source=VMDEVUNIFY\SQLDEVUNIFY"

Response

Tenant created: HS-Test - efcae8f2-8db4-4055-9eb0-331df3b72292

This creates the database objects and adds the tenant to the config file. In all following commands this tenant is used (by id or by name). All other commands will not change the config file any more, but only the database.

User

It is possible to add user with limited access to a tenant. When accessing HybridStore its possible to use the tenant id which gives full right or any client id with limited rights.

The configuration can only be done in the config file directly.

Example:

      <add id="fb4de357-5741-4aa1-94bf-0b9ec5fa2333" name="inPoint"
          allowJobs="false"
          DbType="SQLSERVER" DbScheme="ip" DbConnectionString="Data Source=localhost;Initial Catalog=inPoint;User Id=ip;Password={0}"
          DbEncryptedPassWord="**************">
        <Users>
          <add guid="4800b81c-b4fd-4767-abf3-098de37e983c" rights="Read, Write" />
          <add guid="208ddd6a-6590-4176-888c-f8d3480dc9e0" rights="Read" />
        </Users>
      </add>

guid

The id of the user. A valid guid can be generated by a call in powershell:

New-Guid 

Rights

A combination of rights for this user (separated by comma)

• None: No rights at all
• Read: Only reading is allowed.
• Write: Writing of new files is allowed.
• Delete: Deleting of files is allowed.
• Admin: Administer locations, schemes,...
• Full: All rights, including future ones.

Locations

A location is the place where the files are physically stored. This can be a simple directory on a local disc, an UNC path or a more advanced system. Some provider support retention dates to protect the files against unwanted deletion.

Create a Location

Use /AddLocation to add a new location to a tenant.

HybridStoreSv.exe CONFIG /Tenant HS-Test /AddLocation </other parameter...>

Edit a location

Use /EditLocation <name or id> to edit an existing location. You can use the ID created in during creation or use the name you have chosen.

HybridStoreSv.exe CONFIG /Tenant HS-Test /EditLocation LocalHDD </other parameter...>

Common Parameter

Some settings are supported by all types of storage.

Name*

<text>
The name of this location, all chars are allowed but it’s recommended to use only chars which are also allowed in filenames.

LocationType*

FilePathLocation, CenteraLocation, TivoliLocation, AmazonS3Location EmcS3Location
The type of the location.

ReadOnly

True, False
In case this is true, the location is read-only. All write operations will fail with a predefined error.

Compression

NotUsed, GZip
A file is marked as compressed or not, it’s possible to enable or disable this feature later but it will only affect new files.

• NotUsed: no compression
• GZip: each file is compressed by GZip

CompressionExclude

<list of extension>
A list of file extensions, which should be excluded from compression.

Encryption

• NotUsed: default, no encryption
• Rijndael: each file is encrypted using Rijndael using a 256 bit key and a random salt per file.
• RijndaelGZip: each file is first compressed with GZip and then encrypted!

EncryptionKey

<text>
The Rijndael key is derived from this key (after adding a salt).

IMPORTANT!
If files are already encrypted “NEVER” change the key. If needed create a new location which points to the same storage as the current.

Filepath-Location

Files are stored on an UNC-Path or a local Disk! For shares it's possible to set a password or you need to run the service as a windows user with access rights!

LocationType*

FilePathLocation

Paths*

<directory paths>
One or more paths where the files will be written. If more than one path use “;” to separate. In case of multiple paths, the write is only successful if all paths are written. For reading only one successful read is required!

RetentionType

• None: no retention time at all
• Celerra: Dell/EMC Celerra, using the last-access-date
• Celerra_ReOpen: same as above, but the file will be re-opened
• NetApp: NetApp Snaplock, using the last-access-date with the NetApp specific handling of dates after the year 2038.
• NetApp_ReOpen: same as above, but the file will be re-opened
• LastAccessThenReadOnly: generic method of using the last-access-date and read-only attributes
• LastAccessThenReadOnly_ReOpen: same as above, but the file will be re-opened

ReOpen
All Retention-Types support ReOpen. This will first closed the file normally and then reopen it to set the retention-date (setting the last-access-date and then set the file to read-only).

UserName

When accessing a share, use this credentials!

UserPass

When accessing a share, use this credentials!

Example:
Here we create a simple location for storing files on C:\HybridStore\LocalHDD

HybridStoreSv.exe CONFIG /Tenant HS-Test  ^
  /AddLocation                   ^
  /Name LocalHDD                 ^
  /LocationType FilePathLocation ^
  /Paths C:\HybridStore\LocalHDD

Response:

Configure tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
Location Added: LocalHDD(1)

Remember the id of the new location, it will be required when creating schemes! Newer versions can also use the name instead of the id.

With the parameter "RetentionType" it's possible to configure retention enabled storages like Celerra
or NetApp.

Centera-Location

Store files in an DELL/EMC Centera.

LocationType*

CenteraLocation

Adresses *

<centera connection string>
The connection string to the centera.

Authentication

<file path>
In case authentication is required, this is the full path to a PEA-file, which must exists on the computer.

EmbeddedBlobs

<size in bytes>
All files smaller than this threshold, will be embedded into the clip instead of writing a separate blob.

Example: Here we create a location for storing files on a EMC-Centera, you need a working address and the correct "PEA-file" for it.

HybridStoreSv.exe CONFIG /Tenant HS-Test     ^
/AddLocation                                 ^
/Name CentTest                               ^
/LocationType CenteraLocation                ^
/Addresses "168.159.214.20,168.159.214.21"   ^
/Authentication "C:\HybridStore\TEST.pea"

Response:

Configure tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
Location Added: CentTest(2)

Remember the id of the new location, it will be required when creating schemes!

IBM-Tivoli-Location

Storing files using the IBM-Tivoli interface, a 64-Bit client is required!

As default HybridStore would archive wills with the option ´ENABLEARCHIVERETENTIONPROTECTION=yes´ and would immediately set a Hold on the new files. So that TSM will not delete them, when the files are deleted by HybridStore this Hold is then released.

The files will be stored on TSM, in a directory called H:\HybridStore\HS, the drive and the first directory is fixed, the following part can be configured if required!

LocationType*

TivoliLocation

DsmiDir*

<a valid path>
The directory where the API is located. (e.g. tsmapi64.dll)

DsmiConfig*

<a valid path>
The file to config-file (e.g. C:\dsm.opt)

DsmiLog*

<a valid path>
The path for logging (set to TSM)

LogName

<filename>
The filename for logging (set to TSM); default: HybridStore.txt

ClientNodeName

<string>
The node name for archiving. Default: the machine name

ManagmentClass

<string>
The management class. Default: STANDARD

Options

<string>
Options for archiving. Default is “-ENABLEARCHIVERETENTIONPROTECTION=yes” if Data-Retention is enabled.

UserName*

<string>
The name of the user to connect.

UserPass*

<string>
The password of the user to connect.

FileSpaceDirectory

<string>
The path of the file in TSM. Default: “HS”

LegacyMode

true,false
Enables access to files stored by HSM (creating new files is forbidden).

ArchiveSetRetentionActivate

true,false
During archive the retention is active (default: false)

ArchiveSetRetentionHold

true,false
During archive a “hold” is set on the file (default: true)

DataRetentionModeOff

true,false
Disables data retention (default: off)

Amazon S3-Location (and compatible)

Storing files using the Amazon S3 interface. Files are stored in a bucket defined per location. The structure inside will contain the date (year, month, day) the "SharedId" of the file and the original extension.

Request URLs

There are two ways of creating the URLs to access the data.

• old path-style-access where the bucket-name is a folder in the URL.
  e.g. "https:​//s3.Region.amazonaws.com/bucket-name"
• new virtual-hosted-style-access where the bucket-name is a sub-domain in the URL.
  e.g. "https:​//bucket-name.s3.Region.amazonaws.com"

Requirements for bucket name
To use virtual-hosted-style-access the name of the bucket has some special requirement:

• letters a to z (only lower case)
• digits 0 to 9
• single dots or minus as separators (e.g. 'bucket.name' is valid, but '.bucket' or 'bucket..name' are not)
• does not look similar to an ip4 address (e.g. '1.1.1.1' is not allowed)

If these are not met the S3 client will switch automatically to path-style-access.

The default is to use virtual-hosted-style-access if possible!

When using HTTPS the server side must have the correct certificate to match the access style.

LocationType*

AmazonS3Location

Address*

<url or name>
The http(s)-address of the S3 server or the well-known name of it(e.g "eu-central-1")

AccessKey*

<string>
The access key (user-name) for the connection

SecretKey*

<string>
The secret key (password) for the connection

Bucket*

<string>
The name of the S3 Bucket (this must be created before)

EncryptionMethod

<string>
Method for the optional server-side encryption (Amazon S3 supports only "none" or "AES256")

EncryptionKey

<string>
The encryption key for the optional server-side encryption (encoded in Base64), it must have the correct key size!

ObjectLockMode

• None: no protection against deletion and modification (default)
• Governance: files are protected against deletion and modification, but a special authorized user can still delete/modify a protected file.
• Compliance: files are protected against deletion and modification.

The mode for Object Lock which enables protection using retention periods on an S3 file server. If using 'Amazon S3' the bucket must have Object Lock enabled in the configuration of the bucket.

ForcePathStyle

true,false
By setting this to true its possible to force the S3 client to use the old path-style-access when accessing the data.

IMPORTANT
Several vendors also implement the S3-Interface, please check with their documentation what to enter here!

Hitachi Content Platform (HCP) - S3 Interface

Hitachi implemented the Amazon S3 interface, the parameters are slightly different.

LocationType*

AmazonS3Location

Address*

<url> The http(s)-address of the S3 server, this must include the name of the bucket:
e.g.
http://bucket.tenant.hcp.fqdn

AccessKey*

<string>
The user-name for the connection, encoded in base64.

SecretKey*

<string>
The MD5-Hash of the password for the connection.

Bucket

<not used>
The name of the S3 Bucket (this must be created before)

EncryptionMethod

<string>
Method for the optional server-side encryption (Amazon S3 supports only "none" or "AES256"

EncryptionKey

<string>
The encryption key for the optional server-side encryption (encoded in Base64), it must have the correct key size!

ForcePathStyle

true,false
Should not be used and switched off.

DELL-EMC S3-Location

Storing files using the Amazon S3 interface of EMC ECS. This features supports all the features of the Amazon S3 implementation, but it is storing files with the retention-time-extension of EMC.

LocationType*

EmcS3Location

Address*

<url>
The http(s)-address of the server.

AccessKey*

<string>
The access key (user-name) for the connection

SecretKey*

<string>
The secret key (password) for the connection

Bucket*

<string>
The name of the S3 Bucket (this is created during startup, if it does not exist).

EncryptionMethod

<string>
Method for the optional server-side encryption (Amazon S3 supports only "none" or "AES256")

EncryptionKey

<string>
The encryption key for the optional server-side encryption (encoded in Base64), it must have the correct key size!

RetentionChangeType

• Ignore: Ignore calls to increase the retention-time
• Copy: Create a copy of the item with the increased retention time

Configures what the server should do if the retention time of a document has to be increased.

ForcePathStyle

true,false
true,false
By setting this to true its possible to force the S3 client to use the old path-style-access when accessing the data.

Scheme

The scheme groups several settings (like minimal retention, storage location,...) into one and give it a nice human readable name. When storing a file into HybridStore the caller must always provide the scheme to be used.

Create a Scheme

Use /AddScheme to add a new scheme to a tenant.

HybridStoreSv.exe CONFIG /Tenant HS-Test /AddScheme </other parameter...>

Edit a Scheme

Use /EditScheme <name or id> to edit an existing scheme. You can use the ID created in during creation or use the name you have chosen.

HybridStoreSv.exe CONFIG /Tenant HS-Test /EditScheme LocalScheme </other parameter...>

Parameter

Name*

<text>
The name of this scheme, all characters are allowed.

Locations*

<List of locations (name or id)>
A list of locations ids or names, separate by space. Each id can have a trailing -Enqueue or - Immediately. During write the file will be saved to all locations with a trailing Immediately but at least into one locations, the other locations will be filled by a background job.

Example:
/Locations 1 2
or
/Locations LocalHDD-Enqueue CentTest-Enqueue

SkipForExisting

true,false
If this is true and a duplicate is found in a any location, the file will not be stored again in the location(s) listed in /Locations.

MinRetentionDays

<number> If needed you can set a minimal retention time in days for all file in this location. This will increase the used retention date during archive, changing this value will not affect written files.

Example:
Here we create a scheme for the locations, during archive a scheme will be set as parameter to define where files are saved.

HybridStoreSv.exe CONFIG /Tenant HS-Test /AddScheme   /Name LocalScheme  /Locations LocalHdd

Response:

Configure tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
Scheme Added: LocalScheme(1)

Remember the id of the new scheme, if not there is a WCF method for listing all configured schemes! Newer versions can also use the name instead of the id.

Show Configuration

Everything should be now ready for usage, call ListAll to verify the settings. Here all settings except passwords are shown, so you can see all parameters which exists, even if they were not used!

HybridStoreSv.exe CONFIG /Tenant HS-Test /ListAll

Response:

Tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
  Admin: efcae8f2-8db4-4055-9eb0-331df3b72292 -> Full
  Location: LocalHDD(1) FilePathLocation
    ReadOnly       : False
    Encryption     : NotUsed
    Compression    : NotUsed
    CompressionExclude:
    Paths          : C:\HybridStore\LocalHDD
    RetentionType  : None
    UserName       :

  Location: CentTest(2) CenteraLocation
    ReadOnly       : False
    Encryption     : NotUsed
    Compression    : NotUsed
    CompressionExclude:
    Addresses      : 168.159.214.20,168.159.214.21
    Authentication : C:\HybridStore\TEST.pea
    EmbeddedBlobs  : 0

  Scheme: LocalScheme(1)
    Min Ret. Days  : 0
    SkipForExisting: False
    Location(100) -> Immediately  LocalHDD(1)

  Scheme: CentTest(2)
    Min Ret. Days  : 0
    SkipForExisting: False
    Location(100) -> Immediately  CentTest(2)