Configuration

Installation

HybridStore can be fully configured using its WCF-API, most basic configurations can be done via command line.

Setup

After finishing the setup, you get a service "Inpoint.HybridStore", which is installed in C:\Program Files (x86)\HS Europe\inPoint.HybridStore. Next you need to ensure that a valid license is placed in the directory of C:\Program Files (x86)\HS Europe. After that the service will run and use default connections. That means there is no tenant yet, but it can already be accessed by WCF. The url for the interface are written to the DebugOutput and they can also be seen in the config-file (it is created during first start).

Configure using Commandline

After installing open a command line (with admin rights) in the folder of HybridStore.

For the help of all commands execute

"HybridStoreSv.exe CONFIG"

The basic settings are explained in this documentation.

IMPORTANT In case that existing objects are changed, a restart of the service is required! The command line tool will not do that!

TIP: Use "^" at the end of a line, allows to continue the command in the next line to make it more readable!

Connections

Per default inPoint.HybridStore has 3 WCF connections connections for clients and a 4th connection to be used for configuration by inPoint.Admin. (see the sample configuration below).

The connections can only be configured by editing the configuration file directly.

Performance
The fastest connection at the time of writing is the DefaultTcp connection. It is recommended to use this one for all clients. A local client can use net.tcp://localhost:822/HybridStore as a connection string. A remote client should use the full qualified domain name of the server.

Https

The connection is always secured by WCF at least on the message-level. By enabling https the encryption is moved to the transport layer.

Configure by entering the subject of the certificate from the machine certificates (e.g. the full qualified name of the server) into Certificate.

gRCP

If https is enabled the gRPC interface is enabled as well.

Tenant

This creates a tenant with the given name and database credentials, the id is randomly created. This id is needed for all calls to HybridStore!

Create a Tenant

Use /CreateTenant to create a new tenant.

This creates the database objects and adds the tenant to the config file. In all following commands this tenant is used (by id or by name). All other commands will not change the config file any more, but only the database.

The following parameter can be used on the command line and are then written with the same name into the config file. The config file is case sensitive, but the commandline is not!

name*

<text>
The name of the tenant.

DbType*

SQLSERVER, ORACLE
The type of the database.

DbScheme*

<text>
The name of the database scheme.

DbConnectionString*

<text>
The connection string to the database. It can include the password, which will be extracted and saved encrypted in an own setting.

DbPassword

<text>
The password for the database connection (Can be included in the DbConnectionString instead.)

id

<guid>
The unique id of the tenant. If empty the system will generate a random id.

readOnly

false, true
In case this is true, the whole tenant is read-only. All write operations will fail with a predefined error code. Default: false

disableSingleInstance

false, true
In case this is true, the single instance feature is switched off for the whole tenant. Default: false

tempDirectory

<text>
The directory used for temporary data. If empty the default from the operating system is used.

allowJobs

true, false
In case this is false the HybridStore Jobs will not be started. Default: true

Example

HybridStoreSv.exe CONFIG /CreateTenant ^
  /Name HS-Test             ^
  /DbType SQLSERVER         ^
  /DbScheme hsUser          ^
  /DbConnectionString "User Id=hsUser;Password=hsPass;Initial Catalog=HybridStore;Data Source=localhost"

Response

Tenant created: HS-Test - efcae8f2-8db4-4055-9eb0-331df3b72292

The created config file, will contain these settings (formatted here to keep it smaller).

   "HybridStore": {
    "HybridStoreTenants": [
      {
        "Id": "fb4de357-5741-4aa1-94bf-0b9ec5fa2333",
        "Name": "HS-Test",
        "DbType": "SQLSERVER",
        "DbScheme": "hsUser",
        "DbConnectionString": "User Id=hsUser;Initial Catalog=HybridStore;Data Source=localhost;Password={0}",
        "DbEncryptedPassWord": "uq3wDbqt8A0=",
        "ReadOnly": false,
        "AllowJobs": true,
        "TempDirectory": "",
        "CacheTime": 30,
        "DisableSingleInstance": false,
        "Users": []
      }
    ],
    "Certificate": "",
    "MaxPendingCalls": 90,
    "Connections": [
      {  "Name": "DefaultHttp"      , "Secure": false, "Address": "http://%COMPUTERNAMEFQDN%:821/HybridStore"     , "AllowConfig": false, "WcfTimeOut": 60  },
      {  "Name": "DefaultTcp"       , "Secure": false, "Address": "net.tcp://%COMPUTERNAMEFQDN%:822/HybridStore"  , "AllowConfig": false, "WcfTimeOut": 60	},
      {  "Name": "DefaultTcpConfig" , "Secure": true , "Address": "net.tcp://%COMPUTERNAMEFQDN%:8822/HybridStore" , "AllowConfig": true , "WcfTimeOut": 60  },
      {  "Name": "DefaultNetPipe"   , "Secure": false, "Address": "net.pipe://localhost/HybridStore"			  , "AllowConfig": true , "WcfTimeOut": 60  }
    ]
  }

User

It is possible to add user with limited access to a tenant. When accessing HybridStore its possible to use the tenant id which gives full right or any client id with limited rights.

The configuration can only be done in the config file directly.

Example:

    "HybridStoreTenants": [
      {
        "Id": "fb4de357-5741-4aa1-94bf-0b9ec5fa2333",
        "Name": "HS-Test",
        "DbType": "SQLSERVER",
        "DbScheme": "hsUser",
        "DbConnectionString": "User Id=hsUser;Initial Catalog=HybridStore;Data Source=localhost;Password={0}",
        "DbEncryptedPassWord": "uq3wDbqt8A0=",
        "ReadOnly": false,
        "AllowJobs": true,
        "TempDirectory": "",
        "CacheTime": 30,
        "DisableSingleInstance": false,
        "Users": [
          {
            "Guid": "4800b81c-b4fd-4767-abf3-098de37e983c",
            "Rights": "Read, Write"
          },
          {
            "Guid": "208ddd6a-6590-4176-888c-f8d3480dc9e0",
            "Rights": "Read"
          }
        ]
      }
    ]

guid

The id of the user. A valid guid can be generated by a call in powershell:

New-Guid 

Rights

A combination of rights for this user (separated by comma)

• None: No rights at all
• Read: Only reading is allowed.
• Write: Writing of new files is allowed.
• Delete: Deleting of files is allowed.
• Admin: Administer locations, schemes,...
• Full: All rights, including future ones.

Locations

A location is the place where the files are physically stored. This can be a simple directory on a local disc, an UNC path or a more advanced system. Some provider support retention dates to protect the files against unwanted deletion.

Create a Location

Use /AddLocation to add a new location to a tenant.

HybridStoreSv.exe CONFIG /Tenant HS-Test /AddLocation </other parameter...>

Edit a location

Use /EditLocation <name or id> to edit an existing location. You can use the ID created in during creation or use the name you have chosen.

HybridStoreSv.exe CONFIG /Tenant HS-Test /EditLocation LocalHDD </other parameter...>

Common Parameter

Some settings are supported by all types of storage.

Name*

<text>
The name of this location, all chars are allowed but it’s recommended to use only chars which are also allowed in filenames.

LocationType*

FilePathLocation, CenteraLocation, TivoliLocation, AmazonS3Location EmcS3Location
The type of the location.

ReadOnly

true, false
In case this is true, the location is read-only. All write operations will fail with a predefined error.

Compression

NotUsed, GZip
A file is marked as compressed or not, it’s possible to enable or disable this feature later but it will only affect new files.

• NotUsed: no compression
• GZip: each file is compressed by GZip

CompressionExclude

<list of extension>
A list of file extensions, which should be excluded from compression.

Encryption

• NotUsed: default, no encryption
• Rijndael: each file is encrypted using Rijndael using a 256 bit key and a random salt per file.
• RijndaelGZip: each file is first compressed with GZip and then encrypted!

EncryptionKey

<text>
The Rijndael key is derived from this key (after adding a salt).

IMPORTANT!
If files are already encrypted “NEVER” change the key. If needed create a new location which points to the same storage as the current.

Filepath-Location

Files are stored on an UNC-Path or a local Disk! For shares it's possible to set a password or you need to run the service as a windows user with access rights!

LocationType*

FilePathLocation

Paths*

<directory paths>
One or more paths where the files will be written. If more than one path use “;” to separate. In case of multiple paths, the write is only successful if all paths are written. For reading only one successful read is required!

RetentionType

• None: no retention time at all
• Celerra: Dell/EMC Celerra, using the last-access-date
• Celerra_ReOpen: same as above, but the file will be re-opened
• NetApp: NetApp Snaplock, using the last-access-date with the NetApp specific handling of dates after the year 2038.
• NetApp_ReOpen: same as above, but the file will be re-opened
• LastAccessThenReadOnly: generic method of using the last-access-date and read-only attributes
• LastAccessThenReadOnly_ReOpen: same as above, but the file will be re-opened

ReOpen
All Retention-Types support ReOpen. This will first closed the file normally and then reopen it to set the retention-date (setting the last-access-date and then set the file to read-only).

UserName

When accessing a share, use this credentials!

UserPass

When accessing a share, use this credentials!

Example:
Here we create a simple location for storing files on C:\HybridStore\LocalHDD

HybridStoreSv.exe CONFIG /Tenant HS-Test  ^
  /AddLocation                   ^
  /Name LocalHDD                 ^
  /LocationType FilePathLocation ^
  /Paths C:\HybridStore\LocalHDD

Response:

Configure tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
Location Added: LocalHDD(1)

Remember the id of the new location, it will be required when creating schemes! Newer versions can also use the name instead of the id.

With the parameter "RetentionType" it's possible to configure retention enabled storages like Celerra
or NetApp.

Centera-Location

Store files in an DELL/EMC Centera.

Important!
The client accessing the Centera requires the installation of the 'Microsoft Visual C++ 2005 Service Pack 1 Redistributable Package' It is located in the installation directory of HybridStore in the tools directory (execute 'vcredist_2005_SP1_MFC_x64.EXE').

LocationType*

CenteraLocation

Adresses *

<centera connection string>
The connection string to the centera.

Authentication

<file path>
In case authentication is required, this is the full path to a PEA-file, which must exists on the computer.

EmbeddedBlobs

<size in bytes>
All files smaller than this threshold, will be embedded into the clip instead of writing a separate blob.

Example: Here we create a location for storing files on a EMC-Centera, you need a working address and the correct "PEA-file" for it.

HybridStoreSv.exe CONFIG /Tenant HS-Test     ^
/AddLocation                                 ^
/Name CentTest                               ^
/LocationType CenteraLocation                ^
/Addresses "168.159.214.20,168.159.214.21"   ^
/Authentication "C:\HybridStore\TEST.pea"

Response:

Configure tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
Location Added: CentTest(2)

Remember the id of the new location, it will be required when creating schemes!

IBM-Tivoli-Location

Storing files using the IBM-Tivoli interface, a 64-Bit client is required!

As default HybridStore would archive wills with the option ´ENABLEARCHIVERETENTIONPROTECTION=yes´ and would immediately set a Hold on the new files. So that TSM will not delete them, when the files are deleted by HybridStore this Hold is then released.

The files will be stored on TSM, in a directory called H:\HybridStore\HS, the drive and the first directory is fixed, the following part can be configured if required!

LocationType*

TivoliLocation

DsmiDir*

<a valid path>
The directory where the API is located. (e.g. tsmapi64.dll)

DsmiConfig*

<a valid path>
The file to config-file (e.g. C:\dsm.opt)

DsmiLog*

<a valid path>
The path for logging (set to TSM)

LogName

<filename>
The filename for logging (set to TSM); default: HybridStore.txt

ClientNodeName

<string>
The node name for archiving. Default: the machine name

ManagmentClass

<string>
The management class. Default: STANDARD

Options

<string>
Options for archiving. Default is “-ENABLEARCHIVERETENTIONPROTECTION=yes” if Data-Retention is enabled.

UserName*

<string>
The name of the user to connect.

UserPass*

<string>
The password of the user to connect.

FileSpaceDirectory

<string>
The path of the file in TSM. Default: “HS”

LegacyMode

true,false
Enables access to files stored by HSM (creating new files is forbidden).

ArchiveSetRetentionActivate

true,false
During archive the retention is active (default: false)

ArchiveSetRetentionHold

true,false
During archive a “hold” is set on the file (default: true)

DataRetentionModeOff

true,false
Disables data retention (default: off)

Amazon S3-Location (and compatible)

Storing files using the Amazon S3 interface. Files are stored in a bucket defined per location. The structure inside will contain the date (year, month, day) the "SharedId" of the file and the original extension.

IMPORTANT
Several vendors also implement the S3-Interface, please check with their documentation how to configure it!

Request URLs

There are two ways of creating the URLs to access the data.

• old path-style-access where the bucket-name is a folder in the URL.
  e.g. "https:​//s3.Region.amazonaws.com/bucket-name"
• new virtual-hosted-style-access where the bucket-name is a sub-domain in the URL.
  e.g. "https:​//bucket-name.s3.Region.amazonaws.com"

Requirements for bucket name
To use virtual-hosted-style-access the name of the bucket has some special requirement:

• letters a to z (only lower case)
• digits 0 to 9
• single dots or minus as separators (e.g. 'bucket.name' is valid, but '.bucket' or 'bucket..name' are not)
• does not look similar to an ip4 address (e.g. '1.1.1.1' is not allowed)

If these are not met the S3 client will switch automatically to path-style-access.

The default is to use virtual-hosted-style-access if possible!

HTTPS

When using HTTPS the server side must have the correct certificate to match the access style. See the troubleshooting for details.

LocationType*

AmazonS3Location

Address*

<url or name>
The http(s)-address of the S3 server or the well-known name of it(e.g "eu-central-1"). When using HTTPS make sure that the certificate matches!

AccessKey*

<string>
The access key (user-name) for the connection

SecretKey*

<string>
The secret key (password) for the connection

Bucket*

<string>
The name of the S3 Bucket (in case it's not existing it will be created if allowed)

EncryptionMethod

<string>
Method for the optional server-side encryption (Amazon S3 supports only "none" or "AES256")

EncryptionKey

<string>
The encryption key for the optional server-side encryption (encoded in Base64), it must have the correct key size!

ObjectLockMode

• None: no protection against deletion and modification (default)
• Governance: files are protected against deletion and modification, but a special authorized user can still delete/modify a protected file.
• Compliance: files are protected against deletion and modification.

The mode for Object Lock which enables protection using retention periods on an S3 file server. If using 'Amazon S3' the bucket must have Object Lock enabled in the configuration of the bucket. During startup HybridStore will verify that this setting is enabled, if the verification itself is not possible it can be skipped using SkipBucketCheck.

ForcePathStyle

true,false
By setting this to true its possible to force the S3 client to use the old path-style-access when accessing the data.

SkipBucketCheck

false,true
During startup it verified that the bucket is existing and if ObjectLockMode is enabled it is verified that the bucket has versioning enabled. This is done to provide better error messages in case of a bad configuration. If a user is not allowed to execute this calls on the server side these checks must be skipped.

ETagCheck

The ETag is an MD5 hash of the file which is returned by the S3 server when reading a file. This ETag can be used to verify that the file has not changed before even downloading it. The normal download of a file will also verify the MD5 checksum of the file, so this is an additional check. It's not supported by all vendors.

• auto: (default) If the ETag looks like an MD5 it will be used, otherwise it will be ignored.
• on: The ETag will be used to verify the file, it will throw an error if the ETag does not match the MD5 of the file.
• off: The ETag will not be used.

ChecksumAlgorithm

During upload the data is always protected by the calculation of an MD5 checksum which was generated on the client side. The S3 client can calculate an additional checksum which is stored in the database and can be used to verify the file before reading it (similar to the ETag check).

• None: (default) No additional checksum is calculated.
• CRC32: Use the CRC32 algorithm (also called CRC-32-IEEE) to protect the integrity of the data. (Base64 encoded, 32-bit)
• CRC32C: Use the CRC32C algorithm (also called CRC Castagnoli) to protect the integrity of the data. (Base64 encoded, 32-bit)
• SHA1: Use the SHA1 cryptographic hash algorithm to protect the integrity of the data. (Base64 encoded, 160-bit)
• SHA256: Use the SHA256 cryptographic hash algorithm to protect the integrity of the data. (Base64 encoded, 256-bit)

Amazon S3-Location compatible with special settings

• Hitachi Content Platform (HCP)

Hitachi Content Platform (HCP)

Hitachi implemented the Amazon S3 interface. The settings are very similar, if not mentioned use the same as above.

Retention Time
Versions before 9.6.x do not support the standard retention mechanism of Amazon S3.

Address*

<url>
The http(s)-address of the S3 server. When using HTTPS make sure that the certificate matches!

AccessKey*

<string>
The user-name for the connection, encoded in base64.

SecretKey*

<string>
The MD5-Hash of the password for the connection.

Bucket

<string>
The name of the S3 Bucket (in case it's not existing it will be created if allowed). In previous versions of the S3-SDK it was configured to be empty. In this version that is no longer true!

ForcePathStyle

true,false
Should not be used and switched off.

SkipBucketCheck

false,true
Use true to disable the check, the check requires extra permission and is likely to fail with permission denied.

DELL-EMC S3-Location

Storing files using the Amazon S3 interface of EMC ECS. This features supports all the features of the Amazon S3 implementation, but it is storing files with the retention-time-extension of EMC.

LocationType*

EmcS3Location

Address*

<url>
The http(s)-address of the server. When using HTTPS make sure that the certificate matches!

AccessKey*

<string>
The access key (user-name) for the connection

SecretKey*

<string>
The secret key (password) for the connection

Bucket*

<string>
The name of the S3 Bucket (this is created during startup, if it does not exist).

EncryptionMethod

<string>
Method for the optional server-side encryption (Amazon S3 supports only "none" or "AES256")

EncryptionKey

<string>
The encryption key for the optional server-side encryption (encoded in Base64), it must have the correct key size!

RetentionChangeType

• Ignore: Ignore calls to increase the retention-time
• Copy: Create a copy of the item with the increased retention time

Configures what the server should do if the retention time of a document has to be increased.

ForcePathStyle

true,false
By setting this to true its possible to force the S3 client to use the old path-style-access when accessing the data.

Delete a Location

Use /DeleteLocation <id>|<name> to delete a location given by id or by name. The location must not be in use by any scheme and must not contain any files.

Scheme

The scheme groups several settings (like minimal retention, storage location,...) into one and give it a nice human readable name. When storing a file into HybridStore the caller must always provide the scheme to be used.

Create a Scheme

Use /AddScheme to add a new scheme to a tenant.

HybridStoreSv.exe CONFIG /Tenant HS-Test /AddScheme </other parameter...>

Edit a Scheme

Use /EditScheme <name or id> to edit an existing scheme. You can use the ID created in during creation or use the name you have chosen.

HybridStoreSv.exe CONFIG /Tenant HS-Test /EditScheme LocalScheme </other parameter...>

Parameter

Name*

<text>
The name of this scheme, all characters are allowed.

Locations*

<List of locations (name or id)>
A list of locations ids or names, separate by space. Each id can have a trailing -Enqueue or - Immediately. During write the file will be saved to all locations with a trailing Immediately but at least into one locations, the other locations will be filled by a background job.

Example:
/Locations 1 2
or
/Locations LocalHDD-Enqueue CentTest-Enqueue

SkipForExisting

false,true
Default: false
If a file is archived again (and single instance is not switched off) HybridStore makes sure that the file is saved into all the location(s) listed in /Locations. That means if the file is already in the correct location nothing will happen (except maybe the retention date on the hardware is increase) but if the file is not saved there it will be copied there.

If this is true and a duplicate is found in any location, the file will not be stored again in the location(s) listed in /Locations.

MinRetentionDays

<number> If needed you can set a minimal retention time in days for all file in this location. This will increase the used retention date during archive, changing this value will not affect written files.

Example:
Here we create a scheme for the locations, during archive a scheme will be set as parameter to define where files are saved.

HybridStoreSv.exe CONFIG /Tenant HS-Test /AddScheme   /Name LocalScheme  /Locations LocalHdd

Response:

Configure tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
Scheme Added: LocalScheme(1)

Remember the id of the new scheme for further configuration.

Delete a Scheme

Use /DeleteScheme <id>|<name> to delete an existing scheme given by id or by name. The scheme must be empty!

Show Configuration

Everything should be ready now, call ListAll to verify the settings. Here all settings except passwords are shown, so you can see all parameters which exists, even if they were not used!

HybridStoreSv.exe CONFIG /Tenant HS-Test /ListAll

Response:

Tenant: efcae8f2-8db4-4055-9eb0-331df3b72292 - HS-Test
  Admin: efcae8f2-8db4-4055-9eb0-331df3b72292 -> Full
  Location: LocalHDD(1) FilePathLocation
    ReadOnly       : False
    Encryption     : NotUsed
    Compression    : NotUsed
    CompressionExclude:
    Paths          : C:\HybridStore\LocalHDD
    RetentionType  : None
    UserName       :

  Location: CentTest(2) CenteraLocation
    ReadOnly       : False
    Encryption     : NotUsed
    Compression    : NotUsed
    CompressionExclude:
    Addresses      : 168.159.214.20,168.159.214.21
    Authentication : C:\HybridStore\TEST.pea
    EmbeddedBlobs  : 0

  Scheme: LocalScheme(1)
    Min Ret. Days  : 0
    SkipForExisting: False
    Location(100) -> Immediately  LocalHDD(1)

  Scheme: CentTest(2)
    Min Ret. Days  : 0
    SkipForExisting: False
    Location(100) -> Immediately  CentTest(2)