API
Development
All communication between HybridStore and any caller is done via WCF! For performance reasons use net.pipe if accessing HybridStore locally and prefer net.tcp over http if accessing it remotely!
Adding a reference in Visual Studio
To create a proxy, first configure HybridStore and verify that it is running.
Then add a service reference to your project! e.g. use
net.pipe://localhost/HybridStore as address.
You see two interfaces, the second one is only needed for configuration!
All important functions are used in the sample project!
Interface IHybridStore
This is the main interface for storage and retrieval. All operations here will return an error-code and an error-message. See the enum of HybridStoreStatus for a list of all values. In case there is a normal exception, it will be because of an communication problem between server and client.
REMARK: All times are in UTC!
ListSchemes
This function will return all defined schemes (name and id)
Response
SchemeId
int: The id of the scheme.
Name
String: The name of the scheme, as chosen by the user.
WriteFile
Method for writing a single file.
Request
The parameter is a single HsWriteRequest class.
ClientId
String: Tenant or User-Id e.g. efcae8f28db440559eb0331df3b72292
CheckSum
String: MD5 of the data
Name
String: The Name of the File
RetentionDate
DateTime: The date when the file is no longer protected (in UTC, use the year 9999 for forever)
Scheme
int: The id of the HybridStore scheme
Size
long: The original size of the file
Data
Stream: The file itself, as Stream
Response
Id
long: The Id of the file in HybridStore
ReadFile
Method for reading a single file, this will return also all informations about the file.
Request
The parameter is a single HsReadRequest class.
ClientId
String: Tenant or User-Id e.g. efcae8f28db440559eb0331df3b72292
Flags
int: Special flags, leave zero
Id
long: The Id of the file in HybridStore
Response
CheckSum
String: MD5 of the archived file
CreationDate
DateTime: The date and time when the file was archived (in UTC)
RetentionDate
DateTime: The date when the file is no longer protected (in UTC, use the year 9999 for forever)
FileType
HsFileType: Not used, always "None"
Name
String: The Name of the File, as given in WriteFile
Scheme
int: The id of the HybridStore scheme
Size
long: The original size of the file
Data
Stream: The file itself, as Stream
ReadFileRange
Similar to "ReadFile", but only a part of the file will be returned!
Request
The parameter is a single HsReadRangeRequest class.
ClientId
String: Tenant or User-Id e.g. efcae8f28db440559eb0331df3b72292
Flags
int: Special flags, leave zero
Id
long: The Id of the file in HybridStore
Position
long: The starting position to start reading
Length
long: The number of files to be returned
Response
CheckSum
String: MD5 of the archived file
CreationDate
DateTime: The date and time when the file was archived (in UTC)
RetentionDate
DateTime: The date when the file is no longer protected (in UTC, use the year 9999 for forever)
FileType
HsFileType: Not used, always "None"
Name
String: The Name of the File, as given in WriteFile
Scheme
int: The id of the HybridStore scheme
Size
long: The original size of the file
RangeSize
long: The number of bytes that will be returned, this is equal or less than "Length" from the request.
Data
Stream: The file itself, as Stream
Delete
This deletes a file, this is only allowed if the retention time is not over yet!
Request
ClientId
String: Tenant or User-Id e.g. efcae8f28db440559eb0331df3b72292
Id
long: The Id of the file in HybridStore
Response
Return
Bool: If true, the file was deleted, if false there was no file found. Errors are reported as fault.
Interface IHybridStoreConfig
So far this is not a stable interface and should not be used. In the future it can be used to configure any aspect of HybridStore.
HybridStoreFault
All errors are reported as faults. Normal exception are then always an indication of a communication problem between server and client.
Operation
HybridStoreOperation: The operation which was attempted.
unknown: Unknown what operation was attempted.writeFile: Archive filesreadFile: Retrieve of filesdelete: Deleting of filesextendRetention: Change (increase) the retention timeduplicate: Duplicate an existing filereadConfig: Read information about the configurationadmin: Administering HybridStore
ErrorCode
HybridStoreStatus: The status-code of the fault.
NotSet: This should never occur. The default value of the enum.Success: SuccessSuccessNoWork: Success, but the operation did not do any actual work.UnknownError: Unknown error, see the ErrorMessage for details.FileIsProtected: The operation could not be done, because the file is protected.FileNotFound: The file was not found.UnknownTenantOrUser: The tenant or client is not known.NotAllowed: the operation is not allowed.UnknownScheme: The scheme is not known (e.g. for archiving a file)UnknownLocation: The location is not known (e.g. invalid entries in the database)CheckSumError: The file is corrupt (either on disk or during transport).NotSupported: The operation is not supported.NotRunning: Some part of HsStorage is not running.FileInOnlyStore: A file is only in one store left, it's not allowed to remove it there.LocationReadOnly: The location is read-only.ArgumentError: A generic error with the arguments.UnknownJob: A unknown background job was specified.ConfigIntegrityError: Deleting a configuration object which is still in use. e.g. deleting a referenced scheme, create a scheme with an non-existing storeConfigDbPasswordError: The saved database password is invalid.ConfigInvalidError: The configuration is invalid, see the ErrorMessage for details.
ErrorMessage
String: The long error message.
Sample Program
The sample program uses all base functions, it has a batch-file which executes all functions with dummy values (as long there is a scheme with an id of "1" it will work without any changes). All results are written to StdOut or StdErr.
Common Arguments
/op
The operation which is executed!
lList schemeswwriterreadrpread partddeletemwmass write
/tenant
<string> The id of the tenant or the user.
/endPoint
<string> The name of the endpoint (default is BasicHttpBinding_IHybridStore)
List Schemes -- /op l
This function requires no more properties. It writes all existing schemes to StdOut
Write -- /op w
Writes a given file to HybridStore, the ID is also returned as program exit code.
/scheme
<int> The scheme to be used for writing, the default is "1"
/path
<string> The full path to a file, which will be archived.
MassWrite -- /op mw
Writes all files given in an UTF8-textfile to HybridStore. Please note that the client is reused and continues working, even if there is an error during archive.
/path
<string> The full path to a file, which contains the files to be archived, each line consists of <schemId\>|<path>.
e.g.
1\|C:\\TestData\\File.txt
Read -- /op r
Reads a file and saves it into a directory. Some information about the file is written, the checksum is read again from the saved file.
/hsId
<long> The ID of the file, which is saved.
/path
<string> The path to a directory, where the file is saved.
ReadPart -- /op rp
Reads a part of an archived file, the content read is written to StdOut as ASCII.
/hsId
<long> The ID of the file, which was saved.
/pos
<string> The position in bytes in the stored file, where reading should start.
/len
<id> The length in bytes, which should be returned.
Delete-- /op d
Deletes a file from HybridStore
/hsId
<long> The ID of the file to be deleted!
Source
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace HsCmdSample
{
class Program
{
static void Main(string[] args)
{
try
{
Program prog = new Program();
prog.MainMethod(args);
}
catch(Exception ex)
{
Console.WriteLine(ex);
}
}
#region Params
int GetParam(List<String> argList, String name, int defaultValue)
{
for(int i=0;i<argList.Count-1;++i)
{
if(String.Equals(argList[i], name, StringComparison.InvariantCultureIgnoreCase))
{
String val = argList[i+1];
argList.RemoveRange(i,2);
return Convert.ToInt32(val);
}
}
return defaultValue;
}
long GetParam(List<String> argList, String name, long defaultValue)
{
for(int i=0;i<argList.Count-1;++i)
{
if(String.Equals(argList[i], name, StringComparison.InvariantCultureIgnoreCase))
{
String val = argList[i+1];
argList.RemoveRange(i,2);
return Convert.ToInt32(val);
}
}
return defaultValue;
}
String GetParam(List<String> argList, String name, String defaultValue)
{
for(int i=0;i<argList.Count-1;++i)
{
if(String.Equals(argList[i], name, StringComparison.InvariantCultureIgnoreCase))
{
String val = argList[i+1];
argList.RemoveRange(i,2);
return val;
}
}
return defaultValue;
}
#endregion
String tenant;
void MainMethod(string[] args)
{
List<String> argList = new List<string>(args);
String op = GetParam(argList, "/op" , "<na>").ToLower();
tenant = GetParam(argList, "/tenant", "efcae8f28db440559eb0331df3b72292");
switch(op)
{
case "l": ListSchemes(argList); break;
case "w": Write (argList); break;
case "r": Read (argList); break;
case "rp": ReadPart(argList); break;
case "d": Delete(argList); break;
case "mw": MassWrite(argList); break;
default:
throw new ArgumentException(String.Format("Operation not known => /op {0}",op));
}
}
/// <summary>
/// List all Schemes, use to give meaningfull names to the user, but use ids in config
/// </summary>
/// <param name="argList"></param>
private void ListSchemes(List<string> argList)
{
var ep = GetParam(argList, "/endPoint" , "BasicHttpBinding_IHybridStore");
HS.IHybridStore hs = new HS.HybridStoreClient(ep);
var r = hs.ListSchemes(new HS.ListSchemesRequest()
{
clientId = tenant
});
Console.WriteLine("ListSchemes List");
foreach(var scheme in r.ListSchemesResult)
{
Console.WriteLine(" {0} -> {1}", scheme.SchemeId, scheme.Name);
}
}
/// <summary>
/// Mass Archive, even if there are errors during calls to HybridStore, the "client" stays connected!
/// /path points to a file, which contain schemeid|path
/// e.g.
/// 1|C:\Test01.txt
/// 1|C:\Test02.txt
/// </summary>
/// <param name="argList"></param>
private void MassWrite(List<string> argList)
{
var path = GetParam(argList, "/path" , "");
var ep = GetParam(argList, "/endPoint" , "BasicHttpBinding_IHybridStore");
if(String.IsNullOrEmpty(path))
{
throw new ArgumentNullException("No path given!");
}
HS.IHybridStore hs = new HS.HybridStoreClient(ep);
Console.WriteLine("MassWrite from {0}",path);
foreach(var line in File.ReadAllLines(path))
{
String[] splitted = line.Split('|');
if(splitted.Length != 2)
{
throw new ArgumentException(String.Format("The line must be <schemeID>|<Path>"));
}
var scheme = Convert.ToInt32(splitted[0]);
var file = splitted[1];
using(var fs = new FileStream(file, FileMode.Open,FileAccess.Read) )
{
try
{
var r = hs.WriteFile(new HS.HsWriteRequest()
{
ClientId = tenant, // Tenant or UserId
RetentionDate = DateTime.UtcNow, // RetentionDate in UTC, use year 9999 for "forever"
Name = Path.GetFileName(file), // Name of the file
Scheme = scheme, // The HybridStore scheme
Size = fs.Length, // The Size of the File
CheckSum = MD5Hash.CreateString(fs), // Md5 of the File
Data = fs // The Data, don't forget to reset the stream, after calculation the checksum
});
Console.WriteLine(" {0} => {1}",r.Id, Path.GetFileName(file) );
}
catch(System.ServiceModel.FaultException<HsCmdSample.HS.HybridStoreFault> ex)
{
// Even if there is an error "HS.IHybridStore" keeps beeing valid!
Console.Error.WriteLine(" Error for line => {0}, Status: {1}, Message: {2}", line, ex.Detail.ErrorCode,ex.Detail.ErrorMessage);
}
}
}
}
/// <summary>
/// Write a single file
/// </summary>
/// <param name="argList"></param>
/// <returns></returns>
private long Write(List<string> argList)
{
var scheme = GetParam(argList, "/scheme" , 1);
var path = GetParam(argList, "/path" , "");
var ep = GetParam(argList, "/endPoint" , "BasicHttpBinding_IHybridStore");
if(String.IsNullOrEmpty(path))
{
throw new ArgumentNullException("No path given!");
}
HS.IHybridStore hs = new HS.HybridStoreClient(ep);
using(var fs = new FileStream(path, FileMode.Open,FileAccess.Read) )
{
var r = hs.WriteFile(new HS.HsWriteRequest()
{
ClientId = tenant, // Tenant or UserId
RetentionDate = DateTime.UtcNow, // RetentionDate in UTC, use year 9999 for "forever"
Name = Path.GetFileName(path), // Name of the file
Scheme = scheme, // The HybridStore scheme
Size = fs.Length, // The Size of the File
CheckSum = MD5Hash.CreateString(fs), // Md5 of the File
Data = fs // The Data, don't forget to reset the stream, after calculation the checksum
});
Console.WriteLine("Written: {0}",r.Id);
Environment.ExitCode = (int)r.Id;
return r.Id;
}
}
/// <summary>
/// Delete a file, in case it's protected (return 0 false, means that there was nothing found!)
/// </summary>
/// <param name="argList"></param>
private void Delete(List<string> argList)
{
var hsId = GetParam(argList, "/hsId" , 0L);
var ep = GetParam(argList, "/endPoint" , "BasicHttpBinding_IHybridStore");
HS.IHybridStore hs = new HS.HybridStoreClient(ep);
try
{
var r = hs.DeleteFile(tenant, hsId);
Console.WriteLine("Deleted: {0} -> {1}",hsId,r);
}
catch(System.ServiceModel.FaultException<HsCmdSample.HS.HybridStoreFault> ex)
{
if(ex.Detail.ErrorCode == HS.HybridStoreStatus.FileIsProtected)
Console.WriteLine("Deleted: {0} -> File is still protected!",hsId);
else
Console.WriteLine("Deleted: {0} -> other error {1}",hsId, ex);
}
}
/// <summary>
/// Read the file, remember to close the stream!
/// </summary>
/// <param name="argList"></param>
private void Read(List<string> argList)
{
var hsId = GetParam(argList, "/hsId" , 0L);
var path = GetParam(argList, "/path" , "");
var ep = GetParam(argList, "/endPoint" , "BasicHttpBinding_IHybridStore");
if(!Directory.Exists(path))
Directory.CreateDirectory(path);
HS.IHybridStore hs = new HS.HybridStoreClient(ep);
var r = hs.ReadFile(new HS.HsReadRequest()
{
ClientId = tenant,
Flags = 0,
Id = hsId
});
// the stream should be closed as quick as possible
using(var strm = r.Data)
using(var fs = new FileStream(Path.Combine(path,r.Name), FileMode.Create,FileAccess.Write) )
{
Console.WriteLine("Read of {0}", hsId);
Console.WriteLine(" Name -> {0}",r.Name);
Console.WriteLine(" Size -> {0}",r.Size);
Console.WriteLine(" Ret.Date -> {0}",r.RetentionDate);
Console.WriteLine(" Scheme -> {0}",r.Scheme);
Console.WriteLine(" Saved -> {0}",fs.Name);
Console.WriteLine(" CheckSum -> {0} from archive",r.CheckSum);
strm.CopyTo(fs);
}
using(var fs = new FileStream(Path.Combine(path,r.Name), FileMode.Open,FileAccess.Read) )
{
Console.WriteLine(" CheckSum -> {0} after saving",MD5Hash.CreateString(fs));
}
}
/// <summary>
/// Read only a part of the file, remember to close the stream!
/// </summary>
/// <param name="argList"></param>
private void ReadPart(List<string> argList)
{
var hsId = GetParam(argList, "/hsId" , 0L);
var pos = GetParam(argList, "/pos" , 0L);
var len = GetParam(argList, "/len" ,10L);
var ep = GetParam(argList, "/endPoint" , "BasicHttpBinding_IHybridStore");
HS.IHybridStore hs = new HS.HybridStoreClient(ep);
var r = hs.ReadFileRange(new HS.HsReadRangeRequest
{
ClientId = tenant,
Length = len,
Position = pos,
Flags = 0,
Id = hsId
});
// the stream should be closed as quick as possible
using(StreamReader strm = new StreamReader(r.Data,Encoding.ASCII))
{
var data = strm.ReadToEnd();
Console.WriteLine("Read Part of {0}, from {1} to {2}", hsId, pos, len);
Console.WriteLine(" Size(Part): {0}", r.RangeSize);
Console.WriteLine(" Size(All) : {0}", r.Size);
Console.WriteLine(" Data: '{0}'", data);
}
}
}
class MD5Hash
{
static public Byte[] Create(Stream strm)
{
using(System.Security.Cryptography.MD5 md5 = System.Security.Cryptography.MD5.Create())
{
strm.Position = 0;
var md5Hash = md5.ComputeHash(strm);
strm.Position = 0;
return md5Hash;
}
}
public static String CreateString(Stream strm)
{
var hash = Create(strm);
return ToHexString(hash);
}
public static String ToHexString(Byte [] data)
{
StringBuilder sb = new StringBuilder(data.Length*2);
for(int i=0;i<data.Length;++i)
{
sb.Append(data[i].ToString("X2"));
}
return sb.ToString();
}
}
}