Welcome

The StoreWise Decentralized S3 Object Storage Developer Documentation.

The latest gateway release can be found here: https://dl.bintray.com/storewise/s3-gateway/​

Introduction

StoreWise is a secure & distributed object storage layer for a broad range of applications and use cases.

StoreWise is designed for individuals and organizations that require scale-out data storage infrastructure with a high level of security, privacy, control, sovereignty and availability.

StoreWise's object storage service is built to be 100% bit-compatible with Amazon's Simple Storage Service (AWS S3) including Identity and Access Management (IAM) APIs. With this any existing AWS S3-compatible application or gateway device will work seamlessly with StoreWise.

Although StoreWise is S3 compatible, we also work with our technology partners to design and implement custom storage interfaces for specific applications.

The primary purpose of this guide is to provide a in-depth understanding of the StoreWise technology stack and provide a developer quick-start for using the S3 API.

πŸ‘‰Get Started

Can't wait to try it out? πŸ”₯Jump straight to our Get Started tutorial!

Throughout this guide, for the sake of brevity, the StoreWise API functionality is designated as β€œStoreWise” and the AWS S3 API functionality is designed as β€œS3.”

StoreWise - Infrastructure

Due to the distributed nature of StoreWise's architecture, the deployed infrastructure is quite different from that of AWS S3.

The storage infrastructure consists of four main components:

  • Metadata server

  • Client Gateway

  • Host Selection Engine (HSE)

  • Public blockchain

By default the metadata server and HSE is hosted and maintained by StoreWise while the gateway runs on-prem client-side. Technology partners have the ability to run the entire stack on-prem. Please contact [email protected].

πŸ“‘ Metadata Server
☁️ Gateway
πŸ”— Public Blockchain
⛏️ Host Selection Engine (HSE)
πŸ“‘ Metadata Server

The metadata-server is responsible for forming, maintaining and renewing storage contracts on the blockchain. Multiple metadata servers are available across geographic regions.

Features:

  • Handles all complex tasks associated with storage nodes and smart contracts.

  • Can be called for more granular functionality.

  • Access across different regions to decrease latency.

☁️ Gateway

The gateway is a small client-side application that can either be integrated at the application layer with the AWS-S3 SDK, or, accessed through a self-contained local S3 Server. Features:

  • Client Side encryption.

  • Fully self-contained, continues to operate if metadata-server is down.

  • Storage contracts are stored locally.

  • Data is sharded, erasure coded and transferred directly to the storage nodes.

πŸ”— Public Blockchain

The public blockchain is designed for creating and storing (smart) data storage contracts with storage nodes around the globe and ensure no tampering occurs to both the storage nodes and data itself.

Features:

  • Immutable.

  • Highly secure & reliable.

  • Creation of smart storage contracts.

  • No downtime.

⛏️ Host Selection Engine (HSE)

The host (storage node) selection engine is responsible for finding the best storage nodes for the customer's application.

Features:

  • Select only the best nodes for your business application.

  • Filter nodes across a variety of parameters (for example. Latency, throughput, geographic location etc.) and expose these as granular control leavers to the user.

The gateway is designed to operate at 100% availability even if the StoreWise infrastructure suffers an outage. In case of a unforeseen outages, the gateway can continue to Push/Pull data.

Infrastructure schematic.

Workflow;

  1. Developer set's up an account and creates a bucket with policies (e.g. Geo-fence, Storage class etc.)

  2. A request is sent to the Metadata server for storage contract formation.

  3. The Metadata Server sends a request to the host selection engine (HSE) to parse a list with selected storage nodes.

  4. With the whitelist, the blockchain is asked to form smart contracts with the selected storage nodes.

  5. Contracts are encrypted, downloaded and stored by the gateway locally.

  6. The gateway signs the contracts and runs a local S3 Server.

  7. Gateway encrypts the data and uploads it directly to the storage nodes. To ensure resilience data gets erasure coded and mirrored across different nodes (default 3x).

Operation workflow

Operation workflow during all systems OK & during outage.
  1. Nominal operation. Gateway synchronizes telemetry to the metadata-server akin to performance and the state of the contracts. The metadata-server ensures contracts stay healthy, performant and enough funds are present.

  2. During outage. Although telemetry gets sent to the metadata server, the gateway is a completely self-sustaining, self-contained system. In case the metadata server goes down, no critical failures happens and the gateway can continue transferring data. This is possible because the gateway stores all storage contracts locally in addition to a local DB storing all S3 metadata.

  3. Once the metadata server is back up, metadata gets synchronized again and everything is back to normal operation.

TL;DR Web-console (SETUP) ➑ Metadata server ➑host selection engine ➑blockchain➑Metadata Server➑gateway➑Store Data!

Implemented Methods List

Common Operations

Feature

Status

Remark

GET Service

βœ”οΈ

Returns a list of all of the buckets

Bucket Operations

Feature

Status

Remark

GET Bucket

βœ”οΈ

Lists objects of the bucket

GET Bucket accelerate

✘

Returns the Transfer Acceleration state of a bucket

GET Bucket ACL

βœ”οΈ

Returns the bucket ACL

GET Bucket analytics

✘

Returns an analytics configuration for the bucket

GET Bucket CORS

✘

Returns the cross-region resource sharing configuration set for a bucket

GET Bucket encryption

✘

Returns the server-side encryption configuration of a bucket

GET Bucket inventory

✘

Returns an inventory configuration from the bucket

GET Bucket lifecycle

✘

Returns the lifecycle configuration information set on a specified bucket

GET Bucket location

βœ”οΈ

Returns the region where the bucket is

GET Bucket logging

✘

Returns the logging status of a bucket and the permissions users have to view and modify that status

GET Bucket metrics

✘

Returns a metrics configuration from the bucket

GET Bucket notification

✘

Returns the notification configuration of a bucket

GET Bucket Object versions

✘

Lists the metadata about all of the versions of objects in a bucket

GET Bucket policy

✘

Returns the policy of a specified bucket

GET Bucket replication

✘

Returns the replication configuration of a bucket

GET Bucket requestPayment

✘

Returns the request payment configuration of a bucket

GET Bucket tagging

✘

Returns the tag set associated with the bucket

GET Bucket versioning

βœ”οΈ

Returns the versioning state of a bucket

GET Bucket website

✘

Returns the website configuration associated with a bucket

DELETE Bucket

βœ”οΈ

Deletes bucket

DELETE Bucket analytics

✘

Deletes an analytics configuration

DELETE Bucket CORS

✘

Deletes the CORS configuration of a bucket

DELETE Bucket encryption

✘

Deletes the server-side encryption configuration from the bucket

DELETE Bucket inventory

✘

Deletes an inventory configuration from the bucket

DELETE Bucket lifecycle

✘

Deletes the lifecycle configuration from the bucket

DELETE Bucket metrics

✘

Deletes a metrics configuration from the bucket

DELETE Bucket policy

✘

Deletes the policy from the bucket

DELETE Bucket replication

✘

Deletes the replication configuration from the bucket

DELETE Bucket tagging

✘

Deletes a tag set from the specified bucket

DELETE Bucket website

✘

Removes the website configuration for a bucket

HEAD Bucket

✘

Determines if a bucket exists and you have permission to access it

List Bucket Analytics Configurations

✘

Returns a list of analytics configurations for the bucket

List Bucket Inventory Configurations

✘

Returns a list of inventory configurations for the bucket

List Bucket Metrics Configurations

✘

Returns a list of metrics configurations for the bucket

PUT Bucket

βœ”οΈ

Creates bucket

PUT Bucket accelerate

✘

Uses the accelerate subresource to set the Transfer Acceleration state of an existing bucket

PUT Bucket ACL

βœ”οΈ

Configures the Access Control List of a bucket

PUT Bucket analytics

✘

Adds an analytics configuration to the bucket

PUT Bucket CORS

✘

Sets the cors configuration for your bucket

PUT Bucket encryption

✘

Uses the encryption subresource to set the default encryption state of an existing bucket

PUT Bucket inventory

✘

Adds an inventory configuration to the bucket

PUT Bucket lifecycle

✘

Creates a new lifecycle configuration for the bucket or replaces an existing lifecycle configurations

PUT Bucket logging

✘

Set the logging parameters for a bucket and to specify permissions for who can view and modify the logging parameters

PUT Bucket notification

✘

Enables notifications of specified events for a bucket

PUT Bucket policy

✘

Adds or replaces a policy on a bucket

PUT Bucket replication

✘

Creates a replication configuration or replaces an existing one

PUT Bucket requestPayment

✘

Sets the request payment configuration for a bucket

PUT Bucket tagging

✘

Adds a set of tags to a bucket

PUT Bucket versioning

✘

Sets the versioning state of an existing bucket

PUT Bucket website

✘

Set the website configuration for a bucket

Object Operations

Feature

Status

Remark

GET Object

βœ”οΈ

Lists objects

GET Object ACL

βœ”οΈ

Returns the access control list of an object

GET Object tagging

✘

Returns the tag-set of an object

GET Object torrent

✘

Returns torrent files from a bucket

DELETE Multiple Objects

✘

Delete multiple objects from a bucket

DELETE Object

βœ”οΈ

Removes the null version (if there is one) of an object and inserts a delete marker, which becomes the latest version of the object

DELETE Object tagging

✘

Deletes a tag set from the specified object

HEAD Object

✘

Retrieves metadata from an object without returning the object itself

OPTIONS Object

✘

Allows to send a preflight request to trigger an evaluation of the rules that are defined in the CORS configuration

POST Object

✘

Adds an object to a bucket by using HTML forms

POST Object restore

✘

Restore a temporary copy of an archived object

PUT Object

βœ”οΈ

Adds an object to a bucket

PUT Object copy

βœ”οΈ

Copies an object

PUT Object ACL

βœ”οΈ

Adds an ACL to an object

PUT Object tagging

✘

Adds one or several tags to an object

SELECT Object Content

✘

Filters the contents of an object based on a simple Structured Query Language (SQL) statement

Multipart

✘

Initiates, aborts, completes, lists, uploads, uploads copy

The following AWS documents apply to the StoreWise API design:

AWS S3
AWS IAM
AWS S3

AWS S3 capabilities referenced in this Guide are based on API version 2006-03-01, described in the Amazon Simple Storage Service API Reference: http://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf

AWS IAM

AWS IAM capabilities referenced in this Guide are based on API version 2010-05-08, described in the AWS Identity and Access Management API Reference: http://docs.aws.amazon.com/IAM/latest/APIReference/iam-api.pdf​