Data Service

Data Asset Management System - Architecture Overview

This system provides a secure and scalable way for users to store, manage, and share their data assets within the platform. It leverages multiple interconnected canisters on the Internet Computer to distribute responsibilities and ensure efficient operation.

1. Central Orchestration: The Data Asset Service

At the heart of the system is the Data Asset Service. Think of it as the main controller or traffic director. When a user wants to upload, share, or manage a data asset via the Frontend, the request goes to this service first. It's responsible for:

• Understanding the request (upload, share, delete, etc.).

• Coordinating with other specialized canisters to fulfill the request.

• Managing the overall lifecycle and metadata routing for assets.

• Initiating the creation of new storage units (shards) when needed.

2. Specialized Storage Units (Shards)

To handle large amounts of data and user activity, the system uses different types of specialized storage canisters called shards:

• Data Asset Shards: These store the metadata about each asset – things like its name, description, owner, and importantly, who has permission to access it (Access Control Lists).

• Data Storage Shards: These store the actual content (the file data itself), likely in encrypted chunks. They handle the heavy lifting of data uploads and downloads.

• Shared Activity Shards: These act as logbooks, recording when and with whom assets have been shared, providing an audit trail.

The Data Asset Service directs requests to the correct shard based on the specific asset or user involved.

3. Supporting Infrastructure

Several other canisters provide essential background services:

• Identity Manager: Verifies the user making the request, ensuring they are who they say they are.

• Subscription Manager: Checks if the user has enough storage quota (based on their subscription plan) before allowing uploads or updates. It also tracks how much storage is being used.

• vetkd system api: A system service used for advanced cryptographic operations, potentially enabling secure, time-limited, or conditional access to shared data without revealing master keys.

• IC Management Canister: A fundamental IC service used by the Data Asset Service to programmatically create/update new shard canisters when the existing ones reach capacity.

4. Typical Workflow (Example: Uploading a File)

• The user initiates an upload from the Frontend.

• The request hits the Data Asset Service.

• The Service checks the user's identity (Identity Manager) and storage quota (Subscription Manager).

• If checks pass, the Service tells the appropriate Data Asset Shard to record the new file's metadata and permissions.

• The Service tells the appropriate Data Storage Shard to expect data chunks from the user.

• The Service informs the Frontend which Storage Shard to send the file data to.

• The Frontend uploads the file data directly to the designated Data Storage Shard in chunks.

• The Data Storage Shard assembles the chunks, stores the file, and informs the Subscription Manager about the storage space used.

5. Scalability Through Sharding

The use of separate shards for metadata, data, and activity logs is key to the system's scalability. As more users store more data, the Data Asset Service can automatically create new shard canisters. This distributes the storage load and computational work across many canisters, preventing any single canister from becoming a bottleneck and allowing the system to grow horizontally.