Skip to main content

Vector File System

Introduction​

Shinkai's Vector File System (VectorFS) offers a file-system like experience while providing full Vector Search capabilities. Vector Search is the underlying technology which allows LLMs to find and use needed information from data sources at scale, which is classically called RAG (retrieval augmented generation).

The VectorFS does not replace classical operating-system level file systems, but lives within the Shinkai node, acting as both the native storage solution for all content used with AI in the Shinkai node, while also allowing external apps to integrate it into their stack like a next-generation VectorDB. Internally in the VectorFS data embeddings are stored as Vector Resources, which you can learn more about in this blog post.

Additionally, the VectorFS fully incorporates a global permission system (based on Shinkai identities) thereby securely allowing sharing AI data embeddings with anyone, including with gated whitelists which require delegation/payments (coming in the future).

Understanding the Basics​

At the heart of VectorFS is a hierarchical structure based on paths, where / represents the root directory of a profile's VectorFS in the Shinkai node. This structure allows for an intuitive organization of data, mirroring the familiar file system hierarchy found in operating systems.

FSEntries: FSFolder and FSItem​

Within the VectorFS, every non-root path contains an FSEntry, which can be either an FSFolder or an FSItem. This distinction is crucial for understanding how VectorFS organizes and manages its contents:

  • FSFolders: These are directory-like structures that can exist at any depth within the VectorFS, starting from the root. An FSFolder can contain other FSFolders (subdirectories) and FSItems (files), allowing for a nested, tree-like organization of data. Folders in VectorFS are not just simple containers; they also hold metadata such as creation, modification, and access times, providing a rich context for the data they contain.

  • FSItems: Representing the "files" within the VectorFS, FSItems are containers for a single Vector Resource + an optional set of Source Files the Vector Resource was created from (pdfs, docs, txts, etc). FSItems enables the VectorFS to tie original file formats with their vector representations, enhancing the system's utility for a wide range of applications. All FSItems have embeddings, meaning that they are always able to be found via Vector Search in the VectorFS (and be Vector Searched internally as well).

Understanding Permissions in The VectorFS​

Every path into the VectorFS that holds an FSEntry has a PathPermission specified for it. The PathPermission consists of:

  • ReadPermission: Determines who can read the contents of a path. It can be one of the following:

    • Private: Only the profile owner has read access.
    • NodeProfiles: Specific profiles on the same node have read access.
    • Whitelist: Only identities explicitly listed in the whitelist have read access.
    • Public: Anyone on the Shinkai Network can read the contents.

  • WritePermission: Determines who can modify the contents of a path.

    • Private: Only the profile owner has read access.
    • NodeProfiles: Specific profiles on the same node have read access.
    • Whitelist: Only identities explicitly listed in the whitelist have read access.

Whitelist Permissions​

When either the read or write permission for a path is set to Whitelist, then whether a user (Shinkai Name) has access to the path is determined by the whitelist held inside the PathPermission.

In the whitelist each ShinkaiName can be set to one of Read, Write, or ReadWrite. This mechanism enables the filesystem to grant or restrict access based on the specific needs of each path.

Folder permissions are also naturally hierarchical as one would expect. This means that if a user is whitelisted for /folder1/ then they will be automatically whitelisted for an item that is held at /folder1/my_item. Do note, /folder1/my_item needs to have at least one of its read/write permissions set to Whitelist in order for the whitelist on /folder1/ to apply to my_item.