Storage Architecture

Local-first data persistence with IndexedDB and OPFS

Opal Editor uses a sophisticated local-first storage architecture that keeps all your data on your device with zero backend dependencies. The storage system provides multiple storage backends optimized for different use cases.

Storage Backends

Opal Editor implements three primary storage backends, each with specific advantages:

IndexedDB Storage

Best for: Maximum browser compatibility and general use

IndexedDB storage uses the Lightning FS library to provide a virtual file system backed by IndexedDB, the standard browser database API.

// From IndexedDbDisk.ts
class IndexedDbDisk extends Disk {
  constructor(guid: string, indexCache?: TreeDirRootJType) {
    const lightningFs = new LightningFs();
    const fs = lightningFs.promises;
    // Creates a virtual filesystem in IndexedDB
  }
}

Key features:

• Works in all modern browsers
• No permissions required
• Stores data as key-value pairs
• Supports files up to browser limits (typically 50MB-100MB+)
• Persists across browser sessions

Technical details:

• Uses @isomorphic-git/lightning-fs for file system operations
• Data stored in browser’s IndexedDB under workspace GUID
• Asynchronous operations with Promise-based API
• Automatic garbage collection and compaction

OPFS Storage (Origin Private File System)

Best for: Performance and large files

OPFS is a newer browser API that provides direct file system access with better performance characteristics than IndexedDB.

// From OpFsDisk.ts
class OpFsDisk extends Disk {
  constructor(guid: string) {
    // Gets native file system handle
    const rootDir = await navigator.storage.getDirectory();
    const patchedOPFS = new PatchedOPFS(rootDir);
    const fs = new OPFSNamespacedFs(patchedOPFS, `/${guid}`);
  }
}

Key features:

• Native file system performance
• Better suited for large files and workspaces
• Lower memory overhead
• Direct disk access (outside main thread)
• Isolated per-origin storage

Technical details:

• Requires browser support (Chrome 102+, Edge 102+, Safari 15.2+)
• Namespaced under workspace GUID for isolation
• Supports synchronous operations in workers
• Automatic persistence without quota prompts

Memory Storage

Best for: Temporary workspaces and testing

Memory-based storage keeps data in RAM, useful for temporary workspaces that don’t need persistence.

// From MemDisk.ts
class MemDisk extends Disk {
  // Volatile storage that clears on page reload
}

Storage Architecture

Disk Abstraction

All storage backends implement a common Disk interface, allowing the editor to work with any storage type seamlessly:

abstract class Disk {
  // File operations
  async readFile(path: AbsPath): Promise<string | Uint8Array>
  async writeFile(path: AbsPath, contents: string | Uint8Array): Promise<void>
  async removeFile(path: AbsPath): Promise<void>
  async renameFile(oldPath: AbsPath, newPath: AbsPath): Promise<void>
  
  // Directory operations
  async newDir(path: AbsPath): Promise<AbsPath>
  async copyDir(oldPath: AbsPath, newPath: AbsPath): Promise<void>
  
  // File tree indexing
  async triggerIndex(): Promise<FileTree>
  latestIndexListener(callback: Function): void
}

File Tree Indexing

Opal maintains an in-memory file tree index for fast navigation:

// Indexes the entire workspace file structure
await disk.fileTreeIndex();

// Get all files
const allFiles = disk.fileTree.all();

// Filter by type
const markdownFiles = allFiles.filter(node => node.isMarkdownFile());
const images = allFiles.filter(node => node.isImage());

The file tree is cached in IndexedDB and loaded on workspace initialization for instant access.

Data Persistence

Opal uses Dexie.js to manage structured data in IndexedDB:

class ClientIndexedDb extends Dexie {
  workspaces!: EntityTable<WorkspaceRecord, "guid">;
  disks!: EntityTable<DiskRecord, "guid">;
  builds!: EntityTable<BuildRecord, "guid">;
  deployments!: EntityTable<DeployRecord, "guid">;
  historyDocs!: EntityTable<HistoryDAO, "edit_id">;
}

Stored metadata:

• Workspace configurations
• Disk type and settings
• Build and deployment history
• Document edit history
• Remote authentication tokens

Storage Events

The storage system emits events for file changes, enabling reactive updates:

// Listen for file changes
disk.writeIndexListener(() => {
  console.log('File tree updated');
});

// Listen for specific file changes
disk.outsideWriteListener('/README.md', (contents) => {
  console.log('README updated:', contents);
});

// Listen for file creation
disk.createListener(({ filePaths }) => {
  console.log('Files created:', filePaths);
});

Performance Optimizations

Mutex-based Concurrency

All file operations are protected by mutexes to prevent race conditions:

// File operations acquire mutex automatically
await disk.writeFile('/example.md', 'content');

Lazy Loading

The file tree index is cached but file contents are loaded on-demand:

// Index loads metadata only
await disk.init();

// Content loaded when needed
const content = await disk.readFile('/large-file.md');

Batched Operations

Multiple file operations can be batched for better performance:

// Create multiple files atomically
await disk.newFiles([
  ['/file1.md', 'content 1'],
  ['/file2.md', 'content 2'],
  ['/file3.md', 'content 3']
]);

Storage Limits

IndexedDB

• Typically 50MB minimum per origin
• Can request up to 60% of total disk space
• Subject to browser quota management

OPFS

• Typically larger quota than IndexedDB
• Better suited for large workspaces
• Persistent storage without prompts

Opal automatically selects the best available storage backend. OPFS is preferred when available, falling back to IndexedDB for maximum compatibility.

Best Practices

Choose the right storage type

• Use OPFS for large workspaces with many images
• Use IndexedDB for maximum compatibility
• Use Memory storage for temporary workspaces

Monitor storage usage

Check available storage periodically:

const estimate = await navigator.storage.estimate();
console.log(`Used: ${estimate.usage} / ${estimate.quota}`);

Handle quota errors gracefully

Storage operations may fail when quota is exceeded:

try {
  await disk.writeFile(path, largeContent);
} catch (error) {
  if (error.code === 'QuotaExceededError') {
    // Prompt user to free space
  }
}

Advanced Usage

Custom Disk Implementation

You can implement custom storage backends by extending the Disk class:

class CustomDisk extends Disk {
  async readFile(path: AbsPath) {
    // Custom read implementation
  }
  
  async writeFile(path: AbsPath, contents: string | Uint8Array) {
    // Custom write implementation
  }
}

Cross-Disk Operations

Move files between different storage backends:

// Copy entire workspace to different storage
await sourceDisk.copyDiskToDisk(targetDisk);

Virtual Files

Create temporary virtual files that don’t persist:

disk.addVirtualFile({
  type: 'file',
  basename: 'temp.md',
  virtualContent: async () => 'Generated content'
});