Mark/docker_backup
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Backend Architecture: Multi-Provider Storage System #9

New issue
Closed
opened 2026-03-21 21:20:22 +00:00 by Mark · 1 comment
Mark commented 2026-03-21 21:20:22 +00:00
Owner
Copy link

Overview

This issue tracks the implementation of the backend architecture designed to support multiple storage providers for Docker volume backups.

Architecture Document

Full specification: docs/BACKEND_ARCHITECTURE.md

Storage Providers to Support

Phase 1 - Essential

  • Local filesystem - Basic local storage
  • S3 / S3-compatible - AWS, MinIO, Wasabi, DigitalOcean Spaces
  • Rsync over SSH - Remote servers via rsync

Phase 2 - Enterprise/Cloud

  • WebDAV - Nextcloud, ownCloud, generic WebDAV
  • Google Cloud Storage - GCP native storage
  • SFTP/FTP - Remote file servers

Phase 3 - Consumer Cloud

  • Google Drive - Personal/business GDrive
  • Dropbox - Personal/business Dropbox
  • Box.net - Enterprise Box storage
  • Azure Blob Storage - Microsoft Azure

Technical Approach

Storage Provider Interface

All providers implement a common interface:

interface StorageProvider {
  initialize(config): Promise<void>
  testConnection(): Promise<ConnectionResult>
  upload(options): Promise<UploadResult>
  download(options): Promise<DownloadResult>
  delete(path): Promise<void>
  list(prefix): Promise<StorageObject[]>
}

Factory Pattern

const providers = {
  s3: S3Provider,
  gcs: GCSProvider,
  gdrive: GDriveProvider,
  dropbox: DropboxProvider,
  box: BoxProvider,
  webdav: WebDAVProvider,
  rsync: RsyncProvider,
  azure: AzureProvider,
  sftp: SFTPProvider,
  local: LocalProvider,
}

Backend Tech Stack

  • Runtime: Node.js + TypeScript
  • Framework: Fastify (REST API)
  • Database: PostgreSQL + Prisma ORM
  • Queue: BullMQ (Redis) for job scheduling
  • WebSocket: Socket.io for real-time updates
  • Docker: dockerode for Docker API integration

Related Issues

  • #1: Replace mock API layer
  • #6: Implement actual backup and restore operations

Implementation Notes

Provider-Specific Authentication

Provider Auth Method
S3 Access Key + Secret
GCS Service Account JSON
Google Drive OAuth2 (refresh token)
Dropbox OAuth2 (access token)
Box OAuth2 or JWT
WebDAV Username + Password
Rsync SSH Key
Azure Connection String
SFTP SSH Key or Password

NPM Dependencies

# Core
npm install fastify @fastify/websocket @fastify/cors
npm install prisma @prisma/client bullmq ioredis

# Storage providers
npm install @aws-sdk/client-s3 @aws-sdk/lib-storage
npm install @google-cloud/storage
npm install googleapis  # For Google Drive
npm install dropbox
npm install box-node-sdk
npm install webdav
npm install @azure/storage-blob
npm install ssh2-sftp-client
## Overview This issue tracks the implementation of the backend architecture designed to support multiple storage providers for Docker volume backups. ## Architecture Document Full specification: `docs/BACKEND_ARCHITECTURE.md` ## Storage Providers to Support ### Phase 1 - Essential - [ ] **Local filesystem** - Basic local storage - [ ] **S3 / S3-compatible** - AWS, MinIO, Wasabi, DigitalOcean Spaces - [ ] **Rsync over SSH** - Remote servers via rsync ### Phase 2 - Enterprise/Cloud - [ ] **WebDAV** - Nextcloud, ownCloud, generic WebDAV - [ ] **Google Cloud Storage** - GCP native storage - [ ] **SFTP/FTP** - Remote file servers ### Phase 3 - Consumer Cloud - [ ] **Google Drive** - Personal/business GDrive - [ ] **Dropbox** - Personal/business Dropbox - [ ] **Box.net** - Enterprise Box storage - [ ] **Azure Blob Storage** - Microsoft Azure ## Technical Approach ### Storage Provider Interface All providers implement a common interface: ```typescript interface StorageProvider { initialize(config): Promise<void> testConnection(): Promise<ConnectionResult> upload(options): Promise<UploadResult> download(options): Promise<DownloadResult> delete(path): Promise<void> list(prefix): Promise<StorageObject[]> } ``` ### Factory Pattern ```typescript const providers = { s3: S3Provider, gcs: GCSProvider, gdrive: GDriveProvider, dropbox: DropboxProvider, box: BoxProvider, webdav: WebDAVProvider, rsync: RsyncProvider, azure: AzureProvider, sftp: SFTPProvider, local: LocalProvider, } ``` ## Backend Tech Stack - **Runtime**: Node.js + TypeScript - **Framework**: Fastify (REST API) - **Database**: PostgreSQL + Prisma ORM - **Queue**: BullMQ (Redis) for job scheduling - **WebSocket**: Socket.io for real-time updates - **Docker**: dockerode for Docker API integration ## Related Issues - #1: Replace mock API layer - #6: Implement actual backup and restore operations ## Implementation Notes ### Provider-Specific Authentication | Provider | Auth Method | |----------|-------------| | S3 | Access Key + Secret | | GCS | Service Account JSON | | Google Drive | OAuth2 (refresh token) | | Dropbox | OAuth2 (access token) | | Box | OAuth2 or JWT | | WebDAV | Username + Password | | Rsync | SSH Key | | Azure | Connection String | | SFTP | SSH Key or Password | ### NPM Dependencies ```bash # Core npm install fastify @fastify/websocket @fastify/cors npm install prisma @prisma/client bullmq ioredis # Storage providers npm install @aws-sdk/client-s3 @aws-sdk/lib-storage npm install @google-cloud/storage npm install googleapis # For Google Drive npm install dropbox npm install box-node-sdk npm install webdav npm install @azure/storage-blob npm install ssh2-sftp-client ```
Mark commented 2026-03-22 01:25:46 +00:00
Author
Owner
Copy link

✅ All Storage Providers Implemented

All 9 requested storage providers are now complete:

Phase 1 - Essential

  • Local filesystem
  • S3 / S3-compatible (AWS, MinIO, Wasabi, etc.)
  • Rsync over SSH

Phase 2 - Enterprise/Cloud

  • WebDAV (Nextcloud, ownCloud)
  • Google Cloud Storage
  • SFTP

Phase 3 - Consumer Cloud

  • Google Drive
  • Dropbox
  • Box.net
  • Azure Blob Storage

All providers implement the StorageProvider interface with:

  • Initialize/test connection
  • Upload/download with progress tracking
  • Delete/list/exists/getMetadata

Factory pattern allows easy extension: createProvider(type)

## ✅ All Storage Providers Implemented All 9 requested storage providers are now complete: ### Phase 1 - Essential - [x] Local filesystem - [x] S3 / S3-compatible (AWS, MinIO, Wasabi, etc.) - [x] Rsync over SSH ### Phase 2 - Enterprise/Cloud - [x] WebDAV (Nextcloud, ownCloud) - [x] Google Cloud Storage - [x] SFTP ### Phase 3 - Consumer Cloud - [x] Google Drive - [x] Dropbox - [x] Box.net - [x] Azure Blob Storage All providers implement the `StorageProvider` interface with: - Initialize/test connection - Upload/download with progress tracking - Delete/list/exists/getMetadata Factory pattern allows easy extension: `createProvider(type)`
Mark closed this issue 2026-03-22 01:25:46 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:lint-failed

   
blocked:needs-decision

   
blocked:no-changes

   
blocked:no-procedure

   
blocked:verify-failed

   
bug

Something is not working correctly

  
feature-request

   
in-progress

   
needs-decompose

Ready to be decomposed into sub-issues

  
needs-triage

   
ops-task

Operations task — install, configure, deploy, API calls

  
quickfix

   
security/high

High severity security issue

  
security/low

Low severity security issue

  
security/medium

Medium severity security issue

  
test-coverage

Test coverage review / gap report

  
triage-approved

Approved by triage pipeline

No labels
blocked:gate-fetch
blocked:gate-fetch
blocked:gate-fetch
blocked:gate-fetch
blocked:lint-failed
blocked:needs-decision
blocked:no-changes
blocked:no-procedure
blocked:verify-failed
bug
feature-request
in-progress
needs-decompose
needs-triage
ops-task
quickfix
security/high
security/low
security/medium
test-coverage
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/docker_backup#9
No description provided.