For example, for Amazon S3, the upload is free. You only pay for the storage.So you have the benefit of:

  • Extremely fast access to local SeaweedFS Filer
  • Near-Real-Time Backup to Amazon S3 with zero-cost upload network traffic.

Architecture

  • Message Queue can be: Kafka, AWS SQS, Google Cloud Platform Pub/Sub.
  • Cloud Sink can be: AWS S3, Google Cloud Storage, Microsoft Azure, Backblaze B2.

Configuration

  • Configure notification. use "" to see the notification section.
  1. [source.filer]
  2. enabled = true
  3. grpcAddress = "localhost:18888"
  4. directory = "/buckets"    # all files under this directory tree are replicated
  6. [sink.s3]
  7. # read credentials doc at https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/sessions.html
  8. # default loads credentials from the shared credentials file (~/.aws/credentials).
  9. aws_access_key_id     = ""     # if empty, loads from the shared credentials file (~/.aws/credentials).
  11. region = "us-west-1"
  12. bucket = "your_bucket_name"    # an existing bucket
  13. directory = "/"                 # destination directory
  • Start the Kafka.
  • Start the replication. "weed filer.replicate"

Replicate existing files