Sync data from S3

You use the source S3 connector in Tiger Cloud to synchronize CSV and Parquet files from an S3 bucket to your Tiger Cloud service in real time. The connector runs continuously, enabling you to leverage Tiger Cloud as your analytics database with data constantly synced from S3. This lets you take full advantage of Tiger Cloud's real-time analytics capabilities without having to develop or manage custom ETL solutions between S3 and Tiger Cloud.

Expand image

You can use the source S3 connector to synchronize your existing and new data. Here's what the connector can do:

• Sync data from an S3 bucket instance to a Tiger Cloud service:

  • Use glob patterns to identify the objects to sync.
  • Watch an S3 bucket for new files and import them automatically. It runs on a configurable schedule and tracks processed files.
  • Important: The connector processes files in lexicographical order. It uses the name of the last file processed as a marker and fetches only files later in the alphabet in subsequent queries. Files added with names earlier in the alphabet than the marker are skipped and never synced. For example, if you add the file Bob when the marker is at Elephant, Bob is never processed.
  • For large backlogs, check every minute until caught up.

• Sync data from multiple file formats:

  • CSV: check for compression in GZ and ZIP format, then process using timescaledb-parallel-copy.
  • Parquet: convert to CSV, then process using timescaledb-parallel-copy.

• The source S3 connector offers an option to enable a hypertable during the file-to-table schema mapping setup. You can enable columnstore and continuous aggregates through the SQL editor once the connector has started running.

• The connector offers a default 1-minute polling interval. This means that Tiger Cloud checks the S3 source every minute for new data. You can customize this interval by setting up a cron expression.

The source S3 connector continuously imports data from an Amazon S3 bucket into your database. It monitors your S3 bucket for new files matching a specified pattern and automatically imports them into your designated database table.

Note: the connector currently only syncs existing and new files—it does not support updating or deleting records based on updates and deletes from S3 to tables in a Tiger Cloud service.

#livesync in the TigerData Community

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

• Ensure access to a standard Amazon S3 bucket containing your data files.

  Directory buckets are not supported.

• Configure access credentials for the S3 bucket.
  The following credentials are supported:

  • IAM Role.

    • Configure the trust policy. Set the:

      • Principal: arn:aws:iam::142548018081:role/timescale-s3-connections.

      • ExternalID: set to the Tiger Cloud project and Tiger Cloud service ID of the service you are syncing to in the format <projectId>/<serviceId>.

        This is to avoid the confused deputy problem.

    • Give the following access permissions:

      • s3:GetObject.
      • s3:ListBucket.

  • Public anonymous user.

• File naming: Files must follow lexicographical ordering conventions. Files with names that sort earlier than already-processed files are permanently skipped. Example: if file_2024_01_15.csv has been processed, a file named file_2024_01_10.csv added later will never be synced. Recommended naming patterns: timestamps (for example, YYYY-MM-DD-HHMMSS), sequential numbers with fixed padding (for example, file_00001, file_00002).

• CSV:

  • Maximum file size: 1 GB

    To increase this limit, contact sales@tigerdata.com

  • Maximum row size: 2 MB

  • Supported compressed formats:

    • GZ
    • ZIP

  • Advanced settings:

    • Delimiter: the default character is ,, you can choose a different delimiter
    • Skip header: skip the first row if your file has headers

• Parquet:

  • Maximum file size: 1 GB
  • Maximum row size: 2 MB

• Sync iteration:

  To prevent system overload, the connector tracks up to 100 files for each sync iteration. Additional checks only fill empty queue slots.

To sync data from your S3 bucket to your Tiger Cloud service using Tiger Cloud Console:

And that is it, you are using the source S3 connector to synchronize all the data, or specific files, from an S3 bucket to your Tiger Cloud service in real time.