Upload & Download From the Cloud: S3, Google Drive, and More

Arc simplifies the process of uploading and downloading from secure cloud storage platforms like Amazon S3, Google Drive, OneDrive, Box, Dropbox, and more. This article serves as a guide for configuring automatic uploads and downloads from the cloud with Arc.

• Create a Connector
• Establish a Connection
  • OAuth Authentication
• Configure Remote Folders
• Configure Automation
• Next Steps

Create a Connector

Arc has dedicated connectors for many different cloud storage platforms. To begin integrating with a particular platform, find the appropriate connector in the toolbar on the left of the Flows page and drag it into the flow canvas to create an instance of the connector:

This guide will use Amazon S3 as an example cloud storage platform, but the same principles apply regardless of the specific cloud storage service you want to use.

This connector will handle both uploading and/or downloading from the cloud, depending on how you configure it.

Establish a Connection

The first step in configuring our new S3 Connector is to establish a working connection with the cloud storage service.

For most connections, the only thing that needs to be configured to establish a connection is a set of credentials that grant access to a cloud storage account.

For S3 specifically, these credentials come in the form of an Access Key and Secret Key:

Some cloud storage connectors will have a blue Connect button, indicating that this cloud storage provider requires OAuth authentication. This process is described below.

OAuth Authentication

This section can be skipped for cloud storage providers that do not use OAuth: S3, Azure Blob, and Wasabi.

Using OAuth authentication requires a bit of extra work when establishing the initial connection. However, this applies only to the first time you connect; all successive connections are authenticated behind-the-scenes with no user interaction required.

First, you need to generate two OAuth values: a Client ID and a Client Secret. These values are generated using the console/portal/GUI for the service you are trying to access; for example, if you want to connect to Google Drive, you'll need to generate these values within the Google developer's console.

Once you have the Client ID and Client Secret values, you can plug these into the connector settings in Arc:

Then, clicking the Connect button will take you to a portal where you can enter user credentials directly in the cloud storage platform you want to access.

After authenticating in this portal, the connector is granted an access token (and a refresh token) that it can use in future connection attempts; this manual process is only required once.

Configure Remote Folders

Each connector targets a specific folder on the cloud storage platform to upload files to and a specific folder to download files from.

Some cloud storage platforms, such as S3, also have a 'bucket' concept that groups these folders together. First, configure the bucket that should contain the relevant files and folders (if necessary).

Configuring the remote folders is done in the Upload and Download sections of the Settings tab. The Remote Path field in these sections should be set to the folder path (on the cloud service) where files should be uploaded to (if applicable) and downloaded from (if applicable).

Note: S3 uses a pseudo-folder system with file prefixes, so this Remote Path field is called Prefix in the S3 Connector.

These connectors support additional download options, like filtering files that are downloaded and caching file details to ensure files are not downloaded more than once. You can ignore these options for our simple automated setup.

Configure Automation

With the current configuration steps, the connector is able to communicate with the cloud storage platform and knows where files should be uploaded or downloaded. Next, you'll configure the Automation settings to define precisely how the data flows.

In the Automation tab, there are checkboxes for enabling Upload and Download automation separately. This is how you tell the connector whether its job is to pull files down, push files out, or both.

When Upload automation is enabled, the connector will upload files to the cloud, including any files routed to the connector via the blue flow arrows in Arc's Flows canvas as well as any files added to its Input/Send folder (either via the Input tab or by placing files directly in the folder via the filesystem).

When Download automation is enabled, the connector will poll the configured folder in the cloud at a specified interval to find files to download. Files that are found and downloaded will be placed in the Output/Receive folder for the connector. If the connector is connected to others via blue flow arrows, then these files will not remain in this Output/Receive folder but will instead be passed along to the next connector in the Flow.

If you're ever not sure where the Input/Send or Output/Receive folders are on the filesystem, you can check the Input and Output tabs of the connector, which have these filepaths at the top of the tab panel.

Next Steps

If you haven't already, you can get started with Arc for free by downloading the application here. Arc comes with a free, fully-functional 30-day trial license, so you can confirm for yourself that the FTP and SFTP automation is as simple as described in this article.

Here in the knowledge base, you can also find articles on Getting ArcESB Up and Running, Hosting Your Own FTP and SFTP Server, and more.

If you need help working on your custom Arc solution, please don't hesitate to reach out to our technical team at support@arcesb.com.