SFTP for Amazon S3 with ArcESB
Amazon S3, or Simple Storage Service, is a low-cost, cloud-based object storage service that offers users virtually unlimited storage through affordable, pay-for-what-you-need pricing. The economics, always-on availability, and flexible nature of S3 storage have led organizations to rely on S3 for every kind of storage you can imagine - from point-in-time backups to collecting sensor data and everything in between.
Many organizations seek to leverage the simplicity and security of SFTP as a simple file transfer mechanism to pipe data from enterprise applications into Amazon S3.
Like thousands of organizations and teams across the globe, you can employ S3 as a central repository to store all your data in one place for a range of initiatives:
- Back up your data and retain historical versions of your data
- Share files across the organization through a cloud platform within the AWS ecosystem
- Store audit logs
- Support data governance
- Provide Analytics data access from tools like Amazon QuickSight
- Process distributed queries from business intelligence software like Hadoop or Databricks
- And others
ArcESB: Affordable SFTP to S3 Integration at Scale
SFTP is arguably the most common secure file transfer protocol in use today. Based on strong SSH encryption, SFTP provides a simple channel for securing the transfer of files between systems. What's more, SFTP is almost universally available. Dating back to the late 1990's, SFTP capabilities are commonly available on Linux, Unix and Mac systems, and client applications are easily accessible on modern Windows systems as well.
With widespread availability and robust security, IT administrators have been scripting and automating file transfers through SFTP for years. It's natural that they would want to support the same interface for pushing files to Amazon S3. Unfortunately, it's not that simple. As S3 is technically object storage instead of file storage, the protocols for interacting with S3 are different than working with traditional file systems. So what is the solution?
Amazon offers paid SFTP to S3 integration through their AWS Transfer Family service, however the SFTP interface is an additional service with both hourly and data surcharges. This can make the AWS SFTP service terribly expensive for always on or high-volume file transfers.
The good news is you can avoid these fees and cut your file transfer costs with an intermediary managed file transfer service like ArcESB.
With ArcESB, you can easily automate the SFTP-S3 process and handle unlimited file transfers without scaling costs. Anything that can connect to SFTP can connect to ArcESB, and ArcESB will seamlessly push files into S3. Additionally, the advanced mapping and translation capabilities of ArcESB enable in-flight data processing and data transformations which can be invaluable when moving data to long-term storage like S3.
Build Intelligent SFTP Workflows and Transformations with ArcESB
In addition to its affordability, ArcESB provides enhanced data transformation and automation capabilities to help you streamline and manage your SFTP-S3 file transfers. As a hub for managed file transfer, ArcESB can also serve as your home for all your file transfers to S3, SQL Server, and every other database system of note.
Automation
By providing conditional logic, drag-and-drop integration, webhooks, and API management, ArcESB makes it easy to seamlessly automate and schedule SFTP-to-S3 workflows.
Notifications and Monitoring
ArcESB's notifications and monitoring tools automatically keep you in the loop when any obstacles arise during the file transfer process. You can set custom email notifications so your system admin can respond in real-time to any disruptions.
Automatic Retries
ArcESB supports automatic retries so temporary network outages don't cause headaches. ArcESB only reports a file as transferred once delivery is guaranteed.
Custom Business Logic
Quickly bring intelligence and process automation to your business workflows. ArcESB provides an intuitive visual interface so non-technical users can quickly create file transfers data flows through our drag & drop, no-code workflow designer. You can also leverage advanced developer features to simplify even the most complex workflows. Get the flexibility you need without needless complexity.
Data Transformations, Query, Lookup, Merge
ArcESB provides you easy-to-use tools so you can move specific subsets of your data into S3 and even transform them from one file format into another along the way. With ArcESB, you can use a webhook or API request to query data from any underlying data source or database, Lookup data in a given document, use conditional logic to apply a data transformation, then move it to S3. You can even merge information from multiple tables, data sources, or lines of a document and then apply conditional logic before transferring it into S3.
Connectors for Every Destination
Amazon S3 is far from the only place you need to move your files. Leverage our connectors to create robust workflows, automate business processes, and more. Get started with connectors for:
- Box
- Dropbox
- Google Drive & Google Storage
- OneDrive
- Oracle DB
- SQL Server
- S3
- And More
How to Use ArcESB to Transfer Files from SFTP to S3
Setting up an automated file transfer from SFTP to S3 with ArcESB consists of five simple steps.
1. Set Up an SFTP Server in ArcESB
Start by hosting an SFTP Server that remote clients can use to send files to ArcESB for processing. To get this server up and running, simply use the ArcESB Profiles page to configure necessary attributes, such as the port, server certificate, root directory, and logs.
2. Set User Credentials for Connecting to the SFTP Server
Set user credentials that SFTP clients can use to connect to the SFTP server. Drag the SFTP Server Connector to the ArcESB Flows Canvas and use the settings panel to configure user credentials for one remote user. In addition, use this settings panel to specify authentication modes and set up send & receive subfolders; ArcESB will then route files to the flow automatically.
3. Set Up the SFTP Client
Now it's time to point an SFTP Client to the SFTP Server. Simply drop an SFTP Client Connector onto the Flows Canvas, configure the server & port, and enter the user credentials you created in the previous step. Specify the path to the folder/subfolder where the server expects to receive files. The system will automatically prompt you to test the connection.
4. Select and Set Up an Amazon S3 Connector
Drag and drop an S3 connector to your flow on the Flows Canvas. Enter your S3 access credentials and specify the bucket or folder/subfolder where you want ArcESB to upload your files.
5. Test the Flow
Your SFTP to S3 workflow is now complete. Test the configuration using the Test Files option, which allows you to trace files through the data flow.
For more information on how to set up the flow from SFTP to S3, watch our webinar How to Integrate Managed File Transfer and Cloud Storage.
Download a 30-Day Free Trial of ArcESB
Download ArcESB to start automating your SFTP to S3 file transfer today. You can even run ArcESB direclty inside AWS to fit seamlessly into your AWS ecosystem.