1 December 2020
Demystifying AWS S3 uploads in Ruby on Rails
by Steve Loar
Amazon Web Services include S3 (Simple Storage System) which Amazon describes as “an object storage service” that provides “scalability, availability, security and performance”. While there are many types of objects that can be stored on S3, in this article we will focus on storing a JSON file that will be shared between two physically separated systems. On one side is a consumer-facing website taking orders from customers. On the other side is a 3rd party fulfillment warehouse which is shipping those orders. An API might be a preferred interface between these companies’ systems, however for this example, the fulfillment company does not offer that, so we are uploading JSON files to their AWS S3 account.
Accessing Amazon S3 programmatically is becoming more commonplace, and if you have not already done this, the documentation can seem overwhelming to digest. In this post, I hope to provide some insight from my experience to help you quickly get up to speed on the AWS S3 skill set when using Ruby on Rails.
Before jumping into the code, though, you should take a look at the documentation for S3. A high-level overview can be found here: Amazon S3 FAQs. This is not a conventional starting point, but this FAQ is well organized, and scanning through it will cut the time you need to understand enough S3 to start working with it. We are not covering how to sign up for AWS, how to select the class of S3, and set up a bucket. You will need to do these before starting to write any code. I suggest that you follow Amazon’s S3 setup documentation that will lead you through the setup process.
For this post, you just need to come away from this introductory research with the understanding that Amazon S3 is a system of buckets that contain objects. In more familiar terms, this translates to buckets being like folders, and objects being like files. Furthermore, the buckets have globally unique names that compose a URL to allow access to the objects via HTTP, just like an API endpoint. Uploading the objects will be done by us programmatically using an account that is set up on AWS for such access. Amazon describes that here. When you create this account, be sure to capture the access key and secret key.
I created the example code using ruby version 2.7.1 and rails version 6.0.3. You will want to use something similar to follow along. Amazon has provided a gem for us to access S3. Add the gem to your Gemfile as shown in the following line.
gem 'aws-sdk-s3', '~> 1.78'
As previously mentioned, our example app will be an order handling process. We do not want our customers waiting for the warehouse to get the order, so we are going to do this processing asynchronously using something like SideKiq. Such a job is shown below. It is fairly simple code logic: get the order information, build a document object containing the order information, and send that to AWS S3. We are focusing on the S3 interaction in these examples, so I am not covering Sidekiq or any functionality that might take us off this subject.
class SendOrderJob < ApplicationJob
queue_as :send_orders
def perform(order_id)
order = fetch_order(order_id)
document = build_document(order)
send_to_s3(document, order_id)
end
private
def send_to_s3(document, order_id)
ShipmentOrder::SendShipmentOrderDocument
.new(
document: document,
file_name: file_name(order_id),
aws_client: aws_client
).send_document
end
def aws_client
@aws_client ||= ShipmentOrder::AwsClient.new()
end
def filename(order_id)
#create a filename using the order id
end
def fetch_order(order_id)
# fetch the order information
end
def build_document(order)
# format the order information
end
end
In the above code we are getting the aws_client information needed to talk to S3 from a class whose code is shown below. All of the interactions with S3 will be done via this ShipmentOrder::AwsClient class.
module ShipmentOrder
class AwsClient
def initialize
config_data = get_config_data #from a secrets file
end
def s3_client
@s3_client ||= Aws::S3::Client.new(
region: 'us-east-1',
credentials: credentials
)
end
def s3_bucket
resource = Aws::S3::Resource.new(client: s3_client)
resource.bucket('my_bucket_name')
end
private
def credentials
Aws::Credentials.new(
config_data.access_key,
config_data.secret_key
)
end
end
end
The key take away from the above code is that we are creating an Aws::S3::Client object using our access key and secret key that are in an Aws::Credentials object. These keys need to match the keys you generated when you created the account on AWS that will access the AWS S3 bucket. We will use the Client object to connect to AWS. Finally, we have the bucket as an AWS::S3::Resource object that we will use to put our shipment order documents into.
Now we have all the pieces to push a file to AWS S3. We will do that in the code below which we instantiated and made a call to in the SendOrderJob shown earlier.
The code below is fairly straightforward. We use the aws_client to send the order document to S3 referencing the bucket resource that it is to be placed in. Once the put_object call has completed we will get back a response object from AWS which contains an etag element if we are successful. Anything else is considered an error and our code should alert someone who can take action.
module ShipmentOrder
class SendShipmentOrderDocument
attr_reader :aws_client, :document, :file_name
def initialize(document:, file_name:, aws_client:)
@document = document.to_s
@file_name = file_name
@aws_client = aws_client
end
def send_document
response = aws_client.s3_client.put_object(
bucket: aws_client.s3_bucket,
key: file_name,
body: document,
content_type: 'text/json'
)
if response.etag
return true
else
return false
end
end
end
end
To test this, we need to write a test like the one in the following code. In this example, we are setting up some fake data for our filename and document. Then we set up a mock on our s3_client so that an instance double is returned with the behavior we want. Then in the test we set the behavior of the put_object method to always return an etag value of ‘ok’. While that does not truly test the functionality of AWS S3 receiving a document, that’s OK, as we really want to check that the code on our side is always working. You should also have separate tests around creating the filename and the document, as well.
require 'rails_helper'
RSpec.describe ShipmentOrder::SendShipmentOrderDocument, type: :service do
let(:filename) { 'file_order_id.json' }
let(:document) { 'my_document' }
let(:sender) do
ShipmentOrder::SendShipmentOrderDocument.new(
document: document,
file_name: filename,
aws_client: aws_client
)
end
let(:aws_client) { ShipmentOrder::AwsClient.new }
let(:s3_client) { instance_double(Aws::S3::Client) }
before do
allow(aws_client).to receive(:s3_client).and_return(s3_client)
end
describe '#send_document' do
it 'calls AWS to put a document in the S3 Bucket' do
allow(s3_client).to receive(:put_object).with(
bucket: aws_client.s3_bucket,
key: filename,
content_type: 'text/json',
body: document
).and_return({ etag: 'ok' })
response = sender.send_document
expect(response[:etag]).to eq('ok')
end
end
end
In this article, we have created code to upload a file to AWS S3 and have written tests to cover that code. While there is a lot to learn about working with S3, hopefully, you are now up to speed on uploading files. From here, you should be able to figure out reading files, via get_object and deleting files with delete_object. I hope this article will encourage you to incorporate AWS S3 in your Ruby on Rails app.
