Rails Direct Upload to AWS S3 from React Native

react rails and amazon s3 logos

I recently took on the task of allowing a user of a React Native app I’m helping build upload a custom profile picture. It sounded like a relatively simple task when I was estimating it in our sprint planning. However, I still allowed myself some grace since I’d never done such a thing before and put 8 hours on it. Little did I know what was to come.

See, I knew our backend was running Ruby on Rails (RoR) and I knew that Active Storage is now the thing but I didn’t realize the issues I would run into when I threw Amazon Web Services (AWS) S3 into the mix. I had heard good things bout Active Storage though I hadn’t worked with it any, I know RoR well enough to know that the things they add are intentional and typically well thought out, and I also knew my experience with S3 was while the configuration could be somewhat complex when it comes to IAM roles and things once it was running the way you wanted it should be pretty easy to use. Especially for something that was going to be public.

Early on in my work on this task I was informed by the back end engineer that Active Storage had this pretty neat way of allowing the client application to send files directly to S3 and just sending a reference string to the Rails server. This is preferred because instead of sending the data from the client to the Rails server to Amazon it goes directly from the client to Amazon. Bypassing one step speeds everything up and also saves some load on the server. I thought to myself this was pretty cool. We at Airship had done this before with a web app with solid results. I had that code to reference and base my work off of.

Where things start to go wrong…

This is where things start to splinter. I start to digest the code from the web app we created:

import axios from "axios";
import SparkMD5 from "spark-md5";
const getUploadInfo = async (file) => {
  const checksum = await createFileChecksum(file);
  return axios.post(
    `${process.env.BASE_URL}/rails/active_storage/direct_uploads`,
    {
      blob: {
        filename: file.name,
        content_type: file.type,
        byte_size: file.size,
        checksum: checksum
      }
    }
  );
};
export const createFileChecksum = async (file) => {
  return new Promise((resolve, reject) => {
    const chunkSize = 2097152; // 2MB
    const chunkCount = Math.ceil(file.size / chunkSize);
    var chunkIndex = 0;
    const md5Buffer = new SparkMD5.ArrayBuffer();
    const fileReader = new FileReader();
    const readNextChunk = () => {
      if (chunkIndex < chunkCount || (chunkIndex === 0 && chunkCount === 0)) {
        const start = chunkIndex * chunkSize;
        const end = Math.min(start + chunkSize, file.size);
        const fileSlice =
          File.prototype.slice ||
          File.prototype.mozSlice ||
          File.prototype.webkitSlice;
        const bytes = fileSlice.call(file, start, end);
        fileReader.readAsArrayBuffer(bytes);
        chunkIndex++;
        return true;
      } else {
        return false;
      }
    };
    fileReader.addEventListener("load", event => {
      md5Buffer.append(event.target.result);
      if (!readNextChunk()) {
        const binaryDigest = md5Buffer.end(true);
        const base64digest = btoa(binaryDigest);
        resolve(base64digest);
      }
    });
    fileReader.addEventListener("error", () =>
      reject(`Error reading ${file.name}`)
    );
    readNextChunk();
  });
};
export const uploadFile = async (file) => {
  const uploadInfo = await getUploadInfo(file);
  await axios.put(uploadInfo.data.direct_upload.url, file, {
    headers: uploadInfo.data.direct_upload.headers
  });
  return uploadInfo.data.signed_id;
};

Real quick, getUploadInfo() sends the relevant info about the file the the Rails back end and returns what’s needed to direct upload to S3. createFileChecksum() is used by getUploadInfo() to calculate the base64 encoded md5 checksum of the file being sent. While Amazon does not require this Rails does. Lastly, uploadFile() uploads the file to AWS and then returns the signed_id that is then sent to Rails so it can associate that file with whatever it is in the back end.

I later realized most of this code came from somewhere else, maybe even the @rails/activestorage package. I found similar code living in a <a href="https://github.com/rails/rails/blob/master/activestorage/app/javascript/activestorage/file_checksum.js">file_checksum.js</a> file in the Rails repository on GitHub. No matter the source of the code, there was an issue. I don’t have access to the FileReader api on mobile. I’m working in React Native and not a browser. So now the search commences to doing this exact same thing in React Native.

All the things that didn’t work

Actually, I’m not going to bore you with everything that didn’t work. I honestly don’t think you care. You probably Googled how to do this and it’s NOWHERE TO BE FOUND on the internet. Yet, this direct upload has been a feature in Rails for a while. You might have even landed on the Rails issue Make ActiveStorage work for API only apps and a comment there:

For those on react native, I was able to get direct uploads working using rn-fetch-blob for md5 hashing (which is output in hex), then converting its hex output into base64 using buffer for calculating the checksum. To lookup the content_type, I used react-native-mime-types, and last but not least, used rn-fetch-blob again for calculating the size. Then, just follow the communication guidelines pointed out by @cbothner, and if the files are big, use rn-fetch-blob for efficiently uploading the file.

Samsinite

So, I tried to follow the above thread and I couldn’t get it to work. Granted, that comment is almost 6 months old and in JavaScript time that’s a lifetime ago. The main issue I ran into is I could not for the life of me get the checksum to match up with what Amazon calculated on their side. I kept getting responses of “The Content-MD5 you specified was invalid”. I tried MANY ways of generating the md5 checksum and they all ended up with the same Content-MD5 message being returned from AWS.

So here’s how I ended up getting it to work (why you’re really here):

import axios from "axios";
import Config from "react-native-config";
import RNFetchBlob from "rn-fetch-blob";
import AWS from "aws-sdk/dist/aws-sdk-react-native";
import { Platform } from "react-native";
import { Buffer } from "buffer";
const { fs } = RNFetchBlob;

AWS.config.update({
  accessKeyId: Config.AWS_ACCESS_KEY_ID,
  region: Config.AWS_REGION,
  secretAccessKey: Config.AWS_SECRET_ACCESS_KEY
});
const s3 = new AWS.S3({ apiVersion: "2006-03-01" });

const getUploadInfo = async (fileInfo, file) => {
  const params = {
    Bucket: Config.AWS_BUCKET,
    ContentType: fileInfo.type,
    Key: fileInfo.fileName,
    Body: file
  };
  const psUrl = s3.getSignedUrl("putObject", params);
  const checksum = unescape(psUrl.split("&")[1].split("=")[1]);

  return axios.post(
    `${Config.API_UPLOAD_HOST}/rails/active_storage/direct_uploads`,
    {
      blob: {
        filename: fileInfo.fileName,
        content_type: fileInfo.type,
        byte_size: fileInfo.fileSize,
        checksum: checksum
      }
    }
  );
};

export const uploadFile = async (fileInfo) => {
  const uri =
    Platform.OS === "ios" ? fileInfo.uri.replace("file://", "") : fileInfo.uri;
  const file = await fs
    .readFile(uri, "base64")
    .then(data => new Buffer(data, "base64"));

  const uploadInfo = await getUploadInfo(fileInfo, file);
  const { headers, url } = uploadInfo.data.direct_upload;

  try {
    await axios.put(url, file, { headers: { ...headers } });
  } catch (e) {
    throw e;
  }

  return uploadInfo.data.signed_id;
};

This is definitely not the most elegant solution. I haven’t refactored it at all yet either. However, it works. In the world of code that means something. So what in the world is going on here? I’ll walk through it, although, I’ll jump around the file some. First, I setup the aws-sdk and a new s3 instance. I’m using react-native-config to manage environment variables here. I initially did this to see if I could get the signed_id I needed by just bypassing Rails and uploading directly to AWS, that didn’t work. However, what I noticed when I generated a pre-signed URL for uploading via the aws-sdk was that the URL contained and md5 checksum!

Back to the code

Okay, the code, walk through it, here we go. I call uploadFile() in the response from react-native-image-picker on my screen component. That’s where the fileInfo argument comes from. I then get the proper URI based on the OS, and read the file with rn-fetch-blob. I turn that data into a Buffer because the aws-sdk on accepts certain types of files when creating a pre-signed URL. I then pass the fileInfo and the file along the getUploadInfo(). getUploadInfo() then creates a pre-signed URL using the s3 instance we setup earlier and does some hacky string manipulation (needs a refactor) to acquire the checksum. Now, I can use that checksum (which Amazon code created) to get the direct upload URL and headers from Rails. Lastly, I upload the file to AWS and return the signed_id which I send along to Rails elsewhere in my code.

Ultimately, this was a pretty frustrating problem to fight against. However, it felt so good when I uploaded a file and saw the user profile image change. I actually got up and ran around my home office with my hands in the air rejoicing. I’m also stoked that I can share this solution and see how others might improve on what I did or figure out better ways to go about this. I’m not convinced this is the best solution to this problem, however, it’s a solution that works.

From my yarn.lock:
– react-native v0.60.5
– react-native-image-picker v1.1.0
– rn-fetch-blob v0.10.16
– aws-sdk v2.532.0

Supercluster with @react-native-mapbox-gl/maps

During a recent project in my work at Airship I had to stop using the built in cluster functionality that <a href="https://github.com/react-native-mapbox-gl/maps">@react-native-mapbox-gl/maps</a> provides and utilize Supercluster instead. The reason is we need access to the points that make up the clusters. We had some items that never broke out of their clusters because they had the same exact longitude & latitude combination. As well as wanting to show a slide up view of those locations in a list view. What started me down this path was an issue on the deprecated <a href="https://github.com/nitaliano/react-native-mapbox-gl">react-native-mapbox-gl</a> library which shares a lot of functionality with the new library. You can view that issue here. I’m honestly surprised that this functionality isn’t available in the library since it is supported in the Mapbox JS SDK as documented here with the getClusterLeaves() function. I noticed people asking how to do this so when I nailed it down I knew a how-to was coming.

Continue reading “Supercluster with @react-native-mapbox-gl/maps”

Portfolio Site How-To For New Developers

portfolio

I recently shared my portfolio site with the Free Code Camp Nashville group and got some inquiries into some of the technologies and features I used to build it. So I figured I’d share all aspects of the site and some steps to utilizing the same tools I did.

Should I build my site from scratch?

When I asked this question to the ever helpful NashDev community I received a resounding “No” from Senior devs. This might seem counter intuitive, however, the overall thought process was if you’re just starting out, unless you’re looking to be considered a designer, then using something someone else has already done very well as the base of your portfolio is better than building it yourself. Concentrate on highlighting the things that you are going to be doing in a potential job, not on the overall layout and design of your portfolio site. I decided to go with a template from HTML5 UP for a few reasons. First, they’re FREE as long as you keep the attribution. Second, AJ who creates these amazing templates is based out of Nashville just like me and had in the past connected me with some solid people to have beer/coffee with and discuss my career. Third, it’s a static site. That is, it’s 100% HTML, CSS & JavaScript so I would have many simple options for hosting when I got to that point. Lastly, all of the templates are responsive so if a potential hiring manager clicks through from their phone they’re going to get a great experience.
Continue reading “Portfolio Site How-To For New Developers”

My Learning Hacks

rocks working out

So I wanted to share the hacks that I’m using to upgrade my life some and increase my mental aptitude to crushing this curriculum. There is a certain amount of mental stamina that has to be maintained on a day to day basis to keep up this pace. So… here’s my brain upgrades/biohacks/etc… that keep my moving tip top while doing this bootcamp full time and taking care of kids full time. All with links to the best place to buy them.

First thing is my coffee. I start my day off with my take on, Bulletproof Coffee. I currently use Central American Single Origin Organic whole bean coffee from Aldi. The most important thing for me is my coffee being light to medium roast. Nothing dark. I also like whole bean because coffee really does taste way better if it’s fresh ground. I add to that 1 tbsp of Thrive Market Ghee and 1 tbsp of whatever I have around to up ketones. I prefer Brain Octane Oil because it definitely is nicer on the insides but also use Now Foods MCT Oil or just Coconut Oil if I have neither of the aforementioned. I blend this all up together. This gets me going and curbs hunger for hours. I also pop a couple BioScience Brain Boost and wash them down with my coffee.

I then hopefully do my Gymnastic Bodies Daily Limber routine to get the blood flowing some. I follow that up with some meditation utilizing Calm. I try (but usually fail) to write my daily journal entry utilizing this template I built in Evernote. I very rarely get all of these things done. However, I try to.

When I’m on long coding sessions I supplement in the middle of the day with Four Sigmatic Lion’s Mane which I’ve found can really get me going. I have to ensure I’m stimulated enough mentally or I start getting distracted trying to keep my brain working hard enough. I also diffuse essential oils in my workspace. Specifically Lemon, Rosemary, and Eucalyptus. All three of those are supposed to help with concentration and mental performance. I find the scents definitely keep me alert if not anything else. Something that I recently ran across is Nootrobox Sprint and I’m interested to try it out. I might just order the mini bottle for $15.

That should be it if I remember correctly. If I realize I missed something I’ll update this post.

UPDATE 3/31/17
I completely forgot about how I listen to Brain.fm whenever I’m doing work. It really zones me in more than anything I’ve ever listened to. I used to use Spotify playlists but now this is the ONLY thing I’ll listen to when I need to concentrate.

The Science of Getting Rich – A Classic

the science of getting richEven though The Science of Getting Rich was released over a century ago, there are heaps of ideas inside it if you’re keen to make a seriously better lifestyle for you, your loved ones and the community in general. You acquire what you own because of doing things by a variety of means and the ones that are successful are likely to stick to those reasons and apply them to other endeavors and keep growing rich. People who’ve not worked out the way to use these techniques – and it doesn’t matter how well they perform or how dedicated to their obligations they happen to be, they may never become successful. Wallace Wattles declares in this book that anyone that can follow this formula could definitely grow rich.

So where is This Century-Old Secret?

Being conscious of the correct way to go about getting rich is paramount and the part that is made public by Mr. Wattles, is in your head. The basic beliefs are rather like Steve Pavlina’s concepts in his The Law of Attraction. You are in command over exactly what you do, so for everyone who is continually screwing up and getting things wrong, it’s time to face the reasons explaining why. The causes of an absence of success happen to be within your own unconscious mind. Nonetheless those people that are terribly successful may be seen as being born with this power of effective thinking and they appear to grow into entrepreneurs naturally. The Science of Growing Rich was created in order that everybody could get a positive mind-set, a positive attitude and carry that attitude out into their daily lives to win gigantic success.

The Power of Human Will

Your will is why you choose to do things, and why you do not do things as well . Will is why a large amount of folk leap out of bed before the alarm goes off, whereas others go back to s1eep and dedicate their entire lives being consistently behind. The first kind are full of enthusiasm and expectancies for their day and the second potentially are thinking adversely, unless they’ve already been handed lots of money or won the lotto, those individuals do not end up being rich.

Download Your Free Copy of The Science of Getting Rich Here

The Science of Getting Rich can be discovered here today positively free in .pdf format. This book has empowered lots of folks, as well as Rebecca Fine who’s applied the basic tenets behind the book and made a training course that has been brought to just about every country across the world, inspiring individuals to make an effort and change their method of thinking to positive.

A massive number of individuals have experienced extraordinary benefits as a result of adopting the advice contained in the Science of Getting Rich.

I quickly learned that not only is there a science but a system. The best system to apply The Science of Getting Rich is Biz Builder University. What better place to learn a science then in a university right?

It is easy to become overwhelmed these days with the large number of methods and plans there available about achieving personal success, but this book might have easily fallen into obscurity if it did not contain such a great deal of amazing information. Download your copy of The Science of Getting Rich here today and see why this book continues to be republished after a century.

Seth Alexander

Two people you need in your professional life

My take on Seth Godin’s recent post: Two people you might need in your professional life. I consider them necessary instead of optional.

An agonist. An agonist causes action to happen. They are more than a muse. They force you to take action and that is something I believe most of us need to create our best work. Not that the agonist will get us creating our best work all the time but in the end we will all be better because of them.

And of course, a procrastinatrix. A person whose sole function is to make sure you get it done now, rather than later.

In an online business world where we are our own bosses, these two people are more important than ever. Do not trick yourself into believe you can do these jobs yourself. If you think you do both of these jobs already I challenge you to spend the time and the money to find a person to do these jobs for you. Then compare the before and after results. Neither of these jobs is very hard, however, they are very hard to do yourself.

Seth Alexander