File uploads with Remix

This article features old apis. Read the latest guide here.

As of writing this there is an official guide on file uploads in Remix but it's basically a placeholder until they iron out some APIs and have time to revisit the guide.

This is my unofficial guide.

<Form />

To use the Remix <Form /> component with files you must use the encType prop and update the method to "post" or "put":

<Form method='post' encType='multipart/form-data'>
    <input type='file' name='my-file' />
    <button>Submit</button>
</Form>

Now, in your route's action you can pull that File out of the form data.

export async function action({request}: ActionFuntionArgs) {
    const formData = await request.formData()
    const file = formData.get('my-file')
    if (!(file instanceof File)) throw new Error('Invalid file')

    // process file

    return {
        message: "File received"
    }
}

The uploaded file(s) will be stored in memory and you'll need to decide what to do with them.

Base64

You could convert the upload into a base64 value:

const formData = await request.formData()
const file = formData.get('my-file')
if (!(file instanceof File)) throw new Error('Invalid file')
const arrayBuffer = await file.arrayBuffer()
const buffer = Buffer.from(arrayBuffer)
const base64 = buffer.toString('base64')

But most likely you'll want to store them somewhere...

Local file system

Luckily, Remix has a handy utility for storing uploads locally. Here's what that looks like:

import {
    unstable_createFileUploadHandler,
    unstable_parseMultipartFormData
} from '@remix-run/node'

export async function action({request}: ActionFuntionArgs) {
    const uploadHandler = unstable_createFileUploadHandler({
        // where to save the file
        directory: '/app/storage',
        // use the incoming filename instead of creating a new name
        file: (args) => args.filename
    })
    const formData = await unstable_parseMultipartFormData(request, uploadHandler)

    return {
        message: "File received"
    }
}

unstable_parseMultipartFormData

You might have noticed that request.formData() went away and was replaced with this unstable_parseMultipartFormData function. This is a Remix utility for iteratively deciding where uploads go.

  • request.formData() - read everything into memory and then decide what to do with file data
  • unstable_parseMultipartFormData - decide what to do with file data as it's being read

This allows users to upload big files without the application storing those entirely in memory all at once. Instead, you can stream those files into their destinations.

The unstable_parseMultipartFormData function will iterate over the form data keys and if the key contains a file, it will hand that off to be processed by the upload handler.

Gotcha

One "gotcha" with parsing the form data this way is that you don't have access to the other form values because the FormData instance hasn't been created yet. For example, say you had another input in the same form for the user to select a category for the file that determines where that file lives on disk of if you need to do some validation on the provided category.

const formData = await unstable_parseMultipartFormData(request, uploadHandler)
const category = formData.get('category')

// Now I just need to valida....oh wait, my file has already been uploaded...

To get around this, you can use the useSubmit hook and append values as search params to be read in the action.

const submit = useSubmit()

return (
    <Form method="post" onSubmit={(e) => {
        const formData = new FormData(e.currentTarget)
        const params = new URLSearchParams({category: formData.get('category')})
        const options = {
            action: `/app/upload?${params}`,
            method: "post",
            encType: "multipart/form-data"
        }
        submit(formData, options)
    }}>
)

And then in your action:

export async function action({request}: ActionFuntionArgs) {
    const url = new URL(request.url)
    const cateogry = url.searchParams.get('category')

    // continue on to use unstable_parseMultipartFormData()
}

Custom upload handlers

Remix allows you to create your very own upload handlers.

const uploadHandler: UploadHandler = async ({ data, filename }) => {

    // handle the data and filename however you need to

    // return File | string | null | undefined
    return "https://example.com/assets/newly-uploaded-file.png";
}
const formData = await unstable_parseMultipartFormData(request, uploadHandler);

// Remix swaps in whatever you return from your upload handler
const uploadUrl = formData.get('my-file')

Composing upload handlers

Remix also comes with a utility for using multiple upload handlers to handle certain uploads differently. The following is lifted directly from the Remix docs.

  const uploadHandler = unstable_composeUploadHandlers(
    // custom upload handler
    async ({ name, contentType, data, filename }) => {
        if (name !== "img") {
            return undefined;
        }
        const uploadedImage = await uploadImageToCloudinary(data);
        return uploadedImage.secure_url;
    },
    // fallback to memory for everything else
    unstable_createMemoryUploadHandler()
);

const formData = await unstable_parseMultipartFormData(
    request,
    uploadHandler
);

In this example, the first handler only applies to the upload coming from the input with the name of "img":

<input type='file' name='img' />

By returning undefined, Remix will try the next upload handler. In this case, it's the unstable_createMemoryUploadHandler.

unstable_createMemoryUploadHandler

Wait, why is there a unstable_createMemoryUploadHandler? Wouldn't that be the same as request.formData()? Yep, almost. This utility does come with the ability to limit the upload size. It defaults to 3MB.

const uploadHandler = unstable_createMemoryUploadHandler({maxPartSize: 5000000}) // 5MB
const formData = await unstable_parseMultipartFormData(
    request,
    uploadHandler
);
const file = formData.get('my-file')

// up to you to handle now

Upload to S3

Here's an example that illustrates efficiently streaming the upload to S3.

import {
  type ActionFunctionArgs,
  json,
  unstable_parseMultipartFormData,
  type UploadHandler,
} from "@remix-run/node";
import { S3Client } from "@aws-sdk/client-s3";
import { env } from "~/env.server";
import { Upload } from "@aws-sdk/lib-storage";
import { Readable } from "stream";

// you probably wouldn't actually create the s3 client here...
const s3 = new S3Client({
  region: env.AWS_REGION,
  credentials: {
    accessKeyId: env.AWS_ACCESS_KEY_ID,
    secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
  },
});

export async function action({ request }: ActionFunctionArgs) {
  const uploadHandler: UploadHandler = async ({
    data,
    filename,
  }) => {
    const readableStream = Readable.from(data);
    const upload = new Upload({
      client: s3,
      params: {
        Bucket: env.AWS_S3_BUCKET,
        Key: filename,
        Body: readableStream,
      },
    });
    await upload.done();
    return filename;
  };

  await unstable_parseMultipartFormData(request, uploadHandler);

  return json({
    success: true,
  });
}

Hope this helps!

Integration testing with MSW dynamic mocks

MSW and cypress/playwright are awesome testing tools. However, things can get tricky when you combine them. The crux of the problem is that your test code is not in the same process that your app code is. They are two discreet processes so you don't have direct access to the msw server instance.

With msw, you typically define the "happy path" responses in a mocks/handlers.ts file and this is used in your mock server. But...what if you want to test an error scenario?

You have a few options.

Dynamic mock scenarios

Basically split up the handlers.ts into a scenarios object and add a query parameter to "activate" a certain scenario. You can see this in the msw docs here.

There are two less-than-desirable traits to this approach though:

  • They're separate from the test code
  • You have to come up with your own arbitrary scenario naming convention

Use a development branch of msw

There is work being done to bridge the gap between msw server and e2e test code. It's just not ready for prime time yet. The setupRemoteServer API allows you to communicate to your server over a web socket connection. The idea is very promising and I'm excited for it to get merged! You can help by funding the maintainer's development efforts.

Development only app endpoint

I came up with a low-tech setupRemoteServer alternative to using network behavior overrides in a dedicated test endpoint. I'm using Remix so I simply created a development-only action route that I can make requests to that will interact with the msw server for me.

Now, I can set up a scenario directly from the e2e test.

The loader of my /app/contacts route makes an api call to fetch data from a http://localhost:6000/people api. The above code allows me to mock that request directly in my test code, keeping it all together.

I can further clean this up by abstracting this into a mock() function.

Hope this helps!

Docker app hosting

Like a ton of other people I was recently inspired by listening to Lex Fridman interview Peter Levels. Hearing him speak about the current state of development made me want to simplify.

I was trying out fly.io for an app that I was working on but it was yet another platform to learn, will probably fall out of fashion, and was kinda pricey for what I wanted to use it for. I have already messed around with:

  • heroku
  • netlify
  • vercel
  • others

And those are great for high availability and horizontal scaling but....I don't have those problems and those platforms can get expensive quick compared to having your own VPS and running your apps on it. Granted, I'm familiar with servers and docker. If you're not then by all means use those services. They're great for that. However, since I have that experience, use a wide range of technologies for different apps, then docker with my VPS makes a lot of sense to me. It also helps that it's relatively easy to find Dockerfiles for various frameworks.

Why Docker?

I prefer to not mess around with server packages and updates if I don't have to. It scares me doing that stuff in production. Using docker means I can easily run apps that need varying versions of php, node, python, etc and I don't have to worry about getting all those pieces in place directly on my server. If I just used php and no framework like Peter then I probably wouldn't bother with docker.

Let's go

I just had to figure out how to come up with a github action workflow to deploy to my server. Here's what I came up with. This does require some server set up(below).

The workflow deletes all but the last three images stored on github and on my server so I'm not paying for those on github and not chewing through disk space on my server.

Server setup

First, create a github user account with docker permissions on your server:

sudo useradd -m github
sudo usermod -aG docker github

And created a ssh key/pair for the github user:

# change to github user
sudo su - github

# generate ssh key/pair
ssh-keygen -t rsa -b 4096 -C "[email protected]"

Next, copy the contents of /home/github/.ssh/id_rsa.pub into /home/github/.ssh/authorized_keys and fix the permissions if necessary:

chmod 600 /home/github/.ssh/authorized_keys

Next create the required repository secrets here:

https://github.com/<your-username>/<repository-name>/settings/secrets/actions

The following secrets should be created:

SERVER_HOST=<your servers hostname or ip>
SERVER_SSH_KEY=<the contents of the private key: /home/github/.ssh/id_rsa>
SERVER_USERNAME=github

The deploy script also assumes your container needs env vars set and those should be created in a /home/github/yourappsname.env file. It also assumes the container uses port 3000 and should forward to the same port on the host.

You can then add nginx in front of it if you've configured a domain's DNS to point to your server. Create an A record if using an ip as the value and the domain as the name. Ex: yourappsname.com.

# /etc/nginx/conf.d/yourappsname.conf
server {
    listen 80;
    server_name yourappsname.com www.yourappsname.com;

    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Reload nginx:

nginx -s reload

And then use certbot to add an ssl certificate:

certbot --nginx -d yourappsname.com -d www.yourappsname.com

Result

Now I have a pretty good template to follow to deploy whatever apps I decide to build and not be charged a crazy amount by hosting providers. The only hosting I'm using is a fixed price Hetzner VPS I can resize if/when I need to.

Next steps

Postgres has been my database of choice but I think I want to simplify to just using sqlite and see how far that gets me. A lot of the things I want to build are hobby projects with potential to make money. I don't want to pay for fancy db hosting for these apps. There are some database providers that have free plans but I haven't come across any that offer free backups and that's something I do want. I plan to use docker volumes to store the sqlite database on my VPS host and then start with cron-based backups and then use litestream if I ever feel the need to go down that path.