This is case #2 in coming up with solutions to internal problems to become a more valuable developer.

Case #2: API Documentation

At my current job, we are contracted to develop an internal frontend application for a fortune 500 company. The employees of this company must interact with several applications to do their jobs. The application I lead the frontend development for aims to alleviate that; to have one place for the employees to go to do their work. The application will then communicate with those different applications behind the scenes via api requests. The backend is composed of several different expressjs applications that are all brought together via a webserver. We, the frontend team, just need to interact with a single http port to make our requests against.

Example:

• https://<current-domain>:8080/api/foo -> get's proxied to the "foo" service
• https://<current-domain>:8080/api/bar -> get's proxied to the "bar" service

Each of these api services are developed by different teams within our client company.

Emulation

Developing within this company's network requires VPN access and that comes with a whole host of challenges. Instead, we've set up a mock expressjs server to develop against locally. This allows us to not have any dependency on their backend development environment. All we need is api documentation. We don't need to wait for these apis to actually be developed and deployed. We can just take the api documentation they've provided and create mock endpoints and responses within our express app to develop against. This is really nice because we have complete control over everything and can develop against various responses for any given endpoint(think error responses).

Receiving api documentation

We were/are being given "api documentation" in pretty much any and all formats:

• emails
• .txt documents
• Word documents
• power points
• slack messages

Because the backend consists of several teams maintaining their own api services, there is no "standard" way of giving us api documentations. We have attempted to get this client to provide api documentation in a specific format but because we're just a vendor, we really have no control over them. We just take what we're given and put it in our express app. Because of this, and the fact that they do not maintain any sort of central api documentation, our mock express app has become the only central "api documentation" in existence for their api services.

The problems

At the time, our mock express app contained ~700 unique endpoints. There were a few problems I noticed:

Discarding request payloads

When adding mock POST/PUT endpoints, we were just making them repond with the expected success response, totally disregarding the request payloads.

app.post('/thing-service/thing', (req, res) => {
  return res.json({message: "Created thing successfully!"})
})

After we developed the frontend to send the expected payload based on the api docs we were sent, that request documentation was essentially gone. We'd have to search for when and where those api docs were sent if we ever needed to view them again.

Inability to search

The next problem was that the only "documentation" we had was our express app code which was very hard to search just due to the size it had grown and the inability to exclude search terms from the hardcode responses. We didn't really have a good way to search on urls. If we searched for "thing", that could show up in url patterns or, more commonly, in the hardcoded respones making it really difficult to find anything.

Clumsy and tedious

If we wanted to test various respones from a specific endpoint we'd have to go find it and then hardcode in a new response. This is a very manual and tedious task. Often we would end up losing the original response, which was still valid, because we'd overwrite it to test a new response.

The Solution

I wanted a way to:

• keep various potential responses
• keep request POST/PUT payloads
• easily send/receive api documentation
• easily explore all ~700 endpoints
• keep our mock express app

Route files

That's when I took inspiration from Swagger(which I find very confusing to get started).

My first thought was to identify the most relevant pieces of data for any given endpoint:

• url pattern
• http method
• request payloads - if applicable
• query params - if applicable
• responses
  • success responses
  • error responses

Luckily this is all stuff that can be encapsulated in a json file, which I know our backend api counterparts are comfortable with(as opposed to something like yaml).

{
  "title": "Create thing for username",
  "path": "/thing-service/:username/thing",
  "params": {
    "username": {
      "value": "johndoe",
      "help": "The owner's username"
    }
  },
  "method": "POST",
  "payload": {
    "thing": {
      "name": "Foobar",
      "color": "Blue"
    }
  },
  "responses": [
    {
      "status": 200,
      "response": {
        "message": "Thing created successfully!"
      }
    },
    {
      "status": 400,
      "response": {
        "errors": [
          {
            "detail": "Invalid color submitted"
          }
        ]
      }
    }
  ]
}

I could define several of these route files and then write code in my express app to:

• glob a certain directory for these route files
• parse them into javascript objects
• iterate over them and create functioning endpoints for each

import glob from 'glob'
import express from 'express'

const app = express()

app.get('/foo/bar', (req, res) => {
  return res.json({foo: "bar"})
})

// all the other ~700 routes

const routes = glob
  .sync('api-routes/**/*.json')
  .map(require)

for (const route of routes) {
  app[route.method](route.path, (req, res) => {
    const firstResponse = route.responses[0]
    return res.json(firstResponse)
  })
}

const PORT = process.env.PORT || 8080
app.listen(PORT, function () {
  console.log('Dev Express server running at localhost:' + PORT)
})

This was great! I instantly solved several problems. I communicated to my backend counterparts that I would strongly prefer them to document their apis in these json route files. All I had to do was get the routes files and dump them in my api-routes directory and they'd instantly be avaible to develop against!

UI for the route files

I still wanted to solve searching and make reading these route files easier though. That's when I decided this documentation needed a user interface. The initial idea was pretty simple:

• add an api endpoint to the mock server that
  • get's a list of existing endpoints on the express app instance(method, and url pattern)
  • get all routes from files
  • combine those to come up with a single list of known routes with as much information as possible
  • respond with that list of routes
• create a react app to consume that endpoint and render documentation for the routes
• serve that react app on a mock server endpoint

This whole thing seemed like it would shedding light on my existing express app so it seemed appropriate to call it "headlamp". Link to repo here. It started to take shape. Because I had a complete list of all the routes defined in the frontend, it was trivial to add searching by url pattern and http method.

Clicking on an endpoint would expand to show everything known about the endpoint...along with a few other added features...

Some features I added:

• ability to actually submit the request and see the response in the UI and in the browser's network tab
• ability to toggle different responses directly from the ui
  • we can now toggle the error response on and test that in our application UI
• ability to submit and activate adhoc json responses
  • makes it incredibly easy to debug things if we know a certain production response is causing an issue
• ability to search url patterns by omitting the route param names if you can't remember those
  • searching "/thing-service/:/thing" would find /thing-service/:username/thing
• attempts to find where this endpoint is being used in the source code (see "References" section in the screenshot above)
• shows the location of the route file if this endpoint was created from one
• responds with headers to show
  • what route file is responsible for this request
  • a link to where you can view this request's UI documentation

(Disregard the fact that the link points to a different port than what's in the browser url. This is just a result of running headlamp in development mode to hack on it. It points to the same port when used normally.)

• ability to use .js route files and create dynamic responses. See third code snippet here.
• ability to expand the references to view the acual source code of where this endpoint is used

• ability to upload HAR files if you need to emulate a specific sequence of responses actually encountered in production

Result

This case of scratching our own itch has been immensely helpful and made our development experience so much nicer and more productive than interacting with an express app manually.

Our client now asks us how they can run this to view their own api's documentation.

Is the headlamp code the best, most organized, test-driven developed code out there? Absolutely not, nor does it need to be. This was an internal tool developed to make our own lives easier. It is serving its purpose extremely well for us. I haven't needed to touch it for almost 2 years now and we've used it every day since its inception.

Looking for ways to keep your work interesting while at the same time improving your productivity? Take the time to assess your current development challenges. View them as opportunities to come up with effective solutions. I thoroughly believe this is a solid approach to proving your worth and advancing your career.

Fun fact: At the time of writing this our mock express app now has 1298 unique endpoints.

Link to headlamp