Generating code and the webpack loader

In many frameworks (like Gatsby and Next.js) you don’t necessarily need to know that webpack is there. You’re abstracted away and that’s a great thing. However, sometimes you might need to do something more custom than importing JavaScript or CSS into a project. When that need arises, writing your own webpack loader can be a useful tool.

In some scenarios you might want to take some type of input as a string and transform it to some other type that can be processed or used. This can even mean importing an HTML file and using it to generate metadata dynamically.

Let’s consider a real world use case

A project I work on, Tachyons, has hundreds of components that are written as HTML files. With that HTML we have a few needs before it’s ready to become a page. We want to figure out what Tachyons CSS is being used, we want to crunch the CSS Stats, and figure out what Tachyons modules are being used. Our first approach to this was rather hacky and required custom build tooling to take the HTML file and transform it to another file.

In our new (WIP) version of the docs we’re using React and a webpack-based toolkit so I wanted to tuck away this HTML transformation as an implementation detail. The best way to do this is to create a custom webpack loader for the component HTML files in the project.

Essentially, I want to:

  • load the HTML
  • parse out the metadata (Tachyons modules, CSS Stats, etc.)
  • generate JavaScript that can be consumed

Desired usage

When considering writing a custom loader it’s often helpful to figure out how you want to consume the output.

For this case, the way I wanted to import/use the HTML files looks something like:

import React from 'react'

import ComponentPage from './ComponentPage'
import { css, metadata, html } from '../components/definition-lists/inline.html'

export default () => <ComponentPage css={css} metadata={metadata} html={html} />

This means that I’d need to turn the HTML string that the loader receives into some JavaScript code that would look something like:

export const css = 'SOME STRING'
export const metadata = {}
export const html = ORIGINAL_CONTENT_AS_STRING

Code generation

In the loader, we receive the orignal HTML string, get the data we want and the generate exports.

const fs = require('fs')
const path = require('path')
const getClasses = require('get-classes-from-html')
const getStats = require('./get-stats')

module.exports = async function(content) {
  const callback = this.async()

  const srcPath = path.join(
    process.cwd(),
    'node_modules/tachyons/src/tachyons.css'
  )
  const srcCss = fs.readFileSync(srcPath, 'utf8')

  const classes = getClasses(content).map(c => `.${c}`)

  const stats = await getStats(srcCss, classes)
  const modules = await getModules(classes)
  const meta = { stats, modules }

  const code = `
    export const css = ${JSON.stringify(css)}
    export const html = ${JSON.stringify(content)}
    export const meta = ${JSON.stringify(meta)}
  `

  return callback(null, code)
}

Since we’re then returning JavaScript, we can then use babel-loader afterwards and everything will just work:

{
  test: /\.html$/,
  use: [
    'babel-loader',
    'components-loader'
  ]
}

Now, from the project we can:

import { css, meta, html } from '../some/file.html'

Loaders all the way down

MDX in particular heavily uses loaders (and Parcel plugins) as its interface for being included in a project. This allows it to transpile MDX into a pure JSX string that can then be used by different frameworks. For its React implementation, it wraps the JSX with a functional component, imports the needed dependencies, and specifies it’s custom pragma.

const { getOptions } = require('loader-utils')
const mdx = require('@mdx-js/mdx')

module.exports = async function(content) {
  const callback = this.async()
  const options = Object.assign({}, getOptions(this), {
    filepath: this.resourcePath
  })
  let result

  try {
    result = await mdx(content, options)
  } catch (err) {
    return callback(err)
  }

  const code = `/* @jsx mdx */
  import React from 'react'
  import mdx from '@mdx-js/mdx/create-element'
  ${result}
  `

  return callback(null, code)
}

This approach allows it to also be used in a Vue project by using a Vue specific loader as well:

const { getOptions } = require('loader-utils')
const mdx = require('@mdx-js/mdx')

module.exports = async function(content) {
  const callback = this.async()
  const options = Object.assign({}, getOptions(this), {
    filepath: this.resourcePath
  })

  let result
  try {
    result = await mdx(content, { ...options, skipExport: true })
  } catch (err) {
    return callback(err)
  }

  const code = `import {mdx} from '@mdx-js/vue'
let h
${result}
export default {
  name: 'Mdx',
  render(vueCreateElement) {
    h = mdx.bind({vueCreateElement})
    return MDXContent({})
  }
}`

  return callback(null, code)
}

This is nearly identical to the React loader aside from the fact that it uses slightly different syntax for writing the code for a component.

Aside from the syntactic wrapping, it doesn’t do much else. So we encourage MDX users that have custom needs to layer in a loader that is executed before MDX. It’s a clean interface to operate on, and most importantly, keeps MDX core as lightweight as possible.

Here’s an example of a custom frontmatter loader with MDX:

const matter = require('gray-matter')
const stringifyObject = require('stringify-object')

module.exports = async function(src) {
  const callback = this.async()
  const { content, data } = matter(src)

  const code = `export const frontMatter = ${stringifyObject(data)}
${content}
  `
  return callback(null, code)
}

This is a custom loader that should be executed before the built in MDX loader so that the new functionality is mixed in properly. MDX will ignore the frontmatter in a file because remark will parse it, but the nodes in the AST won’t be processed. However, since this loader runs first and will turn it into an export, this will be properly transpiled by MDX. Pretty neat.

Conclusion

Custom webpack loaders are a powerful tool for handling special cases where you need some custom need for all files with a particular file type. You can use them to read in a different file type and generate JavaScript code that can be imported and used like a normal JS file.