Node Modules at War: Why CommonJS and ES Modules Can’t Get Along

Interop between them is possible, but it’s a hassle

Published in

Code Red

12 min readAug 5, 2020

Dan Fabulich is a Principal Engineer at Redfin. (We’re hiring!)

In Node 14, there are now two kinds of scripts: there are old-style CommonJS (CJS) scripts and new-style ESM scripts (aka MJS). CJS scripts use require() and module.exports; ESM scripts use import and export.

ESM and CJS are completely different animals. Superficially, ESM looks very similar to CJS, but their implementations couldn’t be more different. One of them is a honey bee, and the other is a murder hornet.

A hornet and a honey bee. One of these is like ESM and the other is like CJS, but I can never remember which one is which. Credit: wikimedia, wikimedia

Calling CJS from ESM and vice versa is possible, but it’s a hassle.

Here are the rules, which I’ll explain in more detail below:

You can’t require() ESM scripts; you can only import ESM scripts, like this: import {foo} from 'foo'
CJS scripts can’t use static import statements like the one above.
CJS scripts can use asynchronous dynamic import() to use ESM, but that's a hassle, compared to synchronous require.
ESM scripts can import CJS scripts, but only by using the “default import” syntax import _ from 'lodash', not the “named import” syntax import {shuffle} from 'lodash', which is a hassle if the CJS script uses named exports. (Except, sometimes, unpredictibly, Node can figure out what you meant!)
ESM scripts can require() CJS scripts, even with named exports, but it’s typically not worth the trouble, because it requires even more boilerplate, and, worst of all, bundlers like Webpack and Rollup don’t/won’t know how to work with ESM scripts that use require().
CJS is the default; you have to opt-in to ESM mode. You can opt-in to ESM mode by renaming your script from .js to .mjs. Alternately, you can set "type": "module" in package.json, and then you can opt-out of ESM by renaming scripts from .js to .cjs. (You can even tweak just an individual subdirectory by putting a one-line {"type": "module"} package.json file in there.)

These rules are painful. Worse, for many users, especially newbies to Node, these rules are incomprehensible. (Fear not, I’ll explain them all here in this article.)

Many observers of the Node ecosystem have speculated that these rules are due to a failure of leadership, or even hostility toward ESM. But, as we’ll see, all of these rules are here for a good reason, which will make it very difficult to break these rules in the future.

I’ll conclude with four guidelines for library authors to follow:

Consider shipping only ESM
If you’re not shipping only ESM, ship CJS only
Test that your CJS works in ESM
If needed, add a thin ESM wrapper for your CJS named exports
Add an exports map to your package.json

It’s all gonna be OK.

Discuss on Reddit
Discuss on Hacker News

Background: What’s CJS? What’s ESM?

Since the dawn of Node, Node modules were written as CommonJS modules. We use require() to import them. When implementing a module for other people to use, we can define exports, either “named exports” by setting module.exports.foo = 'bar' or a “default export” by setting module.exports = 'baz'.

Here’s a CJS example using named exports, where util.cjs has an export named sum.

// @filename: util.cjs
module.exports.sum = (x, y) => x + y;// @filename: main.cjs
const {sum} = require('./util.cjs');
console.log(sum(2, 4));

Here’s a CJS example where util.cjs sets a default export. The default export has no name; modules using require() define their own name.

// @filename: util.cjs
module.exports = (x, y) => x + y;// @filename: main.cjs
const whateverWeWant = require('./util.cjs');
console.log(whateverWeWant(2, 4));

In ESM scripts, import and export are part of the language; like CJS, they have two different syntaxes for named exports and the default export.

Here’s an ESM example with named exports, where util.mjs has an export named sum.

// @filename: util.mjs
export const sum = (x, y) => x + y;// @filename: main.mjs
import {sum} from './util.mjs'
console.log(sum(2, 4));

Here’s an ESM example where util.mjs sets a default export. Just like in CJS, the default export has no name, but the module using import defines its own name.

// @filename: util.mjs
export default (x, y) => x + y;// @filename: main.mjs
import whateverWeWant from './util.mjs'
console.log(whateverWeWant(2, 4));

ESM and CJS are completely different animals

In CommonJS, require() is synchronous; it doesn't return a promise or call a callback. require() reads from the disk (or perhaps even from the network), and then immediately runs the script, which may itself do I/O or other side effects, and then returns whatever values were set on module.exports.

In ESM, the module loader runs in asynchronous phases. In the first phase, it parses the script to detect calls to import and export without running the imported script. In the parsing phase, the ESM loader can immediately detect a typo in named imports and throw an exception without ever actually running the dependency code.

The ESM module loader then asynchronously downloads and parses any scripts that you imported, and then scripts that your scripts imported, building out a “module graph” of dependencies, until eventually it finds a script that doesn’t import anything. Finally, that script is allowed to execute, and then scripts that depend on that are allowed to run, and so on.

All of the “sibling” scripts in the ES module graph download in parallel, but they execute in order, guaranteed by the loader specification.

CJS is the default because ESM changes a lot of stuff

ESM changes a bunch of stuff in JavaScript. ESM scripts use Strict Mode by default (use strict), their this doesn't refer to the global object, scoping works differently, etc.

This is why, even in browsers, <script> tags are non-ESM by default; you have to add a type="module" attribute to opt into ESM mode.

Switching the default from CJS to ESM would be a big break in backwards compatibility. (Deno, the hot new alternative to Node, makes ESM the default, but as a result, its ecosystem is starting from scratch.)

CJS can’t `require()` ESM because of top-level await

The simplest reason that CJS can’t require() ESM is that ESM can do top-level await, but CJS scripts can't.

Top-level await lets us use the await keyword outside of an async function, at the “top level.”

ESM’s multi-phase loader makes it possible for ESM to implement top-level await without making it a “footgun.” Quoting from the V8 team’s blog post:

Perhaps you have seen the infamous gist by Rich Harris which initially outlined a number of concerns about top-level await and urged the JavaScript language not to implement the feature. Some specific concerns were:
• Top-level await could block execution.
• Top-level await could block fetching resources.
• There would be no clear interop story for CommonJS modules.
The stage 3 version of the proposal directly addresses these issues:
• As siblings are able to execute, there is no definitive blocking.
• Top-level await occurs during the execution phase of the module graph. At this point all resources have already been fetched and linked. There is no risk of blocking fetching resources.
• Top-level await is limited to [ESM] modules. There is explicitly no support for scripts or for CommonJS modules.

(Rich now approves of the current top-level await implementation.)

Since CJS doesn’t support top-level await, it’s not even possible to transpile ESM top-level await into CJS. How would you rewrite this code in CJS?

export const foo = await fetch('./data.json');

There’s an active debate on how to require() ESM in this thread. (Please read the whole thread and the linked discussions before commenting. If you dive in, you’ll find that top-level await isn’t even the only problematic case… what do you think happens if you synchronously require ESM which can asynchronously import some CJS which can synchronously require some ESM? What you get is a sync/async zebra stripe of death, that’s what! Top-level await is just the last nail in the coffin, and the easiest to explain.)

It’s frustrating, because the vast majority of ESM scripts don’t use top-level await, but, as one commenter wrote in that thread, “I don’t think designing a system with the blanket assumption that some feature just won’t get used is a viable path.”

Reviewing that conversation, it doesn’t look like we’re going to be able to require() ESM any time soon!

CJS Can import() ESM, but It’s Not Great

For now, if you’re writing CJS and you want to import an ESM script, you’ll have to use asynchronous dynamic import().

(async () => {
    const {foo} = await import('./foo.mjs');
})();

It’s… fine, I guess, as long as you don’t have any exports. If you do need to do some exports, you’ll have to export a Promise instead, which may be a huge inconvenience to your users:

module.exports.foo = (async () => {
    const {foo} = await import('./foo.mjs');
    return foo;
})();

ESM can’t import named CJS exports unless CJS scripts execute out of order

You can do this:

import _ from './lodash.cjs'

But you can’t do this:

import {shuffle} from './lodash.cjs'

That’s because CJS scripts compute their named exports as they execute, whereas ESM’s named exports must be computed during the parsing phase.

Fortunately for us, there’s a workaround! The workaround is annoying, but totally doable. We just have to import CJS scripts like this:

import _ from './lodash.cjs';
const {shuffle} = _;

There are no real downsides to this, and ESM-aware CJS libraries can even provide their own ESM wrappers that encapsulate this boilerplate for us.

This is totally fine! But don’t you just… wish it were better?

Out-of-order execution would work, but it might be even worse

A number of people have proposed executing CJS imports before ESM imports, out of order. That way, the CJS named exports could be computed at the same time as ESM named exports.

But that would create a new problem.

import {liquor} from 'liquor';
import {beer} from 'beer';

If liquor and beer are both initially CJS, changing liquor from CJS to ESM would change the ordering from liquor, beer to beer, liquor , which would be nauseatingly problematic if beer relied on something from liquor being executed first.

Out-of-order execution is still under debate, though the conversation seems to have mostly fizzled out a few weeks ago.

Dynamic Modules could save us, but their star is poisoned

There’s an alternative proposal that doesn’t require out-of-order execution or wrapper scripts, called Dynamic Modules.

In the ESM specification, the exporter statically defines all named exports. Under dynamic modules, the importer would define the export names in the import. The ESM loader would initially just trust that dynamic modules (CJS scripts) would provide all required named exports, and then throw an exception later if they didn’t satisfy the contract.

Unfortunately, dynamic modules would require some JavaScript language changes to be approved by the TC39 language committee, and they do not approve.

Specifically, ESM scripts can export * from './foo.cjs', which means to re-export all of the names that foo exports. (This is called a “star export.” 🤩)

Unfortunately, there’s no way for the loader to know what’s being exported when we star export from a dynamic module.

Dynamic-module star exports also create issues for spec compliance. For example, export * from 'omg'; export * from 'bbq'; is supposed to throw when both omg and bbq export the same named export wtf. Allowing the names to be user/consumer-defined means this validation phase needs to be post-handled / ignored somehow.

Proponents of dynamic modules proposed banning star exports from dynamic modules, but TC39 rejected that proposal. One TC39 member referred to this proposal as “syntax poisoning,” as star exports would be “poisoned” by dynamic modules.

This poison star is very angry with you. Credit: seekpng

(In my opinion, we’re already living in a world of syntax poison. In Node 14, named imports are poisoned, and under dynamic modules, star exports would be poisoned. Since named imports are extremely common and star exports are relatively rare, dynamic modules would reduce syntax poison in the ecosystem.)

This may not be the end of the road for dynamic modules. One proposal on the table is for all Node modules to become dynamic modules, even pure ESM modules, abandoning the ESM multi-phase loader in Node. Surprisingly, this would have no user-visible effect, except perhaps slightly worse startup performance; the ESM multi-phase loader was designed for loading scripts over a slow network.

But I don’t feel lucky. The Github issue for dynamic modules was recently closed, because there has been no discussion of dynamic modules in the last year.

Sometimes Node can synthesize named exports from CJS (but it’s unpredictable

The Node team, noticing how painful this is, put in a heroic effort to try to automatically parse CJS files to autodetect possible named exports! 😮

Node’s cjs-module-lexer attempts to automagically detect patterns like these:

exports.a = 'a';
exports['b'] = 'b';
if (false)
  exports.c = 'c';

Object.defineProperty(exports, 'd', { value: 'd' });
module.exports = {e: 'e'}

The problem is, it’s imperfect. It doesn’t work if the CJS uses patterns like these:

Object.defineProperty(exports, 'a', {
  get () {
    return 'nope';
  }
});

Object.defineProperties(...)

module.exports = {
  a,
  ...d, // ignored
  b: require('c'),
  c: "not detected since require('c') above bails the object detection"
}

The good news is, it was specifically designed to target the transpiled outputs of TypeScript, Babel, and Rollup. If you write your code in a TypeScript .cts file, Node will definitely detect all of your named exports!

ESM can require(), but it’s probably not worth it

require() is not in scope by default in ESM scripts, but you can get it back very easily.

import { createRequire } from 'module';
const require = createRequire(import.meta.url);

const {foo} = require('./foo.cjs');

The problem with this approach is that it doesn’t really help; it’s actually more lines of code than just doing a default import and destructuring.

import cjsModule from './foo.cjs';
const {foo} = cjsModule;

Plus, bundlers like Webpack and Rollup have no idea what to do with this createRequire pattern. So what’s the point?

What should you publish?

1. Consider shipping only ESM

ESM is the future, and there’s a solid movement of folks trying to put this whole fiasco behind us, and have everyone writing only ESM code.

The problem with shipping only ESM is that it’s a big inconvenience to CJS users. Your CJS users will have to await import your code instead of being able to synchronously require it.

If you already ship a library with CJS, shipping ESM-only is a backwards-incompatible breaking change. (Note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.)

2. If you must publish CJS, publish only CJS

If you’re transpiling from TypeScript, you could transpile to both CJS and ESM, but this introduces a hazard that users may accidentally both import your ESM scripts and require() your CJS separately. (For example, suppose one library omg.mjs depends on index.mjs, while another library bbq.cjs depends on index.cjs, and then you depend on both omg.mjs and bbq.cjs.)

Node normally dedupes modules, but Node doesn’t know that your CJS and your ESM are the “same” files, so your code will run twice, keeping two copies of your library’s state. That can cause all kinds of weird bugs.

So, if you’re publishing CJS at all, and you’re writing in TypeScript or another language that transpiles to JS, transpile only to CJS.

3. Test that your CJS library works in ESM

Write a little test in a .mjs file, in ESM JS (without transpiling) and make sure you can import your CJS in the way you expect.

If you expect your ESM users to use a default import, like import mylibrary from 'mylibrary', make sure that works. If you export your ESM users to use named imports, like import {foo} from 'mylibrary', make sure that works.

If it doesn’t, you might need to provide a thin ESM wrapper for your CJS.

4. If necessary, provide a thin ESM wrapper for your CJS

(Again, note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.)

import cjsModule from './index.cjs';export const foo = cjsModule.foo;

Put your ESM wrapper in a file using the .mjs extension.

5. Add an `exports` map to your `package.json`

Like this:

"exports": {
    "require": "./index.cjs",
    "import": "./index.mjs"
}

Beware: adding an exports map is always a “semver major” breaking change. By default, your users can reach into your package and require() any script they want, even files that you intended to be internal. The exports map ensures that users can only require/import the entry points that you deliberately expose.

That’s almost certainly a good thing! But it is a breaking change.

(If you allow your users to import or require() other files in your module, you can set up separate entry points for those, as well. See the Node documentation on ESM for details.)

Always include file extensions in export map targets. Refer to "index.cjs" not just "index" or a directory like "./build".

If you follow these guidelines, your users will be OK. Everything’s going to be all right.

P.S. Redfin is hiring.