Interoperability with the Node.js Built-in Test Runner
tl;dr - Use the
taprunner to run tests written withnode:testor runtaptests withnode --testand it Just Works with full interoperability. Mix and match how you see fit.
The best way to appreciate how these two things work together is
with an example. The tapjs/node-test-example
repo is a module
with a shocking number of bugs for how little code it is, with a
test written using node:test and effectively the same test
written using tap.
When you run npm test, it'll run with both tap and node --test.
git clone git@github.com:tapjs/node-test-example.git
cd node-test-example
npm install
npm test # run both with tap, and both with node --testnpm run test:nodeExecutes both test suite files with thenode --testrunner.npm run test:tapExecute both test suite files with thetaprunner.npm run test:mixRun the node test with node, the tap test with tap, sharing a coverage folder.npm run test:crossRun the node test with tap, the tap test with node, sharing a coverage folder.
The test:mix and test:cross show using the node --test and
tap runners so that they dump coverage into the same folder.
Then you can use tap report to report on it.
In all cases, you can see that the results are pretty similar.
Differences#
- When running with the
taprunner, they're almost identical. The main thing is that thenode:testdoesn't provide per-assertion reporting, so you only see a report on the test block, and possibly the first failure, not all the assertions within it. - When running with the
node --testrunner, thetaptest provides diffs and source callsite printing, while thenode:testtest shows aconsole.log()of the thrown Error.
Of course, the two runners produce very different output overall, but they should both be pretty sensible.
Personally, I think the tap runner is a lot more useful, and
certainly if you write tests in TypeScript (or use tap's import
mocking) it's nice to not have to specify the --loader and
--import arguments explicitly.
But on the flip side, that fanciness comes with a cost. With
TypeScript disabled, tap runs these two tests in about 450ms on
my system (350ms or so with coverage disabled), while node --test does it in around 170ms. In both cases, the
test/tap.test.js test takes around 150ms to run, and the
test/node.test.js takes under 10ms.
Real world tests doing complicated stuff would show a less dramatic difference, so this is in no way a representative benchmark, but as always, performance and features are fundamentally opposed, because features require running code, and not running code is always faster.
You Do You#
The goal of the node:test interoperability in node-tap is to
make it possible for you to get the best of both worlds. You
could have part of your test suite written as node:test tests,
if they don't need t.mockImport or TypeScript, and other tests
written in tap that are just more convenient to run with a
runner that knows which loaders to apply.
Enough talk! Show the output!#
Running with tap:
FAIL test/node.test.js 2 failed of 4 6.834ms ✖ suite of tests that fail > uhoh, this one throws ✖ suite of tests that fail > failer FAIL test/tap.test.js 3 failed of 18 340ms ✖ suite of tests that fail > uhoh, this one throws > Invalid time value lib/index.mjs:11:43 ✖ suite of tests that fail > failer > should be equal test/tap.test.js:35:7 ✖ suite of tests that fail > failer > should be equal test/tap.test.js:37:7 🌈 TEST COMPLETE 🌈 FAIL test/node.test.js 2 failed of 4 6.834ms ✖ suite of tests that fail > uhoh, this one throws test/node.test.js 20 }) 21 22 test('suite of tests that fail', async t => { 23 await t.test('uhoh, this one throws', () => { ━━━━━━━━━━━━━┛ 24 assert.equal(thrower(0), '1970-01-01T00:00:00.000Z') 25 assert.equal(thrower(1234567891011), '2009-02-13T23:31:31.011Z') 26 assert.equal(thrower({}), 'Invalid Date') 27 }) error origin: lib/index.mjs 8 9 // This is a function that throws, to show how both 10 // handle errors. 11 export const thrower = (n) => new Date(n).toISOString() ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 12 13 // one that fails, to show how failures are handled 14 export const failer = (n) => String(n + 1) error: Invalid time value code: ERR_TEST_FAILURE failureType: testCodeFailure name: RangeError Date.toISOString (<anonymous>) thrower (lib/index.mjs:11:43) TestContext.<anonymous> (test/node.test.js:26:18) TestContext.<anonymous> (test/node.test.js:23:11) ✖ suite of tests that fail > failer test/node.test.js 26 assert.equal(thrower({}), 'Invalid Date') 27 }) 28 29 await t.test('failer', () => { ━━━━━━━━━━━━━┛ 30 assert.equal(failer(1), '2') 31 assert.equal(failer(-1), '0') 32 // expect to convert string numbers to Number, but doesn't 33 assert.equal(failer('1'), '2') error origin: test/node.test.js 30 assert.equal(failer(1), '2') 31 assert.equal(failer(-1), '0') 32 // expect to convert string numbers to Number, but doesn't 33 assert.equal(failer('1'), '2') ━━━━━━━━━━━━━━┛ 34 // expect to convert non-numerics to 0, but it doesn't 35 assert.equal(failer({}), '1') 36 }) 37 }) --- expected +++ actual @@ -1,1 +1,1 @@ -"2" +"11" error: "'11' == '2'" code: ERR_ASSERTION failureType: testCodeFailure name: AssertionError operator: == TestContext.<anonymous> (test/node.test.js:33:12) TestContext.<anonymous> (test/node.test.js:29:11) FAIL test/tap.test.js 3 failed of 18 340ms ✖ suite of tests that fail > uhoh, this one throws > Invalid time value lib/index.mjs 8 9 // This is a function that throws, to show how both 10 // handle errors. 11 export const thrower = (n) => new Date(n).toISOString() ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 12 13 // one that fails, to show how failures are handled 14 export const failer = (n) => String(n + 1) type: RangeError tapCaught: testFunctionThrow Date.toISOString (<anonymous>) thrower (lib/index.mjs:11:43) Test.<anonymous> (test/tap.test.js:27:13) ✖ suite of tests that fail > failer > should be equal test/tap.test.js 32 t.equal(failer(1), '2') 33 t.equal(failer(-1), '0') 34 // expect to convert string numbers to Number, but doesn't 35 t.equal(failer('1'), '2') ━━━━━━━━━┛ 36 // expect to convert non-numerics to 0, but it doesn't 37 t.equal(failer({}), '1') 38 t.end() 39 }) --- expected +++ actual @@ -1,1 +1,1 @@ -2 +11 compare: === Test.<anonymous> (test/tap.test.js:35:7) Test.<anonymous> (test/tap.test.js:31:5) test/tap.test.js:23:3 ✖ suite of tests that fail > failer > should be equal test/tap.test.js 34 // expect to convert string numbers to Number, but doesn't 35 t.equal(failer('1'), '2') 36 // expect to convert non-numerics to 0, but it doesn't 37 t.equal(failer({}), '1') ━━━━━━━━━┛ 38 t.end() 39 }) 40 41 t.end() --- expected +++ actual @@ -1,1 +1,1 @@ -1 +[object Object]1 compare: === Test.<anonymous> (test/tap.test.js:37:7) Test.<anonymous> (test/tap.test.js:31:5) test/tap.test.js:23:3 Asserts: 17 pass 5 fail 22 of 22 complete Suites: 0 pass 2 fail 2 of 2 complete # { total: 22, pass: 17, fail: 5 } # time=459.924ms
Running with node --test:
$ node --test ✔ add (0.569917ms) ✔ stringOrNull (0.063833ms) ▶ suite of tests that fail ✖ uhoh, this one throws (0.910959ms) RangeError [Error]: Invalid time value at Date.toISOString (<anonymous>) at thrower (file:///Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43) at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:26:18) at Test.runInAsyncScope (node:async_hooks:206:9) at Test.run (node:internal/test_runner/test:631:25) at Test.start (node:internal/test_runner/test:542:17) at TestContext.test (node:internal/test_runner/test:167:20) at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:23:11) at Test.runInAsyncScope (node:async_hooks:206:9) at Test.run (node:internal/test_runner/test:631:25) ✖ failer (0.532708ms) AssertionError [ERR_ASSERTION]: '11' == '2' at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:33:12) at Test.runInAsyncScope (node:async_hooks:206:9) at Test.run (node:internal/test_runner/test:631:25) at Test.start (node:internal/test_runner/test:542:17) at TestContext.test (node:internal/test_runner/test:167:20) at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:29:11) at async Test.run (node:internal/test_runner/test:632:9) at async Test.processPendingSubtests (node:internal/test_runner/test:374:7) { generatedMessage: true, code: 'ERR_ASSERTION', actual: '11', expected: '2', operator: '==' } ▶ suite of tests that fail (1.684292ms) ✔ add (1.774ms) ✔ stringOrNull (1.091ms) ▶ suite of tests that fail ✖ uhoh, this one throws (10.016ms) Error: Invalid time value | // This is a function that throws, to show how both | // handle errors. | export const thrower = (n) => new Date(n).toISOString() | ------------------------------------------^ | | // one that fails, to show how failures are handled at Date.toISOString (<anonymous>) at thrower (/Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43) at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:27:13) { type: 'RangeError', tapCaught: 'testFunctionThrow' } ✖ failer (3.676ms) Error: should be equal --- expected +++ actual @@ -1,1 +1,1 @@ -2 +11 | t.equal(failer(-1), '0') | // expect to convert string numbers to Number, but doesn't | t.equal(failer('1'), '2') | ------^ | // expect to convert non-numerics to 0, but it doesn't | t.equal(failer({}), '1') at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:35:7) at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:31:5) at /Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:23:3 { compare: '===' } ▶ suite of tests that fail (17.681ms) ℹ tests 9 ℹ suites 1 ℹ pass 4 ℹ fail 5 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0 ℹ duration_ms 160.809375 ✖ failing tests: test at file:/Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:23:11 ✖ uhoh, this one throws (0.910959ms) RangeError [Error]: Invalid time value at Date.toISOString (<anonymous>) at thrower (file:///Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43) at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:26:18) at Test.runInAsyncScope (node:async_hooks:206:9) at Test.run (node:internal/test_runner/test:631:25) at Test.start (node:internal/test_runner/test:542:17) at TestContext.test (node:internal/test_runner/test:167:20) at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:23:11) at Test.runInAsyncScope (node:async_hooks:206:9) at Test.run (node:internal/test_runner/test:631:25) test at file:/Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:29:11 ✖ failer (0.532708ms) AssertionError [ERR_ASSERTION]: '11' == '2' at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:33:12) at Test.runInAsyncScope (node:async_hooks:206:9) at Test.run (node:internal/test_runner/test:631:25) at Test.start (node:internal/test_runner/test:542:17) at TestContext.test (node:internal/test_runner/test:167:20) at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:29:11) at async Test.run (node:internal/test_runner/test:632:9) at async Test.processPendingSubtests (node:internal/test_runner/test:374:7) { generatedMessage: true, code: 'ERR_ASSERTION', actual: '11', expected: '2', operator: '==' } test at test/tap.test.js:24:5 ✖ uhoh, this one throws (10.016ms) Error: Invalid time value | // This is a function that throws, to show how both | // handle errors. | export const thrower = (n) => new Date(n).toISOString() | ------------------------------------------^ | | // one that fails, to show how failures are handled at Date.toISOString (<anonymous>) at thrower (/Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43) at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:27:13) { type: 'RangeError', tapCaught: 'testFunctionThrow' } test at test/tap.test.js:31:5 ✖ failer (3.676ms) Error: should be equal --- expected +++ actual @@ -1,1 +1,1 @@ -2 +11 | t.equal(failer(-1), '0') | // expect to convert string numbers to Number, but doesn't | t.equal(failer('1'), '2') | ------^ | // expect to convert non-numerics to 0, but it doesn't | t.equal(failer({}), '1') at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:35:7) at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:31:5) at /Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:23:3 { compare: '===' }