How to Use the Magic String Library
Greetings, friends! Today, I'd like to discuss a cool library called Magic String. This library is a fantastic way to make small modifications to your code and generate source maps, linking back to the original code. It's quite powerful and actually used internally by Vite, a new build tool created by the prestigious author of the Vue.js framework, Evan You.
You may be wondering why we need the Magic String library. After all, tools like Babel and Recast provide ways to generate source maps already. However, these tools rely on using parsers to generate an Abstract Syntax Tree (AST). Then, you have to make modifications to the AST and transpile the code, generating source maps along the way. This seems like a lot of work for replacing simple pieces of your code 🤔.
Enter the Magic String library. 🎉
Getting Started
To get started, let's create a new directory and navigate to it:
mkdir magic-string-tutorial && cd $_
Then, we'll bootstrap a new project with a package.json file:
npm init -y
All we need to install is the Magic String library:
npm i magic-string
Let's look at a simple example. We'll create a file called example.js with the following contents:
const fs = require('fs');
const MagicString = require('magic-string');
const s = new MagicString('problems = 99');
s.overwrite(0, 8, 'answer');
console.log(s.toString()); // "answer = 99"
s.overwrite(11, 13, '42'); // character indices always refer to the original string
console.log(s.toString()); // "answer = 42"
s.prepend('const ').append(';'); // most methods on a MagicString instance are chainable
console.log(s.toString()); // "const answer = 42;"
const map = s.generateMap({
source: 'source.js',
file: 'converted.js.map',
includeContent: true,
}); // generates a v3 source map
fs.writeFileSync('converted.js', s.toString());
fs.writeFileSync('converted.js.map', map.toString());
Before we run this script, let's break down this file piece by piece.
First, we require the fs module so that we can read/write to files.
const fs = require('fs');
Next, we import the Magic String library and create an instance of MagicString:
const MagicString = require('magic-string');
const s = new MagicString('problems = 99');
Using this MagicString instance, we can call the overwrite method. In this example, we are overwriting characters 0 through 8, exclusive, of the string we placed inside MagicString constructor.
const MagicString = require('magic-string');
const s = new MagicString('problems = 99');
s.overwrite(0, 8, 'answer');
// "problems" becomes "answer", since we are replacing characters 0 through 8 (exclusive)
You may be wondering why we need to overwrite the text using the MagicString library. Why can't we use normal JavaScript String methods to replace the text? The Magic String library needs to keep track of each change that is made to the original source string passed into the MagicString constructor. Source maps keep track of the original line numbers and columns (positions in each line) and know how to map it to the transformed code's line numbers and columns.
We can use the toString method on the MagicString instance to see what the manipulated string looks like so far:
console.log(s.toString()); // "answer = 99"
We can replace more characters that we passed in the MagicString constructor:
s.overwrite(11, 13, '42'); // character indices always refer to the original string we passed to the MagicString constructor
console.log(s.toString()); // "answer = 42"
The MagicString library also provides methods for prepending and appending strings before the source string (the string we passed into the MagicString constructor). It also lets you chain these methods together for convenience.
s.prepend('const ').append(';'); // most methods on a MagicString instance are chainable
console.log(s.toString()); // "const answer = 42;"
A source map is generated using the generateMap method on the MagicString instance. If you're not familiar with source maps, this article provides a great discussion on how they work and MDN discusses how use them with your browser's debugger. Source maps expect a name for the file in the source property. In our example, we simply created a string instead of reading from a file. We'll just choose the name source.js, but you can choose whatever name you'd like. The file property specifies the name of the source map that will be created. JavaScript source maps usually end with the js.map extensions. The includeContent property is used to specify whether the original source code (the code you used inside the MagicString constructor) should be placed inside the sourcesContent property of the source map.
const map = s.generateMap({
source: 'source.js',
file: 'converted.js.map',
includeContent: true,
}); // generates a v3 source map
Finally, we use Node.js's writeFileSync to write the modified string and source map to files:
fs.writeFileSync('converted.js', s.toString()); // save to a file the modified version of the string we originally specified in the MagicString constructor
fs.writeFileSync('converted.js.map', map.toString()); // save to a file the generated source map
I use fs.writeFileSync in the example for brevity, but I highly encourage everyone to use the asynchronous fs.writeFile or fs.promises.writeFile methods instead.
Now that we have explained the example piece by piece, let's execute the script. If you run node example or node example.js, then it'll create two files: converted.js and converted.js.map. If we look inside converted.js, we'll see the following contents:
const answer = 42;
If we look inside converted.js.map, you'll find a source map:
{
"version": 3,
"file": "converted.js.map",
"sources": [
"source.js"
],
"sourcesContent": [
"problems = 99"
],
"names": [],
"mappings": "MAAA,MAAQ,GAAG"
}
Notice that the sourcesContent property in the source map contains the original string we passed into the MagicString constructor. Additionally, you will see source.js under the sources property. Browsers such as Google Chrome will display this name in the debugger, so you can look at your original source code. We'll see an example of this later.
Alright. Hope you're still with me! That was a basic example based on the official documentation on the Magic String GitHub page.
Creating a Transformer with Source Map Support
Let's look at a more interesting example and see how we can debug the transformed source code using Google Chrome. First, let's write some code in a file called input.js:
console.log('Pizza makes me feel happy');
console.log('Empty boxes of pizza makes me feel sad');
console.log('Having my pizza stolen makes me feel angry');
console.log('Coding makes me feel happy');
console.log('Finding bugs makes me feel sad');
console.log('Spending too much time on bugs makes me feel angry');
In the following example, we will simply modify code that contains words of emotion (happy, sad, angry) with a corresponding emoji. Then, we'll use the Magic String library to generate a source map that will link back to the original source code.
We will create a file called emotion-transformer.js that will be responsible for transforming our input.js code.
const fsP = require('fs').promises;
const MagicString = require('magic-string');
const emotionMapping = {
happy: '😃',
sad: '😢',
angry: '😠'
};
const runTransformation = async () => {
try {
const code = await fsP.readFile('input.js', 'utf8');
const s = new MagicString(code);
const pattern = /\b(happy|sad|angry)\b/g;
while ((match = pattern.exec(code))) {
const emotion = match[0];
const start = match.index;
const end = start + emotion.length;
s.overwrite(start, end, emotionMapping[emotion]);
}
const outputSourceMap = 'output.js.map';
const map = s.generateMap({
source: 'input.js',
file: outputSourceMap,
includeContent: true,
});
const sourceMapURL = '\n//# sourceMappingURL=' + outputSourceMap;
await fsP.writeFile('output.js', s.toString() + sourceMapURL);
await fsP.writeFile('output.js.map', map.toString());
} catch (err) {
throw err;
}
}
runTransformation();
Again, let's dissect this code before running it.
First, we will import the fs.promises API so we can read files and write to them asynchronously using promises.
const fsP = require('fs').promises;
Let's look at the runTransformation function. We first create a new MagicString instance using the code inside input.js.
const code = await fsP.readFile('input.js', 'utf8');
const s = new MagicString(code);
We will use a regular expression (regex) pattern to grab one of three words: happy, sad, or angry. We use word boundaries, \b, to make sure that we grab whole words only.
const pattern = /\b(happy|sad|angry)\b/g;
Let's look at this weird piece of code:
while ((match = pattern.exec(code))) {
const emotion = match[0];
const start = match.index;
const end = start + emotion.length;
s.overwrite(start, end, emotionMapping[emotion]);
}
The above code is using a trick mentioned on MDN's RegExp.prototype.exec page. It allows us to use the exec() method multiple times to find successive matches in the same string (the contents of input.js in this case). Upon each iteration of the while loop, we can find the start and end indices of where each emotion occurs inside input.js. However, we need to figure out whether the match is "happy", "sad," or "angry" and overwrite those words with the appropriate emoji. We can use this mapping to replace each emotion word with its corresponding emoji.
const emotionMapping = {
happy: '😃',
sad: '😢',
angry: '😠'
}
The next thing we need to do is generate the source map using the Magic String library after we have used regex to replace all the emotion words with emojis.
const outputSourceMap = 'output.js.map';
const map = s.generateMap({
source: 'input.js',
file: outputSourceMap,
includeContent: true,
});
Finally, we write the contents of our transformed code to a file named output.js. It's important to note that we have to make sure to manually add a special comment to the end of this file to tell browsers where to find the source map. We also have to write the generated source map to a file which we will name output.js.map. Make sure you use the .js.map extension, since that is the proper convention for JavaScript source map files.
const sourceMapURL = '\n//# sourceMappingURL=' + outputSourceMap;
await fsP.writeFile('output.js', s.toString() + sourceMapURL);
await fsP.writeFile('output.js.map', map.toString());
Alright. Let's run node emotion-transformer to execute the script. It should generate two files. The first one will be the output.js which contains the code we modified from input.js.
console.log('Pizza makes me feel 😃');
console.log('Empty boxes of pizza makes me feel 😢');
console.log('Having my pizza stolen makes me feel 😠');
console.log('Coding makes me feel 😃');
console.log('Finding bugs makes me feel 😢');
console.log('Spending too much time on bugs makes me feel 😠');
//# sourceMappingURL=output.js.map
Yay! Our emojis are there! Makes me feel happy 😃.
The second file will be output.map.js which contains the source map that links the code modifications between output.js and input.js:
{
"version": 3,
"file": "output.js.map",
"sources": [
"input.js"
],
"sourcesContent": [
"console.log('Pizza makes me feel happy');\nconsole.log('Empty boxes of pizza makes me feel sad');\nconsole.log('Having my pizza stolen makes me feel angry');\n\nconsole.log('Coding makes me feel happy');\nconsole.log('Finding bugs makes me feel sad');\nconsole.log('Spending too much time on bugs makes me feel angry');\n"
],
"names": [],
"mappings": "AAAA,iCAAiC,EAAK;AACtC,gDAAgD,EAAG;AACnD,kDAAkD,EAAK;AACvD;AACA,kCAAkC,EAAK;AACvC,wCAAwC,EAAG;AAC3C,0DAA0D,EAAK;"
}
We can now use this source map in modern browsers such as Google Chrome. First, let's create an HTML file to import our transformed code, output.js.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Magic String Source Map Test</title>
</head>
<body>
<h1>Magic String Source Map Test</h1>
<script src="output.js"></script>
</body>
</html>
We can serve this HTML file using the serve package. Let's install it.
npm i -D serve
Then, we can create a command to start the server in our package.json file:
"start": "serve ."
This will serve all assets from the current directory so we can see the source map in our browser. If we open up the developer tools in Google Chrome, we should be able to navigate to output.js and insert breakpoints. When we refresh the browser, it will automatically map those breakpoints to the original source code inside input.js. Super cool! 😎
If you would like to see all the finished code, please check out my GitHub page.
The "emotional word to emoji" transformer we built is very basic, but I hope you realize that the Magic String library can lead to some interesting applications. You can replace way more code than this and even create a system similar to Webpack loaders without the hassle of dealing with Webpack. However, if you're already using Webpack in your application, then I'd advise that you'd build a Webpack loader instead, since Webpack can already generate a source map that links your bundled output files back to the original individual input files.
Conclusion
The Magic String library is a great tool for transforming code and generating source maps quickly and easily. If you're working on a new JavaScript bundling tool, building a small DSL, experimenting with new experimental JavaScript features, or even creating your own features, then the Magic String library might be right for you. Try to come up with interesting ideas and have fun!