[MUSIC] Hi everyone, my name is Rachel Fenichel. I lead the Blockly team at Google. This video is part of our 2022 Blockly Developers Summit and Im going to tell you about the TypeScript migration that the team worked on in the past year.
In mid-2021 the Blockly team paused all feature work in the core library to convert our code to TypeScript. It was a multi-quarter project, and we touched almost every line of code in the library.
Of course we had reasons for this rewrite – I didn’t just get bored and decide to change everything.
So let’s start with motivation: why migrate to TypeScript? What problems does it solve? TypeScript, and the tools around it, make life easier for us across four areas: type checking, documentation, modularization, and transpilation.
I’ll start with type checking, because it’s in the name.
JavaScript is a dynamically typed language. Plain JavaScript doesn’t check types at all: you find out about type errors at runtime – usually when your code crashes.
This is great for prototyping: as long as you, the programmer, know the types, you don’t have any problems, and you avoid boilerplate for type declarations.
It’s bad for ongoing maintenance: new team members don’t know the types, and they have to rely on comments or other team members to explain. Knowledge is often lost.
TypeScript is a strongly typed programming language that builds on JavaScript. Static type checking catches basic mistakes and helps guarantee correctness without exercising every single line of code in the tests.
Previously Blockly split the difference and used the Closure Compiler and JSDoc comments for type checking. This was a reasonably solid system, but TypeScript’s tooling and IDE support have outpaced our previous tooling. TypeScript’s type system is also more expressive and robust.
Next is documentation, which is a critical part of any library. Documentation communicates information about types, visibility, and the intent of code. It’s read by both humans and computers.
We’ve provided type declaration files for a while now, and developers use those type declarations when adding Blockly to their projects.
But our type declarations have historically been incomplete, and sometimes incorrect, because the Closure Compiler’s type system and the TypeScript type system are not quite the same.
Emitting accurate type declaration files is important: it’s a way to fully describe our APIs and makes it easy to diff APIs between versions and document breaking changes.
During this migration we broke our code into modules.
By default, if you load JavaScript through a script tag, your code ends up on the global namespace.
Again, this is great for prototyping: you don’t have to carefully consider how to pass data around, or whether you can call a function from a given file.
And again, it’s bad for ongoing maintenance: without modules, you have to keep track of how any code, in or outside of your codebase, might access your data. In Blockly this meant that every change was potentially a breaking change. Worse, we never knew, because it depended on code that was out of our control! Modules solve this: anything that isn’t exported is encapsulated in the module, and not accessible externally.
This has huge benefits for us as library authors: we can refactor and clean up code any way we want, as long as the module’s API stays the same. It also makes it easier to optimize code.
Finally, transpilation – which really means compatibility with Internet Explorer 11.
So far, we’ve stayed compatible by only writing code in ES5.
It’s a simple solution. There’s no transpilation; the code we write is the code that runs. You can debug without compiling, no matter which browser you use.
But there are downsides. We have to keep track of browser features and make sure that nothing accidentally slips into the codebase that won’t work on IE11. New developers find the code clunky and unintuitive. We don’t get any of the syntactic sugar offered by ES6 and later versions of the language.
Transpiling solves this: we can write code in any JavaScript dialect and emit ES5 code.
At a higher level I had three goals for this migration: Improve developer workflows for the people who write the library and for the people who use the library; improve correctness; and make the library and our releases stable and predictable.
So, why TypeScript? Most of these problems can actually be solved with careful use of JavaScript. The Closure Compiler supports modules and transpilation, and its type checker is powerful.
But the Closure Compiler is mostly used inside Google: external developers don’t usually know how to use it. Inside Google there are whole toolchains built around the Closure Compiler. As open-source developers we don’t get the benefits of internal tooling, and by using the Closure Compiler we don’t get the benefits of external JavaScript development tools.
TypeScript has good language tools: they are precise, correct, and fast. Autocomplete, code navigation, and other IDE features make writing code faster and more pleasant, and that means happier developers. TypeScript is also the recommended language for new web development at Google.
It became clear that Closure Compiler would not get better for us going forward. And it became a trade-off between the significant work of migrating versus the ongoing effort needed to solve the problems I just discussed using our existing tools.
The next question was: what would migration look like? Would we change everything at once, or creep across the codebase on file at a time? I adopted three principles.
Migration needed to be incremental. We migrated in discrete steps–not by changing every file extension to .ts and debugging from there. I defined a sequence of migration steps and we applied each step to every file in core before moving on to the next step. This wasn’t just a stylistic choice: it meant that many steps were – Automatable–the second principle. There aren’t good tools for jumping from ES5 to TS, but there are plenty of tools for converting code from ES5 to ES6, and for moving from ES6 to TypeScript. Where possible, we automated discrete steps–by writing our own scripts, using external scripts, or using editor tools.
The combination of incremental change and automated transformations made it easy to tell the difference between a mechanical change and a meaningful shift in the structure of a file or module.
The last principle is safety. After a transformation–and before beginning the next step–we verified that the code could still be loaded and run in a variety of contexts. All changes had to pass continuous integration tests to be merged.
So far I’ve covered reasons for migrating and key principles. Next up are the actual changes in code.
First: we moved from var to const and let. Again, the goal of this change was to make our code easy for other tools to parse. The Closure Compiler supports transpilation, so we set it to take in ES6 and output ES5. This was pretty straightforward, and we could do it one file at a time.
Next: modularization, by moving from Closure’s goog.provide to goog.module.
Monica wrote scripts to automate the actual code changes, but we had to explicitly record the API of every module as a list of exports. We started with the easy stuff, and then moved into the tricky bits: is a function part of the public API if it’s marked private and ends with an underscore? Okay, but what if we told someone to call it in a forum post five years ago? What about three years ago? Where possible we added runtime deprecation warnings and documented replacement APIs. We tended toward leniency during this stage. This step was also mostly done one file at a time.
Modules have both explicit imports and exports, so in this step we also resolved circular dependencies and reviewed modules that operate only through side effects. This step was tricky because it involved coordinated changes to multiple files.
Next we converted all files to use named exports as preparation for later steps in the migration. This one also took changes to multiple files at once.
That work took us to the end of the third quarter of 2021, and an important decision: was the code ready for release? The code worked well for basic test cases, but JSDoc generation was a mess, type declarations didn’t compile, and we couldn’t be sure that every possible module format worked.
We split the difference and published a beta instead of a full release. This let us get feedback about our changes but keep the main release stable, and gave us extra time to test before the real release.
Q4 of 2021 was about cleanup and consolidation. Christopher dove deep into the world of JavaScript modules to make sure Blockly worked across a wide matrix of configurations: in compressed and uncompressed mode; loaded through a script tag or a module loader; on browsers old and new.
JavaScript modules are unfortunately complicated and there are multiple competing standards. Fixing issues here sometimes felt like navigating an obstacle course blindfolded, with teammates yelling instructions from the side.
At this point all the files in core used ES6 and goog.module. Next we applied the same transformations to the files in the blocks and generators directories.
And in preparation for the Q4 release, Maribeth and Aaron wrestled with our automated documentation generation until it understood the new shape of our code base–modules and all.
At the end of Q4 we released a new version of Blockly as a major version bump. We saw immediate adoption–and, of course, there were bug reports.
It’s hard to say who uses which version of Blockly. My working hypothesis is that developers who have started to use Blockly in the past 1-2 years are probably using it as a package rather than forking, and that a decent number of those developers automatically and smoothly update as we push new releases, and account for most of the updates in new releases.
Developers who started using Blockly a long time ago, either by forking or by copying in blockly_compressed.js, are more likely using package or private functions. As a result, those developers may have more work to do at each update to the library. If that’s you, let us know! We want to help.
This brings us to the start of 2022. Some of the team returned to their other projects, including work in blockly-samples. The rest of us started the next modernization step: classes.
JavaScript did not have inheritance built into it until ES6. Instead, major libraries implemented inheritance through helper functions. Blockly used a version of the Closure library’s goog.inherits function and Closure conventions for creating classes.
Aaron, Beka and I methodically converted every Blockly class to an ES6 class and replaced goog.inherits with extends. As with previous work, we did this using automated tools under supervision. Some classes were simple; others required substantial reworking to keep the same behaviour while meeting ES6 requirements for constructors.
During Q1 Beka and Christopher wrote and published a script that automatically applies deprecations and renamings to an existing codebase. Developers can use this script to update between versions of Blockly. You can think of it as a fancy find and replace: it parses a JSON file in core Blockly that contains information about renames and deprecations, and it updates code to use the new names if there’s an easy fix. This script represents our commitment to you as developers: we can’t promise we’ll never change things, but we’ll provide tools and support to help you with your upgrades.
And of course, all this while Christopher continued to play whack-a-mole with module loading issues.
The eagle-eyed among you may have noticed that none of this code is actually in TypeScript. In a sense, everything we’ve done so far is pre-migration.
Every error we fixed, no matter how small, moved us closer to an error-free build in TypeScript, as shown by a steadily decreasing list of errors when running the TypeScript Compiler on our code.
In Q2, I think–I hope–we’ll make the jump to TypeScript.
Google has an internal tool for updating large numbers of files from goog.module to TypeScript. Aaron spent the first half of Q2 resolving errors–generally type errors–that block that tool. I’m recording this video in mid-April, but I expect to have most of our files converted to TypeScript and be writing all new code in TypeScript by the end of the quarter.
Automated conversion to TypeScript is the last big change and marks the end of the feature freeze in core.
We still have plenty of work ahead: fixing bugs; tightening up types; adopting new tools to generate human- and computer-readable documentation; and wrestling with the compiler.
But we’ll have more bandwidth to fix bugs and respond to feature requests, and we’ll have a stable, modular codebase with clear API boundaries.
Maybe, just maybe, we’ll be able to do a minor release instead of a major, at the end of the year.
With type checking fully handled by the TypeScript compiler, I can turn my eyes to the rest of the work that the Closure Compiler currently does.
I’ve been exploring other tools for the many tasks the compiler does: transpilation, minification, and dependency management.
It may be time to stop using the Closure Compiler. That decision depends on many factors, with stability high on the list. Closure Compiler has faults, but it has outlasted many JavaScript tools and frameworks. Whatever replaces it needs to be long-lasting.
Restructuring our build system is a lot of work! Definitely don’t want to have to do it again in two years.
It’s been a long journey: modernizing ten years of code doesn’t happen overnight.
JavaScript, and web technology as a whole, changed completely in the last decade. But the result of all this work is clear: clean code, clear types, good tools, and ways to avoid whole classes of mistakes in our code going forward.
Thanks for watching this talk, and I hope to see you at our upcoming developer summit. If you have questions you can post them in our Q&A at the summit; or bring them to Q&A sessions at the summit.