Change language

2022 Blockly Developers Summit: TypeScript Migration

2022 Blockly Developers Summit: TypeScript Migration

[MUSIC] Hi everyone, my name is Rachel Fenichel.  I lead the Blockly team at Google. This video is part of our 2022 Blockly  Developers Summit and Im going to tell you about the TypeScript migration that  the team worked on in the past year.

In mid-2021 the Blockly team paused all feature  work in the core library to convert our code to TypeScript. It was a multi-quarter project, and we  touched almost every line of code in the library.

Of course we had reasons for  this rewrite – I didn’t just get bored and decide to change everything.

So let’s start with motivation: why migrate  to TypeScript? What problems does it solve? TypeScript, and the tools around it, make  life easier for us across four areas: type checking, documentation,  modularization, and transpilation.

I’ll start with type checking,  because it’s in the name.

JavaScript is a dynamically typed language.  Plain JavaScript doesn’t check types at all: you find out about type errors at  runtime – usually when your code crashes.

This is great for prototyping: as long  as you, the programmer, know the types, you don’t have any problems, and you  avoid boilerplate for type declarations.

It’s bad for ongoing maintenance: new  team members don’t know the types, and they have to rely on comments or other team  members to explain. Knowledge is often lost.

TypeScript is a strongly typed programming  language that builds on JavaScript. Static type checking catches basic mistakes  and helps guarantee correctness without exercising every single line of code in the tests.

Previously Blockly split the difference and used  the Closure Compiler and JSDoc comments for type checking. This was a reasonably solid system,  but TypeScript’s tooling and IDE support have outpaced our previous tooling. TypeScript’s  type system is also more expressive and robust.

Next is documentation, which is  a critical part of any library. Documentation communicates  information about types, visibility, and the intent of code. It’s read  by both humans and computers.

We’ve provided type declaration  files for a while now, and developers use those type declarations  when adding Blockly to their projects.

But our type declarations have historically  been incomplete, and sometimes incorrect, because the Closure Compiler’s type system and  the TypeScript type system are not quite the same.

Emitting accurate type declaration files is  important: it’s a way to fully describe our APIs and makes it easy to diff APIs between  versions and document breaking changes.

During this migration we  broke our code into modules.

By default, if you load  JavaScript through a script tag, your code ends up on the global namespace.

Again, this is great for prototyping:  you don’t have to carefully consider how to pass data around, or whether you  can call a function from a given file.

And again, it’s bad for ongoing maintenance: without modules, you have to keep track of  how any code, in or outside of your codebase, might access your data. In Blockly this meant that  every change was potentially a breaking change. Worse, we never knew, because it depended  on code that was out of our control! Modules solve this: anything that isn’t  exported is encapsulated in the module, and not accessible externally.

This has huge benefits for us as library authors:  we can refactor and clean up code any way we want, as long as the module’s API stays the same.  It also makes it easier to optimize code.

Finally, transpilation – which really means  compatibility with Internet Explorer 11.

So far, we’ve stayed compatible  by only writing code in ES5.

It’s a simple solution. There’s no transpilation;  the code we write is the code that runs. You can debug without compiling,  no matter which browser you use.

But there are downsides. We have to  keep track of browser features and make sure that nothing accidentally slips  into the codebase that won’t work on IE11. New developers find the  code clunky and unintuitive. We don’t get any of the syntactic sugar offered  by ES6 and later versions of the language.

Transpiling solves this: we can write code  in any JavaScript dialect and emit ES5 code.

At a higher level I had three  goals for this migration: Improve developer workflows for the people who  write the library and for the people who use the library; improve correctness; and make the  library and our releases stable and predictable.

So, why TypeScript? Most of these problems can  actually be solved with careful use of JavaScript. The Closure Compiler supports modules and  transpilation, and its type checker is powerful.

But the Closure Compiler is mostly used inside  Google: external developers don’t usually know how to use it. Inside Google there are whole  toolchains built around the Closure Compiler. As open-source developers we don’t  get the benefits of internal tooling, and by using the Closure Compiler we don’t get the  benefits of external JavaScript development tools.

TypeScript has good language tools: they are  precise, correct, and fast. Autocomplete, code navigation, and other IDE features  make writing code faster and more pleasant, and that means happier developers.  TypeScript is also the recommended language for new web development at Google.

It became clear that Closure Compiler  would not get better for us going forward. And it became a trade-off between the  significant work of migrating versus the ongoing effort needed to solve the problems  I just discussed using our existing tools.

The next question was: what would migration  look like? Would we change everything at once, or creep across the codebase on file  at a time? I adopted three principles.

Migration needed to be incremental.  We migrated in discrete steps–not by changing every file extension  to .ts and debugging from there. I defined a sequence of migration steps and we  applied each step to every file in core before moving on to the next step. This wasn’t just a  stylistic choice: it meant that many steps were – Automatable–the second principle. There  aren’t good tools for jumping from ES5 to TS, but there are plenty of tools for  converting code from ES5 to ES6, and for moving from ES6 to TypeScript. Where  possible, we automated discrete steps–by writing our own scripts, using external  scripts, or using editor tools.

The combination of incremental  change and automated transformations made it easy to tell the difference between a mechanical change and a meaningful shift  in the structure of a file or module.

The last principle is safety. After a  transformation–and before beginning the next step–we verified that the code could still  be loaded and run in a variety of contexts. All changes had to pass continuous  integration tests to be merged.

So far I’ve covered reasons for  migrating and key principles. Next up are the actual changes in code.

First: we moved from var to const and let. Again, the goal of this change was to make  our code easy for other tools to parse. The Closure Compiler supports transpilation,  so we set it to take in ES6 and output ES5. This was pretty straightforward, and  we could do it one file at a time.

Next: modularization, by moving from  Closure’s goog.provide to goog.module.

Monica wrote scripts to automate  the actual code changes, but we had to explicitly record the API  of every module as a list of exports. We started with the easy stuff, and  then moved into the tricky bits: is a function part of the public API if it’s  marked private and ends with an underscore? Okay, but what if we told someone to call it in a forum  post five years ago? What about three years ago? Where possible we added runtime deprecation  warnings and documented replacement APIs. We tended toward leniency during this stage. This  step was also mostly done one file at a time.

Modules have both explicit imports and  exports, so in this step we also resolved circular dependencies and reviewed modules  that operate only through side effects. This step was tricky because it involved  coordinated changes to multiple files.

Next we converted all files to use named exports  as preparation for later steps in the migration. This one also took changes  to multiple files at once.

That work took us to the end of the third quarter  of 2021, and an important decision: was the code ready for release? The code worked well for basic  test cases, but JSDoc generation was a mess, type declarations didn’t compile, and we couldn’t  be sure that every possible module format worked.

We split the difference and published  a beta instead of a full release. This let us get feedback about our  changes but keep the main release stable, and gave us extra time to  test before the real release.

Q4 of 2021 was about cleanup and consolidation.  Christopher dove deep into the world of JavaScript modules to make sure Blockly worked across a  wide matrix of configurations: in compressed and uncompressed mode; loaded through a script  tag or a module loader; on browsers old and new.

JavaScript modules are unfortunately complicated  and there are multiple competing standards. Fixing issues here sometimes felt like  navigating an obstacle course blindfolded, with teammates yelling instructions from the side.

At this point all the files in  core used ES6 and goog.module. Next we applied the same transformations to the  files in the blocks and generators directories.

And in preparation for the Q4 release,  Maribeth and Aaron wrestled with our automated documentation generation until it understood  the new shape of our code base–modules and all.

At the end of Q4 we released a new  version of Blockly as a major version bump. We saw immediate adoption–and,  of course, there were bug reports.

It’s hard to say who uses which version  of Blockly. My working hypothesis is that developers who have started to use Blockly  in the past 1-2 years are probably using it as a package rather than forking, and that a  decent number of those developers automatically and smoothly update as we push new releases, and  account for most of the updates in new releases.

Developers who started using  Blockly a long time ago, either by forking or by copying  in blockly_compressed.js, are more likely using package or private  functions. As a result, those developers may have more work to do at each update to the library.  If that’s you, let us know! We want to help.

This brings us to the start of 2022. Some of  the team returned to their other projects, including work in blockly-samples. The rest of  us started the next modernization step: classes.

JavaScript did not have inheritance  built into it until ES6. Instead, major libraries implemented inheritance through  helper functions. Blockly used a version of the Closure library’s goog.inherits function and  Closure conventions for creating classes.

Aaron, Beka and I methodically converted  every Blockly class to an ES6 class and replaced goog.inherits with extends. As with  previous work, we did this using automated tools under supervision. Some classes were  simple; others required substantial reworking to keep the same behaviour while  meeting ES6 requirements for constructors.

During Q1 Beka and Christopher wrote and published  a script that automatically applies deprecations and renamings to an existing codebase.  Developers can use this script to update between versions of Blockly. You can think  of it as a fancy find and replace: it parses a JSON file in core Blockly that contains  information about renames and deprecations, and it updates code to use the  new names if there’s an easy fix. This script represents our commitment to  you as developers: we can’t promise we’ll never change things, but we’ll provide tools  and support to help you with your upgrades.

And of course, all this while  Christopher continued to play whack-a-mole with module loading issues.

The eagle-eyed among you may have noticed that  none of this code is actually in TypeScript. In a sense, everything we’ve  done so far is pre-migration.

Every error we fixed, no matter how small, moved  us closer to an error-free build in TypeScript, as shown by a steadily decreasing list of errors  when running the TypeScript Compiler on our code.

In Q2, I think–I hope–we’ll  make the jump to TypeScript.

Google has an internal tool for updating large  numbers of files from goog.module to TypeScript. Aaron spent the first half of Q2 resolving  errors–generally type errors–that block that tool. I’m recording this video in mid-April,  but I expect to have most of our files converted to TypeScript and be writing all new code  in TypeScript by the end of the quarter.

Automated conversion to TypeScript is the last big change and marks the end of  the feature freeze in core.

We still have plenty of work ahead: fixing  bugs; tightening up types; adopting new tools to generate human- and computer-readable  documentation; and wrestling with the compiler.

But we’ll have more bandwidth to fix  bugs and respond to feature requests, and we’ll have a stable, modular  codebase with clear API boundaries.

Maybe, just maybe, we’ll be able to  do a minor release instead of a major, at the end of the year.

With type checking fully handled  by the TypeScript compiler, I can turn my eyes to the rest of the work  that the Closure Compiler currently does.

I’ve been exploring other tools for  the many tasks the compiler does: transpilation, minification,  and dependency management.

It may be time to stop using the Closure  Compiler. That decision depends on many factors, with stability high on the  list. Closure Compiler has faults, but it has outlasted many JavaScript tools and frameworks.  Whatever replaces it needs to be long-lasting.

Restructuring our build system is a lot of work! Definitely don’t want to have  to do it again in two years.

It’s been a long journey: modernizing ten  years of code doesn’t happen overnight.

JavaScript, and web technology as a whole,  changed completely in the last decade. But the result of all this  work is clear: clean code, clear types, good tools, and ways to avoid whole  classes of mistakes in our code going forward.

Thanks for watching this talk, and I hope  to see you at our upcoming developer summit. If you have questions you can post  them in our Q&A at the summit; or bring them to Q&A sessions at the summit.