As uTheory has grown, so has the computer code that powers it. Roughly 185,000 lines of code (LOC) power uTheory's server, webpage, app and administrative tooling. For context, many programs have millions of lines of code. But our codebase is now larger than what any one programmer can fully hold in their head.

To let us move faster at building music learning tools, and not break things, we've spent much of 2021 improving our codebase with more reliable technologies and practices. This post is quite technical, but we wanted to share with you a bit of what we're up to!

TypeScript

Javascript is the only language that runs in web browsers, and when we started writing uTheory, this is the language we chose so that anyone with a phone, tablet or computer could use uTheory. In Javascript, any variable can hold any kind of value: text, numbers, a boolean (true/false), complex objects of data, or nothing at all. And you can change what kind of value a variable holds whenever you want:

let x; // The variable x is empty, or "undefined" in javascript terms
x = 3; // x holds the number 3
x = "3"; // now x holds the text "3"
x = new Date('2003-03-03'); // now x holds a date object for 3/3/3

This makes getting up and running with javascript easy, but it puts the onus on the programmer to track what type of data any variable holds at any time. As codebases grow, that puts significant mental strain on programmers, and makes it harder to make changes without introducing errors. If you don't know if x is a number or some text, you don't know whether you can divide it by something, or if you can convert it to lower case.

So, we've moved the vast majority of our codebase (85%, or 154k LOC) to TypeScript, a language that allows you to specify what kind of data any variable can hold. Typescript provides tools so that whenever you use that variable, you're given hints to prevent doing illogical things (like trying to divide "Hi, Greg!" by 32).

TypeScript can be transformed into Javascript (a process called compiling, or transpiling) so that it can run in web browsers. And, generally, if you're using the strictest verification settings, as we are, a TypeScript program will only compile if it can actually run.

Automatic Unit, Integration and Regression Testing

Alongside of this, we've vastly increased the percentage of uTheory which is automatically tested, focusing on those parts of the codebase which are used most widely. Before we deploy a new version of uTheory, these tests check to be sure things work as expected. Given the musical nature of our codebase, the testing process can be delightfully cacophonous:

But it helps us make sure that by changing line 65, we didn't accidentally cause line 143,201 to break!

Upcoming Data Migration

New Database

Since late 2015, uTheory has used RethinkDB as its primary database. RethinkDB is a natural partner to the Javascript language. It will store deeply nested data of any format, and provides an expressive language for retrieving information from the database. But, like Javascript, it gives no guarantees about the format of the data it stores.

And even early on we had problems with its performance, and turned to work-arounds, like storing data for class grade-sheets separately from the original student data, which added undesirable complexity to our server code. In the years since, we've simply outgrown RethinkDB.

Over the past month, we've completely rewritten uTheory to use PostgreSQL, a battle-tested relational database with strong data type guarantees. We've written and tested code to move existing data to our new Postgres database, and on Sat., January 29th at 10pm Eastern, we'll take uTheory offline for somewhere between six to twelve hours to make the transition to using Postgres.

On the surface, not much will change. At first, all of uTheory's functionality will remain the same. But, under the hood, uTheory will be operating on a chain of technology that is designed to help programmers move quickly in large codebases.

That's because with Postgres we know what the shape of our data is. In combination with Prisma, it enables our developers to rely on autocomplete rather than their own memories. Check out how little typing I actually do when writing a function to load a user's skill progress. The autocomplete (or "intellisense") practically does it all!

What's Next?

In the coming months we'll be rolling out new features and expanding configurability for uTheory. From new rhythm and ear training exercises and lessons, to more robust progress tracking, we think you'll love what our new tech stack is going to make possible!