Unicode, Tolkien, and Privacy

Unicode, Tolkien, and Privacy

When I’m in the mood to write, you can often follow a chain of though in my posts. Recently, a post on LLM tokenization lead to a post on how Unicode characters are tokenized, which led to a post on Unicode surrogates. The latter ended by touching on Unicode’s PUA (Private Use Area), which of course leads to J.R.R. Tolkien and privacy, as we shall see. To back up a bit, Unicode started as an attempt to one alphabet to rule them all, a coding sys…