MuPDF Removes Linearisation

Why has MuPDF removed Linearisation support from its saving code?

In the latest version of MuPDF, we’ve removed the ability to linearise a PDF file as it’s saved. This blog post is a quick explanation of why we’ve done this, and why we don’t think anyone should really mind.

Removing features is something that we never do lightly, but we think it’s the right thing to do here.

First off, what exactly is Linearisation?

Well, when the PDF format was first produced (35 years ago!) the average Internet connection was a lot slower than it is now.

Having to wait for the entirety of a PDF to download to be able to start viewing it was a real imposition when that meant hanging around for 10 minutes on a 16kbps modem; especially when your kid brother would pick up the phone just before you got the last of it, and you had to start all over again!

So, the brains at Adobe invented the idea of allowing a PDF to be ‘Linearised’. This is a posh way of saying that you could view the first page quickly, while the rest of the file continued to download.

That sounds great. Why are you removing that?

Well, yes, it was a neat feature. But it had its limitations; only the first page was accelerated in this way. If you want to access page 2, say, you still need to wait for the entire document to be downloaded.

Yes, it’s nice to be able to see the title page, but if you can’t even get to the table of contents faster, it certainly doesn’t help you look up what you need on page 34!

And Internet connections are much faster these days; even cellphone connections are hundreds of times faster than modem speeds. The issue has largely gone away.

Couple this with the fact that with modern byte-range capable webservers (the norm for the past 20 years), you can get better results with intelligent fetching of normal PDF documents than you ever could with linearised files, it’s really a technology whose time has passed.

Even so, why remove the code?

Well, the code has never really worked terribly well. When we first wrote it, we did our best, and it seemed to work, but, as almost no one actually supported it at all, it was really hard to get it tested.

Even now, decades on, it’s largely an unused part of the spec - very few people actually even attempt to implement it. Those packages that do have their source code sprinkled with comments like “the spec doesn’t say this, but…”.

While the PDF specification is a large document, that’s actually fairly well defined, this particular area of it is, shall we say, “open to misinterpretation”.

There are implementations out there, and it was in attempting to make one of these happy with the files we were producing that lead us to take the decision to remove the code.

We are by no means the only package to have problems here; a certain “well known implementation” can produce documents that upset the validator we were using.

So, why not just fix the code?

Well, we tried.

But in doing so it became clear that different people had interpreted the spec in different ways. Trying to find the “correct” way of working is by no means trivial; indeed, it may not actually be possible to satisfy everyone.

Even if we could find such a way that kept every consumer happy, it’d take us a lot of development effort (time that could be better spent elsewhere), and it’d leave us with much more complicated file writing code than we’d like.

So, we have taken the decision to remove the option entirely.

So, what does this actually mean?

Firstly, this means that we’ll no longer be generating files that claim to be linearised, but actually aren’t. We’d rather not make claims that we can’t back up!

Secondly, it means that our file writing code is simpler, faster, and more maintainable. Less complexity inevitably leads to fewer bugs, and makes it easier for us to add new features in future.

We strongly believe that almost no one will ever notice the difference between working with linearised PDFs and normal PDFs these days. If you are one of the vanishingly small number of people for whom this is a genuine issue, then there are other free tools out there that can post-process a PDF produced by MuPDF into a linearised one(*).

(* Contact us for a recommendation! )