Parsing Other People's File Formats
Kleo reads whatever ebook you throw at it. Almost all the engineering is in the part nobody sees.
Kleo is a web ereader. You upload a book, it opens, your place syncs across devices. The part you interact with is deliberately quiet.
Almost all of the engineering sits in the step before that.
Files are not data
An EPUB is a zip of HTML with a manifest and a spine telling you what order to read the files in. In theory. In practice:
- The manifest disagrees with what is actually in the archive
- Chapter breaks live in the markup, or they don’t
- Images are referenced by paths that do not resolve
- Encoding is declared as one thing and is another
- Two files both claim to be the same chapter
PDFs are worse, because a PDF describes where to draw glyphs on a page. It does not describe what the text is. Reading order is an inference, not a fact. Pull text in document order from a two-column layout and you get the two columns interleaved line by line.
The shape that worked
- Parse. Open the container and enumerate what is actually inside, not what the manifest claims.
- Extract. Pull text and structure out of each part, normalising encoding as you go.
- Structure. Work out real chapter boundaries and reading order.
- Store. Keep the extracted form, not the original, as the thing you serve.
- Serve. Render from the clean version, with reading position synced separately.
The important decision is step four. Parse once, store the result, serve from that. Every time I have seen someone parse on read instead, it ends the same way: slow, and inconsistent across devices because two clients parsed the same file slightly differently and now the user’s position means different things in different places.
The bit I underestimated
Failure handling.
Some percentage of files will not parse cleanly and no amount of work drives that to zero. You have to decide early whether a partially parsed book is better than no book, and then tell the user which one they got.
Silently serving a book with three missing chapters is worse than refusing it. The user finds out at chapter four, blames themselves for a while, then blames you.
Why I keep coming back to this
Every document AI project I have worked on is this pipeline with a model bolted on the end. Ingest messy files, extract structure, store the clean version, then do something clever with it.
The clever part gets the attention. The parsing decides whether any of it works.