Thursday, January 12, 2012

KF8 is nothing more than EPUB with mobi

OK, if you've been reading me a while, you know that I mostly specialize in EPUB, that I have a background in HTML, and that I know the basics of converting EPUB to mobi.

Today, as the Kindle Format 8 specifications were made public, the #eprdctn Twitter group filled up with complaints about what the new Kindle Gen and Kindle Previewer 3 were doing to people's ebooks. It sounded worrisome.

So, I took my “From InDesign CS 5.5 to EPUB and Kindle” book, which I had previously converted to mobi with the old Kindle Previewer 2, and converted it with Kindle Previewer 3 [Note that the new Kindle Previewer 3 contains the new Kindle Gen 2, with a GUI interface. Though it's ostensibly just for viewing mobi files, it also converts them without having to resort to the command line.] And then I unpacked them with my new toy, mobi_unpack, and compared the files inside.

Inside there are old mobi files and new KF8 mobi files (and the original EPUB!).

I started with the old mobi files. What I didn't realize is that Amazon has been generated non-standard hideous code for their ebooks for years. It's like MySpace in here. Look:

Here's my original code:

<p class="caption">The Articles panel starts out empty. You have to add articles to it manually either by dragging frames to it or by clicking the plus sign.</p>

And here's how KP 2 converted it to old mobi:

<p height="0em" width="0" align="center"><font size="-1" face="sans-serif" color="#000000"><i>The Articles panel starts out empty. You have to add articles to it manually either by dragging frames to it or by clicking the plus sign.</i></font></p>

And here's how KP3 converts it to old mobi:

<p height="0em" width="0" align="center"><blockquote width="0"><font size="-1" face="sans-serif" color="#000000"><i>The Articles panel starts out empty. You have to add articles to it manually either by dragging frames to it or by clicking the plus sign.</i></font></blockquote></p>

All I can say is ew. <font>? Really? What is this, 1997? Not only is this garbage in your HTML, when it should clearly be in the CSS, but it's old, and deprecated. Even when I set the Doctype to HTML4 Transitional, about as loose as you can get, the document created with the brand new Kindle Previewer 3, released today, had 3482 errors and 239 warnings. It does not validate. It's garbage.

The only differences I found in files converted with the old Kindle Previewer and the new was that they no longer add !important as an extra value for the align attribute (makes me shudder just to think of it, and I can't imagine it ever did anything in the first place), and the addition of blockquote tags, who knows why (as in the above example).

That was the old mobi content.

You can also find KF8 content in the Kindle Previewer 3 converted mobi file.

First I compared the CSS. It's very similar.

The only difference I found was that in the KF8, the max-width property was eliminated. I imagine as I investigate more, I'll find other things that KF8 won't accept. I'll let you know.

Now on to compare the HTML.

I see that KP3 changes <a id="Anchor-123" /> to <a id="Anchor-123"></a>. Not monumental. Not even significant.

And there were a bunch of spacing things, but really, if people are noticing differences upon viewing documents, I don't think it's because of the conversion.

The KF8 inside the mobi is practically identical to my original EPUB file.

But then, I opened up the new KF8 mobi file in my Kindle Fire. I found one small difference with the indenting.

KF8 indents

The things is, I'm not sure that's KF8's fault. I think it's probably the fault of mobi, which had some weird indenting behavior that I distinctly remember fiddling with a lot to get the original effect (right). And it looks like it breaks in KF8, which, as I noted above, is much more standard and looks just like my original EPUB.

So, I just need to fix the indenting.

All in all, I didn't see a lot of difference in the way this file was displayed, between the old mobi and the new KF8 mobi. Indeed, I think the move away from that disgusting code is a major improvement.

And the KF8 file looked exactly the same on my old Kindle 3 also. No change that I could discern. (In other words when you serve a KF8 file to an old Kindle, it keeps looking at the same old clunky  mobi code that it always has, and the book doesn't break—anymore than it did before.)

I still want to go through each property, bit by bit, to see what's supported and how, but so far, I'm hopeful.

Now, why they just don't call it EPUB, since that's what it is?

37 comments:

  1. So why can't we all get along with EPUB 3?

    ReplyDelete
  2. This gross and unnecessary code in the MOBI format sort of explains the mystery of why the file size is always twice as big as EPUB. Thank you for sharing.

    ReplyDelete
  3. liz said:
    > it's old

    and your point is... what?

    the rendering engine is old.
    so yes, it expects "old" code.

    and the amazon engine is
    "old" because amazon was
    making e-books happen when
    i.d.p.f. was feet-dragging.



    > deprecated.

    what a quaint word.

    .mobi worked well enough to
    have made millions of amazon's
    _paying_customers_ happy...

    and amazon made the format
    work -- consistently! -- on
    a wide variety of machinery.

    the .epub world can't even
    make its format work well
    _or_ consistently on the
    handful of machines from
    companies that declare it
    as their major file-format.

    so you guys got _no_business_
    throwing rocks at anyone else.

    make your own stuff work first,
    and work _better_, and it _will_
    win. but i don't think you can.

    you just want to whine about
    the only guys who really care
    about the e-book experience...


    > It does not validate.

    who cares about "validation"?

    not amazon, because they don't
    have to depend on anyone else.
    they make their own stuff work.


    > It's garbage.

    no, you're myopic. you think
    things like "validation" matter.

    amazon carries the load for
    the entire user experience.

    and yeah, they're "proprietary".

    but apple, and barnes&noble,
    and adobe wrap d.r.m. around
    their products, and thus make
    a mockery of the "open" label,
    so what difference does it make?

    absolutely none.

    indeed, it makes it _worse_,
    because it allows them to
    pass the buck when stuff
    doesn't work correctly,
    whereas amazon makes it work,
    since they take responsibility.

    -bowerbird

    ReplyDelete
    Replies
    1. Well said, couldn't agree more. I love the word myopic. Honestly, how often to you get to use it?

      Amazon have completely altered the landscape of publishing, virtually on their own. Without the Kindle there would be no digital revolution and there would be no one making money working digital conversion houses. Looking back in fifty or a hundred years, this is the moment in time that will mark the shift to digital in the literary world, even though ebooks have been around for thirty years.

      Everyone's just made at Amazon because they're the only business in the business who actually understand that it is business.

      Validation is irrelevant and garbage can be truly beautiful.

      Delete
    2. It's true I was a bit snarky in my description of the old mobi code, and I regret it only because it distracted from a much more important point: Amazon is now using a format that they call KF8 that is virtually identical to EPUB (with the old, non-standard, clunky, and anemic mobi tacked on for legacy Kindles).

      That is a victory for standards and I think a very good move by Amazon. Standards really do matter if you're a book creator, since it means that you don't have to waste your time creating the same book ten times over for different devices.

      And if you're a book reader, that matters too, since there will be more time spent on creating more and better books for you to read.

      Delete
    3. Actually, Amazon's rendering is not at all consistent. It is very easy, using vailid, normal, un-hackish code to get very different results (in margins particularly), between Kindle e-ink devices and some of their apps.

      Delete
    4. …and my limited testing suggests that the formatting on the Kindle Fire reproduces some of the same collapsed margin bugs that exist in the iOS app (at least).

      Delete
    5. If you think standards don't matter or that it's irrelevant that old mobi is garbage, read this about Amanda Hocking, which was part of the reason she left self-publishing: "And what about the hours and hours that she's spent since April 2010 dealing with technical glitches on Kindle" (From The Guardian today: http://www.guardian.co.uk/books/2012/jan/12/amanda-hocking-self-publishing)

      Delete
  4. Keeping files as "mobi" with "epub" inside is a way to control and own their market. They could just as easily accept and distribute epub, but then how would they force people to buy content from them?

    ReplyDelete
  5. Hey Rick, the newest mobi_unpack.py is here: http://www.mobileread.com/forums/attachment.php?attachmentid=80487&d=1325110121

    Here's the description: http://www.mobileread.com/forums/showthread.php?t=61986&highlight=mobi_unpack&page=16

    ReplyDelete
  6. Liz, your article was a bit confusing on the first time through, because you talk about converting with the Kindle Previewer. It should instead say with Kindlegen, right? The Previewer just displays the result.

    Aside from that, very helpful! Thank you!

    ReplyDelete
    Replies
    1. Kindle Previewer actually contains Kindlegen in its package. If you open an EPUB file in previewer it will run it through Kindlegen for you and display the results.

      If you're going to be previewing the file in Kindle Previewer anyway, its easier to just convert that way - less steps to take. Unless I want non-standard Kindlegen compression, it is my main method of running kindlegen nowadays.

      Delete
  7. Sure, there is a mistake for Kindle Preview instead of Kindlegen, but where is the v. 3 of Kindlegen? I can find only the v. 2.3....

    ReplyDelete
    Replies
    1. As Nick replied above, Kindle Previewer 3 contains KindleGen2 inside a lovely GUI interface. I tested the resulting files and they were identical.

      Delete
  8. Liz,

    Don't get hung up on the seemingly large .mobi file size created by kindlegen or Kindle Previewer. Once upload to KDP, Amazon clearly pulls it apart, probably storing the alternate formats so they can provide the customer the correct format for the device/reader they are downloading it to. I have checked the .mobi files we uploaded to KDP and the file sizes of the Kindle books as displayed on Amazon -- prior to KF8, the uploaded .mobi was about twice the size of the actual Kindle book on Amazon. I'd expect the .mobi now to be three times the size of the Kindle book on Amazon (I haven't yet uploaded a new .mobi file created with kindlegen 2). I think this is Amazon's method of ensuring backwards compatibility and future-proofing.

    ReplyDelete
    Replies
    1. Hi Walt. Yes, I'm sure if you're selling through Amazon, you don't have to worry. I'm just less and less willing to put all my eggs in their basket, and I don't want the mobi files that I sell directly to be bloated. I'll figure it out and post instructions :)

      Delete
    2. That raises a good question. If you're selling directly to consumers (rather than through Amazon), would the only version you can produce through Kindle Previewer be the bloated one? Was there a way to "de-bloat" it and then sell 2 separate kindle files on your website: 1 for Kindle 3 and one for the Fire and beyond? Like you, I'm thinking of selling directly to consumers instead of selling through Amazon, but selling a bloated file might have performance problems. (I don't know). I seem to recall that mobie for kindle 3 was just a zip file (I'm not sure), so that would imply that all you'd need to do was to unzip and remove the excess files, then rezip. I'm not sure that this would be a proper Kindle file though.... (Maybe Calibre would offer a solution?)

      Delete
    3. Hi Robert: I'm hoping to go through the process and then describe it here. It sounds like some folks over at MobileRead created a script that you can use called KindleStrip. It looks like they're working on a version that deals with KF8 right now: http://www.mobileread.com/forums/showthread.php?t=96903&page=2

      Delete
    4. Thanks for the info. Yet another reason why it seems ludicrous for Amazon to pioneer another format when another format is the standard. 5 or 10 years down the road, we will probably know whether Amazon's decision to go proprietary really did gain them market share.

      Delete
    5. Liz,

      In no way do we put "all our eggs" in the Amazon basket; we just use their unequaled platform to sell eBooks (and have been doing so since the advent of the Kindle, and selling pBooks through Amazon since 1996). All of our eBooks are available on Amazon, B&N, iBooks, Kobo, and a host of the numerous very minor players.

      We also offer them on our own website, but there's just no way we can realistically come close to the breadth and depth of Amazon's marketing machine. As long as they continue to treat us fairly, we'll keep working with them...and making money doing so.

      Delete
    6. Hey Walt. I didn't mean that you were relying solely on Amazon but rather that many people do, and I think it's a mistake. I much prefer your strategy of having your books available on as many platforms as possible.

      Delete
  9. @Liz, your work is so helpful to everyone. I would love to have a standard way to set up to test on Kindle Fire, Kindle, Nook and iPad where I can test the same content in different devices with a minimum of rework. I'm not doing Fixed Layout. Do you have recommendations for a "test bench" methodology?

    ReplyDelete
  10. Thank you. I don't have a very formal testing system worked out, but maybe I need to figure one out. As we have to test on more and more ereading systems, it becomes more and more important.

    ReplyDelete
  11. Liz, general KF8 question. So we've read the (incomplete--grumble) KF8 publishing guidelines/specs.

    They reference a fixed layout example (top of page 24, "please refer to the attached fixed layout example"). Obviously there is no file attached to the PDF. Searches turn up nothing. Are we missing something? Have you been able to locate a fixed layout example for KF8?

    Among other things, we're looking for a sample OPF file.

    ReplyDelete
    Replies
    1. Found the samples! They're in the sidebar, under "Kindlegen Samples" on the right.

      http://www.amazon.com/gp/feature.html/ref=amb_link_357613502_1?ie=UTF8&docId=1000765211&pf_rd_m=ATVPDKIKX0DER&pf_rd_s=right-4&pf_rd_r=0SY1PX3X16YX5RB413B6&pf_rd_t=1401&pf_rd_p=1343256942&pf_rd_i=1000729511

      Delete
  12. Liz, What is the status of any of the Kindle formats supporting video & audio?

    I see that their samples include an audio example (mp3) but I've not been able to get it to play on the Fire or on the Mac rendition of Kindle. And my small test book with a video clip (based on your AV guide) plays well in ePub but does not run on any of my Kindle readers.

    ReplyDelete
    Replies
    1. Hi Keri: KF8 supports audio and video but only Kindle apps for iOS (that is, on iPad, iPhone, and iPod touch) support them. Kindle Fire does not support audio or video in ebooks, and there is no word on when they will. Further, from what I've heard, KDP, Amazon's self-publishing system, does NOT accept ebooks with audio or video from independent or self-publishers. Sorry to be the bearer of bad news.

      Delete
    2. So if I have a book where the author wants to include music samples readers could play at appropriate points, iBooks is the only way he could distribute such a book? And the new Kindle guidelines for publishers describes explicitly how to embed audio and video.

      Delete
  13. @Doug: He could also sell ebooks with embedded audio from his own website, in either EPUB or mobi formats. If he sells through iBookstore, it will only be for EPUB, not Kindle. Yes, the new Kindle guidelines explain how to embed audio and video (as does my Audio and Video in EPUB miniguide).

    ReplyDelete
  14. Liz,

    The author wants to leverage his (substantial) hard copy sales on Amazon, so he was hoping to offer a Kindle ebook with full functionality. It sounds like that isn't an option. (BTW, I mentioned the specs not because I doubted you but because Amazon shouldn't have put out those specs without a warning that you can't upload to KDP. And I already own the A/V miniguide!) Anyway, to recap: if you're an independent publisher, you can put audio and video in your EPUB ebook, but you'll only be able to sell that book via the iBookstore or your own website. Ditto for an ebook prepared for mobi/KF8, except you WON'T be able to sell that via KDP. Finally, do EPUBs with embedded A/V work in ADE? Thanks as always!

    ReplyDelete
    Replies
    1. Yes, that's all correct. I agree KDP should warn folks that those books aren't allowed from mere mortals like us. I think they originally planned to allow such uploads. Even if they did, you'd want to be very careful about delivery costs, I should've mentioned this earlier. And no, embedded AV doesn't work in ADE.

      Delete
  15. So, thank you for the link to the mobi_unpack. But, how do I repack it?

    ReplyDelete
  16. Hey Liz. I'm having terrible issues getting my KF8 rendered in Kindle Fire. It's actually funny/frustrating because it looks perfectly in my Kindle Previewer. But once I send it to my Kindle Fire it looks like regular HTML, no CSS or custom layout added whatsoever.

    ReplyDelete
  17. Hey Liz,

    I converted my first epub to kf8. I wanted to test for myself which features were available, although they're listed, and I have a question. Although video and audio are supported, they supposedly only play on the kindle App for IOS, well, when I converted it dropped both the audio and video that I had inserted into my epub, and neither are showing up in the code or when I view the mobi in my kindle app on the ipad. Ideas? Does conversion strip the unknown / unsupported tags?

    Thanks :)

    ReplyDelete
    Replies
    1. I believe there's still no way to sideload a Kindle book with audio/video to a Kindle. And no way to publish it unless you're a big publisher (i.e. not through KDP).

      Delete
    2. Thank you very much for the absurdly prompt response :)!!

      Delete

More of my books