Friday, October 29, 2010

iTunes invalidates EPUB files for iBookstore???

Second update:
OK, thanks to more information from Andrew, I've figured out what the problem was. Although it's true that iTunes annoyingly adds that iTunesMetadata file to your original EPUB file when you copy the file to iTunes, that's not what's causing the invalidation. But if you use iTunes to add a cover to your EPUB, as described in this video by Terry White, and then upload that EPUB to iTunes Producer, that EPUB will not validate and thus will not be accepted into the iBookstore, because the cover image is not properly contained in the OPF file.


Instead, the cover should be included in the EPUB file either via Pages' "Use first page as book cover image" option, or via the instructions in my book.




First update: I've just been able to upload an EPUB to the iBookstore with the iTunesMetadata.plist file intact, so I'm not sure exactly what's going on. It's not in the iBookstore yet, but the iTunes Producer says it validates, despite the iTunes file. I'll let you know what happens. It looks like I may have jumped to conclusions that the iTunes Metadata file was the culprit in this story. (It does continue to bug me that iTunes changes my files without asking!)

One of the things that this issue brings up is how hard it is to test things in the iBookstore. First, you have to sign up (which requires a US Tax ID and US Bank Account valid iTunes Store account). Then, you have to have an ISBN for your book, which in the US can cost as much as $125. And finally, you have to actually have the real content of a real book—not a test, according to Apple's rules—to upload. For many of us who are testing the waters and don't yet have the final product in hand, these are difficult impediments to overcome. Apple, couldn't you make it easier for us to test our EPUBs?





I got a note on my blog the other day from Andrew Brooks who is publishing an enhanced ebook to the iBookstore. He has used Apple's Pages software to create the EPUB and even included enhanced elements. But when he went to upload the EPUB file to the iBookstore, he got this error:

“An Apple requirement is that all the files in the ePub be listed in the OPF file. So, the error was that there was an unmanifested iTunes artwork file.”

Little did he know that when he opened the EPUB file in iTunes, iTunes had the audacity to add a little file to his original EPUB document. No, not only to the EPUB that he copied to iTunes, but also to his original EPUB file. The file is called "iTunesMetadata.plist". You won't see it unless you look inside an affected EPUB.

To add insult to injury, when Andrew tried to upload his EPUB to the iBookstore, his ebook was rejected because of the aforementioned error! In short, iTunes adds an erroneous file to his EPUB and then Apple says he can't upload the file because of it.

It turns out that Andrew used iTunes to add a cover to his book, as described in this video by Terry White (and how people have been doing for records for a long time). Unfortunately, although the EPUB and cover will work properly in a local iPad, Apple won't accept it because the EPUB no longer validates. You have to add a cover to your EPUB either with Pages' "Use first page as book cover image" option, or via the instructions in my book.

As I told Andrew, that's pretty ridiculous. We're all testing EPUBs on our iPads and iPhones and we need iTunes not to invalidate our EPUBs while we do it.


There are two solutions. First, keep a clean copy of your EPUB—that has not been copied to iTunes—for uploading to Apple. Or second, crack open your EPUB after copying it to iTunes and eliminate the iTunesMetadata.plist file.


OK, and third, send a note to Apple and tell them that when you copy a file to iTunes, that does not give them the right to alter your original file. And shoot, if they're going to mess with your files, they shouldn't complain about it afterwards.


Note: I have not been able to test this myself, because I don't have the content for an ebook, and so Apple will reject anything that I try to upload based on the fact that it's a test. I'd love to hear if others have had this problem.  I uploaded an old story that I wrote in college to test this issue. I'm going to need some more content for future tests!

13 comments:

  1. I haven't had this problem with iBookstore since I upload only clean files, actually. The epub is invalid because the iTunesMetadata.plist is added to the epub but not listed in opf, and I think that also if it would it will be invalid the same way because of the unsupported media-type.

    That said, I'm curious about those epub in iBookstore including audio&video (if there are any, I'm not so sure): they are invalid too, for what I know, and I wonder how they can be on the store. Do you have any news on that?

    ReplyDelete
  2. HI Letizia:
    Apple changed the rules about validation; EPUB files no longer have to pass ePUBCheck 1.0.5 completely. They are allowed to have "errors" with regard to the audio and video elements. It's all detailed here: http://www.pigsgourdsandwikis.com/2010/08/apple-no-longer-requires-epubcheck.html

    ReplyDelete
  3. Thanks Liz, you confirm what I was imagining. I think they don't use ePubCheck 1.0.5 strictly, since an epub with the iTunesMetadata.plist file inside passes the validation with that validator version. If you test it with 1.0.6 (still for developers, actually) you get the warning we are talking about, instead.

    ReplyDelete
  4. Hi Liz, not entirely on topic but just to clarify a point made in your first update. As a UK publisher www.wearenorthernsky.com I am pretty sure that you do not need a US bank account to do business on the Apple iBookstore, you are required to insert an international banking number if your account is located outside the US. You do however need a valid US Tax ID, but this is fairly easy to get by phoning the IRS.

    ReplyDelete
  5. Thanks, Steve, you're quite right (and I knew that, just got over-excited!) I've updated that bit.

    Also, here's the number that "foreign entities" can call for help expediting the issuance of a US Tax ID: 1-215-516-6999. There's more info here: http://www.irs.gov/businesses/small/international/article/0,,id=96696,00.html

    ReplyDelete
  6. Liz, I can certainly appreciate your frustration with not being able to test the iBookstore system. My wife and I are working on a poetry book that could be more innovative than Apple can tolerate. I wish that I could submit a prototype of the first three poems and get feedback from Apple. The issues I anticipate include:

    o The book is not in English yet is to be sold in the US to Hungarian speakers. The iBookstore is not yet available in Hungary.
    o Each poem will have an .m4v video with a soft subtitle track that can be toggled on/off in iOS devices.

    No doubt there will be issues I haven't anticipated.

    As for the ISBN, I don't think that it is cost-effective to purchase just one at a time. The per unit cost for the smallest batch (10) is much better. We plan to publish two versions, enhanced (with video) and a "regular" version w/o video but with still images instead.

    ReplyDelete
  7. Those who're frustrated by the hassles that Apple seems to be tossing in their way might want to go the Smashwords route. It'll take a Word document and create the proper ePub code for Apple, Sony and B&N, as well as supply it directly to those vendors. And if you want, they'll supply the ISBN. My only Apple title so far, More to William Morris, was done that way.

    The downside is that you lose the ability to control what comes out of Smashword's Meatgrinder Word-to-ePub converter. If there are glitches in the resulting ePub code you can't tweak them as Liz might. Also, you can't add extended features. It's only for uncomplicated books like novels.

    The upside is that Smashwords will let you get a book out quickly. Later, when you've created an enhanced version, you can stop the Smashword distribution to any of the outlets you choose and go directly to Apple with the revised and enhanced version.

    It's not the ideal solution, but in the present, muddled ebook market there is no ideal solution. And it is a way to reach almost everyone but Amazon with one upload.

    ReplyDelete
  8. I noticed iTunes alters the original EPUB file on import but hadn't investigated further. The .plist file includes a hash encrypted key for the cover image, "package" file and a "unique-id" number. I wouldn't be surprised if these are used for iBooks' cache control.

    ReplyDelete
  9. Thanks also for the pointer on the US Tax code. The number has been updated to 00-1-267-941-1099 dialling from the UK. Called and got a code within about 15 minutes. Much easier than I expected. Now just need a Mac Intel...

    ReplyDelete
  10. Hi, Liz:

    I tweeted about this to #eprdctn on the 27th Oct--and it has nothing to do with the Cover being placed via iTunes. I don't place covers via iTunes; I code them the old-fashioned way, and I still had this problem with the ePUB being kicked from the iBookstore queue. Here's the error message I was given:

    Itunes log report stated:
    Unable to authenticate the package: 1234567890.itmsp
    [2010-10-16 09:36:16 PDT] ERROR: ERROR ITMS-9000: "Unsupported mime type for 1234567890.epub: null" at Book (MZItmspBookPackage)
    [2010-10-16 09:36:16 PDT] ERROR: ERROR ITMS-9000: "META-INF/container.xml in 1234567890.epub must contain one and only one root file reference." at Book (MZItmspBookPackage)
    [2010-10-16 09:36:16 PDT] DBG-X: The error code is: 1102

    ==========================

    So--the problem isn't limited to only those who, for whatever reason, upload covers via iTunes; if you're testing an ePUB on iBooks, as I was (for an enhanced iBook--video), the itunesmetadata.plist file will still show up and will still cause problems with the upload.

    Hitch

    ReplyDelete
  11. Any idea of the average time it takes a title to get through Apple' s quality control?

    ReplyDelete
  12. Hi Liz,

    I got some error messages when the ePub (validated with Sigil) was submitted to the itunes store. is there any tool to validate the ePub with the Apple Standard. I read something about an Apple iTMSP validator. Is there any such thing.

    Have a nice day.

    ReplyDelete
    Replies
    1. Yes, use EpubCheck. The so-called "Apple iTMSP validator" is what is used when you submit your book to the iBookstore with iTunes Connect. You can't invoke it independently, and the error messages are opaque at best.

      Delete

More of my books