Opening new windows in iBooks

Rick Gordon mentioned that the ebooks on programming that Apple just published to the iBookstore had some interesting features, among them tables that opened up into a new window, similarly to what happens when you double-click an image:

• the table fills the iPad space, ignoring the typical iBooks margins,
• the regular navigational and presentation features disappear, so you can't move directly to the next page, or change the font or font-size
• if the table doesn't fit in the space, you can pan down or to the right to see the rest of it
• double-clicking the table or clicking Done in the upper-right corner returns you to where you were
• although tables are a natural use for new windows, you can put anything in one

Here is the table as it would appear regularly in iBooks (ew!):


And here is what it looks like in its own window (ah!):


You can pan down to see the lower part of the table:


And when you tilt the iPad vertically, the table resizes to fit:


Since tables are really hard to show well in iBooks (or any ereader), this is a really welcome development. Now, any time you have a complicated table, instead of risking having it mangled, you can just have it show up in its own neat window, and be sure your readers will be able to see it properly.

How do you do it? You simply go into the content.opf file and in the spine section add linear="no" to the document that should open in its own window. Remember that the spine is the place where you define the order of the documents in your ebook. I have complete details about the content.opf file, and the spine in particular, in my book, EPUB Straight to the Point.

I've found that the text in the new window is smaller by default than text in the regular book pages. Again, it is not affected by the text size choice of the reader, so you have to make sure you set it big enough to start with.

You can download my sample EPUB file and look at the code. It's published with a Share Alike and Give Attribution Creative commons license. You'll notice that ePubCheck complains that I didn't fix the Wikipedia footnotes and that I didn't get rid of the iTunesMetadata.plist file. But I'm running out of time and wanted to get this up today. I'll fix those minor things later.

It feels like there's a lot we can do with us. I'd love to hear about what you create!

If you like this article, consider buying my book. It's a great foundation for all things EPUB.

Apple disses ebook developers at its peril

I'm feeling really grumpy today, and I've traced it back to a Twitter conversation last night in which I discovered that Apple has been giving support and information on how to create their new fixed-layout ebooks for iBooks 1.2 only to a select group of ebook producers, and under non-disclosure agreements (NDA).

I just don't understand why Apple wouldn't want to help all ebook developers make great looking ebooks that display beautifully on the iPad and so help sell millions of units. Instead, those of us who aren't on the list have to sleuth out, through time-intensive trial and error, just what will work and what won't.

Sure, that's my job. I'm good at explaining how to do things and figuring what people want and need to know and putting it in a way that makes sense. I don't need them to help me. But what I don't understand is why they don't want to. Why do they want to keep their documentation a secret?

That's not Apple's strategy for Safari, whose documentation is very useful and complete. Perhaps that is because they realize the web does not belong to them? The only way to compete is to offer a powerful browser and to explain to people how to get the most out of it.

For iBooks, though, it's all hush-hush. Perhaps they don't want to admit that they're new fixed layout "illustrated books" are nothing more than glorified PDFs— EPUB files in name only. Perhaps not invalid, but many of these fixed layout books are almost impossible to read because the type is so small, and they are all but unusable on smaller screens than the iPad. What is the point?

And Apple has been less than forthcoming with information about earlier versions of iBooks as well, publicizing books that violate the very requirements listed on their web site, promising to follow the EPUB standard, but picking and choosing just which pieces to support.

The ebook market has taken many years to take off, since both readers and books are needed to make it go, and both were lacking. When ereaders were so expensive, few people had them, so there was little demand and thus little production of ebooks. For me, the iPad was a fiendish way to get ereaders into the hands of millions of people who were curious about ebooks but reluctant to spend $400 on a standalone ereader. iPad buyers could justify the price because of all the other things it could do. All of a sudden, millions of people had an ereader, and were clamoring for more books. It was only after the iPad existed that the price of the Kindle and Nook fell to the ground, encouraging even more people to jump on board. More ereaders means more demand for ebooks, which makes ereaders more attractive, and so on.

Apple's strengths in the ebook market are its beautiful color ereaders (e.g., iBooks on iPad), its support of HTML5 to include audio and video, and its embrace of the standard, non-proprietary EPUB format. These features are a big part of why EPUB is still a viable option, in the face of Amazon's juggernaut.

But an ereader is only as good as the books you can read on it. The iBookstore is practically empty. I hear from more and more people that they would like to buy books from the iBookstore but what they're looking for is not available. No wonder, if Apple is keeping its documentation under lock and key. I would love to help people make gorgeous books for the iPad and iPhone. Apple, you ignore us at your peril.

Page Breaks in iBooks 1.2 (Updated!)

Updated 11 Feb 2011: The bug described in this post has been fixed as of iBooks 1.2.1. The app now recognizes the page-break properties both in local CSS and external CSS. See Page Breaks, Turning off Hyphenation and Horizontal Bolding Bug in iBooks for more details.

One of the most annoying failings of iBooks to date has been its lack of support of one of CSS' most basic controls: when and where to place page breaks in the text. Without this control, you can't force a header to begin at the top of the page, nor keep a caption with its paragraph.

When the new version of iBooks came out yesterday, I heard (through Fabio Brivio on Twitter) that these page break controls were now supported, but I couldn't seem to make it work. I asked Fabio if he would share his EPUB with me, and he kindly emailed it. (Thanks, Fabio!)

After pulling it apart, and applying it to my files, I finally figured it out what the key was: iBooks only supports CSS page break controls when they're in a local style sheet. They won't work in an external one.

So, if you want each of your h3 elements to begin on a new page, in the head section of each of the pages in your EPUB, add:

<style type="text/css">
h3 {page-break-before: always;}
</style>



Other possible values for page-break-before include avoid, if you'd rather a page break did not occur right before the element, and the very interesting, but not yet supported left and right, which push the element in question to the first left or right page, respectively.

You can also create page breaks after an element (or group of elements). Use page-break-after.

Finally, you can use page-break-inside: avoid to keep an element on a single page. For example, you might apply this rule to a div that contained an image and its caption.

CSS also has ways for controlling widows and orphans (stray lines that appear at the top and bottom of pages), but unfortunately, iBooks doesn't yet support them. (Nook, and other ereaders, do.)

I cover these page break features in detail in my book, HTML, XHTML, and CSS: Visual QuickStart Guide, which is a big help for any EPUB book designer who wants to create a really polished ebook. You can learn more about how to get inside an EPUB file, and what to do there, in my EPUB Straight to the Point.

At some point, iBooks will properly support all of the page break values, and will support them both in local CSS as well as the more useful and common external CSS. You'll read about it here first!

Embedding Fonts in iBooks 1.2

Actually, you've been able to embed fonts since iBooks 1.12, but you needed iOS 4.2 to make it work. I'll go over it anyway, because it's awesome.

First, you need to use OpenType Fonts. They have the .otf extension. It looks like TrueType fonts (with the .ttf extension) and on which OpenType fonts are based, are also OK. Remember that getting a font into your ebook is only a part of the battle. You also have to have the permission to distribute the font with your book. Read the font license carefully.

Once you have a font file that you want to embed for your ebook, there are two steps you have to follow. First, you have to use an @font-face rule at the top of your CSS file.

Something like this:

@font-face {
font-family: "FontName";
font-style: normal;
font-weight: normal;
src: url(../fonts/Fontname.otf);
}

I have not found much correlation between specifying a font-style or font-weight in the @font-face rule and success. That is, sometimes I've found that for an italic font you have to specify font-style: italic and sometimes you have to specify font-style: normal, which makes little sense, but that's the way it goes. It does seem necessary to create an @font-face rule for each and every style of every font you want to embed.

The next step is to use the fonts that you've now embedded in the actual style rules of your document:

p {font-family: "FontName";
font-size: 2em;
...
}

It's important that the font name you use be the official one. Any multi-word names should be enclosed in quotes.

The third step is to place the OTF font files in a fonts folder within your OEBPS folder in your EPUB document. If the actual font files aren't in there, they won't be embedded.

And finally, you have to declare those font files in the manifest of your EPUB:

<item id="fontname" href="fonts/fontname.otf" media-type="application/x-font-otf" />

(You can find more details on the manifest in my EPUB Straight to the Point book.)

You can use any id as long as it's unique. Make sure the path accurately reflects where you plced the OTF file. And be sure and use "application/x-font-otf" for the media-type.

And you're done! Here's a screenshot from one of Apple's new illustrated books with a new font. Note that Apple has used the font, Filosofia Regular, for the body text, even though their Publication guidelines expressly discourage such usage. They use various forms of Caecilia for the notes and inline headers.



And herein lies one of the main problems with the new "illustrated books" in iBooks 1.2. The reader can't change the font, and the reader can't change the font size. The text in this particular book is really hard to read. I'm racking my brain trying to figure out why this is better than a PDF, but I don't get it yet.

iBooks 1.2 - Going Back and Forward

There are lots of interesting features in the new version of iBooks 1.2. I'm going to see if I can get through them all today, but instead of putting them all in one post, I'm going to publish them as I finish each section.

You can go back

Now when you click on a link in a book a tiny link appears in the lower left corner of the page allowing you go back to the page from whence you came. When you go back, there is then a link in the lower right corner letting you go back to where you came back from (that is, forward).

This is a huge improvement, and makes footnotes/endnotes and links in general MUCH more useful. I love it.

Imagine you're on a links page in Walden:



You click on the line "I have always been regretting that I was not as wise as the day I was born." and jump to the page that contains that link.

In the bottom-left corner of the new page, you see "Back to page xx".



If you click on it, you'll go back to the page you came from. And now on the page you came from, you'll see a light gray link in the bottom-right corner that lets you go forward again to the page you had clicked to go to.

Apple, stop invalidating my EPUBs with your iTunesMetadata.plist file

Dear Apple:

I'm really tired of iTunes adding an iTunesMetadata.plist file to my original EPUB files every time I test said EPUB files on my iPad or iPhone through iTunes. To add insult to injury, every time you do it, it causes an error in ePubCheck—since the EPUB file now contains a non-manifested file—thus making the file invalid for uploading to your iBookstore.

Could you please quit it?

Sincerely,
Liz Castro

____________

Feel the same way? Feel free to copy the above letter and post it in your own name to Apple's Feedback page for iTunes. Perhaps if enough of us speak up, they'll listen.

For more information, see iTunes invalidates EPUB file for iBookstore???

Reading Library Ebooks on iPad with Bluefire, the Easy Way

There were a couple of people who thought that the instructions I posted for reading library ebooks on the iPad (and similar devices) with Bluefire Reader were a bit complicated, and didn't want to have to download in one place, upload in another, and transfer things around. Bluefire's Micah Bowers offered some help for creating a direct link in the comments, and another reader, Greg from Oregon, wrote a script to make that new link practically create itself.

I'm going to finish off the job by explaining step-by-step how to make a bookmarklet out of Greg's script, following Greg's instructions as well as a process I learned from installing Instapaper (a wonderful, but unrelated app). Don't you love the internet!

1. Open up your iPad (iPod Touch or iPhone) and come back to this page. Remember, the address is http://www.pigsgourdsandwikis.com.

2. Follow this link to jump to a page that contains the script that you'll have to add as a bookmark. (The whole point of this part is to save you the trouble of typing in the script, but feel free if you're so inclined.)

3. You will find a bit of explanation and a block of code. Press in the white space until the Copy command appears. Drag the upper blue dot so that only the red code is selected. Choose Copy.



4. Click the big plus sign in Safari's toolbar and then choose Add Bookmark from the pop-down menu in order to create a bookmark. Of course, you don't really want a bookmark to the instruction page, but we can't copy in the code until the bookmark itself exists. Click Save.



5. Once the bookmark exists, click the Bookmarks icon (next to the Plus sign), and click Edit. Then click the name “Overdrive bookmark” to open up the bookmark we just created.



6. Erase the old link by holding down your finger over the address (the part that starts http://www.elizabethcastro...) and then choosing Select All and finally Cut.



7. Press in the link space one more time until the Paste button appears. Choose it! The script is copied to the bookmark. You can click anywhere in the page to close the Editing window. (You don't have to save.)



The bookmarklet is installed and you never have to look at the code again! Now you're ready to go to your library and find a book. Follow the regular procedure (see Borrowing Library ebooks with a Nook if you need help). Once you've checked out your book and are ready to download, instead of clicking the Download button, click the Overdrive Bookmark we just created.



Rather magically, a new page will appear with a much better link:



And when you click the link, you jump to Bluefire Reader and it downloads your book, verifies your authorization code as normal, and then wonderfully asks you if you'd like to Read it later or Read it NOW!



Ah, you may scoff, that's even worse that what I told you the other day. Perhaps. But once that bookmarklet is installed (it's OK to get someone else to do it for you! or for you to install it for your parents or your kids), reading library ebooks on the iPad is a veritable breeze.

Thank you Micah, Greg, and the guy from Instapaper for working this all out!

And a couple of notes. I haven't tested trying to download multiple titles at once. I suspect it won't work. Just download them one at a time. And I haven't been able to get it to work for PDF ebooks. Not sure why not. Maybe you know?

And finally, this does not remove or bypass DRM at all, since Bluefire requires your iPad or similar to be authorized with your Adobe account. Further, I didn't find any information on Overdrive's site that would indicate they care what device or software you use to read their books, as long as you're not trying to override their protection.

One more finally. I was curious if you could get the book back out, say to read on your computer or copy to another authorized device, and indeed you can. Just go to iTunes, click on the Apps tab, click on Bluefire to see the books it contains, and drag the desired book back out to your computer. Works fine.

Bluefire Lets You Read Library Books on the iPad

Woke up to a lovely morning tweet: "@lizcastro Done.  Update available now in App Store that supports library books." from @micahsb, a developer and entrepreneur. He was referring to the fact that the new version of their Bluefire Reader app for the iPad can display Adobe DRM'd EPUB format ebooks that you've taken out of the library.

And it works! I connected to my library, and quickly did a search for Adobe ePUB, “took out” and then downloaded The Mysterious Benedict Society, which we're in the process of reading, and clicked OK when I was asked if I wanted to open the file in Adobe Digital Editions. This converts the download file into a Bluefire-compatible Adobe DRM'd EPUB file and places it in the Digital Editions folder, within the Documents folder (or My Documents folder on a PC).



Next, I opened iTunes, connected and selected my iPad, chose the App panel, selected Bluefire at the bottom of the window in the File Sharing section, clicked Add..., chose the book file from where it was copied—in the Digital Editions folder within the (My) Documents folder—, and then clicked Sync.



I then opened Bluefire Reader on my iPad, and there was my book! Absolutely lovely.

EPUB Webinar Wednesday, Nov 17 (and more!)

There are a lot of things that have changed about me since I was 15, but wanting to be a teacher has remained constant. OK, it's true I didn't enjoy teaching high-school Spanish (at Princeton Day School in 1986!), but I have always loved explaining thing and feel lucky that I figured out a way to make a living at it.

There is one thing that has eluded me since my days teaching Spanish though: standing up in front of a live audience. I've dabbled with a screencast here and there, but though I've been invited to talk at several different conferences, I've always shied away from to speaking in public. Until now.

David Blatner and Anne-Marie Concepción, InDesign Gurus extraordinaire have invited me to do two EPUB related webinars for InDesign Secrets. The first, InDesign to EPUB with Liz Castro, on November 17 at 10am PT, will be a 75 minute survey course, explaining how to go from InDesign CS4 or CS5 to validated EPUB. I'll show you both how to go from an existing InDesign document originally created for print, as well as what you need to keep in mind in order to create InDesign documents that can be leveraged for both print and digital editions of your books.

Early Bird Tickets are available until Wednesday, November 10th (acks, that's the day after tomorrow!). You can get an extra 10% off just because you read my blog (or follow me on Twitter) by using the discount code GOURDS. (You gotta love David's sense of humor :)

On Wednesday, December 1, also at 10am Pacific (which is 1pm on the East Coast and 7pm here in Central Europe), I'll be doing the second EPUB Webinar: Advanced EPUB Formatting with Liz Castro. In that class, which also lasts 75 minutes, I'll show you how to crack open your EPUB document and add and adjust formatting by hand, including special fonts, keeping elements together, text wrapped around images, and much more.

You can take advantage of early bird pricing by buying your ticket to the second webinar by November 24. And don't forget to get your extra 10% discount by using the GOURDS code!

Ah, you say, that's not standing in front of a live audience! Well, I'm getting there :) For those of you on this side of the pond, I will be giving a talk (in Spanish) in Santiago de Compostela—which many say is the most beautiful city in Spain—on EPUB and how we got here on November 25th at the Trends & Creativity [GL] conference put on by the Clúster do Produto Gráfico e do Libro Galego [GL]. Many thanks to my Twitter friend Tati Mancebo for connecting me up with those folks.

Drumbeat Open Video Lab - Day 2

So I have to preface my take on Day 2 at Drumbeat in Barcelona with a little background. About ten years ago, I dared to write a book on programming, on Perl and CGI in particular. And although I think one of my strengths as a technical writer is bringing a fresh perspective to complicated topics, and indeed the book was quite successful and helped a lot of people add interactivity to their web sites, it was not well received by Perl purists who painted me as an interloper, a gasp!, non-programmer.

Of course, it was and is true that I am not a programmer. Indeed, my angle, if you can call it that, has always been to make technology more accessible for regular people, people who may not code for a living, but who are willing to roll up their sleeves and dip into the underpinnings of a project and not just accept what commercial software allows them to do.

I'm inspired by the spirit of The Macintosh Bible, one of my first publishing projects ever (in particular the Spanish version of the 3rd edition) in which Arthur Naiman made it clear that “easy is hard” and that if you dug a little, you could find a million tricks that would make your computing life that much easier. The Maker phenomenon is a newer reincarnation of this same spirit: that we don't have to settle for what they give us, we can open it up, and build a better version. I love that.

At the same time, I have a hard time shaking off the feeling that I'm just an outsider looking in, not a real programmer, not a real hacker.

So, I'll admit it was with some trepidation that I returned to the Open Video Lab yesterday morning, knowing that they wanted us to build a real project, and that they were going to depend a lot on Javascript and the popcorn.js library that they had built previously. Even their focus on HTML5, which was a big part of why I showed up, was a little daunting. (Does everyone imagine that everyone else knows ten times more than they do?)

I found the whole process fascinating. I'm used to working completely alone, in my own office, doing every bit of my projects, from concept to outline to examples to illustrations to layout to copyediting alone. When I don't know something, I go quietly searching for information, without having to tell everyone I work with what it is that I don't know.

Drumbeat, in contrast, was all laid wide open. About 10 of us returned from the previous day, along with two or three new folks. Dave Humphrey got us thinking about the projects we had talked about on Thursday and helped us both focus on what the projects consisted and what we were trying to solve, as well as what skill sets the members of the group had to offer. There were a lot of Javascript and HTML coders as well as a few designers and video production people. I was totally intimidated by the idea of writing actual code with these guys, and at the same time didn't want to lose the opportunity of learning what they knew. I was also unsure I had enough to offer myself, though thankfully I wasn't the only one to say this.

We ended up dividing into two groups. The first, led by Brett Gaylor, wanted to create a jazzier, prettier demo of popcorn.js, a Javascript library that allows you to line up data to particular points on a video timeline, and thus create mashups like the one they showed the day before in which people in the video tell where they're from and this information is used to trigger Google Maps and Wikipedia giving more information on that location.

A Javascript library is a bunch of code written in such a way that it can be used not just for the original project for which it was conceived, but also easily leveraged for other projects. It's an integral element of the programming community that attempts to highlight and save the best code so that people don't have to continually reinvent the wheel.

The second group, which I was a part of, wanted to figure out a way to expose existing metadata, like the title, creator, or date a video was produced, that was associated with a video either right in the page (with RDFa), or with an external XML file, so that people watching the video could actually see information about the video with a single click.

We then spent about half an hour, with Dave Humphrey's helpful facilitation and focus, discussing the different parts of the project: where we would get the metadata, how we would parse that data once we had it, how users would reveal the data, and then what the data would look like once revealed. The most curious part for me, was then dividing up into the groups that would work on each section of the project.

One group took on the task of figuring out how to pull the metadata out of the HTML file. Although I didn't follow them as closely, I'm pretty sure they used some existing RDFa Javascript libraries. The second worked on writing the code that would make a button appear and disappear when the user clicked. They too were working mostly in Javascript. I was in the third group that worked on displaying the found information right on top of the video. We were writing a combination of Javascript (both with Jquery, a larger web-page related library and without), HTML, and CSS.

I think the part that most surprised me is how small the individual pieces of the project seemed at first, and how much they all realized that they weren't really that small, and would take the better part of the day to create, even divided up between the participants as they were. It was also interesting seeing how they wrote the code in a way that they could test it separately, but that it could later fit together with the other pieces. I loved how they really drew on the different strengths of this randomly assembled group of people and made it possible to all work together. There was really good energy in the room.

We broke for lunch and I was lucky enough to go off with Mark Boas, who is a web developer who has written a popular Javascript plug-in for dealing with audio called jPlayer. He is based in Italy but also has a Catalan connection, and it was great to talk to him and hear what he's working on.

Back at Drumbeat, we all mostly finished up our individual modules, and then began joining them together. Curiously, that was mostly done via USB stick, as the wifi was intermittent at best. Each time the code was joined, we had to test to make sure that what worked separately now would play nicely with the other person's code. The use of jQuery as well as straight Javascrit was problematic, but not crippling.

Once all the code was together, they connected the last computer to a projector so that we could all see and debug the final project together. At this point, the designer folks jumped in and offered suggestions about positioning and font size. What a collaboration!

There were a few hours left before the Drumbeat-wide presentation of projects at 6pm, so I wandered off to find Nicholas Reville who was talking up Universal Subtitles, which is a great collaborative way to add subtitles to videos to make them more accessible to the hearing impaired and to those who don't speak a given language.

And I also found fellow locals Patrick Davenne and Chris Pinchen who offered me beer and conversation, and we lamented the complete disconnect between the festival and Barcelona itself, among other things.

Then back up to the video lab. Our metadata project was finished but the folks doing the popcorn.js demo were still working furiously. They had taken a video of folks at the conference in five languages, edited and compressed it, translated and added English subtitles with the previously mentioned Universal Subtitles, and then used popcorn.js (I think) to pull in and display tweets from the conference on the page as the video plays. The problematic wifi connection had made it hard for them to stream the video in order to synchronize the subtitles, and I was happy to be useful for a few minutes interpreting so that we could finish up the timing.

Perhaps my favorite moment was when about ten of us were hovering over the person doing the final screencast, trying to figure out why the timecodes were appearing on screen. It was crowd-debugging, and we found the extra space and got it out just in time to finish the capture, and run down to the presentation with USB stick in hand. [link coming soon]

So, I'm still not a programmer, but Drumbeat—and the generous, smart people within—let me have a peek into their world that made it feel that much less intimidating. Sometimes when you don't know how something works you imagine it ten times harder and more complicated than it really is. It's probably time for me to work on sharing that idea—particularly with respect to Javascript—with my readers.

Drumbeat Barcelona - Open Video Lab - Day 1

Thanks to @pdavenne and @cataspanglish and their Barcelona Social Media camp, I happened to meet hear Enric Senabre (@esenabre) give a talk on Mozilla's Drumbeat Festival, held this week in Barcelona. Drumbeat is the Mozilla's foundation to go beyond an open source browser and try to find other parts of life that can be enhanced with open source software. The theme of this week's festival is open education and how to facilitate it with open source software. There are 400 attendees, about 75% of which are from out of the country.

Which means I spent the day in San Francisco, practically, complete with a talk by Aza Raskin, but I'll get to that. It was pretty surreal being surrounded by Americans and English speaking Europeans right in the center of Barcelona.

I opted to spend the day in the Open Video Lab. I wasn't quite sure what that was going to mean, and I think a lot of the people, including the organizers, Brett Gaylor, Dave Humphreys and Ben Moskowitz might not have either. That didn't stop them from skillfully guiding and coaching the group to figure out some hands-on projects to work on.

First, they had us go around the room explaining where we came from and what we were working on. There were a fair number of video producers, web designers, educators, and also some coders. I jotted down a few of the topics that people were either working on or interested in: subtitling videos, teaching digital video, knowledge mapping, video production with minority youth, putting videos online, linking existing metadata with video, management consulting, teaching with video, designing and developing video, screen casting, mixing video with education, producing documentaries, video literacy, jplayer javascript library, media studies, documenting systems, figuring out how things relate, web video projects.

They/We came from the UK, Germany, Minnesota, Barcelona, France, Brazil, Galicia, and Madrid, to name just a few.

Brett and Dave began the workshop by asking us "what is possible with open video" and then showing us a few demos that took advantage of HTML5. There was a page with a video of a whale with an overlaid canvas element that mapped the audio to a visual representation. There was a kung fu video that used the browser to add a shading effect in real time. There were even video games, that mixed 3d, Flickr, Twitter, and rendered right in the browser.

Then they talked about popcorn.js, a Javascript library that Brett (I think) and a crew of students from Canada put together in a week to deal with video with HTML5. They showed us a sample that they had created that day with a video from Drumbeat conference attendees saying where they were from, superimposed on a window that was half Google Maps, half Wikipedia entry for that location.

Then Wendy from Bay Area Video Coalition showed us some of the videos that they're producing, with an eye towards promoting social justice.

Ben Moskowitz showed us a demo of MediaThread, a project at Columbia that allows professors and students to reference, annotate, and cite videos available through YouTube or other sources. It looked really valuable. The code is open-source, but the working project is unfortunately available only to folks associated with Columbia University.

Ben also showed us Pad.ma, an Indian site that catalogs videos with all sorts of different kinds of metadata, including name, title, keywords, and words in the transcript, and then lets visitors search through the metadata for particular videos.

Then it was time for the hands-on section. They showed us the example they had pulled together that morning, referenced above, using the popcorn.js library.

But before we got to the code itself, we talked a little about codecs and video editing tools. There was general consensus that despite the existence of some tools, none really offered the same capabilities as Final Cut.

As for codecs, Ben told us that Theora and webM were both open-source, as opposed to the most prevalent codec, H.264, which required paying licensing fees. He said that in about a year we would all be using webM, a standard developed by Google from vp8, and for which Google had paid the licensing fees for everyone (could that be right?), but that it was not yet well supported, and so recommended using Theora, at least for now.

Theora, though, doesn't work in Safari, so you really have to make more than one video source file available in order to be compatible with Apple-based browsers (including on the iPad).

Then he showed us the HTML5 video tag:

<video id="video" src="..." controls data-timeline-sources="locations.xml">

He said you could use HTML and CSS to format the controls, or use the controls attribute. And that attributes that begin with data- are recognized as extensions to HTML5 and thus will not keep the document from validating.  It works because of the popscript.js script being called. He unfortunately did not get to showing us the locations.xml tag.

He then quickly showed us a little tool that he said he had written in a few hours to gather latitude and longitude coordinates from Google Maps that he could then feed into the mashup.

But then we broke up into small groups and we didn't delve further into the code. I hope we get there tomorrow.

In our breakout group, we talked about metadata, and also touched on individuals' particular issues.

We talked about how there are two different classes of metadata... what I like to call formal (dublin core structured) and informal (tags, keywords, made up by individuals), and how they have different purposes (the former for finding the video itself, the latter for pinpointing bits within the video).

Meanwhile, Dave kept helping us focus in on a particular problem... and we decided it was how to expose existing metadata that was already related to a video. There were a few people who had libraries of video as well as XML structured metadata and wanted to be able to overlay the metadata on the video so that the metadata was revealed in the browser.

The other breakout groups came back, and my very shorthand summaries of what they found were:

Students using video to learn, then go back and watch themselves is different than producing video itself (meta-cognitive)

Need one place to put all files, design matters, generational issue

Tools: learner and teacher tools not necessarily the same

Destroy those boundaries, teacher proposes, students remix and turn on head; said another way: teacher create lesson plan, students mess with it

Then we were treated to this great presentation by Aza Raskin on Prototyping.

How to think about Prototyping and why
Hardest part about software development is the people, convincing them to make something
Must build first 100 miles (prototype), then build resort at end (inspiration)
Value of idea is 0
Unless communicated
You want project to be touchable and feelable
Idea < writeup < mockups < prototype < video
Firefox Panorama, organizing tasks spatiallly
Goal of prototyping is to convince yourself of the idea
   0 You will be wrong first time
   1 Complete prototype in a single day
   2 Make a touchable sketch (don't do everything)
   3 Tight feedback loop (dogfood?!)
   4 iterating solution helps illuminate problem
   5 treat code as throwaway, be ready to refactor
   6 steal design, make beautiful

jqueryfordesigners.com

Pitch your prototype
How does it make THEIR life better?
Be dramatic.

Example: Twitter streamer

And then I went back out into Barcelona!

Editing EPUB files right in the iPad

Rick Gordon has long touted his method of editing EPUB files right in a jail-broken iPad. But I've been too chicken. But I am finally getting caught up with some reading I needed to do and found this great article by Keith Fahlgren over on the ThreePress blog (which you should definitely also follow, if you're into EPUB). The article explains how to use the iPhone Explorer program to access the XHTML and CSS files on your iPad to facilitate updating EPUBs. And it's awesome.

Digging around a bit, I found that Macroplant, the folks who wrote iPhone Explorer, also created a little program called Phone Disk, that usually costs $10 but is free until December 1. It's even better than iPhone Explorer. What it does is show you the contents of your iPad (or iPhone or iPod) as if it were any other external drive. Check this out:



You'll notice that there is a Books folder inside the iPad (mine's ambitiously called, “Firstpad”) and that within the Books folder are a series of folders with cryptic names, all ending in .epub. Although you can open up the Books.list file to see which cryptic code is related to the book you're interested in, I find it faster to list the books by Date Modified. That way, the one that I'm working on is always at the very top.

According to Macroplant, you should be able to double-click on a file and open it, and that works for
me with XHTML and CSS with TextEdit, but curiously not with BBEdit, my editing tool of choice, which acts like it's hung before recovering and showing you the file, making it too slow to be useful.

Thankfully, the folks at Barebones gave me a solution. They say that BBEdit looks like it's hanging because it's checking for a few files, and that on a remote file system like the iPad, it takes a longer time. To fool BBEdit into not worrying about where it is, you have to create two empty text files, "bbedit.bbeditSettings" and "tags", and place them in the EPUB folder whose book you want to edit. Now when you click an XHTML or CSS file, it'll open right away in BBEdit.



After you make and save changes in the XHTML file (or CSS), you'll need to close and reopen the book on the iPad to see how the changes look. But this is nothing compared to rezipping, dragging, waiting for it to sync, opening it back up again and so on. It will save you a lot of time.

When you're done testing and adjusting your EPUB file, you need a way to get it out of the iPad so you can distribute it. First, be sure to eliminate both of the BBEdit files that we added. I like to drag them out to the Books level so I'll have them for the next book I'm working on. Then, delete the iTunesMetadata.plist file (which Apple annoyingly adds to your EPUB) as well. It seems like you should then be able to sync the iPad with your Mac via iTunes and then drag the EPUB file out to your desktop but I haven't been able to get that to work (the newer EPUB from the iPad doesn't ever get to the iTunes library).

Instead, still using Phone Disk, drag the entire EPUB folder from the iPad out onto the desktop, and then zip it back up with Terminal. (You remembered to delete the special BBEdit files and the iTunes Metadata file, right?)

Being able to make edits directly to your EPUB on the iPad (or an iPhone) is a huge time saver. Thanks, Keith, for getting me started on this.

Apple releases fix for iPhoto '11 - in the most confusing way possible

Update: On Sunday, Apple fixed the confusing headline in their support article described below. It now correctly says “Update iPhoto '11 to 9.0.1 before upgrading library”. Much better!

---

 Apple has just released an update for iPhoto '11, which they say “in extremely rare cases, could result in data loss when upgrading a library from an earlier version of iPhoto.”

I can't deny that the language really rankles. Frankly, I think they should be worrying more about their customers' photos that were lost and less about how it sounds. There should be a press release that alerts people to the problem and to the solution to make sure that no one else loses this very precious data.

And it doesn't help that they've done it in the most confusing way possible. The headline to the support article says “Upgrade iPhoto library prior to iPhoto '11 Update”:



But that's exactly what you should not do! Indeed, the whole rest of the article is about updating iPhoto '11 before upgrading your iPhoto Library, since upgrading your iPhoto Library with the originally shipped iPhoto '11 is what is erasing people's files.

This is made worse by the fact that they released an update called "9.0.1" for iPhoto "11". It turns out that iPhoto '11 is really version 9.0 (while the last version, iPhoto '09, was really version 8). It's confusing Marketing speak and right now it's pretty annoying. It makes you think that you should go back to iPhoto '09 and update it before installing iPhoto '11. That's not the case. To find out what version you have, choose iPhoto > About iPhoto.

The other problem is the confusion between “updating” the software (iPhoto itself) and “upgrading” your library of photos (the iPhoto Library package, of which there may be more than one, it may have some other name, and is usually found in your Pictures folder).

When you install iPhoto '11, it doesn't automatically or immediately touch your library. So, it's safe to install iPhoto '11 from the disks, then update it with Software Update or by manually downloading and installing the update. Your library will not yet be affected. You should definitely do this first, regardless of the headline of Apple's support article.

Once you open iPhoto '11, it will ask you if you want to upgrade your iPhoto Library. Here's where a lot of people (oh yeah, I forgot, extremely rare cases) had trouble. Make sure you have at least one and preferably two or more full backups of your library or libraries before proceeding. And make sure you've installed the update.

Apple says that updating your iPhoto Library can take “up to an hour or more” and that even if the program looks like it's hung, you should not force quit. Be patient. And contact AppleCare if you get stuck.

iTunes invalidates EPUB files for iBookstore???

Second update:
OK, thanks to more information from Andrew, I've figured out what the problem was. Although it's true that iTunes annoyingly adds that iTunesMetadata file to your original EPUB file when you copy the file to iTunes, that's not what's causing the invalidation. But if you use iTunes to add a cover to your EPUB, as described in this video by Terry White, and then upload that EPUB to iTunes Producer, that EPUB will not validate and thus will not be accepted into the iBookstore, because the cover image is not properly contained in the OPF file.


Instead, the cover should be included in the EPUB file either via Pages' "Use first page as book cover image" option, or via the instructions in my book.

---

First update: I've just been able to upload an EPUB to the iBookstore with the iTunesMetadata.plist file intact, so I'm not sure exactly what's going on. It's not in the iBookstore yet, but the iTunes Producer says it validates, despite the iTunes file. I'll let you know what happens. It looks like I may have jumped to conclusions that the iTunes Metadata file was the culprit in this story. (It does continue to bug me that iTunes changes my files without asking!)

One of the things that this issue brings up is how hard it is to test things in the iBookstore. First, you have to sign up (which requires a US Tax ID and US Bank Account valid iTunes Store account). Then, you have to have an ISBN for your book, which in the US can cost as much as $125. And finally, you have to actually have the real content of a real book—not a test, according to Apple's rules—to upload. For many of us who are testing the waters and don't yet have the final product in hand, these are difficult impediments to overcome. Apple, couldn't you make it easier for us to test our EPUBs?

---

I got a note on my blog the other day from Andrew Brooks who is publishing an enhanced ebook to the iBookstore. He has used Apple's Pages software to create the EPUB and even included enhanced elements. But when he went to upload the EPUB file to the iBookstore, he got this error:

“An Apple requirement is that all the files in the ePub be listed in the OPF file. So, the error was that there was an unmanifested iTunes artwork file.”

Little did he know that when he opened the EPUB file in iTunes, iTunes had the audacity to add a little file to his original EPUB document. No, not only to the EPUB that he copied to iTunes, but also to his original EPUB file. The file is called "iTunesMetadata.plist". You won't see it unless you look inside an affected EPUB.

To add insult to injury, when Andrew tried to upload his EPUB to the iBookstore, his ebook was rejected because of the aforementioned error! In short, iTunes adds an erroneous file to his EPUB and then Apple says he can't upload the file because of it.

It turns out that Andrew used iTunes to add a cover to his book, as described in this video by Terry White (and how people have been doing for records for a long time). Unfortunately, although the EPUB and cover will work properly in a local iPad, Apple won't accept it because the EPUB no longer validates. You have to add a cover to your EPUB either with Pages' "Use first page as book cover image" option, or via the instructions in my book.

As I told Andrew, that's pretty ridiculous. We're all testing EPUBs on our iPads and iPhones and we need iTunes not to invalidate our EPUBs while we do it.


There are two solutions. First, keep a clean copy of your EPUB—that has not been copied to iTunes—for uploading to Apple. Or second, crack open your EPUB after copying it to iTunes and eliminate the iTunesMetadata.plist file.


OK, and third, send a note to Apple and tell them that when you copy a file to iTunes, that does not give them the right to alter your original file. And shoot, if they're going to mess with your files, they shouldn't complain about it afterwards.


Note: I have not been able to test this myself, because I don't have the content for an ebook, and so Apple will reject anything that I try to upload based on the fact that it's a test. I'd love to hear if others have had this problem.  I uploaded an old story that I wrote in college to test this issue. I'm going to need some more content for future tests!

Wait to upgrade to iPhoto 11: Apple says fix is imminent

I just spoke to someone at Apple who called me all the way here in Barcelona to see if my data was OK and if I needed any help. I was pretty impressed. They know how important our photos are to all of us.

They confirmed that there was indeed a window during the updating process in which a force quit could “create data loss”. I asked if I would have had the same problem without force quitting and the answer was probably not, though they didn't really specify how long I would have had to wait. There are people on the Apple Discussion Forums who have let their updates go for days with no apparent signs of life. I asked if RAM could have been an issue, and they said no.

They asked me if I had any projects created with older versions of iPhoto, and I said I had two from iPhoto 5. And they asked me if I had any video playback extensions for QuickTime (I don't). They also asked how big my library was (233gb).

I asked if changing permissions of the iPhoto Library was really helpful (as someone suggested on one of my blog posts), and was told that it was only part of the solution. Deleting old projects does not seem to have any effect whatsoever.

Mostly, and this was the most important thing, they told me that a fix from Apple was imminent and that we would soon have official confirmation of same.

It's probably best to wait for the fix, but if you can't, be sure to properly BACK UP your iPhoto Library before upgrading to iPhoto 11!

DON'T RELY on Time Machine for iPhoto backup before upgrading to iPhoto 11

I got more than 6000 visits to my website yesterday from folks worried about iPhoto ERASING their iPhoto Library, like it did to mine. People are listening. Which is good. But sounding such an alarm is an enormous responsibility that I don't take lightly. I am a very long-time Apple fan and iPhoto is one of my favorite programs.

But every time I think I might be being too harsh on Apple (maybe it was just me, maybe I really could have done something besides force quit when it hung and said it wasn't responding, maybe somehow my 233gb of photos really were shoehorned into 6.5gb etc. etc.), I go to the Apple Discussion Forums and read something like this:

"Well I am one of those who didn't have backup, so thousands of pictures and videos of my kids growing up is gone."

or this

I've lost all the photos of my new son, his Christening, my family over from Australia and my dad's 70th Birthday

and this last one is pretty awful:

iPhoto libraries are not backed up by Time Machine if iPhoto is running during the back up process. I leave my iPhoto application open constantly. I bought the best MacBook Pro money could by with 8GB of RAM so I could have multiple applications open for my design work.  This is an absolute horror show for me. 

and then I'm convinced I haven't yelled enough.

First and foremost, make a BACKUP before you upgrade your software.

And DON'T RELY on Time Machine. Although Apple says it works "in the background" without you having to do anything, it doesn't. I'm afraid I've been ranting about that for a while too. If iPhoto is open (and I leave iPhoto open all the time), Time Machine will skip your iPhoto Library, even as it backs up everything else.

You will think you have a backup, and you will be wrong.

Here are the steps:

1. Close iPhoto. (Again, Time Machine will not back up an open iPhoto Library.)
2. Run Time Machine, or other backup software.
3. Make a separate, additional, independent backup of your iPhoto Library onto an external hard drive, or other media, and store it off site (in some other building). Even if you can't figure out where to take it off site, don't let that stop you from making the second copy. Just drag the iPhoto Library to an external disk.
4. Look at your backups and make sure they have the same size as your original iPhoto Library. To do so, select the iPhoto Library and choose File > Get Info.

Only then should you attempt an upgrade to iPhoto 11.

What if you've already attempted an upgrade, and had your iPhoto Library disappear?

What worked for me was starting up my computer in Safe Mode (with the Shift key down), and then rebuilding Disk Permissions (use Disk Utility, it's in your Utilities folder in the App folder), and then I started up iPhoto with Command and Option pressed down. In the dialog box that appeared, I chose the options 1, 2, and 6. It then asked if I wanted to upgrade my library, and I said yes. And it worked. It took a couple of hours (my library is 233gb) but it worked. It's really slow for me, but so far, and I'm not convinced about the interface, but it looks like my photos are intact.

If that doesn't work, I've also heard of people

eliminating duplicate fonts from the Font Book
Remove DIVX codecs from Library/QuickTime or ~/Library/QuickTime
and even removing files from within the iPhoto Library package
using disk recovery software

Of course, you can also call Apple or post your particular circumstances on the Apple Discussion Forums. There is a wealth of information there.

Good luck. And keep spreading the word:

DO NOT upgrade to iPhoto 11 without a BACKUP!!!

Installing iPhoto 11

I tried to upgrade to iPhoto 11 yesterday, only to find that my 233Gb of photos had been reduced to 6.5Gb, and that my 50,000 photos from the last ten years were gone. Completely. And I was not alone. Reading through the Apple Discussions Forum is heart-wrenching. Post after post from folks who have lost thousands and years of photos. I have done my level best to spread the word and make sure that people have a solid backup (as I thankfully did) before upgrading. If you follow me on Twitter, you're probably sick of me posting about it. But I love my own photos too much to not do everything I can so that no one else loses theirs.

And I guess I love iPhoto too much to give up on it. I thought a lot about whether I really wanted to try again. I even gave Picasa a cursory look as Farhad Manjoo recommends. In the end, I felt like all my public grousing at least demanded that I try again and see if I could find a solution. OK, and I admit it, I still wanted to see it for myself.

Thanks to the folks in the Apple Discussion Forums, I found a solution that worked for me. I restarted in Safe Mode (by holding down the Shift key as the computer starts up). Then I went to Disk Utility and repaired Disk Permissions. After copying my backup of my iPhoto Library back to my computer—which took more than two hours—I launched iPhoto with the Command and Option keys held down. In the alert that appeared, I chose options 1, 2, and 6. iPhoto then asked if I wanted to upgrade my library, to which I said yes.

And then I just waited. And waited. I even left to run an errand. Several hours later, it was done. And all the photos are there. I'll let you know if it was worth it.

But don't forget to make a BACKUP before you try it!

WARNING!!! iPhoto 11 Erases ENTIRE iPhoto Libraries

I was very excited to hear about iPhoto 11. I'm a longtime iPhoto user, with more than 50,000 photos in my iPhoto Library that lately has grown to 233Gb. The Full Screen feature looked great, and I'm dying to see the new books feature. I wrote an entire book on iPhoto book themes and can't wait to see what's new.

Until today. Upon upgrading to iPhoto 11, I got the revolving gray wheel for a very long time, and the Finder said that iPhoto had "stopped responding" so I force quit and started again. I was finally able to upgrade and rebuild my library, but there was nothing there. Every event was labeled properly, and contained placeholders for my images with my labels and even my ratings, but the images themselves were not there. They were all gone.



When I went to see how big my iPhoto Library was, it said 6.3Gb. There's no earthly way my 233Gb of photos could suddenly fit into 6.3Gb. Further confirmation is that suddenly there are 230 extra GB available on my hard drive.

I would be sick (and furious) except that I backed up my library twice before attempting the upgrade. Please pass this information on to everyone you know so that no-one else loses their photos.

Apple's Discussion Forums are full of other people experiencing the same issue. Make sure you have at least one backup before attempting to upgrade. Also, if you depend on Time Machine, make sure Time Machine has been able to complete a full backup with iPhoto completely closed. Note that Time Machine will not properly back up your photos if iPhoto is open.

It's absolutely unconscionable that Apple should release software that would erase an entire iPhoto Library. We must spread the word to minimize the damage. Thanks.

As soon as I figure out how to upgrade successfully, I'll post that information here.

First fiction ebook in iBookstore with video!

I had hoped to write this article using one of my own books as an example. In August, as I wrote here, Apple changed the rules for submitting ebooks with embedded video. If you are interested in publishing a book with video, and downloaded the updated iBookstore Publisher User Guide 1.3.1, you might have been worried about the requirements it contains.

For example, it says you must have a black frame at the beginning and end of every video.

It also says that final files must be produced with Compressor 3.5.2, which, for the record, is only included with Final Cut Studio and Logic Studio Retail, which even when discounted by Amazon, cost lots of money. I count myself among those who thought, "Oh, maybe I don't have the proper tools to put video in my ebooks."

Then, I got a nice note from Jeffrey Dinsmore who had bought my EPUB book and had happily used it to publish an exciting new project that combines short stories with short films. He was thrilled that their new ebook had been accepted by the Apple iBookstore. He says it's the “first fiction ebook in the iBookstore with embedded video”.

It's pretty interesting as a book, that is, enhanced ebook. It's curious to have both media right there in front of you, and to read one and then view the other and think about how and why they were presented differently. You can download a sample and see for yourself, though I don't think any of the videos are contained in the sample. Luckily, you can see them on the Awkward web site.

But that's not why I'm writing this post. I'm writing it because Jeffrey proved, by getting his book into the iBookstore, that those extreme requirements in the iBookstore Publisher Guidelines aren't really requirements. They're recommendations. When I asked him if he had followed the guidelines about black frames, using Compressor, and using a specific resolution of the source video, he replied, “I did absolutely none of the above ... just slapped the .m4v files in and crossed my fingers!”

So, there you go. If you want to publish an ebook with embedded video or audio, all you have to do is follow the instructions as outlined in my EPUB book.

P.S. I wasn't able to get any of my own enhanced ebooks into the Apple iBookstore, not because they weren't technically accurate, but because Apple considers them to be “examples”, despite the copious step-by-step information they contain. I'll keep trying. Meanwhile, if you get your enhanced ebook into the iBookstore, please let me know!

Working at Home

Here's the sort of meme I'd be interested in: what does your workspace look like and how do you make it as efficient as possible? If you decide to join in, please send me a link. I'd love to see where and how you work.

[Cross posted from A Year in Barcelona]

I suppose I should be writing about our road trip through France over the weekend, but instead, as I assembled my office as I do every morning, I thought I might show you my workspace.

You see, here in Barcelona, we have a very small apartment, perhaps about 70 square meters (700 square feet), and I think that includes the hallway and the walls. It's very compact. I don't have a dedicated office space because there simply isn't room for one. Instead, I use the dining room table.

As soon as everyone leaves in the morning, I get rid of all the breakfast crumbs and make sure the table is completely clean. I hate a sticky desk.



Next, I grab the mini Belkin USB hub from the bookshelves, which is already hooked up to my wireless mouse and wired keyboard, and plug it in to the outlet on the floor.



Next, I grab my external monitor from the same bookshelf. I would have liked to have gotten one of those beautiful Apple jobs, but instead, I got a cheap, compact, and light LG (Mine is the M227, very similar to the one linked, but perhaps only available in Europe.) It even doubles as a TV, though I don't use it for that. The monitor goes on the table, and I plug it in as well.



I set my Macbook Pro on the corner of the table with the ports facing the monitor. It's just sleeping, I almost never turn it off. I have to plug it into the wall outlet first, or else the monitor doesn't work. Then I just plug in the monitor cable and the USB hub cable. Voilà, the monitor goes on.



Finally, I grab my fancy mouse pad (read: pad of paper of appropriate size and texture), my wireless mouse, and my stack of papers, from the bookshelf where they live during the evening and set them on the table. I'm ready to go faster than it took to read this article.



In the evening, just before everyone gets home, I reverse the process and put everything in reverse order back onto the bookshelf so they're easier to get the next morning.

I also have a printer and a scanner on the bookshelf, but only put them on my desk when I need them.

What does your workspace look like? How do you make it efficient?

Publishing Ebooks on Barnes & Noble

Barnes & Noble has recently made it possible for self-publishers to upload and sell ebooks for Nooks and the Nook app on the iPad and iPhone—through their Pubit! system. I was particularly interested in this development, because Barnes & Noble supports EPUB format—the open source, non-proprietary format for ebooks.

To sign up for Pubit!, you have to have a US Tax id (a social security number or Employer ID number), a credit-card from a US bank, and a US bank account to which your earnings will be sent. These are almost, but not quite, insurmountable obstacles for non-US residents.

There was one thing in the Terms & Conditions that caught my eye. First, while you set the List Price for your books (upon which your royalties are based), you agree to give Barnes & Noble “sole and complete discretion” over the price they charge your customers. I can’t quite explain why that rankles, though I know I hate giving anyone “sole and complete discretion” over anything of mine!

Next, they’ll pay you 65% of the List Price if that price is between $2.99 and $9.99, but only 40% if the List Price is less than $2.98 or more than $10. It seems that, like Amazon, which has a similar tiered system (70% for the midrange and 35% for very cheap or very expensive), they want to pressure you into keeping the price within a certain range: not so cheap that it’s not worth their while, not so expensive that they end up storing your files for an eternity with no sales in sight.

The excellent news is that you can upload EPUB files directly, which means you have complete control over the layout (and can take advantage of the tips in my EPUB book). However, if you don't have an EPUB file, you can also upload Microsoft Word docs (DOC and DOCX), HTML, RTF, and plain-text documents, all of which will be automatically converted to EPUB.

You can decide whether or not to apply DRM, though once you upload a book, you can't change your mind about DRM. What I'd like to know is whether non-DRM books uploaded through Pubit and bought through Barnes & Noble can really be read on any EPUB compatible ereader (like say iBooks on the iPad). I'll work on testing that and report back.

Book files can't be larger than 20Mb.

It’s nice that you don't need an ISBN, the international book number that booksellers use to keep track of a book within their channels. B&N will assign an “internal” 13-digit number (presumably not an ISBN) to your book.

Curiously, Barnes & Noble says you can only upload books in one of the following languages: English, Spanish, French, German, Italian, Portuguese, Latin, and Dutch. This seems a rather arbitrary limitation. Why not Catalan, or any of the Scandinavian languages, which can all be properly represented with the character set that the Nook already supports?

Why I Crack Open InDesign EPUBs

InDesign CS5 is currently my favorite tool for generating EPUB files from existing documents. It's probably no coincidence that I'm a long time InDesign user in general, quite comfortable with its features and idiosyncracies. In fact, I don't use a standalone word processor at all. I hadn't bought Word for years until I needed it for examples for my latest book. I use InDesign even for the most basic letters. And InDesign's EPUB generation is pretty good: it generates all the persnickety XML files and exports a fair bit of the design from your original document.

But it could do so much more.

Here's why I currently have to crack open InDesign files (solutions to all these problems are in my book). Let's hope the next version of InDesign does all this for me!

1. First and foremost, because the EPUB doesn't validate. To do so, I must add the dc:date element.

2. Because InDesign doesn't properly designate an image or given page as a cover. This is huge!

3. To add metadata. Without standard, accurate metadata, you might as well not publish an ebook because no-one will be able to find it.

4. In order to apply text wrap to images and sidebars. Since InDesign supports the creation of text wrap, around images by themselves, groups of images and captions, and sidebars of text, it should also be able to export it into the EPUB (especially considering that only small adjustments in resulting XHTML and CSS are required).

5. To wrap text around drop caps.

6. To format first line of text (what in InDesign is a Nested Style) so that it maintains formatting in EPUB.

7. To specify which paragraphs should be kept together, for example, images with captions, headers with first paragraph following, etc. Again, InDesign supports keep together but doesn't export to EPUB.

8. To specify page breaks. Although iBooks does not yet support page breaks (which is incredible in itself), other ereaders do. This is a major concern of ebook publishers.

9. To specify widows and headers. One more instance where InDesign already has the functionality but does not support it in export to EPUB. Hugely important for ebook publishers (even though not yet supported in iBooks, it is supported by other ereaders).

10. Because Apple says, on page 15 of their Publishing Guidelines, that "All book elements (for example, the cover, table of contents, first chapter, index, and so on
must be identified in the <guide> block." Note the use of the word "must". Note that <guide> is optional according to the current version of EPUB, and Apple has not yet rejected books without <guide> blocks, to my knowledge.

11. To add pagination guidelines, so that readers can find passages from a particular page of a print edition in their ebook edition (e.g., in a college class).

12. To eliminate erroneous default values that InDesign applies to non-specified properties (which mess up CSS inheritance).

13. To correct/use font names supported by the iPad.

14. To add Zapf Dingbats (which must be added as UTF-8).

15. To add text in foreign languages, like Hebrew, Arabic, Thai, Chinese, Japanese, etc.

16. To fix the spacing bug that occurs when exporting headers (see page 163 in the print edition of my EPUB book).

17. To force iBooks to support left-aligned text (by inserting a span tag within every p element). Granted, this is a hack, but text will continue to look really awful when fully justified until we have decent hyphenation dictionaries in ereaders.

18. To change spacing and margin units from ems to percentages in order to adapt to smaller screens. This is a complicated issue, and will probably get more complicated, but also hugely important.

19. To deal with hyphenation.

20. To fix or insert links. Currently, EPUB export breaks if you try to export a link that contains an ampersand, and most links to Amazon to buy books contain ampersands! <strong>Update (3/Oct/2011): CS5.5 exports links from multiple chapters correctly! </strong>

21. To insert video or audio. I would recommend following Apple's lead on this, using HTML5 tags, but allowing for different formats.

22. To maintain the inheritance among styles in the InDesign document in the generated styles in the CSS. I know this is complicated, but if I can do it with a little bit of GREP, couldn't ID do it automatically?

Most of these issues are described (and solved with workarounds) in my EPUB book. But wouldn't it be nice if InDesign just did this for us? What do you wish InDesign could do?

Pages exports video in EPUB incorrectly

One of my big surprises while testing Pages was the way it treats video. Apple claims on its website that Pages supports exporting video in its EPUB files, and thus, Pages would seem like a great tool for creating enhanced ebooks.

Unfortunately, it's not. And frankly, it just looks like sloppy coding to me. Honestly, I don't understand it.

I created a new Pages document and dragged a video into it, while holding down the Command key, so that it would be "inline" (required for Pages to export to EPUB).

I then exported the resulting document to EPUB. And although the EPUB file passed EpubCheck validation, the video doesn't actually play in iBooks.



That seemed strange to me, so I opened up the EPUB file and looked at the XHTML files generated by Pages. They don't follow Apple's own recommendations!

The problem seems to be that the video is embedded in the wrong format: .mov instead of .m4v. And the file is declared incorrectly in the manifest in the opf file: video/quicktime instead of video/mpeg4.

Indeed, if I make these small adjustments, the video plays perfectly fine. (You can get full details from my book: EPUB: Straight to the Point.)

Also note that although the above referenced document doesn't claim that Pages supports audio, Pages does, sort of. It actually exports audio to EPUB also as a QuickTime file, but somehow it works anyway. Nevertheless, it doesn't follow Apple's own recommendations as outlined in the "iBookstore: Publisher User Guide 1.3.1", which state that audio should be have the .m4a extension. And Apple's sample file uses "audio/mp4" as the proper mime type in the OPF file, while Pages exports as "video/quicktime".

It's downright frustrating that Apple's own tool for creating enhanced EPUB files don't work in Apple's own ereader!

On using Pages for EPUB (part 1)

Apple recently announced that they had added EPUB export to their word-processing software, Pages. If you already own Pages, you just have to go to Software Update to get the new features. My first tests show that it creates perfectly reasonable EPUB files that pass EpubCheck 1.05 with no errors.

Several people have asked me to do a quick rundown on the usefulness of this feature, and so I've been playing with it to a bit. But lately, I've been having trouble doing anything quickly, so instead of obsessing about getting this just right, I'm going to write a series of blog posts as I pull out all the details.

Before I write anything else, I have to tell you that I'm used to laying out technical books in InDesign. With that in mind, I will try not to complain about (or even mention) Pages' weaknesses as a layout program, and instead concentrate how it converts its documents to EPUB format.

Use Word Processing Templates

First and foremost, you should always choose Word Processing templates (or blank ones) when doing an EPUB project. Pages won't let you export documents based on Page Layout templates.

Use Inline Images (not Floating)

Second, most of Pages’ templates include so-called “floating images” which do not export to EPUB. You either have to choose a template with inline images, or convert the images yourself before exporting. For the record, Pages uses the word floating in a way that is quite confusing for those already versed in web design, where floating has to do with text wrap. Indeed what Pages means by floating is much more akin to “absolute positioning” in CSS. Indeed, it means that the image is not linked to the text, and is instead fixed to a particular area of the page (where it “floats”). If you add more text around or on top of the image, its position is not affected. Since there are no physical pages in an exported EPUB, it doesn't make sense to export a “floating” image.

An “inline image” in contrast is anchored to a specific spot in the text. Add more text before or after that point and the image will move to a new position on the page. EPUB documents are rivers of text and inline images.

You can also choose whether or not text should wrap around your inline images, which I describe in more detail in the next section.

Note that to add inline images to your document, you must hold down the Command key while dragging an image from the Media box. You can also select an existing image, go to the Wrap section of the Inspector, and choose Inline (moves with text).

It's rather easy to stick images between words. That'll look OK in Pages,



but it won't look at all right in EPUB.



Be sure that the anchor symbol is at the beginning of the paragraph.

Wrapping Text around Images

Pages does support wrapping text around images, and even exports such wrapping to EPUB (in contrast with InDesign). Simply go to the Wrap section of the Inspector and choose the desired wrap configuration.

Unfortunately, you have to assign the same amount of extra space to three sides of a wrapped image. That is, if the image is wrapped to the left, and you accept the default value of 12pts for the Extra Space in the Wrap Inspector, Pages will export the image with a style that assigns the margin-bottom, margin-top, and margin-right to precisely 2.5641%, which for the margin-top and margin-bottom is exactly 2.5641% too much, especially if the image should be aligned with the top of the paragraph. The result isn't so pretty:



You have to decide whether you'd rather have the text jammed up next to the image on all three sides, whether you'd rather have extra space on all three sides, or whether you're willing to go into the EPUB and manually change the CSS (which you can read about in my book).

Creating a Cover

Pages has the useful but perhaps tricky option of converting the first page of your document into the cover of your EPUB file. It basically takes a screenshot of all the images and text on the first page and saves it in PNG format, and then designates the PNG file as the cover. This is signficant for a few reasons. First, this is the only place in your EPUB document where you can maintain the exact layout of the original page. If your text begins in the middle of the page, it will begin in the middle of the page in the cover as well. If the image takes up half the page, it will continue to do so. This means that you should be careful to choose a section that has a layout that will make sense as a cover. In particular, if you reduce it to a tiny icon, will it still be legible?

And unfortunately, the quality of the resulting PNG file is less than stellar. The only place you can really see the cover in a large size is opposite the table of contents when you're holding the iPad horizontally. In my rough example, it's pretty blurry:



I guess I'd recommend using really big text, and not very much of it.

If you don't use the first page as the cover, Pages makes a screenshot of your first page anyway to display opposite the table of contents, though it uses a generic icon of the book in the book list.

Aligning text (centering, justification)

Here, Apple falls into its own trap. You can set the alignment (centering, justification, or just plain left or right) in Pages, and it even exports it properly, but because Apple displays all text as justified in iBooks unless you use special tricks, the text in the exported EPUB won't be aligned as you wished.

CSS

Pages exports information for the following CSS properties for each and every paragraph (you can forget the Cascade): color, font-size, font-style, font-variant, font-weight, letter-spacing, margin-bottom, margin-top, padding-left, padding-right, text-align, text-decoration, text -indent, text-transform.

Not only that but it gives values to 4 decimal places! Four! A retina display for the iPad must be coming soon...

I write in my book about the perils of specifying too many values in the CSS: basically, you're breaking the Cascade of inheritance on which CSS gets its strength.

Breaks

You can choose to create section breaks, page breaks, column breaks, and layout breaks. The most you'll ever get is a <br style="clear: both;" /> That's not going to get you an actual page break, or any other kind, but it will keep the text that follows from wrapping around any previous elements. I've found that the Layout Break is the most useful for this function as it doesn't add extra page breaks in Pages (which, of course, does support them).

(To be continued)

A Proud Internaut

[Cross posted from A Year in Barcelona.]

When we lived in the States and there were considerably less Catalans around with whom to keep up the language, I used to look around on the internet for ways to practice. Several years ago, I found a radio show called L‘Internauta, whose tagline I just love: “a program about technology, but more importantly, about what people do with technology”. I would download the podcasts, and listen to them in my car, incongruously hearing Catalan while seeing America.

But it wasn’t just language practice. The program is directed by a guy named Vicent Partal, who also began and runs one of the most, if not the most, important source of Catalan news online, Vilaweb. He is a great radio host: knowledgeable and self-effacing, warm and personable. Each week he finds really interesting people to talk with on the show: a group beginning a twitter-based book club, a Catalan programmer-cum-entrepreneur who has written award-winning tools for social marketing, a rural group that used the internet as an organizational tool, and many more. And there are also regular guests, who help direct the conversation to the topics of the week, be they new products, viruses, or general technology news. It always seemed like they were talking about just what I needed to know more about, and I learned a lot. OK, and I can't deny I'm intrigued by how Vicent, who is from Valencia,  conjugates verbs!

Those of you who've read my books know that I tend to use a lot of examples from Catalunya. Usually, I just use my own photographs in order to avoid dealing with rights issues, but several years ago, I needed an example of a video on YouTube. I'd never corresponded with Vicent before, but I thought I'd ask him if he wouldn't mind if I used one of Vilaweb's broadcasts. I'm not sure what he thought of this American stranger writing him in Catalan with such an unusual request, but he kindly gave me permission and you can see the results on pages 306-7 of the Sixth Edition of my HTML book.

Over the next few years, as I listened more assiduously to L’Internauta, and read Vicent's editorials in Vilaweb, often about Catalunya's struggle for independence, I would write Vicent emails from time to time. Probably “fan mails” would be more accurate. I was just an admirer, impressed with the technology, his political insight, and his obvious caring about people. Once in a while, I would suggest some collaboration, and even tried translating Vilaweb articles into English for a time.

It's a curious relationship. There are a few people who follow me in this way too, folks who have used my books to start their careers and then write me letters every once in a while, and update me on what they're doing. I think it's part of how the internet works, and I like it. Somehow it connects us all together.

A few weeks ago, they opened a new Apple Store in Barcelona. I used the purchase of Pages (which can now export to EPUB!) as an excuse to go visit. And there among the throngs of devotees, I recognized Vicent Partal. It took me a while to get up the gumption, but I finally went over and introduced myself. He was perfectly friendly, but distracted with the business he had come to do. I still left happy that I had finally met in person someone I really admired.

When I got home, I was pleasantly surprised to find an email from Vicent. And when I told him about my latest book on EPUB, he suggested I be a guest on L’Internauta. That was today. At first I was really nervous, but it was so fun. It's going to sound stupid, but I love talking about EPUB. I love that anyone can publish a book, that you don't need a publisher, or expensive software, and I love helping people do it. I'm totally intrigued with how the whole market is shifting and how publishing is changing, and I've spent the last six months following all sorts of interesting people on Twitter to learn more about it. And it was really special to be on my own favorite radio program!

I think I've explained this whole thing because this, for me, is the beauty of the internet: making connections with people who might be far away and might speak other languages, but who have a common interest and are willing to share what they know. This is what that blog post the other day about print versus ebooks was getting at: How long are you willing to bet against people using things into beauty? I bet on beauty winning out.