The Blog

The challenge of syncing OmniOutliner

by Ken Case on April 16, 2012

Hi, all!

It’s been almost one year since we shipped OmniOutliner for iPad. Every day since we shipped we’ve received feedback along the lines of “This looks great! But when will it automatically sync with OmniOutliner for Mac?”—and each time we reply back “We’re working on it!” But it’s been almost a year now, and I know you all must be wondering: Why is it taking so long to add automatic syncing to OmniOutliner?

To make documents with large embedded attachments more efficient, OmniOutliner 3 stores its documents as “file wrappers”, which behind the scenes are simply directories holding separate files which represent different parts of the document. (The outline itself is one file, and each attachment is its own separate file.) Unfortunately, most sync services don’t support syncing a directory as an atomic operation.

Wait, hold on, what does “atomic operation” mean? Well, in computer speak, an atomic operation is something that gets done all at once. In this case, we want all of the parts of the document to get synchronized to or from the cloud at the same time, otherwise different parts of the document might not be in sync with itself: it might be missing some attachments, or perhaps some of them will be out of date, or maybe there will be some extra attachments that shouldn’t be there.

In other words, when syncing OmniOutliner, it’s not just important to have all the parts eventually arrive on the other end; we want all of the parts of the document to show up on the other end at the very same time (and we want to know when they’ve all finished arriving) so that we know it’s safe to open and you won’t end up with a corrupt document.

One way to solve this syncing challenge is to change our document format—to stop storing OmniOutliner documents as directories behind the scenes. This change would make it much less efficient for attachments, but would also make it much more compatible with many file servers, email, web forms, source control systems, and so on. We can do that—but such a big change to the OmniOutliner file format won’t work with the currently shipping apps, and doesn’t really make sense to do before shipping OmniOutliner 4. So taking this route to solving the problem means for us to try to ship OmniOutliner 4 as soon as possible (which we’re certainly working very hard to do).

Another way to solve this problem is to help syncing solution providers (such as DropBox and iCloud) improve their support for atomic directory operations so we can sync using their solutions. The most promising option here by far is iCloud, since iCloud’s engineers explicitly do want to support syncing of file wrappers. We’ve had some of our most experienced developers spend much of the past year working on this approach—but much of that time was spent blazing new territory, and most issues we encountered weren’t within our control to fix because we don’t control the technology and servers. Fortunately, we think we’ve pushed through most of the critical issues and we’re hoping this approach will bear fruit soon.

Of course, if it takes too long to change our document format or to get atomic directory syncing working with other people’s cloud servers, another approach would be for us to add syncing support for our own cloud servers. We recently brought our Omni Sync Server out of beta, and we’re using it now to automatically sync OmniFocus and OmniPlan.

One way or another, we’re working very hard to bring automatic syncing to OmniOutliner as soon as possible: we know it’s absolutely critical for anyone who wants to use OmniOutliner on more than one device. We’re very sorry it’s taking so long!

As always, we welcome your feedback! Please feel free to leave a comment here or on our forums, or by sending me a message on twitter (where you’ll find me at @kcase).

Comments

Thanks for giving a detailed, easy to understand explanation on why this is taking so long.

Having co-owned and run an Enterprise software company I can understand the challenges of balancing software development across multiple products and continually having to re-architect products due to technological advances.

Unfortunately the “haters are still going to hate” but I’m sure your loyal fans will hang in there :-)

Looking forward to the next release of OmniOutliner and thanks to you and the team for the great work!

Paul Watts

04.16.12 6:41 PM

I appreciate the update.  Definitely my most wanted feature in OO!

Rich Ruh Ruh

04.16.12 9:14 PM

Good update and explanation.

I’m sure you’ve already considered this, but is there anything stopping you compressing the directory to a single archive file, synching it, then decompressing it on the other side? This would give you just a single file to sync, although I appreciate it would increase the amount of data moving around, as the whole file would presumably have to send/receive each sync.

Mark A

04.17.12 1:10 AM

Thanks for this update.  I’ve been checking the OmniGroup website and browsing the forums fairly often hoping to hear about an update that will allow OmniOutliner for iPad to sync with OmniOutliner for Mac.  Up until I read this blog post, I regretted purchasing OO for iPad, but now that I understand the problem, I’m not at all upset.  I wish this explanation had come out sooner! 

Take your time, and get it right. 

Thanks again for keeping us in the loop!

Chris

04.17.12 1:10 AM

Thanks for the detailed information!

However, I miss an alternative. As far as I know LibreOffice (or OpenOffice) have a file structure too behind the scene and they are wrapping it in a zip file. So couldn’t there be an automatic packing/unpacking of OmniOutliner data?

Bernhard

04.17.12 7:31 AM

Sorry, I must have missed the comment of Mark A.

Bernhard

04.17.12 7:35 AM
Team Member

Zipping up the document wrapper is a perfectly reasonable approach: it’s how we send OmniOutliner documents through email now. That’s the approach we’re looking at for OmniOutliner 4.

But that’s a change to the file format, one which OmniOutliner 3 doesn’t understand today. It works fine for email transfers because those are one-way and opening a zip file will automatically unpack it, but it doesn’t work with syncing because we need every edit to end up zipped back up again automatically.

Ken Case

04.17.12 7:52 AM

I’d settle for a version that didn’t support file attachments but synced with my world of devices in a heartbeat.

Robert

04.17.12 11:29 AM

You OmniGroup guys just steal one patent from Cultured Code, which is—instead of shipping a working syncing solution, you write a long post explaining why you can’t do that, exactly what Cultured Code did 2 years ago—to explain why they can’t make Things Cloud Sync, when OmniFocus already shipped a excellent cloud sync solution.

That’s exactly the reason that I stopped using Things and switch to OmniFocus, now OmniGroup is doing the same thing. This makes me sad.

Mudkip

04.17.12 11:38 AM

Perhaps I’m in the minority here, because I doubt I’m fully exploiting OmniOutliner’s capabilities. Mostly, I just use it for nested lists that I can sort, check off items, and do basic calculations.

I was under the impression that OmniOutliner used standard OPML files, is that not the case?

I realise that you’re an Apple-focused company, but with the Mac Pros being woefully out of date, and considerably overpriced (I don’t need server-grade hardware, but do need a tower with a modern GPU and several terrabytes of internal storage) my desktop is now a Windows-based machine, and my MacBook Pro has been replaced with an iPad. (the MBP is too big & heavy to be portable, the MBA battery life is too short)

In almost every other category of product that I use, I am able to sync files between my PC & iPad via Dropbox effortlessly, as I try to avoid using products which have their own proprietary file formats or sync solutions. (all my documents are plaintext or RTF, for example) OmniFocus & OmniOutliner have been the two exceptions I made to this, based on their reputation, and the promise of sync coming soon.

With Dropbox syncing, I can go to any machine, PC or Mac, log into the website and pull down a file or two that I need access to.

If I want to go a step further, I can install the app which doesn’t just pull down my documents, it also pulls down many of the apps I use to edit them, along with their preferences, so there’s minimal set-up time to start working on files on a new machine the way I prefer. (I get the “portable” version of apps where possible)

iCloud syncing isn’t much use to me, as that is a Mac-only solution, that keeps sync locked within the same application (i.e. to sync between desktop & iPad, you need to own both copies of the same app) and it requires you to log in on the machine to get access to those files. There’s no way to quickly get a document of mine on someone else’s machine, as I need to set up iCloud on another user account, then log into the Mac App store and download the relevant app it syncs with, even if that App is only working with plaintext files. And I don’t especially want to be syncing everything over onto a machine that isn’t mine, just to access a file or two. (even worse if those files were created in separate apps)


Even without considering sync between the desktop & iPad, Dropbox sync lets me move between different Apps on the iPad itself. I use different apps for getting words down on the page than I use for actually editing documents for example. I like to write without distraction in Byword (I prefer iA Writer, but their sync solution is awkward) and then do the actual editing of content in other apps that are more suited to it. (smaller font size, the ability to reorganise sentences, paragraphs quickly etc.)

And while I don’t think it’s likely to be an issue with OmniOutliner, what happens if a company either goes out of business, or decides not to continue publishing on the App store? I have seen several Apps I have purchased disappear off my account when a company has gone under, been purchased by a larger company, decided not to renew their developer license as they weren’t making enough money from it, or simply released a new version and removed the old one from existence. What happens to my documents then, if they’re locked inside the app, and stuck within iCloud, and the app ceases to exist? With Dropbox sync, your files are totally separate from the App that created them, as they should be.

Please Reconsider

04.17.12 12:27 PM

I am disturbed

Pishab

04.17.12 5:25 PM

Hi Ken,

Thanks very much for the explanation.  It really helps me understand the issues involved, the paths forward, and how (characteristically) thoughtfully you’re all approaching this issue.  I encourage you and your staff to post this sort of update more frequently.  It might alleviate some of the frustration many feel from waiting in the dark.

One question: am I correct in understanding from your post that OO4 will have a different “non file-wrapper” document format than OO3, one that allows atomic syncing?

Roger Barre

04.18.12 10:53 AM

So why does the entire transfer operation have to be atomic? Here’s my thought. Call the document Outline.oo3. Under the covers, Outline.oo3 is a directory with lots of other things in it that all need to be consistent. Why not create a temporary directory with a random name (say, Outline-783647364.oo3) and transfer all of the Outline.oo3 contents progressively into it. Only once everything within Outline-783647364.oo3 is transfered and self-consistent, you rename Outline-783647364.oo3 to Outline.oo3. The rename is an atomic operation, and it would make the Outline.oo3 OmniOutliner “file” appear immediately. If there is a conflict (like Outline.oo3 appearing from another device while Outline-783647364.oo3 is being transfered) that can be detected right when the rename would happen. This method ought to be applicable to any sync system that allows hierarchical data, like Dropbox, WebDAV, iCloud, whatever. And it wouldn’t require any file format change by the Omni folks like other solutions do.

Sean

04.19.12 1:02 PM

Hi folks,

one of the main reasons I chose OmniFocus was that it had the ability to sync via WebDAV, because I refuse to give my sensitive data to cloud services I don’t trust. I’m perfectly capable of running my own servers and do this for friends and family. So to make a long story short, whatever you do, please make sure that the syncing solution supports WebDAV in addition to commercial sync services.

Jens

04.21.12 1:58 PM

Why not use ZIP Files to store your folder of information?  This is one standard way of storing directories of files as a single file.  A Zip file, further, can be examined without decompressing, to see its contents.

Apple uses Zip files in Mac OS X.

Why not use Zip files as the file format for OmniOutliner? 

This way, it can be synchronized easily.  And it can be transferred to any file system.

Marianco

05.07.12 9:13 PM

Ken-

I have desktop, iPhone, and iPad versions of OmniFocus, but only the desktop version of OmniOutliner.  When OO offers easy cross-device device syncing, I’ll buy an iPad version, but not before then.

I would also appreciate a universal app.  As much as I like Omni and am willing to pay the $20 that you charge for iOS apps, I won’t buy two versions of it for iOS.

Tom

05.09.12 10:11 AM

Re: Tom’s comment I’m torn.  Certainly I appreciate universal apps, but at least with OmniFocus, it’s worth the money to have the iPad version in addition to the iPhone version.  So would Omni make more money by making the apps universal?  I’d bet that they do a lot better the way they’re doing it with separate versions.  The software is just too good and too crucial to not spend the extra few bucks.

Joshua Reynolds

05.13.12 7:45 AM
Commenting is not available in this channel entry.