Looking for our open source frameworks?

Building Push-Triggered Sync, Part V: The Big Picture

by Tim Ekl on October 12, 2015

At this point, we’re about done discussing Omni’s push architecture! We’ve already seen Go, the APNs data structures and connection, and the pipeline processor that manages hundreds of thousands of push notifications for OmniFocus customers every day. All that’s left is to take a quick tour of some other parts of the stack that support this infrastructure.

As mentioned earlier, much of Omni Sync Server is built atop FreeBSD, and the push provider is no exception. Since Go targets FreeBSD as a “first-class citizen” for compilation and execution, and there’s a port for the Go language, it’s easy to get a Go toolchain up and running. Simply run pkg install lang/go to get all the tools needed for building Go programs.

At Omni, we keep a FreeBSD system in the rotation in our automated build system. Every few hours, it checks out the latest source for the push provider from our internal SVN server, builds it, and archives the result. Along with the actual Go sources for the provider, we provide a Makefile and package information, so that the archived product is an xzipped FreeBSD package. This way, we can take advantage of FreeBSD’s existing package management system for easy deployment and upgrades of the push provider.

Next up, we needed a way to hang on to notification information: what clients were registered with the provider, how they’re grouped, and some statistics tracking. We also needed to integrate with Omni Sync Server for a staged rollout of push: during testing, we enabled push only for some sync servers, in order to measure the kind of extra load that push would levy on our sync system. (Thankfully, this period was very brief, and push is enabled for every customer now.)

All of this information was very well suited for a relational database. However, we’re in a bit of a transitional period there as well. For older applications, like Omni Sync Server, we already have MySQL set up and running well. For newer code, we’re trying to use PostgreSQL wherever possible.

As a result, the push provider wound up talking to both databases! We store all the new push-related information – things like device and group IDs, as well as counts of notifications sent – in PostgreSQL. We also interfaced with a read-only replica of the Omni Sync Server database for checking customers’ sync servers. (While this might not seem like the optimal solution, we’ve been happily using MySQL in production for a few years, and there wasn’t any sense in potentially destabilizing everybody’s sync experience for a PostgreSQL migration. If it ain’t broke, don’t fix it!)

The Web portion of the provider was fairly straightforward. Rather than try to wrap Web access in Apache or nginx, or write a separate Web interface that called a push API, we used Go’s built-in HTTP and HTML templating support to handle all incoming HTTP requests and expose a simple but serviceable administrative interface.

Of course, security – and future compatibility – were big concerns for push. On top of plain HTTP, Go’s standard library also provided a flexible implementation for handling HTTPS through the crypto/tls package. With this, we were able to ensure that the provider passed Apple’s strict ATS requirements, and that all our connections to Apple’s push service were encrypted as well.

In building the push provider, our engineers stood on the shoulders of giants. Lots of hard work went into each of the components mentioned in this post, and thanks in large part to their efficiency and utility, we were able to start work on the provider and deploy it to our customers in less than three months. Of course, our customers were invaluable in this process as well - hundreds of helpful people volunteered for the OmniFocus beta program, and helped us see how push would work together with Omni Sync Server.

As of this writing, the push provider has already sent over ten million push notifications. We’re looking forward to the next ten!

Building Push-Triggered Sync, Part IV: The Notification Pipeline

by Tim Ekl on October 12, 2015

So far in this series, we’ve chosen the Go language to build a push provider; we designed data structures that represent notifications, according to Apple’s specifications; and we’ve connected to APNs so we can send those notifications to Apple (and thus to our customers).

Along the way, though, Omni’s push provider needs to do a fair bit of preprocessing and other work for every notification it prepares to send. There’s also the chance that we won’t actually want to send some notifications, instead preferring to filter them out. In the current provider, we consider each notification for:

  • Its bundle ID – notifications need to be sent across the right connection for their bundle ID, and if the ID doesn’t match up with one of the current versions of OmniFocus, we shouldn’t send it.
  • Logging – we want to keep some debugging information around for notifications, just to make sure they’re all being delivered properly.
  • Statistics – we keep an aggregate count of all the notifications that pass through the provider.
  • Associated Omni Sync Server information – while we roll out the push provider, we want to be able to gradually enable notifications for different sync servers, so as to measure the load that push-triggered sync levies on those servers.
  • Error responses from APNs – Apple may tell us that a notification was malformed, or our connection to APNs may drop unexpectedly.

Overall, we want an architecture that can handle all these different needs in a unified manner. Each of these considerations needs to be its own component, but should have an interface roughly similar to all the others, so that we can debug them separately and also compose them easily to form the entire provider.

Luckily, Go has a wealth of documentation – not only for each API, but also in the form of more general guides and examples. One of these discusses a concurrency pattern that can be built entirely out of Go’s language builtins: that of a pipeline. (Read the full article here.)

The general idea behind a Go pipeline revolves around a combination of two important concurrency primitives: channels and goroutines. At their simplest, a channel is a thread-safe way to pass values between two different parts of a program, and a goroutine is a very lightweight way of running a function on a different thread. (The actual implementation details – and implications – are much more complex, but this should suffice for our purposes.)

If we take both channels and goroutines for granted, we can start setting up a bunch of different pipeline components (called “stages” in the original article) that take in a value on a channel, do some arbitrary thing with it, and push it out on another channel. Let’s consider the example of a Notification instance being logged out for debugging – we’ll want to just print the notification in its entirety, then pass it down the pipeline unchanged. We might write this pipeline component as:

func LogPipe(input <-chan Notification) <-chan Notification {
    output := make(chan Notification)
    go func() {
        for notification := range input {
            fmt.Printf("Notification: %v\n", notification)
            output <- notification
        }
        close(output)
    }()
    return output
}

With this implementation, constructing an instance of this logging pipeline component is as simple as calling a function. It needs an existing channel as input, but gives back a new output channel; this means that we can easily chain multiple different components by passing the output channel from one function to the input of another.

We run the “pump” for this component as a goroutine, so that it’s always running alongside the rest of our program. Inside, we use the range of the input channel, so that the goroutine blocks until a notification comes through. When the input channel closes, this for loop over the input channel terminates. At that point, we’ll close the output channel too, signaling down the pipeline that we’re done handling notifications.

For the push provider’s use, we can do a bunch of different things in each of these components – and logging is only the simplest! The provider itself is structured as a pipeline with nearly a dozen components from start to end:

  • The logging component looks very similar to the above, with just a little bit more configurability about where to send logs (and with what severity).
  • The statistics component is also similar to the above, but instead of logging out a message, it increments a few internal counters depending on the contents of the notification.
  • The OSS component uses the UserInfo map [mentioned previously](TODO%20link%20to %20%20part%202), which we populate with the sender’s OSS username when building some Notification instances. If we need to drop a notification – perhaps because the sending user is on a sync server that’s under heavy load – we can simply refuse to pass it along the output channel from this component.

Even the persistent connection to APNs is handled with a pair of these pipeline components. The first takes in notifications, sends them over to APNs, and forwards them unconditionally; the second then buffers the sent notifications briefly to await an error, then forwards them in turn.

Keep in mind, too, that the pipeline components can have more than one input or output channel! Omni’s provider also includes a pipeline multiplexer and demultiplexer. These components are used to split up the notification stream (based on the associated bundle ID) for transmission to APNs, then rejoin the stream later after the APNs components pass the sent notifications through.

This sort of pipeline architecture is how the provider can handle Omni’s volumes of push notifications. At its peak, our single push provider instance transmits close to 500 notifications every minute – while still using only a few percent of one server’s CPU.

While Go makes this sort of concurrency quick to write and efficiently scalable, it’s not the only piece involved in the puzzle. Next time, we’ll discuss some other technologies involved in the provider stack, including the backing database used to store device registrations.

Building Push-Triggered Sync, Part III: Connecting to APNs

by Tim Ekl on September 21, 2015

Now that we’ve converted our notification data into a format that’s suitable for sending to Apple, our fledgling push provider needs a connection into APNs in order to send that data. Once again, Apple’s documentation comes to our rescue: the chapter “Provider Communication with Apple Push Notification Service” describes how to manage connections to the push notification service.

Among the important bits in this chapter:

  • When connecting, the provider needs the SSL certificate from the Member Center available for authenticating itself to Apple.
  • The provider should maintain a persistent connection to APNs, rather than connecting and disconnecting for each notification. (At higher volumes, Apple might even consider this behavior to be a DDOS attack!)
  • There are two different environments for push notifications: development and production. A connection is specific to one app’s bundle ID, and therefore (at least in Omni’s bundle ID configuration) specific to one of these environments.

To handle all this, we need to build a little flexibility into the provider. We turned to the open-source gcfg library, which can be imported into a Go project simply by running

go get "code.google.com/p/gcfg"

and then importing that same URL at the head of a Go file. With this library, we can define a configuration file that tells the provider about what APNs host(s) it should connect to, and what certificates it should use along the way.

Let’s take a look first at the Go struct that the provider reads its APNs configuration into:

type ConnConfig struct {
    Domain   string
    CertFile string
    KeyFile  string
}

This struct expresses a single connection to APNs. It defines the domain we’ll connect to, letting us switch between sandbox and production environments by changing the host we contact. It also lets us tell the provider where its certificates are, so that it can establish a secure connection and identify itself to APNs, all in one swoop.

For OmniFocus, though, we have multiple bundle IDs – one for the Universal version of the app, and one for the iPhone-only variant – so we’ll need a way to connect to APNs multiple times. To accomplish this, we parse something slightly more complex out of the configuration file:

func ParseConfigFile(filePath string) (map[string]*ConnConfig, error) {
    var configuration struct {
        Apns map[string]*ConnConfig
    }

    err := gcfg.ReadFileInto(&configuration, filePath)
    if err != nil {
        return nil, err
    }
    return configuration.Apns, nil
}

Instead of just reading a single instance of the ConnConfig struct, we’ll read an entire map of them, keyed by strings. The gcfg package uses maps to represent configuration subsections – instead of just specifying a single apns section, we can specify a bunch, each with its own key:

[apns "com.omnigroup.OmniFocus2.iPad"] ; Universal
domain = sandbox.push.apple.com
certFile = debug-universal.cer
keyFile = debug-universal.key

[apns "com.omnigroup.OmniFocus2.iPhone"] ; iPhone-only
domain = sandbox.push.apple.com
certFile = debug-iphone.cer
keyFile = debug-iphone.key

This way, parsing a configuration file can return one ConnConfig struct for each bundle ID that we’ll use to connect. The provider’s connection code can then iterate over these structs, establishing multiple connections along the way. For each connection, that code is fairly straightforward:

func (config *ConnConfig) connectAPNs() (*tls.Conn, error) {
    cert, err := tls.LoadX509KeyPair(config.CertFile, config.KeyFile)
    if err != nil {
        return nil, err
    }

    tlsConfig := new(tls.Config)
    tlsConfig.Certificates = []tls.Certificate{cert}

    host := net.JoinHostPort("gateway." + config.Domain, "2195")
    conn, err := tls.Dial("tcp", host, tlsConfig)
    if err != nil {
        return nil, err
    }

    return conn, err
}

Once this connection is established, the provider can hang on to this tls.Conn pointer as long as it needs to, sending multiple notifications – in the form of the frames described in the previous post – when changes occur in OmniFocus. In the next post, we’ll take a look at how the provider manages all the notifications it needs to send across these connections.

Building Push-Triggered Sync, Part Two: First Steps

by Tim Ekl on August 21, 2015

After researching languages and choosing Go to implement a push provider for OmniFocus, we needed to get started writing code. Apple has set out the interface for a provider in the chapter “Provider Communication with Apple Push Notification Service;” our task then became to write a server that could speak in the binary format APNs expected.

Every notification delivered to Apple is built of a series of items. Each item details one part of the notification, such as its JSON payload or metadata like its priority or expiration date. Items have a common format: each begins with one byte giving the item type, then two bytes giving its total length. All the actual information in the item follows that three-byte “header.”

This is pretty straightforward to represent in Go: we just need a struct whose layout matches the expected format of an item. Let’s write one now:

type Item struct {
    ID         byte
    dataLength uint16
    data       []byte
}

From this basic definition, we can represent all the different kinds of item that might appear in an APNs notification. Since the data in an item is stored in its encoded (byte slice) form, we can also write this item to a binary buffer for transmission to Apple very easily:

func (item *Item) Bytes() []byte {
    buf := new(bytes.Buffer)
    binary.Write(buf, binary.BigEndian, item.ID)
    binary.Write(buf, binary.BigEndian, item.dataLength)
    binary.Write(buf, binary.BigEndian, item.data)
    return buf.Bytes()
}

We can also take advantage of Go’s iota construct inside constant declarations to define names for each kind of item that Apple documents. Since the items are 1-indexed, we simply define the first as 1 + iota and let the language go from there:

const (
    DeviceTokenItemIdentifier = 1 + iota
    PayloadItemIdentifier
    NotificationIdentifierIdentifier
    ExpirationDateIdentifier
    PriorityIdentifier
)

With this infrastructure, defining any given kind of item becomes as easy as providing a short function to populate the struct:

func NewDeviceTokenItem(token [32]byte) *Item {
    return &Item{ID: DeviceTokenItemIdentifier, dataLength: 32, data: token[:]}
}

While some items are a little more complex than this one-liner, in general, creating any single item is simply a matter of parsing its data into a byte slice and returning one of these structs. We can then begin composing these items into larger pieces of data for more convenient use.

The next step is to build an entire frame for transmission to Apple. A frame is simply a list of items, all serialized next to each other and wrapped with a few more bytes’ worth of header data. We can take a similar approach for building frames as we did for items: create a struct that contains a slice of Item structs, then implement a Bytes() function for that struct that recursively encodes all the contained Items and adds the required header data.

Strictly speaking, this is all our push provider needs to start sending notifications! We can construct frames full of items, serialize them, and send them to Apple. However, we’ll probably want a little more metadata internally to help our notifications along. At the very least, we’ll need to know the bundle ID associated with each notification – since the connection we establish to APNs is specific to one app’s bundle ID, and we’re providing push services for multiple apps, we’ll need a way to distinguish between different notifications so we know which connection to use.

To handle this, we wrap up our Frame in one more top-level struct: the Notification. All this struct does is provide information that our provider tracks internally – it doesn’t send anything extra to Apple. We define it as:

type Notification struct {
    Frame *Frame
    BundleID string
    UserInfo map[string]interface{}
}

This last field is a little quirky: it maps strings to interface{}, which is Go’s “anything” type. With this field, the provider yields a bit to Objective-C – we can attach any extra bits of data that we want internally in this UserInfo map, then pull them back out later. We’ll see an example of how this is used in a future post.

At this point, we’re ready to start talking to Apple! We have representations of all the different pieces of data that the APNs binary interface expects, at least for sending notifications. In the next post, we’ll explore how the push provider connects to the APNs gateway.

Building Push-Triggered Sync, Part One: Choosing a Language

by Tim Ekl on August 19, 2015

Here at the Omni Group, we have a long history of writing code in Objective-C. All of the apps we currently sell are written in Objective-C, and those of you that follow our public frameworks will know they’re entirely Objective-C as well.

However, we recently faced a new challenge with our decision to bring Push-Triggered Sync to OmniFocus on Mac and iOS. Apple’s architecture for sending push notifications requires that we use a server-side component called a provider. And while Objective-C was our forte, it was ill-suited for server-side development, since almost none of Omni’s servers run OS X.

Thus, our attention turned to building a provider that could handle the large existing OmniFocus customer base, along with the specific traffic patterns the app generates. (For example, the Omni Sync Server often sees large traffic spikes early Monday mornings, as OmniFocus users get started on their work weeks.) What’s more, we wanted this provider to fit well in our existing server setup, which is gradually moving towards using FreeBSD.

With these constraints in mind, we settled on the Go language for implementing a push provider. Go had several properties that we felt made it a good fit for this feature:

  • It was explicitly targeted at system-level services, rather than our usual focus on apps with rich user interfaces. In particular, Go’s multitasking features (such as first-class functions and goroutines) made it well suited for building a parallelizable server.
  • It has a strong standard library, especially around networking and manipulating arrays of bytes directly — two especially relevant topics for interacting with Apple’s binary notifications interface across the Internet.
  • It is a relatively young language, and so is able to take advantage of recent advancements and best practices in language design. On the flip side, it’s not too young: with Go 1.4, the language has reached a level of maturity sufficient to freeze the API, providing a certain amount of stability in Go projects.
  • It’s a compiled language with FreeBSD as a “first-class” targeted platform, giving us confidence that we could deploy a provider binary across our hosting infrastructure without needing a custom environment or runtime installed up front.

After settling on Go, we also considered various open-source packages for speeding up early development. And while we did wind up using a few (for tasks like MySQL and PostgreSQL database access, as well as logging), existing frameworks for communicating with Apple’s push service (APNs) didn’t quite look as fully-formed as we would’ve liked.

With that, the stage was set: in Go, build an internal framework for communicating with APNs, then use it in a provider daemon that could power push notifications for the hundreds of thousands of customers that rely on OmniFocus every day. We’ll discuss more about specific implementation details next.