Flipboard launched during the dawn of the smartphone and tablet as a mobile-first experience, allowing us to rethink content layout principles from the web for a more elegant user experience on a variety of touchscreen form factors.
Now we’re coming full circle and bringing Flipboard to the web. Much of what we do at Flipboard has value independent of what device it’s consumed on: curating the best stories from all the topics, sources, and people that you care about most. Bringing our service to the web was always a logical extension.
As we began to tackle the project, we knew we wanted to adapt our thinking from our mobile experience to try and elevate content layout and interaction on the web. We wanted to match the polish and performance of our native apps, but in a way that felt true to the browser.
Early on, after testing numerous prototypes, we decided our web experience should scroll. Our mobile apps are known for their book-like pagination metaphor, something that feels intuitive on a touch screen, but for a variety of reasons, scrolling feels most natural on the web.
In order to optimize scrolling performance, we knew that we needed to keep paint times below 16ms and limit reflows and repaints. This is especially important during animations. To avoid painting during animations there are two properties you can safely animate: CSS transform and opacity. But that really limits your options.
What if you want to animate the width of an element?
How about a frame-by-frame scrolling animation?
(Notice in the above image that the icons at the top transition from white to black. These are 2 separate elements overlaid on each other whose bounding boxes are clipped depending on the content beneath.)
These types of animations have always suffered from jank on the web, particularly on mobile devices, for one simple reason:
The DOM is too slow.
It’s not just slow, it’s really slow. If you touch the DOM in any way during an animation you’ve already blown through your 16ms frame budget.
Enter <canvas>
Most modern mobile devices have hardware-accelerated canvas, so why couldn’t we take advantage of this? HTML5 games certainly do. But could we really develop an application user interface in canvas?
Immediate mode vs. retained mode
Canvas is an immediate mode drawing API, meaning that the drawing surface retains no information about the objects drawn into it. This is in opposition to retained mode, which is a declarative API that maintains a hierarchy of objects drawn into it.
The advantage to retained mode APIs is that they are typically easier to construct complex scenes with, e.g. the DOM for your application. It often comes with a performance cost though, as additional memory is required to hold the scene and updating the scene can be slow.
Canvas benefits from the immediate mode approach by allowing drawing commands to be sent directly to the GPU. But using it to build user interfaces requires a higher level abstraction to be productive. For instance something as simple as drawing one element on top of another can be problematic when resources load asynchronously, such as drawing text on top of an image. In HTML this is easily achieved with the ordering of elements or z-index in CSS.
Building a UI in <canvas>
Canvas lacks many of the abilities taken for granted in HTML + CSS.
Text
There is a single API for drawing text: fillText(text, x, y [, maxWidth]). This function accepts three arguments: the text string and x-y coordinates to begin drawing. But canvas can only draw a single line of text at a time. If you want text wrapping, you need to write your own function.
Images
To draw an image into a canvas you call drawImage(). This is a variadic function where the more arguments you specify the more control you have over positioning and clipping. But canvas does not care if the image has loaded or not so make sure this is called only after the image load event.
Overlapping elements
In HTML and CSS it’s easy to specify that one element should be rendered on top of another by using the order of the elements in the DOM or CSS z-index. But remember, canvas is an immediate mode drawing API. When elements overlap and either one of them needs to be redrawn, both have to be redrawn in the same order (or at least the dirtied parts).
Custom fonts
Need to use a custom web font? The canvas text API does not care if a font has loaded or not. You need a way to know when a font has loaded, and redraw any regions that rely on that font. Fortunately, modern browsers have a promise-based API for doing just that. Unfortunately, iOS WebKit (iOS 8 at the time of this writing) does not support it.
Benefits of <canvas>
Given all these drawbacks, one might begin to question selecting the canvas approach over DOM. In the end, our decision was made simple by one simple truth:
You cannot build a 60fps scrolling list view with DOM.
Many (including us) have tried and failed. Scrollable elements are possible in pure HTML and CSS with overflow: scroll (combined with -webkit-overflow-scrolling: touch on iOS) but these do not give you frame-by-frame control over the scrolling animation and mobile browsers have a difficult time with long, complex content.
In order to build an infinitely scrolling list with reasonably complex content, we needed the equivalent of UITableView for the web.
In contrast to the DOM, most devices today have hardware accelerated canvas implementations which send drawing commands directly to the GPU. This means we could render elements incredibly fast; we’re talking sub-millisecond range in many cases.
Canvas is also a very small API when compared to HTML + CSS, reducing the surface area for bugs or inconsistencies between browsers. There’s a reason there is no Can I Use? equivalent for canvas.
A faster DOM abstraction
As mentioned earlier, in order to be somewhat productive, we needed a higher level of abstraction than simply drawing rectangles, text and images in immediate mode. We built a very small abstraction that allows a developer to deal with a tree of nodes, rather than a strict sequence of drawing commands.
RenderLayer
A RenderLayer is the base node by which other nodes build upon. Common properties such as top, left, width, height, backgroundColor and zIndex are expressed at this level. A RenderLayer is nothing more than a plain JavaScript object containing these properties and an array of children.
Image
There are Image layers which have additional properties to specify the image URL and cropping information. You don’t have to worry about listening for the image load event, as the Image layer will do this for you and send a signal to the drawing engine that it needs to update.
Text
Text layers have the ability to render multi-line truncated text, something which is incredibly expensive to do in DOM. Text layers also support custom font faces, and will do the work of updating when the font loads.
Composition
These layers can be composed to build complex interfaces. Here is an example of a RenderLayer tree:
{
frame: [0, 0, 320, 480],
backgroundColor: '#fff',
children: [
{
type: 'image',
frame: [0, 0, 320, 200],
imageUrl: 'http://lorempixel.com/360/420/cats/1/'
},
{
type: 'text',
frame: [10, 210, 300, 260],
text: 'Lorem ipsum...',
fontSize: 18,
lineHeight: 24
}
]
}
Invalidating layers
When a layer needs to be redrawn, for instance after an image loads, it sends a signal to the drawing engine that its frame is dirty. Changes are batched using requestAnimationFrame to avoid layout thrashing and in the next frame the canvas redraws.
Scrolling at 60fps
Perhaps the one aspect of the web we take for granted the most is how a browser scrolls a web page. Browser vendors have gone to great lengths to improve scrolling performance.
It comes with a tradeoff though. In order to scroll at 60fps on mobile, browsers used to halt JavaScript execution during scrolling for fear of DOM modifications causing reflow. Recently, iOS and Android have exposed onscroll events that work more like they do on desktop browsers but your mileage may vary if you are trying to keep DOM elements synchronized with the scroll position.
Luckily, browser vendors are aware of the problem. In particular, the Chrome team has been open about its efforts to improve this situation on mobile.
Turning back to canvas, the short answer is you have to implement scrolling in JavaScript.
The first thing you need is a way to compute scrolling momentum. If you don’t want to do the math the folks at Zynga open sourced a pure logic scroller that fits well with any layout approach.
The technique we use for scrolling uses a single canvas element. At each touch event, the current render tree is updated by translating each node by the current scroll offset. The entire render tree is then redrawn with the new frame coordinates.
This sounds like it would be incredibly slow, but there is an important optimization technique that can be used in canvas where the result of drawing operations can be cached in an off-screen canvas. The off-screen canvas can then be used to redraw that layer at a later time.
This technique can be used not just for image layers, but text and shapes as well. The two most expensive drawing operations are filling text and drawing images. But once these layers are drawn once, it is very fast to redraw them using an off-screen canvas.
In the demonstration below, each page of content is divided into 2 layers: an image layer and a text layer. The text layer contains multiple elements that are grouped together. At each frame in the scrolling animation, the 2 layers are redrawn using cached bitmaps.
Object pooling
During the course of scrolling through an infinite list of items, a significant number of RenderLayers must be set up and torn down. This can create a lot of garbage, which would halt the main thread when collected.
To avoid the amount of garbage created, RenderLayers and associated objects are aggressively pooled. This means only a relatively small number of layer objects are ever created. When a layer is no longer needed, it is released back into the pool where it can later be reused.
Fast snapshotting
The ability to cache composite layers leads to another advantage: the ability to treat portions of rendered structures as a bitmap. Have you ever needed to take a snapshot of only part of a DOM structure? That’s incredibly fast and easy when you render that structure in canvas.
The UI for flipping an item into a magazine leverages this ability to perform a smooth transition from the timeline. The snapshot contains the entire item, minus the top and bottom chrome.
A declarative API
We had the basic building blocks of an application now. However, imperatively constructing a tree of RenderLayers could be tedious. Wouldn’t it be nice to have a declarative API, similar to how the DOM worked?
React
We are big fans of React. Its single directional data flow and declarative API have changed the way people build apps. The most compelling feature of React is the virtual DOM. The fact that it renders to HTML in a browser container is simply an implementation detail. The recent introduction of React Native proves this out.
What if we could bind our canvas layout engine to React components?
Introducing React Canvas
React Canvas adds the ability for React components to render to <canvas> rather than DOM.
The first version of the canvas layout engine looked very much like imperative view code. If you’ve ever done DOM construction in JavaScript you’ve probably run across code like this:
// Create the parent layer
var root = RenderLayer.getPooled();
root.frame = [0, 0, 320, 480];
// Add an image
var image = RenderLayer.getPooled('image');
image.frame = [0, 0, 320, 200];
image.imageUrl = 'http://lorempixel.com/360/420/cats/1/';
root.addChild(image);
// Add some text
var label = RenderLayer.getPooled('text');
label.frame = [10, 210, 300, 260];
label.text = 'Lorem ipsum...';
label.fontSize = 18;
label.lineHeight = 24;
root.addChild(label);
Sure, this works but who wants to write code this way? In addition to being error-prone it’s difficult to visualize the rendered structure.
With React Canvas this becomes:
var MyComponent = React.createClass({
render: function () {
return (
<Group style={styles.group}>
<Image style={styles.image} src='http://...' />
<Text style={styles.text}>
Lorem ipsum...
</Text>
</Group>
);
}
});
var styles = {
group: {
left: 0,
top: 0,
320,
height: 480
},
image: {
left: 0,
top: 0,
320,
height: 200
},
text: {
left: 10,
top: 210,
300,
height: 260,
fontSize: 18,
lineHeight: 24
}
};
You may notice that everything appears to be absolutely positioned. That’s correct. Our canvas rendering engine was born out of the need to drive pixel-perfect layouts with multi-line ellipsized text. This cannot be done with conventional CSS, so an approach where everything is absolutely positioned fit well for us. However, this approach is not well-suited for all applications.
css-layout
Facebook recently open sourced its JavaScript implementation of CSS. It supports a subset of CSS like margin, padding, position and most importantly flexbox.
Integrating css-layout into React Canvas was a matter of hours. Check out the example to see how this changes the way components are styled.
Declarative infinite scrolling
How do you create a 60fps infinite, paginated scrolling list in React Canvas?
It turns out this is quite easy because of React’s diffing of the virtual DOM. In render() only the currently visible elements are returned and React takes care of updating the virtual DOM tree as needed during scrolling.
var ListView = React.createClass({
getInitialState: function () {
return {
scrollTop: 0
};
},
render: function () {
var items = this.getVisibleItemIndexes().map(this.renderItem);
return (
<Group
onTouchStart={this.handleTouchStart}
onTouchMove={this.handleTouchMove}
onTouchEnd={this.handleTouchEnd}
onTouchCancel={this.handleTouchEnd}>
{items}
</Group>
);
},
renderItem: function (itemIndex) {
// Wrap each item in a <Group> which is translated up/down based on
// the current scroll offset.
var translateY = (itemIndex * itemHeight) - this.state.scrollTop;
var style = { translateY: translateY };
return (
<Group style={style} key={itemIndex}>
<Item />
</Group>
);
},
getVisibleItemIndexes: function () {
// Compute the visible item indexes based on `this.state.scrollTop`.
}
});
To hook up the scrolling, we use the Scroller library to setState() on our ListView component.
...
// Create the Scroller instance on mount.
componentDidMount: function () {
this.scroller = new Scroller(this.handleScroll);
},
// This is called by the Scroller at each scroll event.
handleScroll: function (left, top) {
this.setState({ scrollTop: top });
},
handleTouchStart: function (e) {
this.scroller.doTouchStart(e.touches, e.timeStamp);
},
handleTouchMove: function (e) {
e.preventDefault();
this.scroller.doTouchMove(e.touches, e.timeStamp, e.scale);
},
handleTouchEnd: function (e) {
this.scroller.doTouchEnd(e.timeStamp);
}
...
Though this is a simplified version it showcases some of React’s best qualities. Touch events are declaratively bound in render(). Each touchmove event is forwarded to the Scroller which computes the current scroll top offset. Each scroll event emitted from the Scroller updates the state of the ListView component, which renders only the currently visible items on screen. All of this happens in under 16ms because React’s diffing algorithm is very fast.
See the ListView source code for the complete implementation.
Practical applications
React Canvas is not meant to completely replace the DOM. We utilize it in performance-critical rendering paths in our mobile web app, primarily the scrolling timeline view.
Where rendering performance is not a concern, DOM may be a better approach. In fact, it’s the only approach for certain elements such as input fields and audio/video.
In a sense, Flipboard for mobile web is a hybrid application. Rather than blending native and web technologies, it’s all web content. It mixes DOM-based UI with canvas rendering where appropriate.
A word on accessibility
This area needs further exploration. Using fallback content (the canvas DOM sub-tree) should allow screen readers such as VoiceOver to interact with the content. We’ve seen mixed results with the devices we’ve tested. Additionally there is a standard for focus management that is not supported by browsers yet.
One approach that was raised by Bespin in 2009 is to keep a parallel DOM in sync with the elements rendered in canvas. We are continuing to investigate the right approach to accessibility.
Conclusion
In the pursuit of 60fps we sometimes resort to extreme measures. Flipboard for mobile web is a case study in pushing the browser to its limits. While this approach may not be suitable for all applications, for us it’s enabled a level of interaction and performance that rivals native apps. We hope that by releasing the work we’ve done with React Canvas that other compelling use cases might emerge.
Head on over to flipboard.com on your phone to see what we’ve built, or if you don’t have a Flipboard account, check out a couple of magazines to get a taste of Flipboard on the web. Let us know what you think.