For a current project at work, I am adding offline support to a mobile web app. I'm very excited about this as I haven't worked with HTML5's offline support yet. This is an attempt to take notes about my experience.
Before I begin, there are two things specific to this project that might be different from your own offline attempts. For one, as is common here at Applied Geographics, this project uses a map widget which references map tiles that are stored external to the website. Secondly, while there is very little server-side code, this website is hosted on IIS 7 which is common for our clients. That said, if I find myself writing something IIS-specific, I will try to also mention the Apache way.
Let's dive in!
Initial environment
The original site uses a .NET backend to connect to Oracle. It exposes functionality through web services. The frontend is 100% HTML and JavaScript using jQuery Mobile and jQuery Geo. The website is behind cookie-based authentication.
Users of the site can sign in to collect location and other information about features. In this context, a feature is a point on a map. The type of other information collected depends on how an admin configured the app. Since that's not important to adding offline support, I won't get into how these collection apps are configured.
Plan
design
The difference in user experience we have designed for this app is that when an update attempt fails due to lack of Internet connection, the collection app will cache the update request. If there are any cached update requests, a button on the app's home screen should allow the user to re-attempt all updates. Apart from handling failed update requests, the app itself should continue to operate is if it were online.
changes
From the introductory knowledge I have about offline HTML5 I can form a rudimentary plan for what I need to do on the tech-side to add offline support.
- appcache – I need an appcache file in which to list all the files that should be cached and what to do when the user loses his or her connection
- session storage – failed update requests will be stored using HTML5 web storage so they can be attempted later
This post will concentrate on the first part, appcache, which I know almost nothing about.
gotchas
Also from my introductory knowledge, I can guess at some issues I might have implementing this.
- jQuery Mobile's hijax – jQuery Mobile doesn't load pages normally, it intercepts link clicks, pulls in the page with magic and visually slides the page into view; how will this affect the appcache?
- map tiles – map widgets pull tiles from external services and I have heard that using any external file breaks the cache and the site operates as if it weren't cached; is that true?
- web services – the web app issues POST/PUT/DELETE requests to local web services; will this break the cache?
- authentication – the browser will not initially be able to download files in the appcache; will things be ok after login?
other questions
- single file – jQuery Mobile allows all pages to be in a single HTML file to be referenced by id; do I need to do this to use offline support?
- ajax – I'm curious about ajax calls in general; how are web service calls supposed to work with offline support when they are not static files?
- test – what's the best way to test that the cache is successfully being used?
Support
I will be testing on these mobile devices only: Apple iPod Touch 4, Apple iPad 2, HTC Droid Incredible 2, Motorola Droid Bionic.
Appcache
One resource I found especially helpful in constructing this case study is Offline Browsing in HTML5 with ApplicationCache by Malcolm Sheridan. It appears to be a good introduction to the appcache file and I'm going to use it for initial suggestions.
manifest reference
There is a manifest file which needs an appcache extension. I'm going to name mine "col.appcache" to match other names in this collection app. You have to reference the manifest file from the html element of every HTML page.
The catch is that this must be included on every page. If the reference isn't there, the browser won't cache the page.
I assume that means all HTML pages even if they're pulled in by jQuery Mobile. Without even having an appcache file yet, I'm going to go ahead and add the reference to all my html pages.
<html manifest="col.appcache">
I have not added it to the only ASPX page, login.aspx, which does have an html element. My assumption is that this page isn't part of the cache. My hope is that once the user logs in, they will get to index.html and all the supporting JavaScript, image, and CSS files that are part of the cache, and all will be well.
I also have obviously not added the reference to any web service files, which don't have html elements (or any elements at all, because they return JSON).
mime
Next up, I have to make sure the web server returns the MIME type as: text/cache-manifest. The blog post I referenced shows how to do this through IIS 7's GUI. The nice thing about IIS 7 over previous releases is that, finally (!), it's entirely configurable through the static web.config file. Changing the MIME Types setting in IIS 7 is the same as adding the following to your system.webServer section:
<system.webServer> <staticContent> <mimeMap fileExtension=".appcache" mimeType="text/cache-manifest" /> <staticContent> </system.webServer>You can add the MIME type to Apache by editing the .htaccess file and adding this line (if it doesn't already exist):
AddType text/cache-manifest appcache
result
I can immediately tell that the appcache is doing something because after I sign in with Chrome it prints, "Application Cache Error event: Manifest fetch failed (404) http://localhost/mdf/col.appcache" to the console.
Manifest file
Again, I refer you to the blog post linked above for a great introduction to the structure of this file. I'm going to follow along and write down my thoughts.
header
The first line has to be:
CACHE MANIFEST
Easy enough. Next, he recommends a date stamp that you can change to force the browsers to completely refresh all the cached files. He does this with a date stamp in a comment, a pattern I will follow.
# 2011-11-29
When I update the website files in any way, I will update this comment even if I'm not adding or removing whole files.
cache
The CACHE section lists every static file we want in the appcache. My website has:
- 6 html files
- 7 static js files
- 1 static ashx file that returns JavaScript specific to this collection app configuration
- 2 css files
- 5 png files
The remaining files are REST web service endpoints, the login page or server-side code.
I know I should compile some of the JavaScript files into one file but the project is not complete enough for that sort of optimization. In other words, I'm feeling lazy.
My first real question: should I include favicon.ico and apple-touch-icon.png as static files? I do actually reference them in the html files. I'm going to leave them out for now.
CACHE: 401.html 404.html featinfo.html index.html locmap.html notsupported.html Context.ashx js/col.js js/jquery.geo-test.min.js js/jquery.mobile-1.0.min.js js/jquery.webStorage.min.js js/jquery-1.6.4.min.js js/json2.min.js js/proj.min.js css/jquery.mobile-1.0.min.css css/style.css css/images/ajax-loader.png css/images/icons-18-black.png css/images/icons-18-white.png css/images/icons-36-black.png css/images/icons-36-white.png
network
Next up, the NETWORK section which:
defines which resources requires the user to be online
Ah! This is where I put my REST endpoints. That makes sense. However, it raises my second real question. I don't want REST requests to fail behind the scenes if the user isn't online. The stated design is that, when not online, the failed update request will be cached to be attempted later. I'm hoping that by using the regular jQuery $.ajax function, I will still get the error callback, at which point I can cache the original update request myself.
I have three REST endpoints for this website.
NETWORK: GlobalData.ashx MobileFramework.ashx MobileFrameworkData.svc
The two ashx files are generic HTTP handlers that return JSON data based on GET or POST requests. These are no different than, say, a php file doing the same. The svc file is a .NET WCF Data Service which wraps an Object-Relational Model (ORM) that connects to the client's Oracle database. In the end, though, it accepts HTTP requests and returns JSON data like anything else.
Malcolm also lists login.aspx in the NETWORK section. I'm likely wrong about this but I'm going to stick with my initial plan to leave the login page out of the cache. The file is self-contained and, as I see it, a barrier to the application as a whole. I don't want the browser to attempt to download the files listed in the manifest if it doesn't have access to do so.
fallback
I can't entirely picture how this works. The theory is that if a file or image is missing we can specify replacements here. However, all my static files are in the cache, so why would they be missing? It must be for items included in the NETWORK section. Since all of my NETWORK items currently return JSON, I suppose I can have an empty JSON object (or just the keyword: null) in a static file and use that as a fallback. This is something I will look into later.
map tiles
Malcolm's blog post doesn't mention external resources so I went looking around the Internet and found Appcache Facts. The third fact listed explains the sections a little more and says this about NETWORK:
If you want to allow arbitrary URLs to be accessed (scripts, stylesheets, API calls, anything), include *, http://* and https://* in this section.
That sounds like my answer. I should put the root of the map tile URLs here with a * at the end.
http://tile.openstreetmap.org/*
I am a little suspicious that I need to put exactly the three URLs mentioned. I hope specifying an external domain is correct. We shall see. Also, I can maybe put an empty tile image in the FALLBACK section for when the user is offline but goes to the map page.
ssl
The fourth Appcache Fact talks about SSL.
Over SSL, all resources in the manifest must respect the same-origin policy
I can't get into this now because the client doesn't have SSL enabled on the server. However, they do plan to publish the site over SSL. This will cause a problem because the map tiles will likely be on a different subdomain. I'll cross that bridge when I have to and try to come back and write about it.
roll out
That's my full appcache file. I'm ready for my first test.
Chrome
login page
I typed in the app's URL and was taken to the login screen. There's no mention of appcache in Chrome's console.
first load
After logging in, I get the following in Chrome's console:
Document was loaded from Application Cache with manifest http://localhost/mdf/col.appcache Application Cache Checking event Application Cache Downloading event Application Cache Progress event (0 of 22) http://localhost/mdf/css/jquery.mobile-1.0.min.css Application Cache Progress event (1 of 22) http://localhost/mdf/js/jquery-1.6.4.min.js Application Cache Progress event (2 of 22) http://localhost/mdf/js/jquery.geo-test.min.js Application Cache Progress event (3 of 22) http://localhost/mdf/css/style.css Application Cache Progress event (4 of 22) http://localhost/mdf/js/jquery.webStorage.min.js Application Cache Progress event (5 of 22) http://localhost/mdf/js/json2.min.js Application Cache Progress event (6 of 22) http://localhost/mdf/js/proj.min.js Application Cache Progress event (7 of 22) http://localhost/mdf/js/col.js Application Cache Progress event (8 of 22) http://localhost/mdf/js/jquery.mobile-1.0.min.js Application Cache Progress event (9 of 22) http://localhost/mdf/401.html Application Cache Progress event (10 of 22) http://localhost/mdf/404.html Application Cache Progress event (11 of 22) http://localhost/mdf/css/images/icons-36-white.png Application Cache Progress event (12 of 22) http://localhost/mdf/featinfo.html Application Cache Progress event (13 of 22) http://localhost/mdf/notsupported.html Application Cache Progress event (14 of 22) http://localhost/mdf/index.html Application Cache Progress event (15 of 22) http://localhost/mdf/css/images/icons-36-black.png Application Cache Progress event (16 of 22) http://localhost/mdf/css/images/icons-18-black.png Application Cache Progress event (17 of 22) http://localhost/mdf/css/images/ajax-loader.png Application Cache Progress event (18 of 22) http://localhost/mdf/css/images/icons-18-white.png Application Cache Progress event (19 of 22) http://localhost/mdf/locmap.html Application Cache Progress event (20 of 22) http://localhost/mdf/?auth2 Application Cache Progress event (21 of 22) http://localhost/mdf/Context.ashx Application Cache Progress event (22 of 22) Application Cache UpdateReady event
To me, that couldn't look more correct for a first attempt!
When I click on the Resources tab in Chrome's Developer Tools and expand the Application Cache node, I can click on the localhost item. The right panel fills with a list of all files from my cache. One file has Master as the Type, another file has Manifest, and the rest have Explicit. All web service calls are missing but they clearly worked because the app was able to query the database. The web service calls appear in the Network tab like usual.
I successfully added a new non-map feature to the database. I'll try mapping soon, but first...
refresh
I hit F5 and Chrome refreshed the app lightning quick. The console says:
Document was loaded from Application Cache with manifest http://localhost/mdf/col.appcache Application Cache Checking event Application Cache NoUpdate eventI wasn't expecting this part to work so smoothly!
map
Next I added a new feature, but this time clicked a button labeled "Draw on Map" that we have on the page which lets me place a point on a map. Chrome spit a bunch of these onto the console:
x GET http://tile.openstreetmap.org/7/29/46.png
Chrome didn't even request the tiles. There's no 404 status because there's no response at all. Clearly my last NETWORK section line isn't working as I had hoped. I'm going to swap it for the wildcard suggestion in Appcache Facts.
* http://*
Chrome updated the cache when I refreshed the browser. I tried to go to the map again but got the same result, no map tiles. Maybe it's placement in the appcache file. Since the wildcard will cover even my local web services, I'm going to comment them out and just have the wildcard lines.
NETWORK: * http://* #GlobalData.ashx #MobileFramework.ashx #MobileFrameworkData.svc
That worked! So much for being specific. However, I feel the correct way to do this (and the direction I think I will head) is to have a proxy handler to return map tiles. A setup like that will also get me around the SSL same-origin policy issue.
proxy
I wrote up a quick proxy file, GlobalMap.ashx, so that GlobalMap.ashx?/=/7/29/46.png returns the same image as http://tile.openstreetmap.org/7/29/46.png. Now I can put GlobalMap.ashx in my NETWORK section and remove the wildcards.
NETWORK: GlobalData.ashx GlobalMap.ashx MobileFramework.ashx MobileFrameworkData.svc
All is well this way and I feel better about it. The app is working...Time to pull the plug!
offline
I removed my network cable and refreshed the browser. The cached static files loaded fine and quickly. The first web service call failed as planned and jQuery called the ajax error callback. However, the first ajax call asks the web server for some extra information about this specific collection app. In my error callback I forward the user to 404.html. This is not ideal for an app that should seamlessly stay functional in a temporary offline situation.
edit a cached file
I edited col.js, the main JavaScript file to not forward to 404.html if it has retrieved the extra app information at least once this session. Then I hit my second snag: Chrome won't load the updated col.js file no matter how many times I "Empty the cache", update col.appcache, or refresh the browser. I can type the URL for the JavaScript file directly into the address bar and see the old, unchanged file.
It turns out, just having "Empty the cache" checked in the "Clear browsing data" dialog isn't enough. I don't know which one helped, but I checked all of the options and refreshed. Chrome grabbed the updated JavaScript file.
offline for real
With the new code, I again pulled the plug and the first page loaded fine. The first thing it attempts to do is query the database for a list of features already collected. This failed and jQuery called my error function. This couldn't be better as instead of just displaying an error I can now alert the user that he or she may have lost connectivity and will have to try again.
The real test is to try to add a new feature. I clicked the appropriate button and the new page slid in just as it would if I were online. The appcache worked, even when jQuery Mobile was hijaxing the page in!
I entered some information, not testing the map yet, and clicked Save. I got another error callback which is exactly what I wanted. As before, tell the user to try again in a minute.
After plugging the network cable back in, I clicked Save again. The data saved as normal and the app took me back to the feature list which showed the new feature and the old ones.
HTC Incredible 2
Seeing this work in a desktop browser helps determine if things are setup correctly. However, we need to test on real mobile devices to see how it will work for the client.
first load
I opened the default web browser on the HTC Incredible 2 and browsed to the collection app. There is no console to determine if everything is working as planned but the app functioned as normal: login page => feature list => feature info, etc.
airplane
Then I flipped the phone into airplane mode and went back to the browser. After refreshing, things started normally. The feature list page returned empty and the request for previously collected features failed. At the same time the browser alerted me, saying that I should enable some sort of connectivity. I tapped Cancel and went back to the app, which was not happy. When I tapped the New Feature button, jQuery Mobile failed to load the next page and alerted me with "Error Loading Page", even though the HTML file is in the appcache I assume.
I turned airplane mode off and the app got into an odd state. I could refresh and get the list of previous features but attempts to view their info or create new features results in the same jQuery Mobile error notification as when offline. As it turns out, this does not seem to be realted to the app's connectivity. The issue appears to be between appcache on Android and the way jQuery Mobile loads pages. If I were to guess I'd say it has something to do with the login page not being part of the cache but existing on the same directory. Further, and I'm still guessing here, I think leaving the appcached site in any way will cause a jQuery Mobile request to another page in the cache to fail. I would like to test this without AJAX or jQuery Mobile but I don't have time to set that up at the moment.
Android & appcache
I felt the need to test what I did have more. I killed the Internet app, launched again and cleared the browser cache before logging in. At this point the web app functioned fine. I could view existing feature data and add new features. I tapped the sign out button and then signed back in. Without ever having been offline, jQuery Mobile was no longer able to load the secondary pages.I'm not claiming this as an issue with jQuery Mobile, which is only requesting the page from the browser. The browser is not supplying the requested page from the cache. However, it may be something the team would want to look into for potential workarounds.
To give my hypothesis a bit more evidence, I made an exact copy of the website. In the copy, I removed the manifest attribute from the html element of every page and deleted the col.appcache file. After browsing to this new site and following the steps above, the jQuery Mobile app worked perfectly on Android. I can sign out and in as much as I want and still edit features.
This unfortunate behavior might drive me to merge all jQuery Mobile pages into a single HTML file, index.html, which I am able to view even when loading other pages fails.
true offline test
Believing that there's some issue with exiting and then returning to the site, I wanted to test appcache as properly as I could. This meant I had to lose connectivity naturally. Also, to make sure no external forces interfered, I wanted to keep the Internet app running, i.e., without switching to other apps (even Settings). I performed my test while riding the MBTA Red Line.
After clearing the Internet app's cache, I browsed to the web app above ground & waited to lose wireless deep under Central Square. At that point, I tapped New Feature and in slid the data entry page. The page was in the cache, and I could access it. Attempting to save a new feature failed, as expected. When I regained connectivity, I could save. This was a good test, but didn't change the fact that the app would no longer work if the user opened another app.
first conclusion
I don't know where this leaves me. Appcache with multiple pages and jQuery Mobile works fine on Android unless you leave the app? I will test on other Android devices as Android web browsers all seem to work slightly different.
iPod Touch 4
Next up, an iPod Touch 4, with iOS 4. This device is WiFi-only. I opened Safari Mobile and browsed to the web app. It functioned fine so I turned off WiFi in Settings. I tapped the New Feature button and the empty data entry page slid in. Like Chrome, I got an error trying to save this way but re-enabling WiFi and tapping Save again worked as usual.
Once back to the feature list page, I tapped the sign out button. After reauthenticating, I could access the feature list and view feature details.
Mobile Safari appeared to be working well until I tapped the browser's refresh icon. I had actually tested Mobile Safari with older versions of the app, and (to my surprise), an older version of the feature list page appeared. Clearing the cache in Settings didn't help–refresh always shows the old version.
Searching for this issue online led me to this post: http://www.genuitec.com/support-mobione/viewtopic.php?t=2096 on the MobiOne support forum. It says nothing less than iOS 4 doesn't clear the appcache unless you clear the browser cache and then shut the phone off. Oops! After following those steps, I could tell the cache was truly cleared, after which I got the updated page.
I haven't tested on iOS 5 but will provide an update when I can.
Redesign
I needed to fix the code for Android and decided to merge all of the pages into one. My hope was that the roadblock was only the way pages were requested in jQuery Mobile; requests for other static files such as JavaScript and CSS were fine.
Also, having run through the app a bunch of times and become more familiar with appcache, I decided to try a couple other changes:
- add login.aspx to the app cache
- allow anonymous access to all static files; security will be done at the web service level
single page
This change was easy. First, I copied the data-role="page" divs from featinfo.html and locmap.html into index.html. I also removed the two files from the col.appcache manifest. The only thing that remained was that anywhere I referenced an HTML file in a link, I had to change it to use an id. For example, the New Feature button on the main page was:
<a href="featinfo.html" data-role="button" data-icon="plus">New Feature</a>href="#featinfo"<div id="featinfo" data-role="page" data-theme="b">After these changes, I did some quick tests in Chrome and all seemed well. It was actually a little more responsive because the feature info page was already loaded in jQuery Mobile, it didn't have to hijax in a new physical page. Sure, the page was in appcache but there's overhead of having jQuery Mobile request the page.
login page
I'm now following Malcom's blog post's suggestion of adding the login page to the appcache. I get the feeling that these apps should be all or nothing in that every resource that could possibly be requested should be in the manifest somewhere. Since you can't sign in without being online, login.aspx goes in the NETWORK section. I also had to add the manifest reference to the login page's html element. Lastly, I allowed anonymous access to all static pages (pages in the CACHE section of the manifest) so that browsers can get to them right away. Back to Chrome for some tests.
As expected, Chrome now tries to load the whole cache when it gets to the login page. However, loading the appcache failed with a couple of these:
x GET WebResource.axd?xxxxxxxxxxxxxxxxx
Heh, ASP.NET. So, when ASP.NET preparses an ASPX page, it inserts references to core ASP.NET JavaScript. These references all come from a special address: WebResource.axd. This URL doesn't exist as a physical file in the website but the browser will look for it anyway when building the appcache. I had to add this HTTP resource to the NETWORK section of the manifest file.
With this last change in place, the full appcache loaded fine.
manifest
For folks playing along at home, my full manifest file looks like this now.
CACHE MANIFEST # 2011-12-03 CACHE: 401.html 404.html index.html notsupported.html Context.ashx js/col.js js/jquery.geo-test.min.js js/jquery.mobile-1.0.min.js js/jquery.webStorage.min.js js/jquery-1.6.4.min.js js/json2.min.js js/proj.min.js css/jquery.mobile-1.0.min.css css/style.css css/images/ajax-loader.png css/images/icons-18-black.png css/images/icons-18-white.png css/images/icons-36-black.png css/images/icons-36-white.png NETWORK: login.aspx GlobalData.ashx GlobalMap.ashx MobileFramework.ashx MobileFrameworkData.svc WebResource.axd
mobile
I tested this new design with both the Droid and iPod. In short, they both worked perfectly. I can sign in and out while online without issues. I can get to the New Feature page while in airplane mode. Also, the extra responsiveness of not hijaxing in new pages is even more apparent on mobile. If you plan on using appcache and jQuery Mobile, I recommend having only one HTML page.
Offline map
My last task directly related to appcache is handling the map. I need to display something when the user moves to the map page but can't get to map tiles. I'm pretty sure this is where the FALLBACK section of the appcache comes into play. The format of lines in the FALLBACK section is: resource fallback file.
Since I have a proxy for map tiles, I can target that proxy specifically. All the tiles are 256x256 pixel squares. I made an image of that size with some text about having lost connection temporarily and added the following to col.appcache.
FALLBACK: GlobalMap.ashx img/missing-tile.png
mobile
This part worked great on both of my test devices!
For the HTC Android, I started on WiFi and browsed to the map page. On the map page, the user can tap to select a point location for the feature. The map came up as expected. I tapped the browser's back button, and then shut off the wireless router. Re-entering the map page showed map tiles as normal–I assume they're coming from the regular browser image cache. Panning the map kept the old tiles in place and my fallback image appeared for every new tile request...until the 3G connection kicked in. At that point, I started to get real tiles again.
The iPod Touch worked about the same except that there were no cached images when I went back to the map page after shutting WiFi off. I immediately got a full screen of missing tile images. While not ideal, it is functional.
I can think of only one potential problem: what if the browser caches the missing tile image as the real tile? When the user has a connection again, will they still get the missing tile image?
airplane
Testing appcache by putting the phone on airplane mode is less accurate to the real-world environment for the app but I wanted to try it anyway. As it turns out, the Android browser operates quite differently in airplane mode than it does after losing connection naturally. For example, I can tap the "Draw on Map" button and the map page slides in. However, while the map page's header looks fine, I don't get either real map tiles or the fallback image in the map widget itself. It's as if the browser returns 404 to all GlobalMap.ashx requests. Stranger still, the map page doesn't go away when I tap the back button. The URL changes, but the empty map page stays. I had to manually tap refresh to see the feature info page.
The iPod operated normally when I put it in airplane mode. I got all missing tile images and was still able to select a point and use the back button.
I'm glad I tested airplane mode again. I at least learned that Android behaves oddly. This does not feel like it will affect real-world usage of the app.
Round trip
conclusions
Appcache is an excellent new tool in web your development pocket. It has development quirks and implementation issues on mobile devices but adding it is well worth the effort. I hope this stream-of-consciousness post helps a few people. Have fun!
- Appcache is fast, not too hard to setup, and works mostly great!
- As of this writing, if you are using jQuery Mobile, I recommend rolling all your pages into a single, cached HTML page for Android support.
- iOS 4 doesn't properly clear the appcache; point users to http://apps.ft.com/ftwebapp/troubleshooting.html.
- In spite of my initial thoughts, all resources should be included in the appcache in either the CACHE or NETWORK section.
- If you are using ASP.NET and you add any ASPX page to the manifest, e.g., a login page, also add WebResource.axd to the NETWORK section.
- The FALLBACK section is perfect for missing map tiles.