The Missing Bit

Some details about PWA

2021-05-11
web pwa javascript

The term Progressive Web App has been around for a few years, I am not sure, but I think that it was introduced in this blog post.

It took some years to be properly supported, and as of 2021, it is quite well supported on Android and iOS.

For the past month, I have been working on transforming an hybrid app, composed of a web version, a native Android version and a Native iOS version to a PWA.

There are a few things I'd like to share, about some misunderstanding I had and some technical details about PWA.

What is a PWA

If you search for PWA, you find definitions like this: "Progressive web applications, or PWAs, are apps built with web technologies that we probably all know and love, like HTML, CSS, and JavaScript. But they have the feel and functionality of an actual native app."

To mean, this doesn't really mean anything, it is a bit like saying your computer is in a bad mood today while the real issue is that your neighbor's new Wi-Fi is conflicting with your wireless mouse.

But what are they really? They are regular web apps, but with a callback for network requests, so when the browser requests something, your code is called.

That is basically it.

So, how can this "changes everything"? Well, it doesn't really. We've been able to use web technologies for years, but PWA makes it easier. Well, not PWA per se, but everything that is "graviting" around it.

Let's explore the APIs that will make our lives easier.

API enabling PWA

Service Worker

The main API is called Service Worker. I heard that name multiple times before figuring out what it was. In my opinion it should be called "Proxy API" or "Proxy Worker" because that would make their use obvious. I think I would have used them earlier if I knew at a first glance what they were for.

That being said, Service Workers work in the following way:

  1. Registration: navigator.serviceWorker.register('/sw.js')
  2. Event listening: self.addEventListener('...', (evt) => {})
  3. The install event is called.
  4. The activate event is called.
  5. The fetch event is called for each request in scope.

Cache API

Another important element of Servie Workers, but not directly tied to them as you can use the API in regular Window code, is the cache API. It is an API like local storage, but for Response and it works in Service Workers (local storage does not).

The cache API is very simple to use with simples put and match APIs.

Manifest

Not an API, but a central piece of what makes a PWA. The manifest is a JSON file describing the App, with things like it's name, description, icons, …

The manifest location is specified by a link tag in your HTML's head:

<link rel="manifest" href="manifest.json">

IndexedDB

While local storage is not available in workers, IndexedDB is. The API is quite complicated and unless you have specific needs, I suggest you use a simple wrapper like localforage.

A few gotchas

HTTPS

For service workers, EVERYTHING should be served via HTTPS. Firefox has an option to allow HTTP for development, and chrome allows HTTP by default for localhost.

Mime types

Another problem, especially with dev servers, are MIME types. Browsers will refuse to execute the script if the MIME type is not right. Firefox has an about:config override if your dev server is not well behaved security.block_Worker_with_wrong_mime. I did not find equivalent setting in Chromium.

Scope

That one is a bit weird, but workers have scope. Remember that register call from above? You pass an absolute URL to it, and service workers will only proxy requests BELOW that scope.

So if you do:

navigator.serviceWorker.register('/assets/sw.js')

The service worker fetch callback will ONLY be called for requests starting with /assets/. The register function also take a scope option, but that scope CANNOT be broader than the script location. So if your script lives in /assets/sw.js and you do

navigator.serviceWorker.register('/assets/sw.js', { scope: '/pages'})

it will fail install.

This behavior can be altered by adding the Service-Worker-Allowed header.

The simplest configuration is to just put your JS file at /sw.js and it will be able to proxy the whole site.

Webpack

Webpack 5 can magically detect service workers without adding another entry point. But it works under certain conditions and I had it break a few times.

I recommend simply adding another entry in your config like so:


  entry: {
    app: './assets/app.js',
    sw: './assets/sw.js'
  },
  output: {
    filename: '[name].js',
    path: path.resolve(__dirname, './static/')
  },

This will compile them separately and you will be able to load the /sw.js file with the register call outlined above.

Delayed promise

Some promise libraries have delayed promise. Basically they do setTimeout(nextFunc, 0). I'm sure they had their reasons when promises where not baked in browsers, but this WILL break service worker event handlers.

If you have something like this:

self.addEventListener('fetch', (evt) => {
  evt.respondWith(somePromiseThatIsDelayed())
})

The browser will consider the event not handled, and will fire a regular non proxied request. You have to ensure that the promise passed to respondWith is a continuous chain.

That one bite me, and it took a while to debug. I saw the promise code executing, but the browser was also making a regular request, so my app was not working when offline.

Self

What is self? Well, self is the name given to the current global scope. It is like window but generic to both workers and windows. You can use self instead of window in regular scripts too which makes them more portable.

Upgrade

The service worker upgrade process is automatic when the server detect that the /sw.js file has changed. It does so by doing a bytes comparison with the current file.

When an upgrade is triggered, the install event is fired again, followed by activate event. But the new worker WILL NOT be active right away. To activate it, you need to claim the clients with clients.claim() is the activate event handler.

Cache versioning

The easiest way to handle cache versioning is to do something like this:

const CACHE = process.env.CACHE_NAME
//...

caches.open(CACHE).then((cache) => { cache.addAll(files)})
//...

// in activate event handler:

  const keys = await caches.keys()
  for (const key of keys) {
    if (key === CACHE) continue
    caches.delete(key)
  }

Then, if you use Webpack, you can do something like

new webpack.EnvironmentPlugin({ CACHE_NAME: `cache-${Date.now()}` })

and the cache will be busted on every Webpack build.

Of course, this might be a bit extreme, but you can adjust it the way you need.

Redirected response

You cannot put redirected response into the cache. For example, in my app, I had many links pointing to /posts/foo instead of /posts/foo/. The first one redirects to the last one. What I ended up doing is checking if my request should be cached, and if not, adding a slash and checking again (see example below).

Console

In Firefox, console.log don't log to the regular console when called from service workers. To access service worker console, you need to go to about:debugging and click the inspect button corresponding to your service worker, this will open a separate debugger and console.

Response consumption

A response's body can only be consumed once. If you want to cache a response and return it, you need to call clone() on it.

Small example

The following code is testing quality. I am in early stage of learning about PWA.

This example is commented inline:


// Babel runtime for async code
import 'core-js/stable'
import 'regenerator-runtime/runtime'

// The file list is generated by an external program, but any approach is fine
// This is a list of absolute paths, ["/foo/", "/bar/"]
import files from '../data/files.json'

// Local forage for simple key/value store
import localforage from 'localforage'

// Force local forage to use indexed db
localforage.config({
  driver: localforage.INDEXEDDB,
  name: 'cache'
})

// Dynamic cache name, changes every build
const CACHE = process.env.CACHE_NAME

self.addEventListener('install', (evt) => {
  // Do not wait, just install and activate regardless of client state
  self.skipWaiting()
  console.log('Installing PWA')

  // Wait until promise is done
  evt.waitUntil(
    // Open cache and empty the index database
    caches.open(CACHE).then(localforage.clear()).then((cache) => {
      const now = Date.now()
      for (const f of files) {
        try {
          // We set the items that should be cached with when it was cached as
          // now
          // The cache below might fail, but we don't care, because we check
          // the time only on match success (for refresh)
          localforage.setItem(f, now)
        } catch (e) {}
      }
      return cache.addAll(files)
    }).then(() => console.log('Installation successful'))
  )
})

self.addEventListener('fetch', (evt) => {
  evt.respondWith(resolveRequest(evt.request))
})

self.addEventListener('activate', (evt) => {
  evt.waitUntil(activate())
})

async function activate () {
  // Claim all clients/windows (upgrading them)
  await clients.claim()
  // Clean older cache
  const keys = await caches.keys()
  for (const key of keys) {
    if (key === CACHE) continue
    caches.delete(key)
  }
}

async function resolveRequest (request) {
  let response
  let shouldCache = shouldCacheRequest(request)

  if (shouldCache) {
    response = await tryCache(request)
  } else {
    // Because of redirects, try with slash
    const requestWithSlash = appendSlash(request)
    shouldCache = shouldCacheRequest(requestWithSlash)
    if (shouldCache) {
      request = requestWithSlash
      response = await tryCache(request)
    }
  }

  // If we have a cached request, refresh it if needed (here 24h hard coded)
  if (response) {
    refreshIfNeeded(request, response)
    return response
  }

  // No response, try network
  response = await tryNetwork(request)
  if (response) {
    // Network was successful, update the cache, clone response
    updateCache(request, response.clone())
    return response
  }
  return await fallback(request)
}

async function tryCache (request) {
  return await caches.match(request)
}

function appendSlash (request) {
  return new Request(request.url + '/')
}

// Wrapper to return null on fail
async function tryNetwork (request) {
  try {
    return await fetch(request)
  } catch (e) {
    return null
  }
}

async function fallback (request) {
  const url = new URL(request.url)
  // Here you might use the request url.pathname to adjust the offline page
  // You can also reply with a manually crafted Response()
  let path = '/offline/'
  return await caches.match(path)
}

async function refreshIfNeeded (request, response) {
  const url = new URL(request.url)
  const path = url.pathname
  const when = await localforage.getItem(path)
  const now = Date.now()
  // Daily refresh
  if (!when || now - when > 24 * 3600 * 10 * 1000) {
    const res = await fetch(path)
    updateCache(request, res)
  }
}

function shouldCacheRequest (request) {
  const url = new URL(request.url)
  const path = url.pathname
  // Do not cache not listed in file list
  return files.indexOf(path) >= 0
}

function shouldCacheResponse (response) {
  const url = new URL(response.url)
  const path = url.pathname
  // Do not cache not listed in file list and do not cache other status than 200
  return files.indexOf(path) >= 0 && response.status === 200
}

async function updateCache (request, response) {
  // We don't cache response that are not supposed to be cached
  if (!shouldCacheResponse(response)) return
  const url = new URL(request.url)
  const path = url.pathname

  const now = Date.now()
  const cache = await caches.open(CACHE)
  await cache.put(path, response)
  await localforage.setItem(path, now)
}
If you wish to comment or discuss this post, just mention me on Bluesky or email me.