COVID Alert App — Part 7, When We Talk About the App Not “Working”
Public Health Efficacy vs. Successful Tech Deployment
On May 4, 2022, Canadian Senator Colin Deacon stood in the senate and spoke about COVID Alert, asking a colleague of his responsible for govt policy: “does our government have a plan to retire this application, and importantly, conduct a postmortem?”
In early thinking on this idea of a postmortem (something I’ve been talking about with my colleague Sean McDonald for a while now, more from us on that in a bit) there is one core issue that sticks out, and it has been sticking out for a while now, which is how we think about this app working or not working, and also if/how the governments understood how it was, or was not, working over the course of the months that is has been active.
Where is Health Canada’s Audit Report and Who Saw It?
I received an email back from the Office of the Privacy of Commissioner of Canada last week, stating that they did indeed take part of an audit of the app, as they said they would when the app was launched. But that was as far as the details went from that office — they told me that: “The evaluation was led by Health Canada’s Audit and Evaluation Branch. Our office did provide input to the government during the initiative. Given that the evaluation was led by Health Canada, you may wish to reach out to them for more information.”
I have not, and I’m coming up on a month now, found someone at Health Canada that will talk to me about the app. It was somewhat asinine that the Office of the Privacy Commissioner of Canada gave me a link to a generic media relations email address, when they would clearly have a better idea of exactly who at Health Canada worked on this file.
Before I move on, it feels important, to the discussion of a postmortem, to reiterate that both the Office of the Privacy Commissioner of Canada and the Information and Privacy Commissioner of Ontario, supported the app at launch AND said that oversight and monitoring of efficacy was important post-launch. I’m being repetitive about this point because it highlights how thinking about this app so much through the lens of privacy, and counting on privacy offices for oversight, has failed us here. Moving on.
In any case, the Office’s reply led me to this page, Health Canada Evaluation Reports: “The Office of Audit and Evaluation conducts evaluations that provide credible, timely, neutral evidence to support government accountability and decision-making on policy, expenditure management, and program improvements within Health Canada and the Public Health Agency of Canada.”
“Publication of these evaluation reports is consistent with the Government of Canada Policy on Results which requires government organizations to make reports and Management Response and Action Plans (MRAPs) accessible to the public…..Effective February 2013, all Health Canada Evaluation Reports are posted on this website.”
The report is not listed on the website.
If the privacy commissioner(s) took part in the audit, one might assume they saw the final report? I’m not sure. I’ll ask. One also wonders if the now disappeared advisory council had a chance to review it?
What Does Working/Not Working Mean?
As you’ll recall, the thing that brought my attention back to this app after not paying attention for a while was realizing that it was significantly broken — the app, which is now live, requires a PCR test results, which the general public hasn’t had access to for months now. But that does not mean that the app was “working” before testing capacity was overloaded.
If and when we do a postmortem on this app, we have to understand that we’re not doing a tech assessment, we have to ask ourselves what the app did in terms of public health impact.
Here’s one way to think about this, drawing on a time in my career when I knew more about online ad sales. Here’s a basic formula we could borrow from to think about public health impact. And I’m open to all suggestions on if/how this is wrong, of course. I’m just trying to think about how to explain the problem we have here. So here’s the basics of the online ad formula:
ad impressions — → number of click-throughs — — -> conversion rate
When you purchase online ads, you (used to?) buy them in something called “impressions”. So you can buy hundreds of thousands of impressions which will be displayed against the media of your choosing, say a website or a newsletter. Easy to measure and to keep track of.
Then, you can get information about how many people “click-through” on your ad. This number is generally dismal and is full of what we all do which is click on those ads by accident. In any case, that lets you know of all those hundreds of thousands of ads, how many people actually clicked on them.
Finally, and this is the efficacy part, for online ads, you can follow that click-through all the way to the user’s behaviour. In online ad world this might be buying something, signing up for something, etc.
Those are the three parts. So if we take this idea and consider the COVID Alert app, it looks like this:
Downloads are the ad impressions. Without the app or the ad, there is no starting point for tracking efficacy. So the download is the starting point. It is absolutely NOT efficacy. Next there is the one-time keys used, this is comparable to a click-through. The app has received the key, which then allows it to do what it is designed to do, which is alert others. This one-time key used number is also NOT efficacy.
We have to stop here to reiterate that these are the two pieces of public data the governments have made available to us, and to the media, to talk about how the app is or was performing. But this is, at best, incomplete information and at worst misdirection.
Impact of the app requires the equivalent of the conversion rate part. What happened after the one-time keys were uploaded? What did the app accomplish when sending out those notifications? Did it stop exposures? Did it increase testing? Did it strain testing? Did it cause harm or stress or confusion? Did it cause a loss of trust in government? What did that stage of the app’s function actually do? But most importantly, what did this app actually do, in terms of a public health response? And also very importantly, what were the known limitations on understanding the app’s efficacy at the time of launch, based on its design? This second one relates, in some ways, to the focus put on privacy in the design stages.
Frames for Efficacy and Understanding Power/Control
We know that industrial enthusiasm for a techno-solutionist approach ran high when this app was launched. But where did that enthusiasm go, industry-side? As noted in this post of April 21, 2021, Apple and Google did not seem to maintain much interest in evolving the app as knowledge evolved during the pandemic.
I’m still not sure how many people realize that there is exposure notification in the operating systems for both Apple and Android phones. The operating system. Not the app. You can delete the app and the operating system part is still there.
As I continue to read news and review documents the jurisdictional crash that occurred with COVID Alert becomes more and more significant. In the equation above, the one-times keys part is the province.
The two parties that have been impossible to get answers from so far are Health Canada and the Ministry of Health. Note that neither of those parties are the tech teams that worked on the app, though at the provincial level I’ve been directed to the Ontario Digital Service for answers about lab roll-out processes, which does not track with what makes sense to me but I remain open-minded.
Yesterday, I received an email from the province asking about their data, and they also redirected me to Health Canada, saying: “ Thank you for contacting the Ministry of Finance regarding the COVID-19 Data Alert. Uploads of the data set are created and managed primarily by Health Canada. Ontario receives this data set from Health Canada for inclusion in the province’s data catalogue. You may wish to contact Health Canada for more information about how this data set is created.”
The point here is this: if efficacy is framed as how the app did or did not technically function, it’s missing the point. The answers about how the app did or did not help us in the pandemic would not come from that realm, they need to come from an understanding of public health.
It seems telling that there is so much tech documentation about the app and so little documentation from Health Canada. We’ll see.
