How I fixed a bug in India's national vaccination system

One line of code generated lakhs — possibly crores — of vaccination certificates with incorrect ages printed on them, and the power of open-source came in to find a fix

In a capital city ravaged by the second wave of the pandemic and a household swept along with it, the 4th of May 2021 brought with it hope and relief in the form of a vaccination appointment.

Both of those had been hard to find in the preceding three weeks. During the first wave we had the privilege of looking at graphs and numbers with a sense of detachment; in the middle of April this year —the onset of the second wave — it became deeply personal. My family became pixels on a curve that was rapidly dwarfing last year’s peak, with no bend in sight.

The appointment, therefore, represented the possibility of a return to "normal", however fleeting. India had decided to extend its vaccination program to those in the 18-45 age-group only days earlier. The administration and management of this drive was being done through "CoWin" - a government portal where citizens could register and book slots, and on successful vaccination, generate digitally verifiable certificates.

With some persistence, and a bit of luck, I managed to writhe through crowds, albeit virtual, and book a slot. The first shot was had, and I generated my certificate of partial vaccination through a two-step process on the portal. All relatively smooth, except one issue: my age was printed incorrectly.

Once the situation at home became a bit stable, though, I decided to dig deeper.

#CoWin: Under the hood

India's central vaccine administration and credentialing system is based, in large part, on an open-source project called "DIVOC¹" built and maintained by the eGovernments foundation⁶, offered under the MIT license.

What that essentially means is that it is free to use for anyone: be it small, private entities or massive ones like the Government of India. What that also means is that its code is freely available for anyone — including you and me — to access.

The adoption of "open-source" technology for government has not happened overnight in the country. The beginnings were made when Aadhar, a national unique identity project, demonstrated that it was possible to handle "India scale" without licensed, proprietary products to hold it up.

The transition has not been easy, and it is far from even being considered close to complete: selling software to government has long been the arena of international technology giants and consulting firms that remain difficult to dislodge, as the MIT Technology Review ² explains:

No one gets fired for hiring Deloitte or IBM. And when vendors keep getting the same kind of work they’ve done badly, there’s no incentive for them not to build a shitty system. Government requests for proposals are often written so they only fit one or a few vendors. You might see a yes or no box for, “Vendor must have worked on a healthcare system that serves over 500,000 people.” I don’t care whether that system exists, I want to know whether people who have to use it hate it.

A recent, relevant example is "VAMS" — The Vaccine Administration Management System — built for the Centers for Disease Control and Prevention (CDC) of the US by Deloitte, as part of a $44 million no-bid contract. What the software does is trivial at best — it does not include certificate generation — and has been met largely with indifference or criticism, as the MIT Technology Review ³, again, describes:

Instead, “VAMS has become a cuss word,” Marshall Taylor, head of South Carolina’s health department, told state lawmakers in January. He went on to describe how the system has badly hurt their immunization efforts so far. Faced with a string of problems and bugs, several states, including South Carolina, are choosing to hack together their own solutions, or pay for private systems instead.

#Open-source GovTech

Invariably, governments are reliant upon legacy software: technology that was purchased and installed decades ago, with costly annual maintenance contracts. To disrupt this is to attempt a notoriously difficult task of software archaeology, but there is a clear framework to evaluate its benefits:

• Ownership: The primary, and very significant, shift that open-source brings inside government technology departments is one of ownership: instead of vendors on whom all responsibility for failure can be pinned, they must step up and own their own software in letter and spirit. It necessitates the cultivation of expertise inside the government instead of it remaining perennially in the custody of those outside - people with no incentive to ever transfer it.

• Extensibility: Ownership breeds extensibility. Open-source software owned by the government can be suitably modified, extended and re-configured to adapt to additional use-cases without tedious contract renegotiations, or worse, additional tendering.

• Community: As XKCD wryly depicts here, almost all of modern digital infrastructure is supported by

  
  open-source software, maintained by communities of contributors. It is perhaps the most under-appreciated success of the internet: how so many people collaborate remotely to craft software for little monetary gain. For government technology, leveraging this aspect of open-source culture remains the most elusive. Countries like Singapore have taken strong strides to engage the community with their "GovTech" setup.

A beginning was made by India's National eGovernance Division recently by launching its own version of GitHub called "OpenForge", but it remains far from being a vibrant repository of government software that it was envisaged to be.

A nascent GovTech ecosystem has emerged in India in the recent past: eGovernments, EkStep and — my own organisation — Samagra, have made rapid strides in building open-source software rapidly deployable to state and national scale. Projects like Sunbird are notable examples.

To engage the community is to be able to build features fast, and crucially, fix issues faster. Issues like an incorrect age being printed on millions of vaccination certificates.

#Not just a number

Days after the appointment, I stumbled upon DIVOC's issue board to find an issue titled "Age is reflecting incorrect⁴". As you'd expect, it rang a bell.A discussion was already underway; a few hours of looking into the code and some help from friends later, I was able to identify a solution. It was simply the matter of taking one's birth month into account, which the existing code had missed. A single-line fix.

I proposed it to the maintainers of the project, not entirely sure if a response would be received.

Within two days, additional members of the community chimed in to suggest alternate implementations, different potential sources of error. It was pointed out that this fix would work only for a certain proportion of users; nevertheless, it would be an improvement. A final fix was agreed upon, and the submission — the single-line fix — was merged into the main codebase. The next day, it was released.

#The FOSS way forward

Software written for population scale in India is rapidly following this lead: the best traditions of free and open-source software (FOSS) are being leveraged to create software robust enough to withstand scrutiny and gracious enough to accept contributions that can improve efficiency, and solve issues of any scale. From critical bugs that affect performance and optimise efficiency to relatively minor ones that result in millions of incorrect certificates.

Not anymore, hopefully.