Last week, we all saw how software projects, when not properly implemented, can create havoc on our lives and institutions. If you've been living under a rock, the Iowa Caucus, also known as the kick-off to the presidential election season, was conducted and quickly turned into a debacle when the mobile application precincts were using to report results started to become unstable, and ultimately fail.
We've spoken before about projects gone awry. Last year, Hertz sued their agency for $32M after it was unveiled that the project to redesign their digital projects went massively wrong, costing the company millions of dollars and preventing them from going live with their new website and applications. In this case, however, this is even more tragic. One poorly designed app has implications for the future of the Iowa Caucus, may have influenced the election in the short term in terms of the Democratic candidates, and gave the opposition party a talking point and fodder from which to sow the seeds of doubt on the entire process. Given that presidential campaigns are nearly billion-dollar enterprises, this app could be one of the most significant digital failures of all time.
So what went wrong?
Well, more than a few things. Rather than recount the story, which you can read about here or here, I want to focus on the key tidbits that led to this disaster. Since the dust has sorta settled, let's look at some of the problems with this app, how it was built and how it was deployed.
The "Focused" Development Firm
There is a massive movement within the digital services space to be as niched as possible, focusing on one particular area. Admittedly, we are part of that movement, focusing on custom CMS development and custom software development. However, while we maintain a focus on a type of project, many firms focus on particular industries or verticals. This is a smart practice for gaining lead generation, but the trend of being an industry expert has also led to companies such as Shadow, which was hired to develop this application. The fact is, while the company positioned itself as a technology provider for Democratic causes, it clearly did not have the knowledge or capability to pull of an application of such importance and scale. Had they the background in developing such an application, they wouldn't have made the simple mistakes which we will cover in a bit.
Looking at the capabilities of the company, one observer noted:
"The caucus app is firebase / react app built by one senior engineer who's not done mobile apps and a bunch of folks who were very recent code academy graduates who as of a couple months ago worked as a prep cook for Starbucks and receptionist at Regus."
That's crazy! And why was it allowed to happen? Because the senior principals of the agency were insiders and had networked their way to being hired for the project. The fact is, the senior staff had no idea how to execute every aspect of such a valuable application, and they had hired junior staff who had no experience.
Now, I'm in no way saying that focused firms don't know what they are doing. In fact, I believe quite the opposite, when that expertise is authentic. But, I am sure that the focus of this agency had set the Democratic Party of Iowa at ease and made them feel there was a minimal risk when clearly the inverse was true.
The lesson here: niched doesn't mean experience, and it certainly doesn't ensure success, especially when the stakes are high and the project is outside the comfort zone of the agency.
Budget Deficiency
It's been reported that Shadow was paid $60,000, or thereabouts, to complete this application. This amazes me because the value of this project is a multiple of that. The candidates, based on TV spend alone – not counting digital, print, travel expenses, or any other tertiary expenses – spent almost $60M in Iowa. Yet, the party put only $60,000 into building an application that was going to calculate the return on so large an investment. That is .1% of the value of the TV buy alone, and even less when you add in all the other ways candidates spent dollars on this election.
$60,000 is not enough to ensure a proper product development lifecycle, comprehensive quality assurance and testing phases, a series of comprehensive audits to assess security, stability, and scalability of the application, and the proper deployment, training, and support that the application would require. An application like this, where the stakes are so high, requires an approach that considers all of these factors, and allows time and resources (such as budget) to complete each step at an expert level.
At this point, I think it's fair to say that each campaign would've contributed $60,000 to ensure the votes were tabulated correctly.
Improper Project Execution
At the end of the day, the project failed because the company producing it didn't take the proper steps to execute on the requirements. We aren't privy to the internal workings of the agency, nor do we understand the communication between the client and the agency. But we do know two things. The agency was unable to execute effectively, and the client bears responsibility for hiring the agency. It wasn't the agency principal or developers who were apologizing on TV.
Reports are that the app was being developed down to the wire. It was distributed via testing channels and not even published to the proper app stores or distributed in a manner consistent with internal-use applications. The night before, according to internal sources, the application was already failing. We can't be sure when the project started or what went into producing it, but we can be assured that it failed miserably, as so many projects are prone to do. So what failed, specifically? It seems a few things.
In an interview, the agency principal seemed to indicate the app worked well with one half of its mission – to collect data. It was the second phase, relaying the data to the state committee, where the app failed. This may be true, but everything else about the project indicates that the agency was outside their lane in completing the project. The most stunning revelation is that the app, as I mentioned above, wasn't even distributed via the usual means but rather via a testing service called TestFlight. This is a normal process for the development stage of an application, but never for deployment. This alone is an alarm bell for anyone who has ever developed and deployed a mobile app – it's a horrible practice.
The agency also admitted they had not adequately tested the application, and that the primary bug was completely avoidable. Given what we know, it's readily apparent that the project was not properly executed by the developers and that most likely had an adverse effect on election night.
Was The Use-Case Over-Engineered?
At first glance, it seems that the use-case which was being addressed, and the problem being solved, was a good candidate for a software solution. Many remote operators are collecting data, with a need for fast tabulation… It screams technology solution. However – was a technology solution warranted?
This will be a debate for a long time.
Personally, I feel that this is a case where a technical solution, even based on a mobile application, makes a lot of sense. However, given the risks of this particular event, and the fact that this is a one-off application – something that is used once every four years at most – it may have made sense to stay low-tech.
This debacle should remind us all that software is a tool, much like medicine. You should utilize the tool when the benefits outweigh the risks. Doctors use data to aid in making those decisions. As software professionals, we need to use a bit more common sense. Indeed, when the stakes are so high, perhaps we shouldn't revert to over-engineering a solution, but rather keeping with what is tried and true.
I'm unsure anyone watches The New Pope on HBO – but if you watched recently – you could see the "technology" they use in the conclave. It hasn't changed in a thousand years, and it works. So why are we always seeking to streamline? As another client of mine said last week – they could have made a Google Docs / Sheets mashup that got the job done. Why build some complicated application when the stakes are so high?
What Happens When Tech Fails?
The worst part of this entire incident is the various outcomes that present themselves. In this case, we have an electorate that is already distrustful of the process. This may be less so on the Democratic versus on the Republican side, but it does exist. This incident has only worked to reinforce that distrust.
Furthermore, it has created a new level of conspiracy for those who are inclined to believe in those sorts of things. The Buttigieg campaign utilized the same agency for digital work, paying them over $40,000 for another project. With Mayor Pete winning (or in a dead heat…) the caucus, conspiracy theorists are now quick to say that he fixed the results. So much of the opposition's strategy is dependent on casting doubt on any opponent (yes, I'm referencing Burisma and that mess!), and now this incident has given a new storyline to put doubt into yet another campaign. It doesn't help that Pete was adamant early that he was the winner…
Finally, all of the candidates have a bone to pick with the Democratic Party in Iowa. The next election was two weeks after, and campaigns count on a bounce from the results of the first contest. In this case, that bounce may be affected, though no one is sure how. So, candidates like Buttigieg, who put their entire campaigns into Iowa, may not realize the gains they were hoping for. However, in the inverse, it does help some of the underperformers if they can point to irregularities in the results.
The Case for Licensure
I've said this before, and I'm going to repeat it… Is this a case where it makes sense to reconsider the idea of licensing software developers? Software projects fail quite often. Usually, it's to the detriment of a private organization. But in this case, it's affecting our democratic process. Is there an argument that it's time for the software development industry to have some level of standardized licensure, especially when projects may affect the public and our institutions?
The industry is full of self-licensed "experts" and "specialists" who are practicing their craft with no real level of oversight. Perhaps it's time we focus on this problem, in addition to privacy and data concerns, and start to weed out folks who don't necessarily know what they are doing…
Wrapping Up
There are many lessons we can learn here. Hiring the right team to complete a project. Performing the proper steps and stages to complete and deploy an application. Properly testing and securing. And, of course, learning to determine if we even need a software solution in the first place.
If you are still with me, the number one takeaway should be that software is an impactful medium that needs to be taken seriously at every stage because the consequences of a poor implementation can oftentimes be devastating.
