The US government spends a lot of money on contractors. I'm talking a lot a lot. Like, enough to support an entire industry.

        According to companiesmarketcap.com, the combined market cap of the top 10 defense contractors in the United States was ~$750 billion as of the time of this writing (April 2022). The combined value of these massive companies happens to come pretty close to the US Defense budget per year which, while consistently taking half of the budget from all other agencies, has climbed to an astronomical $704 billion for 2021. According to the White House, that number is only expected to grow in the coming years.

 

Figures 1a, 1b: Figure 1a (top) shows the total United States Department of Defense discretionary budget while Figure 1b (bottom) shows this budget as a percentage of the total discretionary budget available for all United States Agencies. Data points after the dashed line represent the White House forecasts of this discretionary budget.
Source: whitehouse.gov; Table 5.4—Discretionary Budget Authority by Agency: 1976–2027

        As I began to find out in the summer of 2021 these million (and sometimes billion) dollar contracts could be quite competitive. I was 6 months into my first official data science job at Booz Allen Hamilton, a large defense contracting company, when I enrolled in a fancy employee training program called the Data Science 5K (DS5K).  One of the cornerstones of this program was the Data Science Fundamentals course with General Assembly. As part of the course all students were required to complete a capstone project where we picked a dataset/problem and explored some solutions using what we learned. Eager to add value to Big Booz, I decided to research how to gain a competitive edge when applying to government contracts.

        The details of bids for defense contracts are not made public. Companies aren't going to share exactly how they won a contract, much less how they lost one. However, every now and then a company that lost a bid may feel that the bid was unfairly awarded. If they want another shot at the contract, they can submit a bid protest with the Government Accountability Office (GAO). According to the GAO, once a bid protest has been submitted the GAO will have 100 calendar days to respond in one of the following warm, snuggly ways:

• Dismissing the protest
  • Technical errors with the protest; protest won't be considered
• Denying the protest
  • Tough love, your protest wasn't persuasive enough
• Sustaining the protest
  • Your arguments are very compelling and your competitor will be legally spanked

        In the midst of this riveting bureaucracy there is one other slightlyteensietiny important feature of this system. Regardless of whether the protest was sustained or denied the GAO will release a report about exactly why your competitor won and you didn't. Also, this information will be public.

While you might not see your competitor's exact application, these summaries often include apples to apples comparisons of your application. This often includes

• technical scores,
• what the government thinks about your company's past performance, and
• the price your competitors bid for the contract

among other interesting data. Sometimes they even organize this data into a nice neat table for your viewing pleasure. We see this in the table below, where we saw that Dataminr was able to secure the same contract at 4x the bid price as Babel because of their better technical approach and reputation.

 

 

Figure 2: Babel vs Dataminr bid protest. Here Dataminr took the contract at almost 4x the price because of their technical approach and good reputation

Source: Bid Protest B-418730.5,B-418730.6

        If your scores were virtually the same but your asking price for the work was too high, you might need to bid slightly lower. If your competitor's advantage was in only one key area then that's likely the bottleneck your team should focus on. All kinds of interesting insights could be derived from the ~30,000 reports on the docket history. What is a nightmare for the average government administrator is an opportunity for the enterprising data scientist.

        I used Scrapy to collect the HTML of every bid protest since the dawn of time. I wanted to prototype an analysis and present it in time for my capstone so I decided to focus on reports that included tables like the one depicted in Figure 1. These types of tables are nice because they are neatly organized in tree tags of the HTML structure making the information easy to group together. I could determine if an HTML page contained a table by parsing the HTML strings for <tr> tags. No fancy deeplearning transformers applied for this prototype!

Then, I began hacking away at this monstrosity of a dataset, starting with something like this:

 

 

Figure 3: Raw output of Scrapy data in a pandas dataframe, the bid_info column holds the entire HTML string of each the bid protest page

and developing into something like this

 

 

Figure 4: Cleaner output of bid protest data in a pandas dataframe. Here, the "Filer" of the bid protest is the one who did not win the contract. "Loser" felt a little harsh.

        The capstone was in 3 days and I had spent the last 11 scraping and cleaning the data. Such is the life of a GaoGetting webscraper. The cleaning funnel for this dataset was pretty brutal because I decided to be the MVP (most valuable programmer) by delivering the MVP (minimum viable product) of analyzing a dataset with more than 500 rows for the capstone. Figure 4 shows how each step in the cleaning process eventually wittled down the dataset from ~13,000 gibberish rows to 671 clean and pristine rows for analysis.

 

 

Figure 5: Cleaning Funnel for the scraped dataset

        Notice that I only chose to analyze protests that identified a winner and contained a specific bid price (denoted by a "$). I chose to only analyze denied protests because denied protests indicated a "true" winner of a contract. In other words, the government was certain that the winner of the contract had what it took to secure the contract and stuck by that decision. 

        Finally, there are many different ways to skin a cat. Descriptions of the same attribute (technical skill, past performance, etc.) had a number of different descriptors. Since the dataset was small enough, I simply mapped each unique descriptor to a numerical value between -1 and 1 to represent the sentiment of the rating. This was more of a heuristic and there are certainly more elaborate ways to apply sentiment analysis to this data, but again, tick tock tick tock.

 

 

Figure 6: Examples of different positive and negative ratings given to different bid protests

Which resulted in the final cleaned dataframe:

Figure 7: Examples of different positive and negative ratings given to different bid protests

       Now this is something we can start working with! Overall, the total money on the line in this prototype analysis was $128,954,104,688. A drop in the bucket in terms of all US defense contracts but I certainly wouldn't turn it down.

        First, I wanted to know how much of a role the bid price played in securing a contract. As we saw in Figure 2, Dataminr was able to secure a contract even though they bid 4x higher than their competitor. I performed a two-tailed null hypothesis significance test (NHST) to check whether the winner of the contract had a different average bid price than the loser of the contract. For this test, the null and alternative hypotheses were

• H₀ - There is no significant difference between the average bid price of winning and losing bids (null hypothesis)
• H_A - Winning bids a different average bid price than losing bids (alternative hypothesis)

  A NHST tests the difference between the means of two distributions. If the two distributions have a high amount of overlap then the analysis will return a high p-value and we will fail to reject the null hypothesis. If there is good reason to believe that there is a difference between the average bid price of winners and losers of government contracts, the analysis will return a low p-value and we will be able to reject the null hypothesis. How low the p-value needs to be to determine statistical significance depends on the social circle your roll with, but the lower you set this threshold the higher confidence you will have in your result.

        In this case, I chose to declare any p-value below 0.05 as statistically significant (statistically significant with 95% confidence). The NHST returned a p-value of 0.97 (meaning that the bids from winning and losing companies are really really similar) meaning that we failed to reject the null hypothesis.

 Figure 8: Results of the null hypothesis significance test with a histogram comparing contract bid distributions. When visualized we can see that the distribution for winning and losing bids are very similar to one another.

 

        In plain English, the bid price doesn't seem to be what's determining a winner of a government contract. Instead, the other factors at play really do influence whether a company wins their bid. After mapping the sentiment to each bid proposal factor, I was able to find the average difference in sentiment value between winning and losing contracts to find the discover the following minor and major differences between winning and losing contracts (displayed in Figures 9a and 9b below)

 

Figures 9a, 9b: The minor average differences between winning and losing contracts (9a, top) and major average differences between winning and losing contracts (9b, bottom). Winning contracts had higher sentiment traits for each factor, but the largest differences were in technical, management, and personnel factors.

 

        The government isn't playing around when it says it wants the right person for the job. As long as you can prove that you have a qualified team, you can rest assured that you will secure the government contract (even if it's at a slightly higher bid price than your competitor).

        Some of the primary takeaways from this analysis are listed below in Figure 10 

Figure 10: Summary of the results of this analysis

        If I were to continue with this type of analysis, I think that future work might follow the type of timeline featured in Figure 11:

 

Figure 11: Possible timeline for future work, a roadmap for transferring this analysis from prototype to a larger project.

 

        Eventually this data could serve as the input to a machine learning model that could predict whether a company would secure a government contract. Furtnermore, one might be able to predict the premium a company could add to their bid price if they were confident in their technical/management/personnel evaluation. Unfortunately I was not able to pursue this prototype as the contract I was placed on had other priorities. However as far as I am aware there is still a golden opportunity for defense contractors to analyze the reports with more refined cleaning methods, improve sentiment analysis of data, and implement a machine learning model to create a more comprehensive and directly applicable machine learning product. Of course, this will be an exercise left to the reader. :)

 

All data collected has been posted to my github here. 

Source I used for stylizing my presentation and images for this blog post here.