The Matchability Score quantifies award potential between contractors and agencies in the federal marketplace. But just how well does it do this? Is it truly successful in being able to capture such an incredibly valuable assessment? And can we translate this into actionable predictive insights?

In order to answer these pressing questions, we conducted several rigorous, intertemporal validation experiments. In this post, we show one of such experiments, using 2016 award data from the Department of Defense to generate a scoring model using the Matchability Score Algorithm.

Out of the 3,293,696 active contracts of the year 2017, a total of 2,266,558 were newly awarded contracts. These contracts were awarded across 40,127 contractors, 21,146 of which appeared in 2017 Department of Defense award data and not in the 2016 award data. Our testing data consisted of this subset of federal contractors. This ensured that the information used to build the model was separate from the data used to test it, eliminating any possibility of deriving unsound conclusions from this experiment.

Using this model, we obtained Matchability Scores for all contractors in the testing set with respect to all agencies of the Department of Defense.

Then, we scanned 2017 Department of Defense award data for the contracting activity of the testing set. For each contractor in the testing set, each awarded contract was obtained, along with the awarding agency and the corresponding 2016 Matchability Score. Thus, we obtained a distribution of the Matchability Scores of all contractors in the 2016 testing set across all Department of Defense agencies with whom they contracted in 2017.

One of the key statistical properties of the Matchability Score is that it is uniformly distributed for the complete population of the pairwise combinations of contractors and federal entities (Departments, their constituent Agencies – and indeed, the Federal Government at large as well). This complete population of awarder-awardee pairs also includes pairs for which no contract activity has taken place. However, such a pair will still have a Matchability Score that quantifies the award potential that exists between them. For example, a company that never contracted with the Defense Logistics Agency will still have a Matchability Score for it. In this way, even in the absence of contracting activity, the Matchability Score provides highly informative market discovery capabilities.

The Matchability Scores are distributed uniformly at a global scale. If we were to find only an insignificant difference between the distribution of the Matchability Scores for each agency and a comparable uniform one, then we would be forced to conclude that the Matchability Score is a meaningless metric. We generated a random sample from the uniform distribution of the same size and range as the testing set of contractors to represent this null distribution.

However, we find that the distribution of the Matchability Scores from the testing set with respect to specifically only those agencies of the Department of Defense that awarded contracts to them is significantly different from the comparable uniform distribution.

The significance of these results is made clearer by its cumulative frequencies.

Contractors from the testing set who scored between 70 and 100, a subset which amounts to 30% of the population of contractors as a whole, were the recipients of nearly 60% of the contracts awarded in 2017 by the various agencies of the Department of Defense.

As a result, contractors with higher Matchability Scores with respect to a particular federal agency are overrepresented among that agency’s pool of awardees. Conversely, contractors with lower Matchability Scores with respect to a particular federal agency are underrepresented among that agency’s pool of awardees.

In conclusion, the Matchability Score is indeed meaningful, insightful, and possesses immense predictive power for the successful capture of federal business opportunities.