Experts question the credibility of “Demographic Disparity Analysis - Duluth Police Department” by Police Strategies LLC

On March 30th, 2022, the Duluth Police Department (DPD) held a presentation in which Bob Scales, CEO of Police Strategies LLC, presented findings from his 172 page “Demographic Disparity Analysis.” Police Strategies LLC performed the analysis in response to data showing racial disparities in Duluth policing that LEAN Duluth released. The DPD did not inform LEAN about the study prior to the March 23, 2022 public invitation.

Following Mr. Scales’s presentation, the LEAN team has reviewed the report and contacted policing experts to verify its findings and methodologies. In this research, LEAN uncovered significant concerns about the credibility of Police Strategies's data methodologies that, in the opinion of LEAN Duluth, raise significant ethical concerns. Mr. Scales has  no academic degree in criminology or statistics and has a track record of using non-academically reviewed methodologies to minimize the perception of racial disparities.

This report shares key takeaways from Police Strategies’s report, provides an academic analysis on Police Strategies’s methodologies, and fills in background information regarding Mr. Scales’s career as a prosecutor and reputation amongst criminologists. Quantitative data has the power to both inform and mislead. Thus, LEAN seeks to ensure that communities know the risks of hiring groups like Police Strategies in the future. LEAN also acknowledges the DPD’s interest in addressing racial disparities in their practices by using the best possible data as a benchmark. We encourage the DPD to use academically reviewed methodologies and trusted third parties in order to ensure transparent results that enable meaningful communication with the community. 

Key takeaways from “Demographic Disparity Analysis” report and presentation

• Chief Tusken acknowledges racial disparities within Duluth Policing:

• “This information is not a revelation. The systems in place, and those in power, know and understand the problem. And yet, solutions are few, painfully slow and often inadequate, and leave so many behind in the tumult of the wake. The information you will hear presented tonight will not solve systemic racism. But it does help us understand the data driving disparities and as a stepping off point to begin greater community conversations.”

• Racial disparities exist in the reporting of crimes, crime victimization, and use of force:

• “Black and Native American subjects were more than nine times more likely to be identified as a suspect in a reported crime than we would expect based on their percentage of the population”
• “Black and Native American subjects were five times more likely to report being the victim of a crime than we would expect based upon their share of the population of Duluth” 
• “Black suspects are about 70% more likely to have force used against them than would be expected based on the number of arrests, while Native Americans are 40% more likely”

Academics question Police Strategies  methodologies

A large aspect of the “Demographic Disparity Analysis” report and presentation was a critique of LEAN’s use of population benchmarking. Population benchmarking uses census data to compare the racial make-up of a larger population to the racial make-up of people impacted by a given policy. For example, LEAN used population benchmarking in its findings that in 2019, people of color made up 50% of the DPD’s Use of Force Incidents despite only making up 10% of the population. 

As an alternative to population benchmarking, Police Strategies promotes using activity based benchmarking. This method compares the racial make-up of people involved in certain law enforcement activities to the racial make-up of people involved in other law enforcement activities to check for disparities, as opposed to using the general population for a benchmark. For example, in Duluth’s “Demographic Disparity Analysis,” Police Strategies compares the racial make-up of arrests to the racial make-up of reported offenders to determine disparities in arrests. 

While activity based benchmarking seeks to pinpoint disparities where they may exist, research from experts in the field does not align with Police Strategies’s critique of population benchmarking. Population benchmarks are helpful as a general rule and are a widespread practice as a first metric. Dr. Frank Baumgartner, a criminal justice expert, writes about how police departments have used population benchmarks successfully to look at racial disparities in traffic stops. In fact, in terms of traffic stops, population benchmarking is often an understatement of disparity because it does not consider who drives and owns cars.

Police Strategies’s alternative methodology, activity based benchmarking, does not appear to be used elsewhere in the field, is not a peer-reviewed methodology, and has a number of potential flaws and areas of concern. The closest parallel methodology appears to be the Risk-Adjusted Disparity Index, a non-population-based benchmark developed by Dr. Lawrence Sherman, an expert criminologist. This methodology compares police activity to crime victimization, using crime victimization as an estimate for the population “at-risk” for violent crime. Dr. Sherman talks about this measure as a second step in understanding whether there is equity in police activity. This appears to be a credible, academically reviewed, and useful second step in understanding disparities in policing. This is not the methodology that Police Strategies is using.

The most troubling aspect of Police Strategies’s methodology is that they inappropriately use crime reports as a denominator. This methodology masks potential police bias because it compares racial disparities in police activity to a benchmark that also includes racial disparities. For example, the report uses the racial make-up for reported crime suspects as the denominator when assessing whether or not there is disparity in arrests. Considering the finding that there was significant disparity in who was being reported as crime suspects, this masks any potential disparity in arrests considering that it compares arrests to an already disparate baseline. In other words, this methodology essentially accepts bias as the status quo rather than illuminating bias as a problem that the department must confront head on.

Another troubling aspect of Police Strategies’s methodology involves the discretion matrix on page 96 of the report. This type of matrix does not follow with research from the field on the types of police actions that involve high and low discretion. In fact, it suggests that all stops that end in arrest are low discretion, regardless of whether the stop began as a traffic stop or a warrant stop. Traffic stops are widely considered to be one of the most discretionary police actions. A traffic stop does not become a low discretion police action simply because the outcome was an arrest. The discretion matrix developed by Bob Scales does not follow with evidence from the field and generates a false narrative that officers are not engaging in racially biased policing. 

Police Strategies has written similar reports around the country

Further research into Police Strategies and its members illuminates their framework of analysis. The two directors of the company, Bob Scales and Mike Sanford, do not have any academic background in statistics. Mr. Scales, a longtime prosecutor, and Mr. Stanford, a Chief of Police for multiple Washington state departments, both have lifelong careers of representing law enforcement. Mr. Scales’s 2019 analysis of the Spokane Police Department featured in the Spokane Inlander utilized many of the faulty data methodologies that were also used in Duluth. 

Local experts in Spokane were especially concerned by how Police Strategies’s methodologies both rely on and further incite racist assumptions that inequitable policing is a result of minority racial groups committing more crimes. Following the release of Police Strategies’s report in 2019, Spokane City Council President Breean Beggs asserted that through contracting Police Strategies, the department was able to find "a methodology to advance that narrative."

Furthermore, local Spokane data expert Ed Byrnes, a PhD in Social Work and professor at Eastern Washington University, comments in the 2019 article that Police Strategies’s methodologies are reminiscent of an "old dog whistle." Prior to the Spokane Police Department hiring Police Strategies, Dr. Byrnes had worked directly with the department as a community member, volunteering his professional and academic skills. The reports that he collaboratively wrote with the department repeatedly uncovered inequity in practices, until the department chose to use alternative methodologies from Police Strategies. Byrnes states in the article, “When folks start saying, ‘We looked at it this way and got this other result that people like better,’ well, the result doesn't justify the methodology. I would just caution anybody to beware of being enamored with the result and therefore justifying the methodology for it.”

LEAN Duluth contacted Dr. Byrnes following the release of the “Demographic Disparity Analysis” for Duluth. During the call, he emphasized, “There is no methodological or statistical reassurance that Bob Scale’s numbers are reliable.” He added:

“I’m really glad that we are talking because there’s a push back going on against racial justice in the criminal justice system. It gets framed as ‘the community’s data is flawed.’ But they don’t take into account that our methods are peer reviewed. Alternatively, Bob Scales is a retired prosecutor and he works with a retired Police Chief. You have to consider who is doing these reports and who they are doing them for. His methodology does not have any peer review at all. This is basically an old narrative dressed up in a veneer.”

How did Duluth end up with Police Strategies and how do we move forward?

Considering the short notice that community groups and the press received upon the release of the report, community members are concerned about whether they were properly involved. Duluth Branch NAACP President Classie Dudley stated, “Nobody from Duluth Police Department or the City collaborated with the NAACP in any way regarding this study. There was no input from our community yet we were named several times throughout the release of the report.” 

When questioned by the press at the “Demographic Disparity Analysis” presentation about whether the community was involved in hiring Police Strategies LLC, Police Chief Mike Tusken redirected, commenting, “We hired police strategies to do this work. Again, we couldn't find others that did the same or similar work. Next question.”

In response to LEAN’s questions, city administration claimed that the Request for Proposals (RFP), a process that announces a project, describes it, and solicits bids from qualified contractors, was bypassed because there were no other competitors that do comparable work to Police Strategies. In reality, there are many organizations and private consultants currently working in police accountability and reform processes. Since there are other options available, it is unclear why there was no RFP process for the $10,000 contract. 

No matter DPD’s motives in hiring Police Strategies, in his opening statements at the March 30th presentation Police Chief Tusken made clear his commitment to do better when acknowledging the presence of systemic racism in policing. Thus, LEAN encourages the city to properly respond to this data, acknowledge its faults, and commit to finding a more appropriate contractor for the upcoming DPD racial bias audit for which the city recently released an RFP. Robust community involvement in hiring the contractor can avoid subsequent disagreements over methodologies after the fact.

Reports from groups like Police Strategies are somewhat helpful in that they call for more data collection. However, they don’t provide the department with clear direction on what to do next. While the report finds some disparities, it primarily focuses on explaining that the disparities are smaller than the community previously found. Police Strategies are repeat offenders that stand in the way of a necessary reckoning with racial discrimination in the criminal justice system.

Published May 3, 2022.