How NFL Teams Scout The Refs

Ramon George is lined up 15 yards behind the line of scrimmage, a little to the left of Dak Prescott. As the umpire, George’s job is to watch the center, left guard, and left tackle for false starts, and after a running play he’s the one who will spot the ball. But neither George nor the Niners defense are expecting a run here. The Cowboys are down 23-17 on the Niners’ 41 with 14 seconds left in the game. San Francisco defenders crowd the sidelines. They aren’t planning on letting Dallas get out of bounds to stop the clock. 

But Dallas does run. With only three Niners left in the box, Prescott takes a wide-open lane and runs 17 yards, sliding to the ground with nine seconds left. The Cowboys’ offensive linemen hurriedly stumble to their positions and Prescott and his center set the ball on the ground, seemingly forgetting that the next play cannot start until an official touches the ball. 

Four seconds left. George comes sprinting in from the backfield and he squeezes his way—just barely—through the A gap between the center and the right guard. George is blocked by the big linemen and he uses both hands to push the center out of his way so he can touch the football and place it on the ground for the official spot. He backpedals into the defense as the clock runs out.

Niners defenders stand up and point at Prescott, realizing they’ve just won the wild card game. 

“Oh my gosh!” Tony Romo shouts on the CBS broadcast. Jim Nantz yells, “The official gets in the way and the game is over! The game is over!” 

“The umpire has to touch the ball, of course that’s ridiculous for a game to end like that, Jim!” Romo says. 

I couldn’t stop thinking about this play. I watched George elbow his way through the offensive line over and over again. He’s running so fast until he thuds against a wall of giant humans who will not let him through. This was a situation where an official impacted the game without even making a decision that anyone could argue with. Was the play-call to blame? Or the Cowboys players? And how much did George’s actions matter here? I couldn’t let it go, so the next week, I called an NFL game management coach to help me understand what went so wrong. 

This coach was shocked by the play-call. He explained that their team wouldn’t have run that play without more time on the clock, because however long they think they have, everything goes faster and smoother in practice, where equipment staffers who know what the play is are standing in for the officials. I expected that. But what he said next surprised me. 

“The crazy thing is that Ramon George is the fittest umpire in the NFL,” this coach said. “He’s in the best shape of any of them. So if he couldn’t keep up with this play, who could have?”

Wait a second, I interrupted. How do you know he’s the fittest? 

“I know who trains each official,” the coach told me. “I know where they went to school, I know what kind of shape they are in. I know everything about them.” 

This coach watches every play of every game to see how the officials move and react. He tracks how long crews take between plays to know how to time out end-of-game scenarios, and they give a scouting report to players on each official’s athletic capabilities. If the official is older, they’ll instruct players to hand them the ball to make it quicker and easier for them to spot the ball. And in the case of this Dallas play, the coach said they could tell that the Cowboys staff didn’t give George a heads-up on what they’d be doing on the play, or George would have been following more closely behind it. 

I knew teams scouted their opponents with obsessive levels of detail, but I didn’t know they scouted the officiating crews just as closely. 

“The Dallas play is exactly why teams scout officials,” this coach told me. “This is exactly why.”

If you ask any data or analytics staffer or game management coach about how they scout the officials, they are all eager to discuss this growing field, and vent about the lack of information available and the inconsistencies among crews, but they’ll only do so anonymously. Every request I made to speak to an NFL team employee was denied by that club’s PR. It seems like every team is expanding in this area of research, but nobody wants their opponents to know exactly what work they’re doing.

“The guy you talked to is absolutely right,” says Dean Blandino, who was the NFL’s VP of officiating from 2013 to 2017. “Ramon George is the most athletic of all the umpires. … They are tracking everything, they are tracking the crews, the calls, whether it’s this crew, this official, this umpire calls holding at a rate twice as much as the average of the other officials, so we know that we have to be more in tune with our offensive lineman, and the close line play and blocking because this umpire calls more holding fouls.” 

When Blandino first started working for the league office in the late ‘90s, he says that the extent of what teams knew about officials were their names and a few facts about their personal life, like their spouse’s name or their job outside of football, simply for the purpose of buttering them up.

“Walt Coleman is like an eighth-generation dairy farmer. Something crazy like that,” says Bears backup quarterback Trevor Siemian, easily pulling this fun fact from his mental archive. (Retired from the NFL in 2018, Coleman is actually a fifth-generation dairy farmer. Close enough.)

Coaches still teach their players those little biographical details, but there’s so much more to the education on officiating now. Later on in his role leading the league’s officiating, Blandino started interacting more with a new type of team employee—game management coaches and analytics staffers.

“The people that are in this role, the people I was dealing with, that was part of their job to get in the rulebook and understand the technical minutiae,” Blandino says. “Most coaches will ask me a question and off the top of my head I can answer it pretty quickly, but some of these questions would come in and I would have to think about it, because they are looking at it like, how do they gain an advantage?” 

Josh McCown played 19 NFL seasons, and he experienced this shift as a quarterback. During the first half of his career, he says coaches talked in general terms about a crew’s reputation. “It was more, they aren’t very good or they are good. There was no detail to it.”

But in 2015 in Cleveland, he remembers then–Browns head coach Mike Pettine standing up in front of the team and presenting a chart of every type of penalty and where that crew ranked amongst the 17 officiating crews for each call. McCown says Pettine’s presentation soon became the norm, and in the last few years of his career, coaches started showing clips of specific penalty calls when they presented the crew stats to players. 

“This crew likes to call this, this crew likes to call that, so let's look at some clips of that, see what they perceive,” McCown says. “It changed dramatically from 2002 until 2020, I promise you, and I would say it was definitely more of a factor in the game plan.”

A second game management coach says that 10 years ago, only a few teams in the NFL had someone tasked with watching every coaching challenge from each week, and tracking the result of each challenge. Now, every team does it. “If you don't, you're putting yourself at a real disadvantage,” he says.

Gene Steratore, a former NFL referee who now serves as CBS’s rules analyst, remembers former Steelers head coach Bill Cowher would ask him the same question before every game he officiated: “Geno, are you going to call the hand checks today?” But the first time Steratore can remember a coach approaching him with specific data was when then–Vikings head coach Mike Tice came up to him before a game. “He told me, ‘You call the least amount of holds of any crew in the NFL. Which I didn’t know. But it makes sense [that he’d know]. Why wouldn’t you do that if you were a team? … As we know in the NFL, [teams] are that meticulous and go that far down the rabbit hole with anything they can possibly gather to gain an advantage.”

Every year at the NFL scouting combine in Indianapolis, the analytics company Pro Football Focus meets with their clients and prospective clients to hear feedback on their product. And over the last five or six years, PFF analyst Steve Palazzolo says teams kept bringing up officiating. Specifically, they wanted PFF to track penalties by the individual official who threw the flag. The league only provides teams with data on penalties by crew, which isn’t granular enough for NFL coaches, who want to know which side or field judges call the most PI, or which line judge calls the most false starts. (The answer? “Jeff Bergman,” says a third game management coach. “If you have him, he calls it tight.”)

“Almost every team said, Hey, yeah, we do this on our own, if you guys could do this, it would be really helpful,” Palazzolo says. “There was need, and demand was high.” 

So this year, for the first time, PFF has added this stat to its NFL product. Teams can search by official, by penalty, or by position, like back judge or side judge. And within that database, each penalty is linked to the film. 

“The refs aren’t too happy about it,” Palazzolo says. 

That’s because every official has their own specific reputation among NFL teams, and it’s not just fueled by personal vendettas or outlier performances. It’s in the numbers; an NFL team shared with me its penalty data, which you can view here.

Here are two examples from opposite ends of the reputational spectrum:

Referee Shawn Hochuli

• “Just talks a lot, he over-officiates the game,” said the second game management coach. 
• “He’s known for wanting to interject in the game, he has loquacious explanations on the mic,” says the third game management coach. 
• “There’s a tendency there, and it’s genetic,” says a former NFL head coach. (Hochuli’s father Ed, who retired after the 2018 season, had the same reputation.)
• The data: Ranked first in penalties per game in two of the last four seasons (‘18 and ‘19). Hochuli’s 20.88 penalties per game in 2018 is the highest season average by any referee in the last four seasons. He also has the second-highest average over the four-year period for anyone to lead a crew in any season during that time (16.83 per game, trailing Walt Anderson, who is now the league's senior VP of officiating) and that number is the highest average among crews active in all four seasons in the period.

Referee Bill Vinovich

• “Vinovich calls much less, in general,” says the second game management coach. “Vinovich's crew is very light on penalties, which people like more than the game being over-officiated.”
• “Least flags, least penalties, fast-paced game,” says the third game management coach. “He's always in primetime.”
• The data: Last (17th) in total penalty count for 2021, with 167 total and an average of 11.13 per game, about five penalties per game fewer than Hochuli’s crew. He’s ranked last in penalties per game in each of the last four seasons, and his average (12.42) in that period is also last among the 23 different referees who have led a crew in at least one of the last four seasons, a full penalty lower than the next-closest referee, Brad Rogers (13.41).

Teams scout even more granularly when it comes to specific calls that tend to be more subjective decisions, like personal fouls and roughing the passer; these are easier to coach a team to avoid. “If you need your team to turn up its discipline level for a certain game because you have a guy that is going to call those a lot,” the second game management coach says, “then that is in your control.”

Take Land Clark and John Hussey. In his two seasons as a referee, Clark has led the league in roughing-the-passer calls. In the last four seasons, Hussey ranked in the top two crews for unsportsmanlike conduct calls three times. He’s also finished in the top three in taunting calls in each of the last four years. If you get either of these guys calling your game, you need to stay clean.

Holding and pass interference are also a focus for NFL coaching staffs. If a crew skews toward one extreme or the other, coaches can prepare their players with how much or how little they can get away with. For example, Jerome Boger’s crew called the fewest offensive holding penalties in 2021 with 1.86 per game. On the other side, Brad Rogers’s team called a whopping 4.13 per game. 

“If they call it a lot, you are probably going to see it a lot,” says Browns guard Joel Bitonio. “I think it is a reminder during the week like, all right, these refs are going to look for holding, let's keep our hands inside, let's be a little more fundamentally sound.” 

Steratore says the league aims for consistency, but each referee has his own philosophy of how he will approach the game. Steratore’s own motto: Make all the fouls big fouls. “We fish for whales and not for minnows,” Steratore says he told his crew. “When you get a little technical, then almost subconsciously, I am looking for a potential foul. That is how I teach young officials: You never go to a play looking for a foul.”

A referee’s philosophy dictates the behavior of their crew, so even though every official is graded each week by the officiating office, crew tendencies become a real thing. But this can have non-obvious effects. A former NFL head coach says one of the most important things his staff learned when they started scouting officials is that some crews are aware of their numbers and rankings. “They are what we call a ‘self-scout crew’ where they are going to go in and they are going to see what they've been calling and then try to go the other way to balance it out,” the head coach says. 

I spoke to four current players with three different teams for this story, and all four said they’d been coached on officiating stats for their whole careers, which span from five to nine seasons of NFL experience. But it can be a challenge for players to recall useful information in the heat of the moment. “I always thought it was funny because the game is so fast, that it's tough as a player to take those kinds of things into account,” says Siemian. 

“Once you get on the battlefield, it goes so fast, so you aren't thinking about, like, Wait, this crew has the ninth-most holding calls? I can get away with something here,” says Bitonio.

In his role now as a rule analyst for Fox Sports, Blandino is invited to visit several training camps each summer and meet with players to answer their questions on officiating. So even if players aren’t necessarily taking the crew rankings into account while they are getting set at the line of scrimmage, Blandino says he’s noticed that players are more educated about the rules of the game, the specific roles of each official, and crews’ general tendencies—a sign that coaches are incorporating the information into their regular meetings.

“There is the fine line of paralysis by analysis,” says McCown. “You don’t want to be there going, All right, who is the line judge right now? All right, well let me throw to his side, and it’s not the best look. You have to be careful of that stuff. But at the same time, I do think there is a level of detail that it communicates to your team that says we are looking for every edge we can get to get better. The takeaway is you should be high on belief and trust in your coaches and what you are doing because they are detailed and take those nuggets.” 

“Ten-second run-off!” Mike Vrabel furiously pointed his index finger and rolled his hand to signify a running clock. “Ten-second run-off!” 

The Titans head coach paced the sideline yelling. There was 1:25 left in this home game against the Saints. His team had an eight-point lead, but the Saints were deep in Tennessee territory. Vrabel has just noticed the officials missed a game clock enforcement that would aid him in his mission to run out the clock: Taysom Hill and Mark Ingram didn’t get set on the line of scrimmage before the play, and the Saints offense was called for an illegal shift, two men moving at the snap. 

According to the rulebook (Rule 7.4.2, Item 6), with the game clock running after the two-minute warning, this should have been enforced with a 10-second run-off if the other team chooses. But when referee Jerome Boger turned on his mic to announce the five-yard penalty and repeat of down, he didn’t announce a run-off. The clock stayed where it paused when the play was whistled dead. 

Two officials on Boger’s crew walked over to Vrabel on the sideline. “What?” Vrabel yelled, his eyes squinting in confusion as they talked to him. He knew he was right, but nobody was listening. 

The Saints would score but fail the two-point conversion, and the Titans held on to win. The CBS broadcast didn’t catch the mistake, and Vrabel wasn't asked about it after the game. The officiating mishap didn’t alter the game this time around, but situations like this happen every week in the NFL. Sometimes, a coach will quote the rulebook to an official in these arguments, and other times they’ll say more generally, Are you sure? My guy upstairs says otherwise.

“The best game management teams check the officials from the box,” the second game management coach says. “Probably two to three times a game, I’ll see if [my head coach] wants to go to this official and ask why this was done this way or ask, did he see this? Why was it enforced this way? The rules are too complicated so the enforcement is inconsistent.” 

The 2022 rulebook, which also includes scenarios and replay rules, is 245 pages long. Then there’s the philosophy document, another 150 pages or so, an informal text that explains the reasoning behind when a foul should be called and when it shouldn’t. (Example: An official shouldn’t call holding that occurs on the opposite side of where a running play went, because it didn’t affect the run.) There are also mechanics manuals for each officiating position and the Hopperbook, a manual created by Ed Hochuli that sorts penalties into 15 metaphorical buckets to help officials with penalty enforcement procedure. “You have 500 pages of data that officials use to prepare for each season and each game,” notes John Parry, former NFL referee and currently ESPN’s officiating analyst. 

“I'm willing to bet most people in all of humanity have not read the whole rulebook,” the former NFL head coach says. 

“Very few people on our team have actually read it,” the second game management coach says. “The rulebook is dense. It’s not a very coach- or player-friendly document.” Their job, put simply, is to study the rulebook like it’s a legal text, so their head coach doesn’t have to.

As I was reporting this story, Vrabel and the Titans came up again and again as an example of a team that plays situational football well and exploits the rarity of certain penalties, like forcing opponents into false starts by having a defensive player bark out the offense’s cadence without getting caught (disconcerting signals), or lying on an offensive opponent for a teeny bit longer after a tackle but not too long (defensive delay of game). 

Vrabel is on the league’s competition committee, so he’s well-versed in the rulebook’s minutiae, and he’s built a staff that specializes in it. Several people in the field interviewed for this story mentioned Vrabel’s right-hand man John Streicher as a prime example of a game management coach who has made their players and coaching staff more knowledgeable in this area of the game. (The Titans PR staff denied my request to speak with the duo for this story.) 

Streicher’s title is football development coordinator, an arrangement of three words that tells us next to nothing about what he actually does. There is no league-standard title for these types of coaches, whose job is to advise the head coach during the game on specific scenarios, like challenges, clock management, and rulebook esoterica, and to coach the players to be aware of the rules and officiating as well. Some of these coaches don’t like the term “game management” because they think it’s insulting to the head coach.

Some are slotted under football operations and go by data-centric titles like Director of Football and Research Strategy or Statistical Analysis Coordinator, while others are listed on the coaching staff under names like Offensive Assistant/Game Manager, or Quality Control/Analytics. Sometimes the job is split among coaches on either side of the ball. 

Some people in these roles are quality control coaches (one of the lowest on the totem pole NFL coaching jobs), some work primarily in the analytics department, and some are executives in the front office, like former GM Brian Xanders, who does replay and rules for the Rams. There are even a few, like Philadelphia's Jon Ferrari and Chicago’s Harrison Freid, who were hired by clubs directly from the league office specifically for their expertise in officiating and replay rules. 

At least two teams looking for an inside track on the officiating mindset have even hired former zebras: The Bucs hired Larry Rose for the 2019 season, and the Raiders had Gerry Austin as a consultant from 2018 to 2020. 

The job is a time-consuming one. Some coaches in this role spend a couple days each week watching every play of every game—not just their upcoming opponent—and noting each flag, each crew’s administration of the game (are they correct? Are they slow or fast in making decisions?), and any reviews and challenges. They’ll then make a list of any potential issues with their upcoming opponent—do they get away with things that should be a penalty? Then they’ll alert their players and coaches of any relevant tendencies, and even mention it to the officiating office to give them the heads-up of what to watch for in the game. 

“When we played an AFC division rival that ran a lot of screens, their linemen would be downfield prematurely a lot,” says the third game management coach. “We warned the crew and got it called in the game.” 

Every coaching staff can submit up to 10 questions to the officiating office after every game. The league responds to each question, sometimes admitting errors or explaining why a certain decision was made. Not that it’ll put the horses back in the barn: In reporting this story, I heard of several successful NFL head coaches who don’t even bother to submit inquiries because they’re sick of finding out what they already knew. They—or their guy upstairs—were right.

Other teams value the feedback from the complaints they submit to the NFL, because it's another datapoint and because it’s the closest they will get to ever finding out which calls the league considers correct or incorrect. “In effect, they are comping for mistakes, confirming mistakes every game,” says an analytics staffer for an NFL team. 

“There are two ways to look at it,” says a second former NFL head coach. “You can get them to say, yeah they missed that call, and a lot of times they do, but really I think, you are better off asking about plays that didn’t get called. Or, you called a foul on us in this situation, but here was that same chance for you to call that foul against that team and it didn't get called. This looks more egregious than ours, can you explain? Stuff like that. We always asked as many as you could.” 

It’s useful, but it’s a small sample size. In an ideal world, an analytics staffer would have access to every team’s complaints and response from the league office. That’s only happened once, in 2019, when the Competition Committee passed a resolution: “For one year only, Clubs will receive the League’s post-game responses to officiating inquiries submitted by any club.” 

But after ‘19, it wasn’t brought up for another vote to extend it beyond that year, and the league went right back to the norm: no transparency. 

“It’s one side of the coin vs. 32 sides, which is what we get for most of the other information,” says an NFL analytics staffer. “I don't have to guess on injuries. I know who was injured and on the injury report. But I do have to guess whether or not the NFL said [a given call] was a penalty or not after the fact. I only know that for my team, I don't know that for others.” 

Like NFL players, NFL officials also wear a tracking device that records their movement on the field during games. But unlike with players, the officials' data isn’t shared with teams. The analytics staffer said they’d love to have the stats on each official, to see how much they run each game and how fast. 

“If you had a profile for an official,” the analytics staffer says, “like, this guy typically moves this amount and now he has run a thousand yards over that, then I'm gonna call a timeout [in a late-game situation] and not wait for him to run up to the line of scrimmage.”

When Dean Blandino thinks back to the first coach he noticed pioneering the area of officiating analytics, one name comes to mind: Mike McCarthy. 

“In Green Bay, they really started diving into the officials and putting together reports,” he says. “My first exposure to a team that put together a comprehensive review.”

McCarthy, as you probably know, was also the head coach presiding over the Cowboys’ 14-second clock-spot-snap snafu. Dallas also led the league last season with 127 penalties. Which is perhaps a good reminder that scouting the officials is just like any other of the hundreds of tactics in a team’s toolbox, with which they try to gain the slightest of edges: It’s only as useful as what you do with it.