But many fans felt Olsen, who has won an Emmy for his work since joining the network in 2021, was hard done by.
Their case was strengthened when Brady struggled to find the right tone early this season, although he seems to be hitting his stride.
Olsen has been paired with Joe Davis in the No. 2 team but was involved in another reshuffle in Week 5.
Davis is Fox's play-by-play broadcaster for the MLB playoffs and called the New York Mets vs Philadelphia Phillies series.
Versatile announcer Jason Benetti joined Olsen in the booth for the Cleveland Browns' loss to the Washington Commanders on Sunday.
The duo had plenty of chemistry and fans think they should be promoted ahead of Brady and Burkhardt.
"Why aren’t @jasonbenetti and @gregolsen88 the primary booth? What a duo," said one fan.
"Jason Benetti and Greg Olsen sound like they've been doing NFL games together for 10 years. Impressive level of chemistry for a temporary outfit," added another.
"Jason Benetti-Greg Olsen broadcast booth as A team as it gets for a non A team duo," said a third.
Benetti covers a wide range of sports for Fox, including baseball, basketball, and college football.
We need Benetti on more NFL games," begged another viewer.
"Even though it kind of sucks that Olsen was 'demoted,' he's working with same amazing announcers this year," suggested a fifth.
In one memorable exchange, Benetti showed familiarity with his new partner.
"You going for this?" Benetti said as the Commanders went for it on fourth-and-two.
"I am. I'm aggressive by nature. Jason, I know we haven't worked a lot together," Olsen replied.
"I can tell. It comes through the TV," Benetti quipped.
Another standout feature was a striking height difference.
Olsen stands at 6-foot-5, while Benetti is 5-foot-11.
Thankfully opposites attract.
While fans enjoyed the new pair, Fox did not spend over $37million a year on Brady to use him as a backup.
Olsen, meanwhile, is reported to have seen his annual salary cut from $10million to $3million.