In his first two seasons as a defensive assistant with the New Orleans Saints, there were no indications — not outwardly, anyway — that Dennis Allen was ready to begin a rapid climb up the NFL coaching ladder.

Internally, coaches he worked for and with — including Sean Payton, assistant head coach Joe Vitt and former defensive coordinator Gregg Williams — believed Allen was on the fast track to becoming a defensive coordinator.

What they didn’t know at the time was Allen, who joined the Saints in 2006 as a part of Payton’s original staff when he was just 33, was on a faster track to becoming a head coach.

At 39, however, Allen this spring made the leap to head coach of the Oakland Raiders after spending just one season as the Denver Broncos’ defensive coordinator following his stint with the Saints from 2006-10.

“You could tell he’s very talented and was on the fast track to becoming a coordinator and a head coach,” Drew Brees said. “Obviously, it happened very quickly. It’s well-deserved because I think we all saw that.

“His DBs loved him as a position coach.”

While his old coaching associates and players he was around with the Saints are happy for him, they’ll put those memories aside for Sunday when they meet Allen, the Saints’ assistant line coach from 2006-07 and secondary coach from 2008-10, and the Raiders.

One of the best memories they all have is of Allen helping them win Super Bowl XLIV, a victory clinched by one of his players when cornerback Tracy Porter returned an interception 74 yards for a touchdown with just more than three minutes to play.

Little did anyone know that night a little more than two years later, Allen would be where he is today.

“Anybody that’s worked with DA or anybody who’s played for DA in ours or Denver’s system isn’t surprised at all,” Vitt said. “He’s smart, he has great people skills, is detail-oriented (and) truly understands the game.”

Even Allen admits his rise to head coach was surprisingly fast — especially when you consider that Vitt, a football “lifer” who’s been in the league for 34 years, has only served as a head coach on an interim basis twice.

“I try not to think about it all that much,” said Allen, 40. “I’ve always believed that if you work hard at what you are doing and try to do the best job you can, then eventually certain opportunities come along.”

Naturally, Allen credits Payton with helping him along the way considering he was only in the NFL a couple of years with the Atlanta Falcons when Payton picked him to join his staff.

“I don’t know if there’s just one thing you look at and say, ‘That’s what it was,’” he said when asked what he took from Payton. “Just Sean’s strong belief in what wins and loses football games.

“Also, the fact that he was able to just stand strong on what those beliefs were and he wasn’t afraid to make tough decisions. That’s what you have to do as a head coach.”