After being turned down twice by a trustee he handpicked, Tom Benson on Wednesday filed suit in federal court asking a judge to allow him to extract his professional sports teams from trusts set aside for his estranged daughter and her children.

The lawsuit comes after Robert A. “Bobby” Rosenthal — the lawyer overseeing trusts benefiting Renee Benson and her children, Rita LeBlanc and Ryan LeBlanc — recently rejected an effort by Benson to remove ownership shares in the Saints, Pelicans and other assets from the trusts in exchange for $449 million in secured promissory notes and the cancellation of $94.5 million in debt.

The 87-year-old Benson announced earlier this year that upon his death, he wants his third wife, Gayle Benson, to inherit the reins of his business empire. He said he no longer wants either Renee Benson or her children to have any more role in running his business conglomerate, which Forbes estimates is worth almost $2 billion, before or after his death. He claimed that he reached that decision after years of worrying about their professional competence, something the relatives dispute.

However, it will be difficult for Benson to carry out his wishes if he doesn’t succeed in removing the various ownership shares in his businesses from his relatives’ trusts.

Rosenthal — whose duty as trustee is to watch out for the best interests of Renee and her children — said he could not approve any deal until Benson provided updated valuations for the teams and other assets. He also rejected an earlier proposal by Benson to pull the teams out of the trusts in exchange for unsecured promissory notes worth about $427 million.

Rosenthal is an attorney at the San Antonio law firm of Stanley Rosenberg, a longtime Benson legal adviser.

Benson’s lawyers have repeatedly said he is the only person who possesses controlling, or voting, shares of the teams, and that those assets do not reside in his spurned relatives’ trusts. The documents also say Rosenthal’s approval is not required for a swap of assets that are equal in value.

The latest lawsuit, filed Wednesday in U.S. District Court in New Orleans, alleges that Rosenthal had previously performed transactions as trustee without questioning the estimated value of the assets involved. It says that Empire Valuation Consultants produced those estimates.

According to the suit, Benson hired Empire in January to update its valuations of the assets in the relatives’ trusts.

Benson’s lawyers previously said the valuations were ongoing and declined to identify who was responsible for them.

Rosenthal said he reserves the right to have separate valuations done by a firm other than Empire. Meanwhile, Benson’s lawsuit claims that Rosenthal had not previously objected to Empire’s valuations in the course of his business as trustee.

Empire has offices in Manhattan’s Empire State Building; Rochester, New York; Boston; Cleveland; and West Hartford, Connecticut.

At least seven trusts were created in 2009 and 2012 in an effort to protect Renee Benson and her children from having to pay estate taxes on their inheritances. Rosenthal is the trustee of those trusts. He may challenge the venue of the latest suit, given that he is in Texas, where the trusts were set up.

The trusts are irrevocable, meaning that Benson cannot add or remove beneficiaries but is allowed to remove assets from the trust, provided he replaces them with other assets of equal value.

The most recent proposed asset swap offered the Saints and Pelicans themselves — as well as holding companies for properties such as Champions Square, Benson Tower and five car dealerships — as collateral for the promissory notes. Benson also said he would forgive loans he made to the trusts in recent years, and he included a 50 percent stake in a couple thousand acres he owns in Texas, plus full ownership of a house on the property.

Part of Renee Benson’s response to being cut off by her father was to ask the probate court in Bexar County, Texas, to suspend her father as overseer of a trust set up by her late mother and instead to give that function to her. The judge suspended Tom Benson but did not transfer oversight of that trust to Renee Benson.

That trust is separate from the ones involved in the federal lawsuit in New Orleans.

Benson is appealing his suspension as overseer of the trust established by his first wife, Shirley, who died in 1980.

Separately, Renee and her children filed a lawsuit in New Orleans Civil District Court aiming to have Benson declared mentally unfit to make business decisions. Judge Kern Reese has directed three physicians to conduct a mental evaluation of the billionaire. They are expected to report their findings this month.