The No. 2 House Republican leader said Wednesday that party leaders would “take action” against Rep. Matt Gaetz if the Justice Department formally moves against the Florida lawmaker, who is under federal investigation for alleged sex trafficking.

The remarks by Rep. Steve Scalise of Louisiana made him the latest congressional Republicans to opt against springing to the defense of the three-term lawmaker from Florida’s panhandle. Federal investigators are probing Gaetz over allegations that include sex with a minor, according to two people who spoke on condition of anonymity because they could not discuss details publicly.

Scalise said in recent years, party leaders have removed Republican lawmakers facing serious charges from their committee assignments. That was essentially a restatement of the House GOP’s internal rules, which require lawmakers to resign from committees if they are indicted for felonies with penalties of at least two years in prison.

“If something really formal happened from Justice, we would of course react and take action,” Scalise told reporters.

A frequent face on conservative television networks and an ardent ally of former President Donald Trump, Gaetz has broadly denied the accusations and not been charged with any crimes. He serves on the Armed Services and Judiciary committees, and critics have said he should immediately be removed from the Judiciary panel because it oversees the Justice Department.

Scalise did not directly answer questions about whether he has confidence in Gaetz. Scalise said he’s not discussed the allegations with Gaetz but expected to meet with him this week.

“It’s serious things alleged. Obviously we want to get the facts,” Scalise said.

Gaetz did not attend a weekly closed-door meeting of House Republicans on Wednesday, Scalise said. But he did appear at Judiciary and Armed Services committee hearings and appeared relaxed.

At Judiciary, several conservative lawmakers chatted amiably with him, including Reps. Jim Jordan, R-Ohio, the ranking member, and Andy Biggs, R-Ariz., who leads the hard-right House Freedom Caucus.

Asked for comment Tuesday after returning to Washington for a vote after Congress’ two-week Easter recess, Gaetz referred reporters to a column he wrote last week saying he had not paid for sex and accusing his critics of being corrupt.

Florida’s two Republican senators have also steered clear of voicing support for Gaetz. Most Republicans have taken neutral stances or said nothing about the three-term House lawmaker.

“Sure, and I don’t think anyone’s saying they’re not,” Sen. Marco Rubio said Tuesday when a reporter suggested the allegations are serious. “And he’s pretty firm in his denial, so we’ll have to wait and see how it plays out.”

On Monday, Sen. Rick Scott said Gaetz faces “pretty serious allegations” but didn’t say what should happen to him. “I think we’ve got to get all the facts,” Scott said.

Both senators are considered potential 2024 presidential hopefuls.

Few GOP lawmakers have voiced support for Gaetz, who has said he will not resign. Trump issued a brief statement last week that said Gaetz “has totally denied the accusations against him.”

Gaetz is also the focus of a bipartisan investigation by the House Ethics Committee of accusations including sexual misconduct, illegal drug use, spending campaign funds for personal use and accepting a bribe or impermissible gift. The committee has provided no additional detail.

Gaetz, 38, became engaged on New Year’s Eve at Trump’s Mar-a-Lago compound in Florida.

On Sunday, No. 3 House GOP leader Liz Cheney of Wyoming called the accusations against Gaetz “sickening” but didn’t suggest he should resign. Gaetz has called for Cheney’s defeat after she was among 10 House Republicans to vote for Trump’s House impeachment in January.

Rep. Adam Kinzinger of Illinois last week became the first GOP member of Congress to say Gaetz should surrender his House seat, tweeting, “Matt Gaetz needs to resign.” Kinzinger, a frequent Trump critic, also voted to impeach Trump in January on a charge of inciting the Jan. 6 attack on the U.S. Capitol.

Gaetz and other Florida Republicans and political allies are being examined as part of a broad public corruption inquiry by federal investigators that began months ago, people familiar with the probe have said.

Prosecutors are said to be examining whether Gaetz and Joel Greenberg, a former county tax official, paid underage girls or offered them gifts in exchange for sex.

One person said investigators were looking at trips Gaetz and other men took and whether women were paid or received gifts to have sex with them or later received government jobs.

It was also revealed in a Florida court last week that Greenberg was working toward a plea deal, which might mean he would provide prosecutors with information about the congressman. Greenburg pleaded not guilty to charges including child sex trafficking and fraud.