President-elect Joe Biden said Friday that President Donald Trump isn’t “fit for the job,” but he repeatedly refused to endorse growing Democratic calls to impeach him a second time.

House Speaker Nancy Pelosi said in a letter to members of her chamber that lawmakers could move as early as next week to impeach Trump for inciting a violent mob that overran the U.S. Capitol if the president didn’t “immediately” resign. Pelosi and Democratic Senate leader Chuck Schumer also have called on Vice President Mike Pence and the Cabinet to invoke the 25th Amendment to force Trump from office — a process for stripping the president of his post and installing the vice president to take over.

Addressing reporters in his home state of Delaware after an event introducing some of his Cabinet choices, Biden noted that a key reason he ran for president was because he’d “thought for a long, long time that President Trump wasn’t fit for the job.”

“I’ve been saying for now, well, over a year, he’s not fit to serve,” Biden said. “He’s one of the most incompetent presidents in the history of the United States of America.”

But he refused to back efforts to remove Trump from the White House and insisted that impeachment was up to Congress. Instead, Biden said he was focused on the start of his own administration on Jan. 20, and he said his top three priorities are beating back the coronavirus, distributing vaccines fairly and equitably and reviving the struggling economy.

His comments laid bare the political balance Biden has worked to strike in the months since winning the presidential election. He has continued to sharply criticize Trump and nearly every facet of his administration but also worked to keep the public’s attention focused on what the new administration will do rather than indulging recriminations against the last one.

Biden nonetheless conceded that Trump “exceeded my worst notions about him. He’s been an embarrassment” and likened the “damage done to our reputation around the world” to “tin horn dictatorships.” The president-elect also suggested that a key hurdle to removing Trump was that he has less than two weeks remaining in his term.

“If we were six months out, we should be doing everything to get him out of office. Impeaching him again, trying to evoke the 25th Amendment, whatever it took,” Biden said. “But I am focused now on us taking control as president and vice president on the 20th and to get our agenda moving as quickly as we can.”

Trump would be the only president to be impeached twice. The House impeached him in late 2019, but the Republican-led Senate acquitted him. Removal from office could also prevent Trump from running for president in 2024, or ever holding the presidency again.

Most Democrats, and many Republicans, put the blame squarely on Trump after hundreds of protesters broke into the Capitol on Wednesday and caused destruction and mass evacuations. The president had urged his supporters to protest as Congress was counting the electoral votes that confirmed Biden’s win. Five people died, including a Capitol Police officer.

Biden called what happened a “god-awful debacle” and said it had “the active encouragement of the president of the United States.”

The president-elect’s comments came hours after Trump tweeted that he planned to skip Biden’s inauguration, becoming the first president in more than 150 years — and just the fourth in U.S. history — to do so. Biden said he’d be “honored” to have Pence at the swearing-in, but didn’t feel the same way about Trump.

That’s “one of the few things he and I have ever agreed on,” Biden said. “It’s a good thing, him not showing up.”

Also Friday, Biden called on the Senate — which Democrats won narrow control of thanks to a pair of runoff election victories in Georgia earlier this week — to confirm his Cabinet choices “promptly and fairly.”

“Given what our country’s been through the last few days,” Biden said, “they should be confirmed as close to Jan 20 as possible.”