Stephen Crabb has been appointed as the new work and pensions secretary, after Iain Duncan Smith resigned on Friday.
Mr Duncan Smith had said the latest planned cuts to disability benefits were "not defensible" in a Budget that benefited higher-earning taxpayers.
David Cameron said he was "puzzled and disappointed" that Mr Duncan Smith had decided to go when they had agreed to have a rethink about the policies.
Alun Cairns is replacing Mr Crabb as secretary of state for Wales.
An ally of Mr Duncan Smith suggested he had been driven to resign by a "high-handed and short-termist political approach" to cuts by Chancellor George Osborne.
Labour leader Jeremy Corbyn said Mr Duncan Smith had "done the right thing", having "presided over some fairly appalling policies".
He said the government's latest moves to reduce disability payments were "simply not fair, not right" and it was wrong to finance tax cuts for the better off by "taking money away" from those with disabilities.
Mr Crabb, 43, who has been MP for Preseli Pembrokeshire since 2005, said on Twitter: "A privilege to be appointed new Work & Pensions Secretary this morning.
"Looking forward to working with my new team."
In his Budget on Wednesday, Chancellor George Osborne said the government would be spending an extra £1bn on the disability budget but changes to benefits announced a few days earlier had suggested the government would save £4.4bn on this by 2020-21.
They included changes to Personal Independence Payments (PIP), due to replace Disability Living Allowance (DLA) in January 2017, that were expected to save £1.3bn a year, but sparked an outcry from opposition parties and some Tory MPs.
But on Friday, before Mr Duncan Smith resigned, a government source had said the planned changes would be "kicked into the long grass".
Baroness Stroud, who worked with Mr Duncan Smith at the Department for Work and Pensions (DWP) and now heads the Centre for Social Justice think tank, which he founded, told the BBC: "He always used to say to me, 'I'm here in order to deliver reform and to protect the poorest'.
"Yesterday he felt that he could no longer protect the poorest... This is a step too far."
Conservative MP Bernard Jenkin, an ally of Mr Duncan Smith, said: "A lot of colleagues will be extremely annoyed at this explosion but they will not be surprised.
"I think the high-handed and short-termist political approach that the chancellor has tended to take to the management of other departmental budgets has tested the patience of more than one minister."
But another Conservative MP, Stephen McPartland, said on his website he would "not be shedding any tears for the evangelical, aggressive and routinely failing welfare reforms that were the personal fiefdom" of Mr Duncan Smith.
"I don't always see eye to eye with the Treasury, but it is disingenuous to blame them," he said.
Commons Leader Chris Grayling told the BBC Mr Duncan Smith had done "some really good work" on welfare reform and he was "surprised" and "very sorry" to see him go.
Asked if this was really about Europe, Mr Grayling - who like Mr Duncan Smith is campaigning for the UK to quit the EU, in oppositon to the prime minister - said: "I don't think this makes any difference at all to the referendum campaign."
Sources close to Mr Duncan Smith also say his resignation was not about Europe.
Labour MP Frank Field, who chairs the Commons work and pensions committee, said: "Behind this is a much, much bigger drama... What he crucially cared about was the balance of resources going to families and children and older people.
"The pensioner element, the biggest part of the Budget, was safeguarded and in fact increased... all these cuts were on people of working age and it's this point that Iain thinks the social contract between generations is being broken."
In his resignation letter, Mr Duncan Smith said: "I have for some time and rather reluctantly come to believe that the latest changes to benefits to the disabled and the context in which they've been made are a compromise too far.
"While they are defensible in narrow terms, given the continuing deficit, they are not defensible in the way they were placed within a Budget that benefits higher earning taxpayers."
Replying to Mr Duncan Smith, Mr Cameron said there had been collective agreement between "you, No 10 and the Treasury" that "the increased resources being spent on disabled people should be properly managed and focused on those who need it most".
"[On Friday] we agreed not to proceed with the policies in their current form and instead to work together to get these policies right over the coming months.
"In the light of this, I am puzzled and disappointed that you have chosen to resign."
Guto Bebb becomes under secretary of state at the Wales office, replacing Mr Cairns, the MP for Vale of Glamorgan.
