Director of supervision Tracey McDermott will take over as acting chief executive on 12 September until a permanent replacement is appointed.
In a statement, Chancellor George Osborne said the regulator needs “different leadership”.
Mr Osborne said: “Britain needs a tough, strong financial conduct regulator. Martin Wheatley has done a brilliant job of launching the FCA in tough circumstances.
“Now that phase is complete, the Government believes that different leadership is required to build on those foundations and take the organisation to the next stage of its development.
“The Government is launching a worldwide search; Martin’s replacement will – like him – need to be passionate about protecting consumers, promoting competition and completing the job of cleaning up the City, so it is the best-regulated market in the world.”
Mr Wheatley, who became the newly-constituted City regulator’s first CEO, will depart with a £490,000 payoff, equal to a year’s pay, though he will continue to act as an adviser to the board until 31 January. He will focus in particular on the implementation of the Fair and Effective Markets Review, a review of the wholesale financial markets established by the Government.
He became a controversial figure, warning the banks in 2012 he would “shoot first and ask questions later”, a statement he later said he regretted.
He had to forfeit his bonus for the 2013-14 financial year after the FCA was criticised for mishandling the announcement of a review into life insurance policies.
However, he also oversaw record fines on banks after they were caught trying to rig the Libor interest rate benchmark and currency markets.
Under his watch, helped by McDermott’s stint as head of enforcement before she moved to supervision, the regulator has levied record fines on banks after they were caught trying to rig the Libor interest rate benchmark and currency markets
He said: “I am incredibly proud of all we have achieved together in building the FCA over the last four years. I know that the organisation will build on that strong start and work so that the financial services industry continues to thrive.”
FCA chairman John Griffith-Jones said: “Martin has done an outstanding job as chief executive setting up and leading the FCA over the last four years. We owe him a lot and I and my board would like to thank him for his great efforts in setting up the organisation and for the contribution he has made to putting conduct so firmly at the top of the financial services agenda.
“We all wish Martin well and I am pleased that we will continue to benefit from his wisdom and expertise over the next few months.”
Mr Wheatley joined the FSA in September 2011 as managing director of its conduct business unit, before becoming chief executive of the FCA in April 2013.
He previously served as chief executive of the Hong Kong Securities and Futures Commission for five years, and has also been deputy chief executive of the London Stock Exchange.