The Archbishop of Canterbury has insisted he was not picking on Wonga after it emerged the Church of England invests in the payday loan firm.
The Most Reverend Justin Welby admitted being "irritated" and "embarrassed" by the revelation but went on to heap praise on Wonga and its management.
Mr Welby hailed the company for its professionalism and suggested it was far from the worst organisation in the loan sector.
The link between the Church and the firm emerged hours after the Archbishop said he wanted to force Wonga out of business by expanding credit unions.
The Financial Times found the Church's pension fund had put money into Accel Partners, a US venture capital firm that led Wonga's 2009 fund-raising efforts.
Until the report emerged, Mr Welby had no idea about the connection.
Sources suggested he was "furious" but on Friday, in a lengthy interview, he merely said: "I was irritated for a few minutes but, you know, these things happen."
Justin Welby: 'It's very embarrassing'
He did admit the affair was "very embarrassing" and vowed to investigate, signalling there could be a review of the Church's entire investment portfolio.
But he said: "I never took on Wonga in particular. The context was talking about the entire payday lender movement.
"Wonga is actually a very professionally managed company. Errol Damelin, the chief executive is a very clever man, [who] runs it extremely well."
Despite praising the company, he said he was still unhappy about the Church's investment.
"They shouldn't be investing in Wonga. We don't think that's a good thing," he told the BBC's Radio 4 Today programme.
And he insisted he was not backtracking on his commitment to clamp down on the industry, which is currently the subject of a Competition Commission probe.
"We need to provide a proper alternative to these very, very costly forms of finance. The worst people are not Wonga. There are plenty of others much worse," he said.
Mr Welby said Church policy allows investments in a company where 25% of its business is in the loan area, indicating the arrangement with Wonga may be against its rules.
"I think we have to review these levels and make sure we are consistent between what we're saying and what we're doing," he said.
The Archbishop conceded that it was almost impossible for the Church to make an investment that was not somehow tainted.
He said: "If you exclude any contact with anything that directly or indirectly gets in any way bad, you can't do anything at all."
Lambeth Palace has said it will ask the Assets Committee of the Church Commissioners to investigate the link to Wonga and review the holding.
It added: "We will also be requesting the Church Commissioners to investigate whether there are any other inconsistencies as normally all investment policies are reviewed by the Ethical Investment Advisory Group (EIAG)."
Mr Welby is seeking to expand the reach of credit unions as part of a long-term campaign to boost competition in the banking sector and clamp down on short-term loan firms.
The Government announced an investment of £38m in credit unions in April to help them offer an alternative option to payday lenders.
The Office of Fair Trading referred the entire payday lending industry, which is worth £2bn, to the Competition Commission last month after finding "deep-rooted" problems.
It said it decided to make the referral because it continues to suspect that features of the market "prevent, restrict or distort competition".
Wonga said in March that it welcomed any attempt to encourage responsible lending and that it has been "instrumental" in helping to raise industry standards.
Mr Damelin, its founder, said: "The Archbishop is clearly an exceptional individual and someone who understands the power of innovation.
"There is mutual respect, some differing opinions and a meeting of minds on many big issues.
"On the competition point, we always welcome fresh approaches that give people a fuller set of alternatives to solve their financial challenges. I'm all for better consumer choice."
The company has launched a new advertising campaign setting out "ten commitments" about its lending practices in an apparent tongue-in-cheek reaction to the Archbishop's original remarks.
Mayor of London Boris Johnson backed the Archbishop's plans and said it was an "interesting interpretation of the gospels".
He told Channel 4 News: "I think it's a wonderful idea.
"Wonga is a perfectly legitimate business but there's no doubt their rates are usurious. There are people who find it very, very difficult to repay the loans once they get into trouble.
"He's not turning over the tables of the money lenders, he's bringing in his own money lending tables."