McCarthy was appointed to Sainsbury's operating board in June 2004 with responsibility for its portfolio of 297 convenience stores. Prior to taking up the role, he had been working as a consultant for the retailer after leaving Tesco.
Before Tesco, he was chief executive of T&S Stores, during which time the company grew from a minor regional player to the largest c-store multiple in the UK until it was taken over by Tesco in 2003.
He has more than 30 years experience in the retail industry, having also worked for high street clothes retailer Next and the Birmingham Post and Mail, where he was managing director of its retail estate.
Sainsbury said that McCarthy had been instrumental in the development of its c-store division.
Earlier this week, in its full-year results to March 25, Sainsbury revealed that the division had experienced a good year with sales uplift of more than 20%. This year it also opened 20 stores and refurbished 94 stores - 34 Sainsbury's Locals and 60 'Sainsbury's at'.
In his new role at Poundland, McCarthy will replace Dave Dodd, who revealed in November last year that he was stepping down in order to focus on new product development and sourcing.
McCarthy said: "I am delighted to be joining Poundland at such an exciting stage in its growth.
"I look forward to working with the team to cement its position as the UK's leading single-price discount retailer."
Colin Smith, chairman of Poundland, added: "Jim brings vast experience of the retail sector as well as a proven track record in achieving growth. We welcome him to the team."