Moussa Sow scored with the last kick of the game at the Estadio de Mongomo to see Senegal to a 2-1 win over Ghana while South Africa missed a penalty and scored an own goal as they allowed Algeria to fight back and win 3-1.
The victories left both Algeria and Senegal on course for a knockout place if they win their next group games on Friday.
Senegal were behind after just 13 minutes when their captain Bouna Coundoul brought down Christian Atsu and Andre Ayew converted the penalty.
But patient play allowed Senegal to take command in the second half as they out-muscled a Ghanaian side who had new coach Avram Grant in charge for the first time.
Mame Biriam Diouf grabbed the second half equaliser when he reacted quickest to a rebound before Senegal snatched the win deep in stoppage time.
A long upfield punt by Coundoul fell kindly for substitute Henri Saivet, whose crisp one-two with Diouf released Sow in the heart of the penalty box and the striker gave goalkeeper Razak Brimah no chance with a first-time shot into the far corner.
“We might have been behind at half-time but we always knew we had the potential to win the game,” said coach Alain Giresse.
South Africa coach Ephraim Mashaba declared the “best team lost” after watching his men squander an early lead and lose in front of a capacity 15,000-strong squad.
But that was unfair on the top-ranked Algerians, who finished strongly to confirm their status among the favourites.
Thuso Phala scored after a slick passing move for South Africa in the 51st minute and they then won a penalty just minutes later for a chance to go 2-0 ahead.
But Tokelo Rantie blasted his kick against the crossbar as a set of calamitous errors cost his side dearly.
Thulani Hlatshwayo mistimed a clearing header to concede an own goal in the 67th minute to allow Algeria back into the game. An innocuous-looking chip shaved the top of his head and beat goalkeeper Darren Keet.
Algeria were ahead four minutes later through as full back Faouzi Ghoulam scored from the right side in the 71st minute.
Islam Slimani made sure of the win with a third goal in the 82nd minute as the ball slipped under Keet’s body in another error from the South Africans.
“It was an intense game and it could have been catastrophic. But things turn quickly in football and we played well to come back and win,” said Algeria coach Christian Gourcuff.