The Mystic River star criticised guests at a fundraiser event in Cannes held to raise money for victims of the 2010 earthquake in Haiti.
Penn, who was guest of honour, said during a press conference before the party on Friday (May 18) that many of his peers have lost interest in Haiti in the years since the devastation.
"It's not only celebrities who went [to help] for a day. It's the whole f**king world. It's all of you. The reason we have Haiti fatigue is because there was never a commitment in the first place," he said.
"People have to understand this is not about diving into the water. You're going to have to swim."
Penn has been heavily involved in relief work in Haiti since the earthquake and created the J/P Haitian Relief Organisation to help rebuild schools and homes in the weeks after the disaster.
Over $2 million (£1.3 million) was raised during the event from ticket sales and a charity auction.
Penn will next be seen in Gangster Squad alongside Ryan Gosling and Emma Stone, and is currently in talks to star in The Secret Life of Walter Mitty.