Bernie Madoff, who orchestrated the biggest Ponzi scheme in history, has died while serving his prison sentence in North Carolina. He was 82.

Madoff, a legendary figure on Wall Street, amassed billions of dollars in investments over decades by swindling thousands of clients out of their savings, according to the USA Today Thursday.

Madoff had been serving a 150-year sentence after he pleaded guilty to running a Ponzi scheme in 2009. Madoff's lawyer and the Federal Bureau of Prisons confirmed his death, of natural causes, Wednesday.

Last year, Madoff's attorneys unsuccessfully asked a court to release him from detention during the COVID-19 pandemic, saying Madoff suffered from end-stage renal disease and other chronic medical conditions.

The people that Madoff has scammed included Hall of Fame baseball player Sandy Koufax, Hollywood actors John Malkovich and Kevin Bacon, and movie director Steven Spielberg's charitable foundation, Wunderkinder.

British banks also fell victims. HSBC Holdings said it had lost around $1 billion. The Royal Bank of Scotland and Man Group and Japan's Nomura Holdings also lost huge amount of money. Farmers, school teachers, mechanics and many others also added to his list of victims.

A former chairman of the Nasdaq stock exchange, Madoff's investment advisory business was exposed in 2008 as a fraud that wiped out people's investments and ruined charities. He became so hated he was forced to wear a body armor in his appearances in court.

"His life has ended, but his larcenous legacy has been indelibly etched in investment lore," Sam Stovall, author of "The Seven Rules of Wall Street" and chief investment strategist at CFRA Research, said.

Over the years, authorities have recovered more than $14 billion of an estimated $18 billion investors placed into Madoff's business. At the time of his arrest, bogus account statements showed clients had holdings valued at $60 billion.

At least two people who invested with Madoff's business committed suicide after their losses. His son Mark also took his own life on the second anniversary of his father's arrest. Andrew, Madoff's other son, died of cancer in 2014.

Madoff is survived by his wife, Ruth, who maintained she had no knowledge of her husband's business and was spared from charges, according to BBC.