Neil Stein, 77, the trailblazing, jet-setting restaurateur who helped ignite a sleepy Center City dining scene and brought sidewalk dining to Rittenhouse Square, died Thursday, Oct. 25, at his Center City apartment, apparently of natural causes, said one of his daughters.

Mr. Stein — son of a grocer from Mount Airy, and a whippet-like semipro baseball player in his youth — created such destinations as the Fish Market, Marabella's, Rock Lobster, Striped Bass, and Rouge in a career that spanned decades. His empire began collapsing in late 2001, when a bank and, later, tax authorities came calling. Mr. Stein pleaded guilty to tax evasion and in 2006 served 10 months in federal prison. In recent years, he had been plotting a return to the business, which would have been yet another comeback for a man whose successes often were followed by addiction-fueled defeats, as he readily acknowledged.

Friends and competitors described him as a visionary, obsessed with the details.

"He began the real renaissance of Center City," Stephen Starr said Friday. "Without Rouge [which opened in 1998], there would have been no Parc," Starr's $14 million-a-year restaurant at 18th and Locust Streets. "Rittenhouse Square was barren."

"He was the man who made restaurants a destination before food and restaurants were cool," said Michael Schulson, whose restaurants include Sampan, Harp & Crown, and Double Knot. "He was the man that paved the road for people like myself and Starr to be able to do what we do in this city."

Neil Stein at Avenue B in 2001.
Mark Ludak / File Photograph
Neil Stein at Avenue B in 2001.

Mr. Stein cut a dashing figure in a sleepy restaurant town. He was 25, a young father selling men's clothing, in 1966 when he got the idea to open a supper club at the Cedarbrook Hill Apartments in Cheltenham (now the Towers at Wyncote) with $5,000 from his father, Moe.

Mimi Says, named after his oldest daughter, indulged his taste for the good life. He drove a Jaguar and a Mercedes and partied hard. His first wife, Angel, divorced him after an impromptu, alcohol-fueled trip to Las Vegas, and his world collapsed, Mr. Stein told an interviewer. He later remarried, but had been single for 35 years, said daughter Maggie Wasserman, who with her husband, Rob, owns the Rittenhouse Square bistro Rouge — the last of Mr. Stein's restaurants. He lived alone in later years.

Licking his wounds from his first marriage, Mr. Stein came back in 1973 with Fish Market at 18th and Sansom Streets, which had a fish counter — an homage to Moe Stein's shop, the Fruit Basket on Wadsworth Avenue. There he worked with the woman who would become his second wife, Cyndi.

His addictions, however, led to divorce and killed the restaurant, as he told an interviewer. In 1981, he owned a short-lived Lower Merion takeout shop called the Icehouse.

He came back again in 1983, when he helped the Marabella family create a chain of modern Italian restaurants called Marabella's. Then three moneyed partners came to Mr. Stein about a pier on the Delaware Riverfront. Rock Lobster effectively created a new destination for outdoor destination dining. But disagreements followed, and Mr. Stein was fired.

Joe Wolf (left) and Neil Stein at Striped Bass in 1994.
Akira Suwa / File Photograph
Joe Wolf (left) and Neil Stein at Striped Bass in 1994.

Shortly after, Mr. Stein's childhood buddy Joe Wolf called with the idea for a restaurant near the Philadelphia Museum of Art. Wolf had recently closed Corned Beef Academy, a modest deli chain for which Mr. Stein had written the early menus as a favor. "There's something here big enough for the both of us," Wolf recalled saying to Mr. Stein.

The two, who attended Leeds Junior High and Germantown High together, could not be more different. "He's Armani," Wolf said he likes to quip. "I'm Polo."

That deal at 2601 Parkway did not happen, but the Butcher & Singer brokerage house at 15th and Walnut Streets was available. With its 28-foot ceilings — and with the Convention Center and its attendant spike in tourism business on the horizon — Mr. Stein and Wolf decided to create a lavish, all-seafood restaurant.

Striped Bass, which opened in 1994, lit up a Center City corner after dark and helped cement Walnut Street as Philadelphia's restaurant row, including such other destinations as Le Bec-Fin, Susanna Foo, and Circa. "He made Philadelphia a more cosmopolitan place," said Alison Barshak, the opening chef at Striped Bass, Mr. Stein's greatest success.

Mr. Stein bought out Wolf in 1997, and set out to open a smaller but posh bistro called Rouge 98 in the Rittenhouse Claridge, on the 18th Street side of Rittenhouse Square. (Its French parlor chairs cost $1,200 each.) Asked one day at Striped Bass to describe Marguerite Rodgers' in-progress design, he simply scribbled "Sexy" on a cloth napkin.

Rodgers said in 1999 about Mr. Stein's gut: "When a concept works, it has to smell like it, sound like it, feel like it, taste like it. "

Rouge — the "98" soon was dropped — opened in summer 1998, ushering in the era of sidewalk dining that Stein had fought the city to achieve.

Neil Stein on the oyster bar of Fishmarket.
ERIC MENCHER / File Photograph
Neil Stein on the oyster bar of Fishmarket.

Then came a slew of openings: In 1999, he revived Fishmarket on 18th Street near Sansom, next door to the former Fish Market. In July 2000, he opened Bleu, a companion bistro to Rouge, at 18th and Locust Streets, where Parc is now. In late 2000, he took the northwest corner of Broad and Spruce Streets, while the Kimmel Center was under construction across the street, for a grand restaurant called Avenue B.

The crowds never materialized for Avenue B, and Mr. Stein found himself in a hole. By the end of 2001, Hudson Bank was after him for missed payments on a $1.3 million note. That case was settled after he received new loans from Commerce Bank, but in 2003, the City of Philadelphia sued him over $1.36 million in business, liquor, and wage taxes. Common Pleas Court ordered him to pay $68,000 a month to satisfy the debt. He filed for bankruptcy and closed Avenue B. Vendors were owed hundreds of thousands of dollars, and employees scrambled to settle tax issues on their own.

His life in shambles, Mr. Stein checked himself into the Caron Foundation in Wernersville, Pa., which treats drug and alcohol abuse. "I believe he was sober from then," daughter Maggie Wasserman said. "In rehab, the kids loved him when he stood up and told his story."

Starr, meanwhile, bid $1.3 million in bankruptcy court to buy Striped Bass. Starr reopened it in early 2004 and later changed the concept to a steak house — reviving the Butcher & Singer name.

But Mr. Stein's troubles were not over. In September 2004, he was arrested for tax evasion. He pleaded guilty and in 2006 began serving a sentence of a year and a day. He was released after 10 months.

In an interview from the prison in Schuylkill County, Mr. Stein accepted blame for his failure and acknowledged how his addictions had not only destroyed his life but those of his loved ones.

"He always held himself out with dignity — no matter what the situation," said Nino V. Tinari, who was not only Mr. Stein's criminal attorney but a close friend.

Prison changed Mr. Stein, he acknowledged in 2008.

"I don't think driving a 911 Porsche is as important as it was 15 or 20 years ago to me," he said. "I lived in an apartment that was $6,000 a month and I drove [a] $100,000 car and I went to St. Bart's every single winter for a month and spent $30,000 or $40,000 in St. Bart's. And those things aren't important to me anymore."

Mr. Stein walked the city in later years. He was frequently accompanied by his boxer, Striper, before the dog's death in 2011.

In addition to daughters Maggie Wasserman and Mimi Milou, he is survived by sons Eric and Perry Milou; nine grandchildren; and a sister, Sheryl Borish, owner of the Marathon Grill chain.

Wasserman said a memorial service would be private.

Donations may be made to Caron at caron.org/give-to-caron.