After police arrested 28 people during an EDL protest in Walsall, West Midlands, shopkeepers and landlords in the town centre said they were lamenting the damage caused to the town and its reputation.
Up to 1,000 people gathered at the event and at a counter-demonstration on Saturday.
Market traders stayed away over safety fears as about 400 EDL members were penned in to two neighbouring streets outside several pubs.
The counter rally was held in another part of the town.
Trouble seemed to erupt after speeches were made in the streets and a small number charged the police cordon and threw bottles, bricks, and dustbins, witnesses said.
Ricky Suman, manager of St Matthew's Hall, Leicester Street
They arrived at about 11:00 BST and there was quite a good atmosphere at first.
Landlord Ricky Suman praised the police and council for the way they handled the protest
But then later, there were a few speeches by the EDL and maybe the police thought that was that but then the confrontation started and our beer garden was used as ammunition.
We'd taken a lot of the furniture inside the night before but they still managed to rip up the nailed down stuff - a window got smashed, plant pots thrown, bottles...
The heated exchange lasted for nearly an hour but the police and the council did a fantastic job, they should be very proud.
They dispersed them one-by-one, taking mug shots of individuals and then the clean-up started. But people stayed away at night. We reopened at about 20:00 BST and normally we have about 400 in but it was only about 200.
I didn't believe it would be that bad - everyone can express their opinion but shouldn't it be done in a proper manner?
Council workers were cleaning the streets outside the pubs on Monday
Lindsey Wakeman, manager of Yates's, Leicester Street
We followed all the police advice and got more security in and used plastic glasses.
I can understand why they penned them in the two streets here by licensed premises as it keeps them in one area and under control.
There were no issues inside the pub but when they all went outside things to change and suddenly there was trouble.
We shut at 13:30 BST and didn't reopen until 18:00 BST. I can't say it was a profitable day for us - football was on too which would have brought in trade.
We had just over 1,000 in on the evening when normally we have about 1,500.
The town just gets negative publicity and a huge police presence put people off.
Jason Stevens, Lunnon fruit and veg store, Walsall Market
Mr Stevens (l) say he is angry at losing a day's trade
We lost a valuable whole day's trade.
It was optional to open or not but they said they couldn't guarantee our safety so we decided not to do it.
It's disgusting - we should all get on together shouldn't we?
Whatever side you're on, a good 60% are up for fighting and jumping on the bandwagon I think.
We were moved when the (Olympic) Torch relay was on and what with the weather this summer as well, it could be better.
It does damage. People see it on the news and stay away.