I want to try and explain why that man reacted so poorly to Lawrence having Holo with him. Essentially, in this man's eyes, it doesn't make sense for a beautiful young woman to be hanging around Lawrence (an unmarried man in debt) unless...
A) Holo was hired as an escort, which costs money.
B) Lawrence is trying to woo Holo with gifts, which costs money.
In other words, this man doesn't think Lawrence is taking the situation seriously, which means loaning money to Lawrence would be risky. If someone truly needs a loan, they shouldn't be wasting their money or getting further into debt by paying for non essential things (like gifts to win a woman's favor). And, as a particularly devout man, this guy is offended that an unmarried Lawrence is potentially spending money on an escort.
During these past 2 episodes, the show has made it a point to put emphasis on Lawrence's undefined relationship with Holo, and now, we're starting to see why. It's also one of the reasons why so many people in this town have been staring at Lawrence and Holo (the other reason being the large number of armors they have, which everyone else knew were worthless now that the Northern Expedition had been canceled). And, as you correctly pointed out, word travels fast, so the rumor mill at this point isn't painting Lawrence in a good light. People are very judgmental.