Japanese is arguably one of the least religious counties in the world but…*
Although Japanese people have almost no interest in religion (reference), the sign prohibiting mendicant monks (not priests) is actually an anti-Chinese sign.
Almost all Japanese believe begging monks are fakes—Illegal Chinese immigrants(reference) without any skills including language and who are just wearing cheap robes and cleverly pretending an oath-of-silence. Since the 1970s, Japan has become so wealthy that begging is a rarity. Nowadays most Japanese would rather (and do) starve rather than ask for a handout or charity.
†↩Technically speaking, these beggars are almost always silent mendicant monks and not “priests” per se. That is, on occasion priests can “beg” for money but generally they actually have to verbally announce themselves in groups with much chanting as well as banging-&-clanging of drums and bells. This practice (shakedown) of we-will-shut-up-if-you-pay-us-for-a-blessing has mostly died out.