If someone is tall and big, people follow them even if they act selfishly, right?

Even if they speak and act selfishly without consideration, people follow them.
But if someone is short and looks weak, don't they easily get ostracized even if they have a good personality?