A party of adventurers (player or non-player) which sustains casualties in an encounter will ultimately break or rout if it does not achieve victory.
At the point in which 25% of a party are unconscious or killed, the party must begin throwing for morale. Average morale throw is 7+ to stand, or not break. Valiant parties may have a higher throw. DM's are allowed: +1 if the party is a military unit, +1 if a leader (leader expertise) is present, +1 if the leader has any tactical expertise; -2 if the leader is killed (for two combat rounds, and then until a new leader takes control), -2 if casualties exceed 50%.
And that's all there is to it. It's a great set of concise rules that are easy to tweak. For example, I might add a provision covering situations where the characters are outnumbered by 2 to 1 or facing overwhelming odds (a mecha, Godzilla, etc.). This rule never should have been left out of the newer editions, but that's what house rules are for.