Then they're inconvenienced and discomforted but otherwise fine. The first job of a pilot is "keep the plane flying." Any avoidance maneuver the pilot undertakes should be within the flight envelope of the plane. And planes should be separated enough that emergency maneuvers do not put the plane at risk of immediate collision with another plane.

If a pilot saw a balloon, freaked out, nosedived / stalled the plane and it crashed, the investigation would conclude "Pilot error," much the same as it would if a pilot saw a flock of geese and did the same thing.