Most would say 4.
But, while 4 did have that creepy Land of Mists and Dil and Ichy, the 5th one had threat of starvation, separation of the Gang forever, a tsunami, a shark, Chomper (yes, he was scary), a Sharp Beak, Chomper's parents, and, of course, the three-claw Sharptooth (who rivaled the original guy).
In IV, other than Dil and Ichy, not much else threatened the Gang.
In V, we have to fear for their safety constantly. (i..e., after Chomper almost attacked them, we did have to wonder if Cera would turn out to be right), we had to wonder if his parents would find the Gang, if the Gang would be stuck on the Island for the rest of their lives.
The only other danger lurking over the Gang in IV, other than Dil and Ichy, was the threat to Grandpa Longneck's survival.