3RFan...the reason it is tougher is because the defender only has one responsibility and that is to deny the ball from the offensive player, whereas if you are playing 5 on 5 man defense, you are responsible for the ball and not just your man.
Often you will see the defensive team attempt to guard a great offensive player like it is box and 1 even when the other 4 are also guarding someone with the idea that the one defender has no help responsibilities...and that can also be effective stopping the one offensive player...but the weakness of that is it provide driving lanes for the other 4 offensive players and it does allow for more back door cuts or allows you to set back screens or other screens off the ball to get that great offensive player open where as when you always have that zone in the back, it is much tougher to do that.
Also, as it was mentioned above, it can really confuse the offense. I personally think you should just run your man offense against it, but the truth is, if it is executed by the defense, neither work very well.
What works is the other 4 players have to be good enough to make the defense pay for playing the box and one because there will be some wide open opportunities for the other 4 players.
The team playing the box is betting the other 4 can't beat them, otherwise they would not play that defense.