I think we have all rode too fast and feel like we are "in control". I do find that riding in a large group (4+ riders), the guys at the back always want to keep pace and push it too far when they fall behind. That is where I have always run into problems, as it's usually the last few riders that cut the corners trying to catch up.
I have ridden fast and felt ok doing it, but it is stopping that I feel is the problem. If you come around a corner and/or have something unexpected on the trail, even the "best" riders are going to have an issue.