You obviously weren't on the infield...it isn't about the race so much as the crazy ass party in the middle of what many consider a 'high-class' event. I had a blast this year, it was my first Derby.
I went to the betting window and had the girl there give me Big Brown and 5 other random horses in an exacta...and she gave me 8 Belles. I was really excited about it ($141) until I heard about her having to be put down...then I got over it again.
Whether or not you like horse racing it is still a great time, I recommend it.
No, I was definitely in the infield, and I was damn drunk when I got in there, but it was sooooo damn crowded by that point that myself and the two other people I was with did little else besides stumble around, not know where we were going assuming we knew where the hell we where, and bump into people. Guess you gotta show up several hours before the race starts on the infield to get the "party" atmosphere cause I definitely didn't sense it at all being there from 4:30 to 6 when the race started.
The parking lot tailgate I was at was more of a party than anything I saw on the infield... it was mostly just people walking around with drinks in their hands and old people sitting on fold-up chairs
Still fun to go no doubt, and I can say I've been there.