A big problem were the tons of feats and the classes, so if you got rid of those, that's a big step. I wouldn't call it a d20 game anymore like that ;)
The other big thing was the linear distribution you get with a 1dX die-system. It's just too swingy and "unnatural"...it does not model processes well.
I guess if you have dumped classes, you also will have to work the skills differently, yes? That's another thing that bugged me with dnd 3e and 3.5... fighters were just bad at skills (the only thing they were good at was...well...fighting) and others, like thieves, were "the king" of skills...just wasn't my kind of thing. Also, miniatures-focused combat...ugh (that was even worse in 4e).