I hope that all the current principle actors are kept around for Avengers 3. After that I would prefer any new Avengers movies have an entirely new cast. I know recasting is inevitable, but I think it would be weird to have, say, Avengers 3 and 4 have the same Black Widow, Hawkeye, Hulk and Ant-Man, but different Captain Americas, Thors and Iron Mans.
I agree with you. While I like the X-Men, frankly I do not like that they inhabit the same universe as the rest of the characters. It never made sense to me. People fear mutants, but if you got your powers through cosmic rays like the Fantastic Four, then you're celebrities? Odd.
But I don't expect X-Men will ever revert anyway. Similar with Spider-Man. I would have thought Fantastic Four might, but they are rebooting it in 2015. In terms of commercial drawing power, Wolverine and Spider-Man are the only characters I could see added to the Avengers lineup that would excite general audiences, but it just never will happen. Fantastic Four would be cool for its villains, and that adding them would allow for bigger scope plot lines like Civil War that would be hard to do otherwise.