The land that made up much of Annandale was occupied by the Fitzhugh family for over six generations and eventually, it became one of the largest tobacco plantations in the area. Interestingly, at one point, Fitzhugh leased some of the property to French Huguenots, who had fled religious persecution in France.