The basis of the present Manor House was built, probably on the site of a timber Saxon house, between 1250 and 1280, in limestone rubble, as a central hall with two-storey transverse wings at either end, together with various outbuildings. This roughly symmetrical shape became in two succeeding centuries the typical plan of the English Manor House, but is seen here a great deal earlier than elsewhere, an open plan without a surrounding moat, protected by its religious association in a period when houses of secular lords needed keeps and baileys.

The Central Hall has been divided, but extended originally between what are now the further walls of the Sitting Room and Library, and was the width of these.

This would have been the community centre of the village, where all would come for announcements and feasts, and for the Manor Court to judge their failings. It would also be the room where most of the living-in members of the demesne ate, sang and slept, round the central fire where smoke drifted out through louvres under the eaves of the roof. The North wing, opening off this, would have contained various stores, buttery, pantry, larder, and so on, which occupied a very considerable part of the house when there were no shops to supply one's needs from day to day.

In Elizabethan times the roof of the Hall was raised to allow the insertion of a second storey, and the area was divided into smaller rooms, under the surviving Stonesfield stone slate roofing. A little later the west wall was rebuilt with the newly invented sash windows.

In the 19th century these windows were replaced by the present stone mullions, no doubt considered by their designer to be more in keeping with the antiquity of the house.

At the same period the North wing was pulled down and rebuilt to provide modern kitchens and bathrooms, the little quatrefoil window in the west gable end being all that obviously survives of the old building. Also the corridors on the east side were added on the exterior of the old Hall wall, and include, as a piece of Victorian whimsy, a cell door from old Exeter gaol.

In the west porch, also added at this time, is now set a modern relief plaque by Langford-Jones of Blew-bury, representing Christ weeping over Jerusalem (Luke 19, 41-42), given to Friends by the artist's widow. The faded inscription on the face of the porch quotes Vergil (Georgics, 11, 458), and is perhaps not so apt today as when it was inscribed: "O happy husbandmen, did they but know the blessings they possess, for whom, far from the din of war, the kindly earth pours forth an easy sustenance".