Cumnor Place was the wealthiest and grandest of the granges of Abingdon Abbey. It was always a favourite with the abbots who would retire there in times when plague or other disease was rife in Abingdon. After surrendering his abbey at the Dissolution in 1538, it is not surprising, therefore, that, when offered a life interest in one of the Abbey's manors, Abbot Thomas Rowland chose to live here. Six years after his death (1547), the building was granted George Owen, the King's Physician. His son leased Cumnor to Sir Anthony Forster, the friend and chief controller of Lord Dudley (later Earl of Leicester)'s private expenses. Dudley's wife, Amy Robsart, stayed with Sir Anthony, in 1560, when her husband was away at court, cosying up to Queen Elizabeth I. One day, the family went off to Abingdon Fair and returned to find poor Amy dead at the foot of the stairs! There were many whispers of murder.

Not long afterward, Cumnor came into the hands of the great Norreys family of Rycote (Oxfordshire). It was a secondary home and little favoured, apparently because of its oppressive atmosphere. The place was a magnificent 14th century stone structure, though considerably altered by Sir Anthony in 1571. It stood just south-west of the present churchyard forming a complex of buildings enclosed within a large quadrangular courtyard. The 44ft long medieval great hall (see illustration) was on the west side. Other attached buildings included a vaulted Tudor gatehouse with long gallery and a small chapel (22ft long). By the time the Lysons brothers visited, in 1806, it was being used as a granary. The Earl of Abingdon had the buildings demolished four years later.

Cumnor Place no longer stands, although parts of the old building have been incorporated into Wytham Church. A nearby 18th century house has also taken on the name.