It may have been later in the Anglo-Saxon period that the settlement shifted focus to a site around the parish church. This may have been after Abingdon Abbey took possession of the manor and dammed the land to the west of Barrow Hills to form their fish ponds behind Daisy Bank. The old village was apparently between the church and park, and the River Thames. The timber-framed vicarage, adjoining the church, dates from the late 13th century when it was the home of a chaplain sent from St. Helens's in Abingdon. These days the village mostly stretches south-east from the church, down to the railway station at Lower Radley where 15th century buildings still survive. The station was the nearest to Abingdon until a branch line (now gone) opened in the 1850s. The name Radley is generally thought to be Saxon, possibly meaning 'Hretha's Sacred Grove'. Hretha was the Saxon goddess of Fame. Although it might just mean 'Red Clearing' or 'Raedda's Clearing': Raedda being the chieftain who founded Reading. The prefix could also be Celtic Rhydd, meaning 'ford', presumably over the Thames.

The abbey continued to hold Radley Manor after the Norman Conquest. However, it was not a grange producing agricultural produce but a hunting lodge with a deer park, managed by a park ranger. The abbots used the wood in the park for fencing and fuel, but they also entertained important visitors there, such as King Henry VII. The house was probably nearer the church than the present Radley Hall. In the late 14th century, it was in the custody of Thomas Golafre, and his more famous son, the Royal servant John Golafre of Fyfield Manor, was possibly born there. The manor passed through a number of hands after the Dissolution, but stories of Princess Mary - later 'Bloody' Queen Mary Tudor - having lived there seem to be apocryphal. It was actually in the hands of her sister, Elizabeth I, until she sold it to George Stonhouse in 1560.

Stonhouse was a Clerk of the Green Cloth, one of the officials controlling the Royal Household below stairs. After he had managed to eject the tenant, he built himself a fine mansion near the walled garden of the present Radley Hall. His son, Sir William, was made a baronet and has an excellent monument in the parish church. His son, Sir George, was a Royalist during the Civil War. This was OK whilst Radley was an outpost of the Royalist garrison at Oxford, but later he had to pay heavy fines to keep his estate. He managed to avoid further disaster with the help of his friend and neighbour, the Parliamentary Speaker William Lenthall. There was an armed struggle between the two sides in the village during 1643, when the Roundheads from Sunningwell attacked. The Cavaliers found safety for a while after having managed to barricade themselves into the church. The exact outcome is unknown, but it could not have been good for them as the north aisle was certainly destroyed by cannon fire. Several men were killed and were apparetly buried in a large mound against the church wall. This was eventually removed when the damp began to affect the superstructure. Sir George didn't get on with his eldest son and so secured a second baronetcy for his younger son, John. His descendants later held both titles. By coincidence, the Bowyers, who inherited Radley Hall from them, also had two baronetcies! The most famous of them was Admiral Sir George Bowyer. His son lost all his money and was forced to rent out his home.

Radley is, of course, most famous as the home of Radley College, originally St. Peter's. Dr. William Sewell spent a vast fortune founding the place and almost went bust from debts of over £40,000. Luckily, he was bailed out by Lord Addington to whom he owed the most. He did secure the lease of Radley Hall for the school in 1847 and they were able to purchase the estate in 1889, although the manor rights remained with the Bowyers until 1901. Former pupils have included actors, Desmond Llewelyn and Dennis Price; comedian, Peter Cook; and poet laureate, Andrew Motion.