The Fyfield estate was inherited by Sir John Golafre Senior (d. 1363) from his mother-in-law, Juliana, widow of Sir John de Fyfield, in 1336. It was probably in this year that he rebuilt any pre-existing house and, surprisingly, a substantial part of his 14th century building survives in the Fyfield Manor of today. The porch and cross-wing at the end of the hall certainly date from this time, although his medival hall has been rebuilt. The entrance arch even has typical Early English ball-flower decoration. The three stone ground-floor rooms of the crossway were originally the kitchen, service room and staircase to the upper floor. This was the timber-framed solar, Golafre's private rooms. It has a fine original wooden roof. Sir John's son, also John, was the father of King Richard II's great friend, Sir John Golafre of Wallingford Castle, but he was illegitimate and his cousin, John, inherited Fyfield. This man was one of those who built Abingdon Bridge and is buried under a stone cadaver in the parish church. Golafre descendants lived in this house until 1448.

In 1554, the place was sold to Sir Thomas White who gave it to his new foundation, St. John's College in Oxford. The College immediately rented it back to his brother, Ralph. It was their nephew, George, who rebuilt the hall about 1580. The ground and first-floor walls are part of the original house, but they were extended upwards to form a second upper floor and the whole block was extensively remodelled. The whole building was sympathetically restored in 1868 by JH Parker, the Oxford booksller. It was for many years the Antiquarian books department of Blackwell Books, but is now a private house once more.

Fyfield Manor is a private residence. It can be seen through its entrance gates, next to the parish church.