For most of its life, Barkham Manor was known as Barkham House. The first house on the site was probably built in the 1640s to replace the old medieval Barkham Court which stood half a mile away within the moat that survives today next to the church. It was built as a new residence for William Standen II who had inherited the manors of both Arborfield and Barkham from his father's cousin, William Standen I,  in 1639. William evidently wanted to put his stamp on his estates with a large new house of nine fireplaces. Arborfield Hall, which had been the main family home, was only around 40 years old, but the previous manor lord's widow, Mary, was living there anyway; so he put up a new place in Barkham and moved his family in.

Upon Mary Standen's death in 1645, William moved into Arborfield Hall and let his 2nd cousin, William Ball, live at Barkham House. Ball was a well-known political pamphlet writer during the Civil War, but no relation of the Barkham Balls who were erroneously thought to be ancestors of George Washington's mother. It was subsequently let to the Stronghill family from Sonning. However, in 1700, the estate was sold to William Waterman of Southwark in Surrey. His son, Rev. John Waterman, married one of the Kingsmills from Sydmonton Court in Hampshire and this became his main residence as his wife looked after her aging mother there, followed by her mad brother. Barkham must have acted as a welcome retreat. After his death, his daughter, Rebecca, lived at Barkham House with her husband, Lawrence Head Osgood from Winterbourne in Chieveley. In 1757, the house was sold to a Wokingham solicitor, then to the Pitts of Binfield Manor who rented it out to Rev. John Gabriel. In 1783, it was sold again, this time to the Fonnereau brother, Thomas and Henry, who appear to have built the present house. Possibly the old building had fallen into disrepair over twenty years of tenancies.

In 1787, Barkham House estate was purchased by Adm. Hon. John Leveson-Gower of Bill Hill in Hurst as a future dower house for his wife, which is exactly what it became after his death five years later - although he rented it to John Nares in the meantime.