Historically, the Grazeley area has been a region of complicated borders and boundary changes. The villages of Grazeley and Poundgreen have always been part of Shinfield parish. Grazeley Green and Goddard's Green are now in Wokefield parish. However, before the end of the 19th century, they were in a detached part of Sulhamstead Bannister parish called Sulhamstead Bannister Lower End. There was also a detached section of Sulhamstead Abbots parish around Grazeley Manor Farm and Burnt House Farm, the north-west portion of which had half of the Royal Ordnance Factory built on it. Despite having few inhabitants, this became the independent parish of Grazeley for a while. It is now also incorporated into Wokefield parish. The little village church was built at Grazeley in 1850 but was converted into a domestic home in 2017.

The manors of Diddenham and Moor Place are at Grazeley. The former was first established as a small Anglo-Saxon settlement at Dydda's Water Meadow. Dydda was the name of the father of St Frideswide, the patron saint of Oxford. He was a minor king in the Oxfordshire/Berkshire area in the 7th century, but whether this was the same man is unknown. The manor was owned by the De Diddenham family in the 13th century, and later passed to the Woodcocks who also owned Moor Place at Poundgreen.

The manor house at Moor Place stood just east of Poundgreen. Its successor was called Grazeley Court. This house was purchased, with the winnings from an Irish lottery ticket, by Doctor George Mitford for himself and his family, which included his daughter, the authoress Mary Russell Mitford. He totally rebuilt the house, renaming it Bertram House, but he had a serious gambling habit and Mary had to turn to writing to support the family. At Bertram, she wrote:

• Miscellaneous Verses (1810)
• Christine (1811) and
• Blanche (1813)