The land surrounding Waltham Place supported a manor from the earliest days. In the Domsday Book, there is mention of a manor at Waltham, which was held by Chertsey Abbey. It answered for ten hides and included a small church that is thought to be incorporated into the current Church of St. Mary's.

By the 15th Century Waltham Manor, or Waltham, was held in the possession of Roger Waltham and this manor house occupied the site of the present house. From Roger Waltham the manor passed into the hands of the Windsor family.

Renamed the Manor of Windsors it remained in the Windsor family until 1589 when Henry, Lord Windsor, sold it to Roger Higgs. In the inquisition taken upon Roger Higgs' death in 1612 the property was described as 'the capital messuage or farm called the Hill Farm or Windsors'. Higgs held a further messuage called Fishers or Thame and another called Cusmans, with land called Pinknors, Shepferd's Croft and Hawkyns in White Waltham. His son, Roger, then fourteen, was given as his heir, but Windsors, alias Waltham, was bequeathed to John Higgs, also known as Redferne, the eldest son of his wife Alice. He, in turn, conveyed it to John Sharp and others in 1615 to pay his father's debts.

In 1624 William Blake, who conveyed it that year to Roger Gardiner, a citizen of London, held the property. Gardiner rebuilt the house in about 1634 and leased it in 1653 to Sir Paul Neile, the son of Richard Neile, Archbishop of York. Paul's son, William Neile, was the astronomer and mathematician and he built an observatory at Hill House, as Waltham was then called, and died there in 1670 of a broken heart over an unhappy love affair. Following his death Gardiner sold the estate to Edward Pownall and from him it passed to Francis Cherry in 1678. His son then resold the property in 1744 to James Theobold who changed the name from Hill House to Waltham Place. More recent owners were the Ellis and Byam Davies families.