Queen Elizabeth I's treasurer, Sir Francis Knollys bought the place in 1542 but was not able take possession until over forty years later.  It was at this time that he tore down the old fortified house and built the first great mansion there, on the higher ground where it is likely the park keeper's lodge had previously stood. Sir Francis' nearby homes in Reading and at Rotherfield Greys (Oxfordshire) were his main residences, but, in his later years, seems to have favoured Caversham and is described as of that place in his will. His son, William Knollys, the Earl of Banbury, entertained Queen Elizabeth I and Queen Anne (of Denmark) there.

It later became the home of the Earl of Craven. During the Civil War, his absence in Europe along with his Royalist payroll led to its confiscation and the house was used, for a time, as the luxurious prison of King Charles I. After the War, however, it was in such a bad state of repair that it was pulled down.

There have been a number of subsequent mansions on the site, but unfortunately, they have been susceptible to frequent fire damage! In 1718, Lord Cadogan started to rebuild the house. He was a good friend of the Duke of Marlborough and tried to make Caversham's gardens rival Blenheim Palace. This building was burnt down in the late 18th century and replaced with a smaller house. Major Marsack much enlarged this in the 1780s until it became a fine Greek Temple style mansion. The present building was erected after the great fire of 1850. Its Welsh Iron Baron owner gave it a metal frame, but this did not stop another fire in 1926. The house was then a school, but is now the BBC World monitoring station, as well as the home of Radio Berkshire.