During the Commonwealth, Colonel Whichcote bought the old garden, or vineyard at Windsor Castle, to the south of the upper ward. At the Restoration, he seems to have sold it to the Windsor Corporation, but they made over most of their rights in the property to King Charles II. Charles built a new tennis court on one portion and used much of the rest as the site for a fine house, which was completed before 1678, Verrio having decorated the staircase with "Stories from Ovid." By a deed dated 13th and 14th September 1680, he conveyed this building to the Earl of Dorset and others, in trust, first for Nell Gwynne, and afterwards, for her son, Charles, Earl of Burford and, later, Duke of St. Albans, and his heirs male. It became known as Burford House. A letter written there on Nell's behalf, on 14th April 1684, and addressed to "Madame Jennings, over against the Tub Tavern, in Jermin Street, London," remarked: "There is a sad slaughter at Windsor, the young men taking their leaves and going to France. Though they are none of my lovers, yet I am loath to part with them."

The year before Nell Gwynne’s death, in 1687, she had rented Burford House to Princess (later Queen) Anne and her husband, Prince George of Denmark for five years at a cost of £260 per annum. They, however, gave up the property for the ‘Garden House,’ just to the north, in 1691 when the new owner, the Duke of St. Albans, came of age. This nobleman, who is said to have been very like his father, Charles II, quickly took up residence. He was a fine soldier and therefore in great favour with King William III. He acted as Lord-Lieutenant of Berkshire, and married the fair Diana, daughter and sole heiress of Aubrey de Vere, last Earl of Oxford, and a famous beauty. Lady Hertford described a visit to the widowed Duchess at Burford House many years later. Charles, the eldest of the Duke's eight sons, succeeded to the property in 1726, and subsequently became Constable of Windsor Castle, Warden of the Forest and High Steward of Windsor. In 1749, the house was described as "a stately and handsome seat with beautiful gardens that extend to the park wall…..his Grace is at present making farther improvements by opening a view into the High Street of the town". The Duke died two years later and his son, apparently a great spendthrift, sold Burford House to King George III a quarter of a century later.

Early in his reign, George III chiefly resided at the palaces of Kew and St. James. However, gradually he acquired the habit of coming to Queen Anne’s ‘Garden House’ at Windsor. To the east of it, in a line with the South Terrace, the King, in 1778, began to build the much more extensive ‘Queen's Lodge’; and, to complete the Royal complex, he also purchased the neighbouring Burford House, for £4,000, the following year. It was renamed the ‘Lower Lodge’ and became the home of the younger princesses. Unfortunate refurbishments appear to have included covering the building with stucco, raising and castellating the parapet and bringing forward and modernizing the dormer windows.

After Queen Charlotte’s death in 1818, the Lower Lodge became the property of Princess Sophia who gave it up to her brother, the Prince Regent. During his extensive transformation of the castle and park, including the demolition of the Queen’s (or Upper) Lodge in order to improve the view of the Long Walk, it was expected that the Lower Lodge would also be torn down. It survived, however, and, under its original name of Burford House, was transformed into the married quarters for workers in the Royal Mews. It remains today incased somewhere in the present building, though indications of its 17th century origins are distinctly lacking.

Old Burford House no longer really exists, although parts are incorporated into the present Burford House at the Royal Mews.