The earliest recorded Ilsley was Reginald de Ilsley who lived in the mid-13th century. He was the sub-tenant of the Manor of Ilsley in East Ilsley, based around what became Manor Farm. It is possible that he was a descendant of Roger, the sub-tenant of Henry de Ferrers, recorded in the Domesday Book (1086). Sir Robert Ilsley, who lived about a hundred years later, appears to be the main founder of the family fortunes. Perhaps he was a great-grandson of Reginald.

Sir Robert  married Isabel, the widow of Sir Richard Handlo of Boarstall Castle in Buckinghamshire. She was the daughter of John, 2nd Baron St. Amand, of West Woodhay Castle and his wife, Margaret, daughter of Hugh Le Despenser, Earl of Winchester. She was ancestress (through her first husband) of the De la Poles, Jameses and Reades of both Boarstall Castle and Rush Court, near Wallingford. Sir Robert served King Edward III when he invaded Southern Scotland in 1356, retaking Berwick, sacking Haddington and burning Edinburgh in the so-called ‘Burnt Candlemas’. He was probably also involved in other English campaigns in Scotland and France. He was Sheriff of Gloucestershire from 1356 to 60.

The exact descent of the manor of Ilsley is then unclear, although Sir Robert seems to have been followed by William Ilsley and then there were certainly two generations of John Ilsleys in the 15th century. We then come to a second William Ilsley who, though he retained Ilsley Manor, seems to have favoured his secondary estate, Hall Place Farm at Beenham, for most of his life. He had three sons. The youngest, George, settled in Brimpton where he had a large family of mostly daughters. The middle son, John, who was Yeoman of the Longbows to King Henry VIII, took on the Beenham estate and the eldest, Edward, stayed with Ilsley. John and George Ilsley both attended the Herald’s Visitation of 1566, shortly before John’s death. John had his family of six children recorded there, including two sons. The youngest, another John, lived in Aldermaston, followed by his son, William, in Wokingham. The family then returned to Beenham for another two generations. The eldest son of John of Beenham, also called William, stayed in Beenham. He was present at the famous mass at Lyford Grange in 1581, after which St. Edmund Campion was captured. William was amongst those arrested that night. He seems to have been banished abroad, but later returned and was caught smuggling Catholic books from Amiens in Picardy to Corpus Christi College in Oxford. William's branch of the family stayed in Beenham for another three generations.

The head of the family, Edward Ilsley, though he continued to own Ilsley Manor but, by the 1520s, had also acquired the sub-tenancy of a second estate, the manor of Crowmarsh Gifford in Oxfordshire, just over the bridge from Wallingford. The manor house may have been what is now Nos. 17 & 19 The Street, but the family lived, instead, at Newnham Farm in Newnham Murren. John Ilsley (commonly spelt Hilsey), the Bishop of Rochester, may have been Edward's younger son. He was certainly one of this family, for the staff of his predecessor, the martyred Bishop John Fisher, which he acquired, was owned by them for several generations before it was given to the Eystons at Hendred House. At the Reformation, Edward’s eldest son, William, would not give up the old religion and the family became prominent Roman Catholic recusants. He married Margaret Stonor of North Stoke, a cousin of the Stoners of Stoner Park, and attended both Oxfordshire Heralds’ Visitations in 1566 & 1574. William’s rather damaged memorial brass figure and inscription (1576) may be seen in Crowmarsh Church; Margaret’s is similarly at East Ilsley Church, although her figure has been completely lost. The couple were succeeded at Crowmarsh by their eldest son, Walter, who married and had a family, but his lands were seized by the Crown, under the recusancy laws, in the 1580s & 90s. Meanwhile his younger brother, William, lived at Manor Farm in East Ilsley and Littlestoke House in Checkendon, where he was followed by his family for another three generations into the early 1700s. His grandson, Francis, attended the Herald’s Visitation of 1664. Four of Francis’ sisters were nuns at Liege in Wallonia. His eldest son, William, and family, attended the chapel of St. Amand at Hendred House for mass. His younger son and namesake was Father Francis Ilsley. He studied with the Jesuits at Liege at the time of the hysteria surrounding the supposed Popish Plot to assassinate Charles II, as invented by Titus Oates. In 1680, he was examined as a witness for the five martyred Jesuits to help disprove Oates' evidence. In 1696, he was Superior of the Residence of St. Mary (in Oxfordshire).