In the same year, Sir John sent a petition to Pope Alexander IV, asking that he might build a chapel and keep a chaplain at his manor of Swallowfield, pleading the dangers which he and his family had to encounter in going through the forest to attend mass at the minster church in Sonning. These included both winter floods and highway robbery! Windsor Forest was formerly of much greater circuit than it is now, extending into Buckinghamshire and Surrey and over the whole of the south-eastern part of Berkshire as far as Hungerford. The circuit as described in Roques' map appears to be about fifty-six miles, including the whole parish of Swallowfield. The forest was a refuge for robbers and, at this time, most famous amongst them were names like Adam de Gurdon and Richard Siward. The Pope granted Sir John his petition and issued two bulls, from Anagni, addressed to the Bishop of Salisbury, in whose diocese Swallowfield was then situated.

Thus armed with Papal authority, Sir John Le Despenser, in 1256, built the church of All Saints, which now stands on the edge of Swallowfield Park. The style of architecture, of some parts of the edifice, however, indicates a much earlier date (circa 1135) and it seems, therefore, probable that the remains of a ruined church of anterior date may have been used by John Le Despenser in building the present one.

Sir John Le Despenser, followed his elder half-brother and overlord Roger St. John, in joining the Barons in their great civil struggle against King Henry III and they both attended the celebrated Council summoned at Oxford in 1258, and commonly known as 'the Mad Parliament'. Early in 1264, Sir John and his young son, Adam, formed part of the force left by Simon de Montfort to hold Northampton. It was, however, taken when the Royalist monks from a Cluniac monastery abutting on the walls of the town, undermined the said fortifications, putting wooden props as a temporary support. By this means the Royalist forces made an easy entrance, whilst a feigned assault was made on the other side of the town. Sir John Le Despenser and his son were taken prisoners and placed in the custody of Reginald Waterwill; but three months later, after the victory at Lewes, they were released "by the King's writ, to Roger Mortimer, who was ordered to bring them among other prisoners to London to be set at liberty." Early the following year, according to the household roll of Eleanor, the wife of Simon de Montfort and daughter of King John, this Royal lady sent a present of wine, from her demesne of Odiham Castle (Hampshire), to Sir John's wife, "the lady of Swalfelde".

In 1265, Roger St. John was killed at the rebellious Battle of Evesham. His estates were confiscated and "Swaluefelde and one messuage and one carucate of lands in Beaumys with woods and rents and all appurts" were granted to Roger de Leybourne, and in a Charter Roll we read that "only a messuage in Beaumys remained the property of Sir John Le Despenser", his half-brother. The latter was probably therefore obliged to retire to nearby Beaumys Castle, though he did hold estates elsewhere. He died in 1274.

During the restoration of Swallowfield Church in 1869, some remains, supposed to be those of Sir John Le Despenser, were discovered about halfway between the south door and the chancel screen, in a stone coffin surmounted by a large flat cross. The skeleton was quite perfect, but the head was separate and outside the coffin, which evidently had previously been disturbed. A quantity of cloves were found, surrounding the body, and the remains of a wooden dish, which had probably held salt, rested on the breast. The lid was carefully refixed and cemented and the coffin was placed in its present position under the west window.

In an inquisition, taken in 1276, the Jurors say that "John Dispensator has made encroachments in the vill of Shenyngefeld" (ie. has enclosed three tenements in Shinfield). By an inquisition, taken the next year, at Gertre in Leicestershire, he was found to have died possessed of the manor of Beransby and the moiety of Wigan-de-la-Mare and several other lands, as also the hundred of Beaumaner, held of Hugh de Spencer in socage and of the house and park there. And by another inquisition, taken at his house at Martley in Worcestershire, he is said to have died possessed of that manor with the advowson of the church, which his father had by gift of Henry III. In right of his first wife, Joan the daughter of Robert de Lou, Sir John Le Despenser also possessed Castle Carlton and Cavenby in Lincolnshire, but, his wife dying childless, these manors went, at his death, to her cousin, John de Merieth. He married secondly, Anne, but died without children, his heir being Hugh, the son of the Justiciar and the favourite of Edward II.

The name of Spencer still survives in the parish of Swallowfield, both amongst the people and also in the village called 'Spencer's Wood'.