There was supposed to have been a Romano-British settlement of some kind near the site of Poughley Priory, but references to the finding of such remains are very dubious and much over-rated. However, there is some possibility that part of the name of the parish does in fact originate in this era.  Chaddleworth is usually said to be Saxon for 'Ceadela's Farm', but this personal name is otherwise unknown. The initial Chadd- could well be a Saxon interpretation of the Celtic word, Coed or 'Wood'.

The manor was, for most of the middle ages, owned by the Priory of Amesbury in Wiltshire. It had been given to this monastic establishment by Queen Eleanor in 1284 as she wished to help support Eleanor of Brittany, one of the nuns. After the Dissolution, it eventually came into the hands of William nelson, Chief Prothonotary of the Court of Common Pleas. His family continued to own the manor for over two hundred and fifty years. George Kerr Nelson built the present Chaddleworth House in 1809, but his son sold up to the Wroughtons of nearby Woolley.

There is a curious legal custom in Chaddleworth parish. If a copyholder’s widow remarried, she forfeited her rights to her husband’s lands. However, if she rode into court backwards on the back of a large black ram repeating certain ludicrous lines that end in a petition to give back her lands, the manor steward would be obliged to return them. This was also true of Enborne.

The village has a fascinating round house with a thatched roof. It was built by the local schoolmaster, James Whistler, in the mid-18th century. Being worried about French invasion, he had his pupils dig out huge cellars for the whole village to hide in! Later, when Mr. Whistler's daughter was resident, the living room became the Wesleyans first preaching room.

The southern portion of the parish is called Poughley. This is a strange name, said to be Saxon for 'Clearing by a feature resembling a bag' which sounds very unlikely. Poughley Priory was founded in 1160 by Ralph De Chaddleworth on the site of an old hermitage in the parish. It stood in a wood, then called Clenfordmere or Ellenfordmere. The place must have been quite important for Wolsey swapped the priory with the Abbot of Westminster for St. James’ Park! Its 13th century west range is now incorporated into the facade of Poughley Farm where a small reset figure of a monk can also be seen. A stone coffin-lid of Robert the Prior can be seen at East Hendred Church, while the elaborate zig-zag Norman doorway of Chaddleworth Church is said to have been brought from the priory. This, however, seems rather doubtful.

The whole area is sometimes called Chaddleworth-cum-Woolley, in reverence to the northern part of the parish, called Woolley, meaning 'Wolf Clearing'! This was a medieval village that was deserted,  like so many, because of plague and unemployment. Only a farm and manor remain. However, it appears to have originally had its own motte and bailey castle, now only visible as a cropmark showing a 46m wide ring work. A collection of 12th century pottery found there might suggest it was built during the Civil War of King Stephen's reign as a southern outpost for Richard de Rivers and his wife, Alice Peverel. It was apparently used as a Royal hunting lodge by King John who visited in 1207. It is not clear when the manor house moved to the present site. Woolley Park is a late 17th century mansion with a fine walled garden. It was the home of the Tippings and the Wroughtons, whose family 'pew' and wall monuments can be seen in the parish church. The area lost its own little chapel in 1759. A second family pew was for the Blandys of Oakash, a large house in the parish dating back to Elizabethan times that never became a manor.