La Beche Castle once stood on the site of what is now merely Beche Farm in Aldworth. This was the main residence of the De La Beche family, after whom it was named. They were a well-known family of medieval knights holding many high positions at court since at least 1260. Their grand monuments can still be seen in the church. Known as the "Aldworth Giants", they were possibly erected by the most influential member of the family, Lord Nicholas De La Beche in the 1340s. The most tangible link with the castle however is seal of the latter's sister-in-law that was dug up at the Farm.

The family's heirs, the Langford family, moved their residence to Bradfield Place. La Beche Castle was left to decay and, by the time of the Civil War, was in ruins. These had disappeared by the 1670s.