Grimsbury Castle sits just south of the village. An ancient hillfort, it was given its name by the Saxons who saw it as a mystical place that must have been built by their chief god, Woden otherwise known as Grim. It was actually occupied by Iron Age man between the third and second centuries bc.

East of the village is the hamlet of Wellhouse. A considerable Roman villa complex was partially excavated there in the mid-19th century. No plans were drawn, but there appear to have been a number of buildings, perhaps forming a farmhouse complex around a courtyard. One building had plain red tessellated floors and a circular cellar at the bottom of a series of steps. Many small objects and bones were recovered, including a perfect decorative vessel of blue and white Durobrivian ware and a bronze armilla. It was certainly occupied in the 3rd century and probably the 4th as well. There were indications that the whole site was sacked and burnt in the late Roman period: the remains of the last inhabitants were found face-down in the mud!

Hermitage was always historically in the parish of Hampstead Norreys. It is supposed to have grown up around the old hermitage of a Saxon or medieval religious recluse. However, until the end of the 19th century, the main village was at Little Hungerford, to the north. The local administrative Hundred of Faircross held its courts between the junctions of Yattendon Road & Marleston Road and Old Street & Long Lane. These four roads appear to have formed the Fair-Cross whose name is still retained in the nearby plantation and farm. It was formed some time before 1256 on the border of the Hermitage part of Hampstead Norreys and Chieveley, out of the Hundred of Rowbury. There is a supposed Norman motte and bailey castle on Oare Common, now within Hermitage, but originally in Chieveley. Almost six hundred years later, Norman knights were replaced by Civil War soldiers, when the Parliamentarian army, under Waller, Cromwell and Skippon, came through Hermitage on their march to avoid Donnington Castle and attack the Royalists from the west during the Second Battle of Newbury.

The church was built in 1835 and was originally known as the 'Adelaide Chapel' after William IV's queen. Hermitage became its own parish, five years later.