The Glanusk family owned Hay Castle while William Latham Bevan and his family lived here - they were cousins and there was no vicarage in Hay. When William was made Archdeacon of Brecon the Bevan family moved out and the building was updated and redesigned by the architect W D Caroe. The entrance hallway was opened up, panelled walls removed and a line of faux Jacobean pillars were put in their place. The Lady Dowager Glanusk moved back in and lived here as her home.
In about 1928 Lady Glanusk employed a young lad called Bramwell Bradley as a house servant. Bramwell was living with his grandparents in Hay and was about 13 at the time that he started work. He recalled his time working for Lady Glanusk in a memoir written when in his 80s - a piece of writing which gives a fascinating insight into life in Hay Castle in the first part of the 20th century.
Bramwell worked for Lady Glanusk untill her death in 1938 when Hay Castle was emptied of furniture and left empty for rental.