Oliver Castle was a small tower house, located in the upper Tweed Valley in the Scottish Borders, within the bounds of the village of Tweedsmuir. The castle was originally part of the line of peel towers along the Tweed Valley. It was replaced in the seventeenth century by a house on the same site, and in the 18th century by a house on lower ground.