One of the advantages of living in the local village is that I could ask around to see if anyone knew of a kiln near the castle. There are others scattered around the peninsula, all exploiting the same outcrops of Jurassic limestone - for example, there is a well-preserved one in the roadside at the top of the hill at Swordle.
It would be good if someone who knows about these kilns could have a look at it. The idea that it might be the original kiln, dating back to before 1300, is exciting.
For a brief overview of pre-industrial lime kilns, see the English Heritage site here.