Knights Yard, Castle Cary

It's Monday, so I was working in Castle Cary this morning.
There used to be a castle in Castle Cary, but according to an information board I saw this morning, after it fell into disuse, much of the stone was robbed away and reused in buildings in the town. I wonder if the stonework in this little street began life as part of the castle?