Found an article from 1830 (Journal Herald) documenting some changes that were happening in the town at the time.
The Hearts of Oak must have been a lot bigger back then, occupying a chunk of what’s now the taxi rank in Boutport Street. That would partly explain why the building is wedge-shaped today – a portion of it was chopped off around 1830