Brown family 19th century

According to an old family Bible my ancestors lived in Foresters Falls, Ontario (then Upper Canada) in the mid-19th century. I know that by the 1890's they had moved to Sarnia, Ontario. I have always suspected my forefathers were United Empire Loyalists, though I have no proof. I did, however, once met a woman in Brockville, Ontario who was the spitting image of my sister. This suggests to me that my family had pass through Brockville, leaving some family members there, before moving on to Foresters Falls. 

I know no one in Foresters Falls will be able to remember back to 1850, but, I wonder if there is a family named Brown there or if there are resources so I can look up who lived there in the 1850's.