Stop meeting people who recognize Malcolm.
When they see you these people, they report that you haven't left Kyrandia yet: