You will lose followers only to an extent and that is why extra followers are added.

So don't worry