What happens if I configure group A to be able to search for members in group A, but group A is also in “Not permitted group” ?

Search permissions that you configure has more priority than what you set up in “Not permitted Groups”. To make it easier to understand, if there are conflicts between two configurations, we will prioritize search permissions. In your case, users in Group A are able to search other members in Group A.