User picker component with group filter doesn't populate all users

I have used pickerfielusers in which the group filter is mentioned as "All Groups". 

"All Groups" has sub groups inside it called Group A, B , etc.

The security of All Groups is "Group Admin" where no member of sub groups are available.

Now, as only a member of Group A, when I access the picker, the picker shows me suggestions from only Group A and not the other group members. 

All the groups here are "Public", "Closed" and "Low".

Is it security of this "All Groups" blocking them from seeing each other?