Recently I have seen an issue on a SharePoint site which was not showing the default security groups and due to that site owners were not able to approve the access requests. I investigated the issue and came to know that site do not have default security groups.
This happened because site was migrated from an older version to SharePoint Online, and user has removed some of the security groups and mistakenly deleted some groups which are the default groups for few sites.
In above screenshot you can see the Access Request Settings page showing the option that No site owner group but the site has unique permissions and there was a security group with full access but that was not the site default owner group.
Now the issue was to restore the default site groups for Owner, member and visitors. The easiest way is to inherit the permissions from parent site and then break the inheritance which can take you to the page to create default security groups. But in this way, you might lose the existing custom security on list and libraries.
The proper solution for this is to access the Permission setup page, you cannot access that page through some link, you can only access it through URL /_layouts/15/permsetup.aspx
Add this url after your site url name like https://<mysiteurl>/<subsite>/_layouts/15/permsetup.aspx. This will take you the site permission page to setup the default security group. You can create new groups or can use the existing one. In this way you will not lose any of the permissions.