There are a large number of ways to provide custom permissions in SharePoint Online e.g. so that a sub-site of your site visitors have access to restricted materials.

iSolutions general advice is that you should endeavour to manage permissions either by having different sites e.g. if you have a lot of materials or by having different document libraries e.g. where there are fewer files needing restriction. You can restrict access at folder and individual file level but this is not recommended.

How to restrict a library with different permissions

1. First create the document library that you want to have different permissions to your main site.
2. Note that all users of this restricted document library must already have access to your site as members or visitors.
3. View the document library, click the cog in the top-right and select library settings.
4. Select permissions for this document library
5. In the ribbon at the top of the page select stop inheriting permissions - normally a document library (or list) will have the same permissions as the site it sits in. By stopping this inheritance it means you can change the permissions just for this library (or list). Note that this means changes you make to the permissions of the site might not apply to this library (you can use the check permissions button to confirm if an individual can access the library).
6. Once you've stopped the inheritance you now have two options, you can either add individuals or groups to have specific permissions or you can edit permissions of existing users/groups.
7. To add individuals or groups click the grant permissions button, enter the details of the person or group you want to have access, click the show options button and then select the level of access they should have.
8. To edit existing permissions tick the box next to the existing group/person and then click the edit user permissions button (or remove user permissions if you wish to remove access altogether). On the following page tick the permissions they should have, note that the highest permission applies so if you tick full control and view only they will, in effect, have full control permissions.