Task Notification Link not working with SAML SSO

Hello Team,

I have SAML SSO successfully configured and running in terms of Authentication but when a notification of task is received and I click on that link it takes asks me whether I am an SSO user or non SSO (which is as per configuration of IdP Choices).

Now the problem, when I click on SSO User 

1) If I am not already logged in previously in another tab, it successfully takes me to the task.

2) If I am not logged in it asks me to enter my credentials --> I enter correct credentials but it gives me 401 unauthorized error.

Can anyone suggest for point #2?