Triple hop issue with ASP.NET delegation Part I: Our Windows XP Pro desktops
Last Friday we had an issue in production: we have a very simple web application with one single page on our intranet that consumes an array of web services. These web services talk to a back end SQL Server.
All in all this is a very typical scenario and like most companies with .NET technology we have web applications using ASP.Delegation in the intranet, the only particular point regarding this web page is that it is called inside an old legacy windows application (not a .NET app). For remote users, this old legacy application is used via Terminal Services.
For our remote users also, the application didn't work and our DBA was registering a bunch of anonymous requests coming from the web server box...
On the other hand we set up our web services tracing to debug and were able to see the end user credentials on each HTTP request, so the end user had managed to authenticate using Integrated Windows Security on our web box and the web service trying to open a SQL connection to the back end.
We used impersonation and Integrated Windows Authentication on our web application and web services (this is an intranet after all). ASP.NET impersonation gave us the chance to restrict the access on the back end based on AD groups and at the same time gave us the ability to audit the user's actions to a very fine grained degree (user name).
The PROBLEM with our Windows XP Pro desktop users
The application worked for our desktop users if and only if they had logged off and on their desktops in the past 48 hours. If the desktops users hadn't logged on for a while, like me, that I lock my computer instead of logging myself off, the application didn't work either and the sql box passed an anonymous login attempt back to our web tier. The web services then passed a SOAP Exception with the NT Service/Anounymous user error message to our web app...
System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.Data.SqlClient.SqlException: Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'.
At first we thought it was the same problem, but it turns out the TS users couldn't use the application even when they logged of and back on, not even when the TS server was restarted, hrm....
By dividing and conquering we applied the kerbtray.exe tool on our web server and one of the desktops and enabled Kerberos logging on both boxes. We noticed that when the application worked the user logged in the web server box used Kerberos, but after a few days the logging defaulted to NTML.
SOLUTION for the Windows XP Pro Desktops
It turns out this was a bug in the kerberos.dll running on Windows XP SP2, SP3 has this problem solved. More information can be found on this MSDN thread. Also the hotfix for Windows XP Professional SP2 can be found on this Microsoft Knowledge Base article. Although this article describes a different problem the hotfix provided here contains the fixed kerberos dll.
There are quite a bit of articles regarding ASP.NET delegation
And quite a few MSDN forum threads, like this one I initiated and has a heated discussion with the moderator, my fault most of it.
The best resources I have found so far, and I hope this digested summary will help you if you have the same double/triple hop issue, are:
Ken Schaefer's blog post regarding IIS and Kerberos Part 5 - Protocol Transition, Constrained Delegation, S4U2S and S4U2P.
Keith Brown's article on MSDN: Credentials and Delegation
and
nunos's Blog: Concerning the credentials double hop issue
and the best of all is a webcast by Yung Chou *all kudos to his explanation of Protocol Transition*
MSDN Webcast: Getting Delegation to Work with IIS and ASP.NET:
The Ins and Outs of Protocol Transition (Part 1 of 2) (Level 300)
This webcast specifically helped us troubleshooting and fixing the second part of our problem, our failed connection when the end users connected remotely via terminal servers.
I'll post more of the problem and the resolution on Part II...
...stay tuned.
All in all this is a very typical scenario and like most companies with .NET technology we have web applications using ASP.Delegation in the intranet, the only particular point regarding this web page is that it is called inside an old legacy windows application (not a .NET app). For remote users, this old legacy application is used via Terminal Services.
For our remote users also, the application didn't work and our DBA was registering a bunch of anonymous requests coming from the web server box...
On the other hand we set up our web services tracing to debug and were able to see the end user credentials on each HTTP request, so the end user had managed to authenticate using Integrated Windows Security on our web box and the web service trying to open a SQL connection to the back end.
We used impersonation and Integrated Windows Authentication on our web application and web services (this is an intranet after all). ASP.NET impersonation gave us the chance to restrict the access on the back end based on AD groups and at the same time gave us the ability to audit the user's actions to a very fine grained degree (user name).
The PROBLEM with our Windows XP Pro desktop users
The application worked for our desktop users if and only if they had logged off and on their desktops in the past 48 hours. If the desktops users hadn't logged on for a while, like me, that I lock my computer instead of logging myself off, the application didn't work either and the sql box passed an anonymous login attempt back to our web tier. The web services then passed a SOAP Exception with the NT Service/Anounymous user error message to our web app...
System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.Data.SqlClient.SqlException: Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'.
At first we thought it was the same problem, but it turns out the TS users couldn't use the application even when they logged of and back on, not even when the TS server was restarted, hrm....
By dividing and conquering we applied the kerbtray.exe tool on our web server and one of the desktops and enabled Kerberos logging on both boxes. We noticed that when the application worked the user logged in the web server box used Kerberos, but after a few days the logging defaulted to NTML.
SOLUTION for the Windows XP Pro Desktops
It turns out this was a bug in the kerberos.dll running on Windows XP SP2, SP3 has this problem solved. More information can be found on this MSDN thread. Also the hotfix for Windows XP Professional SP2 can be found on this Microsoft Knowledge Base article. Although this article describes a different problem the hotfix provided here contains the fixed kerberos dll.
There are quite a bit of articles regarding ASP.NET delegation
And quite a few MSDN forum threads, like this one I initiated and has a heated discussion with the moderator, my fault most of it.
The best resources I have found so far, and I hope this digested summary will help you if you have the same double/triple hop issue, are:
Ken Schaefer's blog post regarding IIS and Kerberos Part 5 - Protocol Transition, Constrained Delegation, S4U2S and S4U2P.
Keith Brown's article on MSDN: Credentials and Delegation
and
nunos's Blog: Concerning the credentials double hop issue
and the best of all is a webcast by Yung Chou *all kudos to his explanation of Protocol Transition*
MSDN Webcast: Getting Delegation to Work with IIS and ASP.NET:
The Ins and Outs of Protocol Transition (Part 1 of 2) (Level 300)
This webcast specifically helped us troubleshooting and fixing the second part of our problem, our failed connection when the end users connected remotely via terminal servers.
I'll post more of the problem and the resolution on Part II...
...stay tuned.
Labels: ASP.NET, Delegation, Kerberos, Windows XP Professional
posted by Lizet Pena de Sola at
8:27 PM
1 Comments:
Hi,
I've just read the MSDN forum posts and quite frankly the advice you were being given by so-called experts was, quite frankly, ridiculous !
Your decsription of the problem was entirely accurate and the Moderator's suggestion to use the Network Service account clearly showed that he didn't know what he was talking about.
If you are using delegation you only need to grant the end-users access to the SQL layer - that's the whole *point* of delegation.
We have been experiencing the *exact* same issue where users stop being able to obtain Kerberos tickets after a period of time. The hotfix you mentioned fixes this issue - we certainly didn't have to start granting unecessary permissions to the Network Service account !
The other issue you mentiond - the 'KRB_ERR_RESPONSE_TOO_BIG' event log entry can safely be ignored. This 'error' basically means that a UDP packet has exceeded the 'MaxDatagramReplySize' value configured in the registry and is therefore switching over to TCP. It isn't really an error as such - more of a warning at best.
Post a Comment
<< Home