Repository navigation
.NET Core 2.1 SocketsHttpHandler does not use Negotiate / SPNego #27415
Description
Overview
While testing PowerShell Core 6.1 I ran into an issue with not being able to authenticate to a Kerberized REST API running on Linux unless I disable SocketsHttpHandler.
PowerShell/PowerShell#7801
It seems like that when the server responds with both Negotiate and NTLM, the SocketsHttpHandler picks NTLM which in my case results in a 401 as the service in question is really expecting Negotiate / SPNego and is not working with NTLM.
As requested by @karelz in https://github.com/dotnet/corefx/issues/30166
I've reproduced it on the daily builds without PowerShell Core involved and same results, so submitting a new issue for this.
Expected result
When server sends multiple auth schemes like Negotiate and NTLM, pick the strongest one which in this case is Negotiate.
Dotnet Info
C:\dev\test\httpclient-spnego\test2>dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.403-servicing-009270
Commit: def6c5f48d
Runtime Environment:
OS Name: Windows
OS Version: 10.0.15063
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.403-servicing-009270\
Host (useful for support):
Version: 2.1.5-servicing-26911-03
Commit: efdba896f7
.NET Core SDKs installed:
2.1.201-preview-007614 [C:\Program Files\dotnet\sdk]
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.400 [C:\Program Files\dotnet\sdk]
2.1.402 [C:\Program Files\dotnet\sdk]
2.1.403-servicing-009270 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5-rtm-31008 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5-rtm-31008 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5-servicing-26911-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
Example repro
var handler = new HttpClientHandler
{
UseDefaultCredentials = true,
AllowAutoRedirect = true,
};
using (var client = new HttpClient(handler))
{
var res = client.SendAsync(new HttpRequestMessage(HttpMethod.Get, uri)).GetAwaiter().GetResult();
System.Console.WriteLine(res);
}Result: 401
HTTP traffic from packet capture
GET / HTTP/1.1
Host: mykerberossite.lab.local
HTTP/1.1 401 Unauthorized
Date: Mon, 17 Sep 2018 21:31:42 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 0
GET / HTTP/1.1
Authorization: Negotiate ****
Host: mykerberossite.lab.local
HTTP/1.1 401 Unauthorized
Date: Mon, 17 Sep 2018 21:31:42 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: NTLM
Content-Length: 0
Workaround is to disable SocketsHttpHandler
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
var handler = new HttpClientHandler
{
UseDefaultCredentials = true,
AllowAutoRedirect = true,
};
using (var client = new HttpClient(handler))
{
var res = client.SendAsync(new HttpRequestMessage(HttpMethod.Get, uri)).GetAwaiter().GetResult();
System.Console.WriteLine(res);
}result: 200
HTTP Traffic
GET / HTTP/1.1
Connection: Keep-Alive
Host: mykerberosite.lab.local
HTTP/1.1 401 Unauthorized
Date: Mon, 17 Sep 2018 21:30:27 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
GET / HTTP/1.1
Connection: Keep-Alive
Host: mykerberossite.lab.local
Authorization: Negotiate ***
HTTP/1.1 200 OK
Date: Mon, 17 Sep 2018 21:30:27 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate ***
Cache-Control: no-cache
Expires: -1
Content-Type: text/plain;charset=UTF-8
Content-Length: 103
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Activity
davidsh commentedon Sep 22, 2018
Your attached output from the wire trace doesn't show that. The client-side HTTP is picking Negotiate.
However, the server is still not authenticating and returning back a 401.
CJHarmath commentedon Sep 22, 2018
Hmm, could be a copy paste error. Will re-test on Monday and update.
CJHarmath commentedon Sep 27, 2018
Hey, sorry for the slow response...
So I've checked again and it's indeed using NTLM with that Negotiate response ( NTLM_NEGOTIATE).
It was just not obvious with my *-ed out HTTP traffic as it didn't show the size of that blob.
NTLM is way smaller than Kerberos so you can just eye ball it.
Wireshark can actually dissect and show that it's NTLMSSP_NEGOTIATE as show below.
And here is the snip when I disable SocketsHttpHandler
davidsh commentedon Sep 27, 2018
So, what you're really saying is that the HTTP stack correctly responded with Negotiate scheme. But Negotiate ended up "negotiating" NTLM instead of Kerberos.
That happens a lot when the requirements for a valid Kerberos infrastructure don't exist. For example, if the client machine is not joined to the Windows Active Directory (or Linux Kerberos) domain of the server, or timestamps aren't matching etc.
In your example, the server is a Linux machine using Kerberos. I'm assuming that the Linux machine is also the Kerberos ticketing server, or is there a separate machine for that?
Usually, this kind of problem is a configuration problem with the machines and not a problem with the client-side HTTP stack since it is picking Negotiate scheme correctly.
cc: @wfurt @geoffkizer @karelz
CJHarmath commentedon Sep 27, 2018
TLDR: Unlike the rest of the Windows web clients (browsers, .NET full, etc) SocketsHTTPHandler is not canonicalizing the given host when trying to request the SPN which breaks Kerberos if the Url has a CNAME and the SPN is only on the DNS A record of the host.
Details
While writing a pretty long reply linking to docs on MSDN about how Kerberos is implemented on windows in terms of SPNego's selection of auth mechanisms, etc etc and preparing the example traces I just found the issue why it's not working.
phew.
This is what's happening in very high level when it's working as expected ( after purged krb tickets and flushed DNS cache - which actually helped me nailing it down - always purge and always flush!)
And this is what's happening with SocketsHttpHandler
So the reason SocketsHTTPHandler is not working is because it's trying to find the SPN for the CNAME instead of canonicalizing it with a forward lookup.
While Windows SSPI is not canonicalizing, all applications are. IE, Firefox, Chrome, .NET web clients, etc are all canonicalizing.
i.e.: coming from .NET Full this is a breaking change.
So SocketsHTTPHandler would need to stand in line and stay consistent.
Perhaps allow a flag to not canonicalize but have the default to do whatever IE,Chrome, etc .NET full web is doing.
Linux
On Linux this is configured via krb5.conf so SocketsHTTPHandler should honor that setting.
rdns = true|false
davidsh commentedon Sep 27, 2018
Thanks for the additional details.
SocketsHttpHandler on Windows uses the Windows SSPI libraries for doing Negotiate and NTLM protocols. I do not think it is something that is directly controllable by SocketsHttpHandler. It would need to be investigated further.
We have tested Linux clients against a Windows server/ActiveDirectory domain. We've demonstrated that Negotiate scheme will use Kerberos if all the machines are configured properly. See: #26418.
But we have not extensively tested a Windows client using Negotiate (Kerberos) into a Linux server environment.
mattpwhite commentedon Sep 27, 2018
In Linux, canonicalization is controlled by configuration knobs in
krb5.conf. Applications are explicitly not supposed to do any kind of canonicalization there and rely on the GSSAPI implementation and its configuration.In Windows, SSPI does not canonicalize, and AFAIK, there are no global knobs to turn. But many applications, including all of the major browsers, WinHTTP (and thus legacy .NET HTTP clients), and most (but not all) OS components do forward canonicalize CNAMEs prior to calling into SSPI. People/things now generally expect that behavior, witness the outcry when Chrome accidentally stopped pre-canonicalizing recently: https://bugs.chromium.org/p/chromium/issues/detail?id=872665
CJHarmath commentedon Sep 27, 2018
My guess is that this will reproduce on windows->windows as well ( haven't tested it yet).
i.e.: Setup IIS with negotiate:kerberos as the only windows auth provider. Configure the Restrict NTLM GPO, then have an SPN for for the A and use a CNAMe to access it.
If configured correctly SockesHttpHandler will break as it will try the wrong SPN then can't use NTLM while the legacy .NET client will work as is.
davidsh commentedon Sep 27, 2018
So, this scenario does work using WinHttpHandler (WinHTTP) on the client-side? So, did you turn off SocketsHttpHandler (via AppContext switch for example) and demonstrate that the scenario works?
See: https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1.0.md
If this works with WinHttpHandler, then it should be possible to fix it for SocketsHttpHandler. WinHttpHandler uses native WinHTTP which uses the same Windows SSPI libraries as SocketsHttpHandler for doing Negotiate and NTLM.
davidsh commentedon Sep 27, 2018
@stephentoub
CJHarmath commentedon Sep 27, 2018
Yep. see my very first post with the below workaround to make this work
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);geoffkizer commentedon Sep 27, 2018
I think we just need to use the canonical host name here (as returned by Dns.GetHostEntry) instead of the hostname in the uri. Correct?
https://github.com/dotnet/corefx/blob/2c8be59ed4405dbdd91777f79763cb2f1384729e/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs#L80
mattpwhite commentedon Sep 28, 2018
Almost. That would handle CNAMEs and partially qualified names of As (that become fully qualified when the OS resolver appends one of the configured search suffixes). But...
davidsh commentedon Oct 19, 2018
@mattpwhite Thank you for the added details regarding CNAMEs and the issues with LLMNR, NETBIOS, etc.
Since you observe the correct behavior in .NET Framework, we will look at that implementation to see where it differs from .NET Core SocketsHttpHandler. That will give us more insight into the correct implementation.
14 remaining items
karelz commentedon Nov 14, 2018
@csharmath we do not port changes to servicing branches unless there is a very good reason - i.e. impact on larger set of customers, without reasonable workaround. Is that the case here?
CJHarmath commentedon Nov 14, 2018
It's hard for me to tell the impact of this, but surely impacts all shops using Negotiate with CNames.
Plus this is a breaking change for Negotiate auth introduced with SocketsHttpHandler requiring a workaround to disable it and loosing on perf.
The workaround is to either use the DNS A record or disable SocketsHttpHandler (more preferable in cases when CNames can change).
or set the env var
Once set, these settings have a potential to be easily forgotten to be undone by dev teams after moving to 3.0 and missing out on the new perf improvements (unless these settings will be ignored with 3.0).
karelz commentedon Nov 14, 2018
@csharmath correct, disabling SockertsHttpHandler is not something we recommend. However, porting every fix into servicing would basically make the servicing branch new master (incl. instability, lower quality, higher chance of other regressions, etc.)
If we hear from more customers, we can consider porting it back.
Does it have specific impact on your environment(s)? Is the first workaround reasonable / acceptable in the meantime for you?
CJHarmath commentedon Nov 14, 2018
To be clear I wasn't trying to propose to port all fixes, but I would consider anything security related as important to evaluate for backporting consideration.
The impact of this issue could be a downgrade from Kerberos to NTLM.
If NTLM is configured and allowed, then it's very unlikely that most of your customers will notice this unless they explicitly audit NTLM ( which we do) as it's still authenticating, but not with the recommended and desired mechanism.
If the webserver is running on Linux or in an environment where NTLM is explicitly disabled, then it breaks auth so those customers will notice it immediately.
If a company already made investments to setup and use Kerberos, but for reasons did not completely disable NTLM then this change weakens their security with them potentially not even realizing it.
If you put your security hat on, then it's a less optimal situation.
So my angle on this is security and would recommend to review your position based on that.
Let me paste from MSDN:
https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/network-security-restrict-ntlm-ntlm-authentication-in-this-domain#security-considerations
davidsh commentedon Nov 14, 2018
There is another workaround/mitigation that can be considered.
This problem only occurs if a CNAME is used in the URI and that CNAME is not registered as an SPN in Kerberos. So, the workaround is to register this additional CNAME SPN in Windows Active Directory / Kerberos environment.
NigelWhatling commentedon Nov 15, 2018
I will be forced to deploy the current project I am working on with SocketsHttpHandler disabled. I'm delivering a Web API that needs to pass through delegated auth to a SAP OData service.
Fix SPN used for Negotiate authentication (dotnet#33426)
NigelWhatling commentedon Jan 3, 2020
Hit this issue in another project for another client. Forced to disable SocketsHttpHandler. :/
karelz commentedon Jan 3, 2020
So practically, 1-2 affected projects per year so far.
Did you consider updating to .NET Core 3.0 or 3.1 (if you prefer LTS versions)? It is fixed there.
NigelWhatling commentedon Jan 3, 2020
Fair enough. Small numbers. For me, that represents the last two projects I've worked on.
I'm not making the call on version for this one. And the workaround is easy enough once you finally figure out (remember) where the problem lies. Just seems weird to leave the bug there.
karelz commentedon Jan 3, 2020
Isn't every bug weird to have? That is their definitions - they are bugs, unexpected behaviors. Fixing is usually done based on wide-spread impact.
You can use existence of this bug as reason to upgrade to newer version - take it up with decision makers. If they don't care ... the bug is likely not such high priority for them, or get info from them why they cannot upgrade.
NigelWhatling commentedon Jan 4, 2020
I guess it's partly frustration on my part. Both projects have burnt many hours over days going back and forth with infrastructure people to try to get the magic soup of SPNs, etc right for Kerberos to work in their environment. Something that certainly isn't my specialty and in these cases don't have the level of access to directly tinker with myself.
Bugs like this one are fun because they don't (to me, at least) immediately point to the code. Authentication just doesn't work. So you go back and beg people to triple-check SPNs and firewalls and who-knows-what-else to figure out what part of the environment is not configured right. Until someone finally ran across this issue in GitHub, it never occurred to me that there might be a new network handler in play (by default) that just didn't do the same thing that the old one did.
Anyway, the bug has been dealt with. It just requires upgrading. Or a workaround if that is not feasible.