DotNetCore 中的 HttpClient.SendAsync - 是否可能出现死锁？

如何解决DotNetCore 中的 HttpClient.SendAsync - 是否可能出现死锁？

我们偶尔会收到一个 TaskCanceledException 调用，根据我们的日志，它在我们为请求配置的超时内完成。第一个日志条目来自服务器。这是该方法在返回 JsonResult（MVC 4 控制器）之前注销的最后一件事。

{
    "TimeGenerated": "2021-03-19T12:08:48.882Z","CorrelationId": "b1568096-fdbd-46a7-8b69-58d0b33f458c","date_s": "2021-03-19","time_s": "07:08:37.9582","callsite_s": "...ImportDatasets","stacktrace_s": "","Level": "INFO","class_s": "...ReportConfigController","Message": "Some uninteresting message","exception_s": ""
}

在这种情况下，请求大约需要 5 分钟才能完成。然后 30 分钟后，我们的调用者从 HttpClient.SendAsync:

抛出任务取消异常

{
    "TimeGenerated": "2021-03-19T12:48:27.783Z","time_s": "12:48:17.5354","callsite_s": "...AuthorizedApiAccessor+<CallApi>d__29.MoveNext","stacktrace_s": "TaskCanceledException    
        at System.Net.Http.httpconnection.SendAsyncCore(HttpRequestMessage request,CancellationToken cancellationToken)\r\n   
        at System.Net.Http.httpconnectionPool.SendWithNtConnectionAuthAsync(httpconnection connection,HttpRequestMessage request,Boolean doRequestAuth,CancellationToken cancellationToken)\r\n   
        at System.Net.Http.httpconnectionPool.SendWithRetryAsync(HttpRequestMessage request,CancellationToken cancellationToken)\r\n   
        at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)\r\n   
        at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)\r\n   
        at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,CancellationTokenSource cts,Boolean disposeCts)\r\n   
        at ...AuthorizedApiAccessor.CallApi(String url,Object content,HttpMethod httpMethod,AuthenticationType authType,Boolean isCompressed)\r\nIOException    
        at System.Net.sockets.socket.Awaitablesocketasynceventargs.ThrowException(SocketError error,CancellationToken cancellationToken)\r\n   
        at System.Net.sockets.socket.Awaitablesocketasynceventargs.GetResult(Int16 token)\r\n   
        at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap,ValueTask`1 task,Int32 min,Int32 initial)\r\n   
        at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter,Memory`1 buffer)\r\n   
        at System.Net.Http.httpconnection.SendAsyncCore(HttpRequestMessage request,CancellationToken cancellationToken)\r\nSocketException","Level": "ERROR","class_s": "...AuthorizedApiAccessor","Message": "nothing good","exception_s": "The operation was canceled."
}

鉴于在发出请求的过程中我们阻止了异步调用（.Result - 遇到了不支持异步的棕地缓存实现），我的第一个猜测是我们遇到了死锁，如Stephen Cleary。但是调用者是一个 dotnetcore 3.1 应用程序，所以这种死锁是不可能的。

我认为，我们对 HttpClient 的使用非常标准。这是最终进行调用的方法：

private async Task<string> CallApi(string url,object content,bool isCompressed)
{
    try
    {
        var request = new HttpRequestMessage()
        {
            RequestUri = new Uri(url),Method = httpMethod,Content = GetContent(content,isCompressed)
        };

        AddRequestHeaders(request);

        var httpClient = _httpClientFactory.CreateClient(HTTPCLIENT_NAME);
        httpClient.Timeout = Timeout;

        AddAuthenticationHeaders(httpClient,authType);

        var resp = await httpClient.SendAsync(request);
        var responseString = await (resp.Content?.ReadAsstringAsync() ?? Task.Fromresult<string>(string.Empty));

        if (!resp.IsSuccessstatusCode)
        {
            var message = $"{url}: {httpMethod}: {authType}: {isCompressed}: {responseString}";
            if (resp.StatusCode == HttpStatusCode.Forbidden || resp.StatusCode == HttpStatusCode.Unauthorized)
            {
                throw new CustomException(message,ErrorType.AccessViolation);
            }

            if (resp.StatusCode == HttpStatusCode.NotFound)
            {
                throw new CustomException(message,ErrorType.NotFound);
            }

            throw new CustomException(message);
        }

        return responseString;
    }
    catch (CustomException) { throw; }
    catch (Exception ex)
    {
        var message = "{Url}: {HttpVerb}: {AuthType}: {IsCompressed}: {Message}";
        _logger.ErrorFormat(message,ex,url,httpMethod,authType,isCompressed,ex.Message);
        throw;
    }
}

我们对这种行为的理论一无所知。我们已经看到，在几百个成功请求中，每个月有 3-5 次任务被取消，所以这种情况是间歇性的，但绝非罕见。

我们还应该在哪里寻找类似死锁的根源？

更新

可能需要注意我们使用的是标准 httpclienthandler。最近添加了重试策略，但我们不会在长时间运行的 POST 上重试，这就是上面的场景。

builder.Services.AddHttpClient(AuthorizedApiAccessor.HTTPCLIENT_NAME)
    .ConfigurePrimaryHttpMessageHandler(_ => new httpclienthandler()
    {
        AutomaticDecompression = System.Net.DecompressionMethods.Deflate | System.Net.DecompressionMethods.GZip
    })
    .AddRetryPolicies(retryOptions);