HttpClient spike in memory usage with large response

Refresh

December 2018

Views

548 time

3

I'm working on a console app that take a list of endpoints to video data, makes an HTTP request, and saves the result to a file. These are relatively small videos. Because of an issue outside of my control, one of the videos is very large (145 minutes instead of a few seconds).

The problem I'm seeing is that my memory usage spikes to ~1 GB after that request is called, and I eventually get a "Task was cancelled" error (presumably because the client timed out). This is fine, I don't want this video, but what is concerning is that my allocated memory stays high no matter what I do. I want to be able to release the memory. It seems concerning that Task Manager shows ~14 MB memory usage until this call, then trickles up continuously afterwards. In the VS debugger I just see a spike.

I tried throwing everything in a using statement, re-initializing the HttpClient on exception, manually invoking GC.Collect() with no luck. The code I'm working with looks something like this:

consumer.Received += async (model, ea) =>
{
    InitializeHttpClient(source);
    ...
    foreach(var item in queue)
    {
        await SaveFileFromEndpoint(url, fileName);
        ...
    }
}

and the methods:

public void InitializeHttpClient(string source)
{
    ...
    _client = new HttpClient();
    ...
}

public async Task SaveFileFromEndpoint(string endpoint, string fileName)
{
    try
    {
        using (HttpResponseMessage response = await _client.GetAsync(endpoint))
        {
            if (response.IsSuccessStatusCode)
            {
                using(var content = await response.Content.ReadAsStreamAsync())
                using (var fileStream = File.Create($"{fileName}"))
                {
                    await response.Content.CopyToAsync(fileStream);
                }
            }
        }
    }
    catch (Exception ex)
    {

    }
}

Here is a look at my debugger output:

memory spike

I guess I have a few questions about what I'm seeing:

  1. Is the memory usage I'm seeing actually an issue?
  2. Is there any way I can release the memory being allocated by a large HTTP request?
  3. Is there any way I can see the content length of the request before the call is made and memory is allocated? So far I haven't been able to find a way to find out before the actual memory is allocated.

Thanks in advance for your help!

1 answers

2

Если вы используете HttpClient.SendAsync(HttpRequestMessage, HttpCompletionOption)вместо GetAsync, вы можете поставить HttpCompletionOption.ResponseHeadersRead, (в отличие от по умолчанию ResponseContentRead). Это означает , что поток ответ будет передан обратно к вам до того , как тело ответа скачал (а не после него), и потребует значительно меньше буфера для работы.