How does garbage collection work in .NET, with generations and the large object heap, and how do you find a memory leak?

Say it out loud, then check

One statement request, every idea

The BMW FS statement worker in time order. Watch where the PDF lives, what the collection checks, and what keeps it alive.

The story

1. Before

   BMW FS customers get a monthly statement: a PDF with their payments and payoff. Our statement service on Azure Container Apps (Azure's service that runs a container for you) renders it from the contract data and used to cache the last few in memory, so a reprint did not re-render. Each PDF is about 130 KB, so it goes on the large object heap (LOH), the heap's shelf for objects of 85,000 bytes or more. In release 4.2 a developer replaced that small cache with a static Dictionary (a key-value map that belongs to the class and lives as long as the app), keyed by (looked up by) contract id plus the current time in ticks (the time as a long number that changes on every call), and called it "a fast cache". A static field belongs to the class itself and lives for the whole process.

   Statement servicerenders one PDF

   pdf: 130 KB

   1/4A request renders one statement PDF: about 130 KB of bytes, held in a local variable.It goes into the static cache. The request ends, but the static field still points at the PDF.The next request builds the same PDF. A new timestamp means a miss, so another entry.By morning the cache holds thousands of PDFs. Memory is at 912 MB of the replica's 1 GiB limit.

   BackPlayNext
2. Why it had to change

   A cache is only useful if a later request can hit it, and a key with the current time can never be hit: every request is new. Nothing removed an entry either, so the static dictionary grew for as long as the process lived. Each PDF was over 85,000 bytes, so it went on the large object heap, the expensive part of the heap. The garbage collector was not broken: the static field is a root (a live pointer the collector starts from), every PDF was still reachable from it (a root could still get to it), and the collector rightly freed none of them. The replica hit its memory limit, was killed, restarted with an empty cache, and the cycle began again. That nightly restart is what hid the bug.

3. How it works now

   Here is what the runtime does with that heap, and why calling the collector more often would not have helped. New small objects start in gen 0. When gen 0 fills, a collection checks the roots and frees the unreachable young objects; the survivors move to gen 1. An object that survives again goes to gen 2, which is checked rarely. A big object, 85,000 bytes or more including its header, starts on the large object heap, and only a full gen 2 collection sweeps it. The walk always starts at the roots: the static cache, the request's local variables, and anything the runtime holds.

   Statement jobone request

   Static cachea GC root

   GCfrees unreachable

   pdf: 130 KB

   1/4The job renders a 130 KB PDF and hands it to the cache.The key has a new timestamp, so every request adds an entry: 1, then 2, then thousands.A full collection starts at the roots. The static cache is a root, and it leads to every PDF.Every PDF is reachable, so the collector frees none of them. The heap only grows.

   BackPlayNext
4. What we gained

   The fix was to remove the root, not to call GC.Collect. The cache is keyed by contract id, holds at most 500 entries, and evicts the oldest; the PDF is also written to Blob storage (Azure's object store for files), so the cache is only a fast path for a recent reprint. A test keeps a weak reference to a PDF, evicts it, forces a collection and asserts the reference is dead. Memory settles around 240 MB, and the nightly restarts stop. The same rule covers a static event: the publisher roots every subscriber, so a view that subscribes must unsubscribe in Dispose.

In the lending API

The four marked lines: where a small object starts, where a big one goes, and how a weak reference proves a PDF was freed.

Only the key lines are shown; Deep walks the whole file.

The code, file by file

Three xUnit tests (xUnit is a test tool for .NET) that prove the generation rules and the leak test. Tap Next to walk the scene with the code.

What they ask next

The follow-ups interviews really ask. Answer out loud, then tap to check.

• ReportedHow do you find a memory leak in production?

  I start with the trend, not a guess. dotnet-counters on the live process shows gc-heap-size, gen-2-size, loh-size, alloc-rate and time-in-gc; a line that climbs while traffic is flat is the signature. Then I take two heap dumps half an hour apart with dotnet-gcdump and compare the biggest types: the one that grew is the suspect. Collect runs on Linux and Windows, but the .gcdump file opens in Visual Studio or PerfView only on Windows, and collect triggers a gen 2 collection that can pause the app, so I take it from a replica that is out of the load balancer. The last step is the path to root: static field, event, cache or timer. That path names the code to fix.

  dotnet-counters monitor \
    -p 4112 System.Runtime

  Next questions, in Deep →Accenture: GC, and where you used it
• ReportedWhy are there three generations, and not one?

  Because most objects die young. A gen 0 collection only walks the newest, smallest area, so it is cheap enough to run often. An object that survives is likely to live longer, so it is promoted and checked less often: gen 1, then gen 2. The expensive full collection, which also sweeps the large object heap, runs rarely. The trade-off is memory: a promoted object takes space in an older generation until a collection of that generation runs.

  Next questions, in Deep →r/csharp: why three generations
• ReportedWhat is the large object heap, and when does it hurt?

  Objects of 85,000 bytes or more, including the object header, are allocated on the large object heap. It is part of gen 2, so it is swept only by a full collection, and it is not compacted by default: freed gaps stay, and a busy process can fragment. That makes big arrays, PDFs and very large strings expensive to churn. I stream instead of ReadAllBytes, and I rent buffers from ArrayPool when I need a big temporary array.

  Next questions, in Deep →Gif Technologies: generations and influence

Follow-up tree

Answer, then open the next level. That is the order an interviewer digs in.

Reporteda candidate was asked Intervieweran interviewer asks it

• ReportedHow do you find a memory leak in production?

  I start with the trend, not a guess. dotnet-counters on the live process shows gc-heap-size, gen-2-size, loh-size, alloc-rate and time-in-gc; a line that climbs while traffic is flat is the signature. Then I take two heap dumps half an hour apart with dotnet-gcdump and compare the biggest types: the one that grew is the suspect. Collect runs on Linux and Windows, but the .gcdump file opens in Visual Studio or PerfView only on Windows, and collect triggers a gen 2 collection that can pause the app, so I take it from a replica that is out of the load balancer. The last step is the path to root: static field, event, cache or timer. That path names the code to fix.

  dotnet-counters monitor \
    -p 4112 System.Runtime

  Accenture: GC, and where you used it
  • What if the GC heap is flat but the process still grows?

    Then the memory is outside the managed heap: native buffers (memory allocated outside the .NET heap), thread stacks, sockets or memory-mapped files (files mapped into the process's address space). I compare the process working set (the memory the OS shows for the whole process, not just the .NET heap) with the GC heap size. If the working set climbs and the GC heap does not, a heap dump will not show it; I look at handle counts (open file and socket handles), thread counts and the native libraries, and dotnet-counters still helps separate CPU from memory.

  • Why not just raise the memory limit?

    It buys hours and moves the cost to the platform. The leak still grows, the replica still restarts, and the limit is now higher for every healthy replica too. Raising it is a mitigation while we find the root, not a fix.

• ReportedWhy are there three generations, and not one?

  Because most objects die young. A gen 0 collection only walks the newest, smallest area, so it is cheap enough to run often. An object that survives is likely to live longer, so it is promoted and checked less often: gen 1, then gen 2. The expensive full collection, which also sweeps the large object heap, runs rarely. The trade-off is memory: a promoted object takes space in an older generation until a collection of that generation runs.

  r/csharp: why three generations
  • Can I choose which generation to collect?

    GC.Collect(0) and GC.Collect(1) exist, and tests use them to make behaviour predictable. Application code should not: a forced collection is a pause with no guarantee the object you care about is unreachable, and it usually hides the root instead of removing it.

• ReportedWhat is the large object heap, and when does it hurt?

  Objects of 85,000 bytes or more, including the object header, are allocated on the large object heap. It is part of gen 2, so it is swept only by a full collection, and it is not compacted by default: freed gaps stay, and a busy process can fragment. That makes big arrays, PDFs and very large strings expensive to churn. I stream instead of ReadAllBytes, and I rent buffers from ArrayPool when I need a big temporary array.

  Gif Technologies: generations and influence
  • How do I measure LOH churn?

    dotnet-counters shows loh-size and alloc-rate; a dotnet-trace capture shows allocations and GC pauses per request. I compare before and after with a load test, not with a stopwatch on one request.

• A static event is the same shape as the cache: why does it leak?

  Because a static event is a root too. The publisher keeps a delegate for every subscriber in its invocation list; a delegate holds the target object. If a request-scoped view subscribes in its constructor and never unsubscribes, the root keeps every view alive for the life of the process. The fix is to unsubscribe in Dispose and let the DI scope call it, or to use a weak event pattern (a subscription that does not root the subscriber) when the subscriber's lifetime is shorter than the publisher's.

  See the bug →
• Does the GC call Dispose for me?

  No. The collector frees managed memory; it does not know that a connection, a file handle or a socket needs returning now. Types that own those implement IDisposable, and using calls Dispose at the end of the block. A finalizer is a late backup that runs on its own thread at an unpredictable time, so it is not a plan. The Dispose page has the full answer.

• What is a finalizer, and why avoid it?

  A finalizer is a method the GC calls before it frees an object, written with a tilde before the class name. It runs on the finalizer thread at an unpredictable time, and an object with a finalizer survives at least one extra collection while it waits. That delays reclamation, and the docs say it can promote the object to gen 1 or gen 2, with no ordering guarantee. Use Dispose for deterministic cleanup; keep a finalizer only as a safety net for a class that owns a raw handle, and suppress it when Dispose has run.

• ReportedHow can you influence the GC?

  Mostly by reducing allocations, not by tuning knobs. Remove boxing in hot loops (boxing wraps a value type like int in an object, which allocates), pool big buffers, stream instead of loading a whole file, and avoid a new HttpClient per call. At the runtime level you choose workstation or server GC, set a memory limit the container respects, and can ask for LOH compaction when a one-off big allocation fragmented the heap. I do not call GC.Collect and I do not use GC.TryStartNoGCRegion (an API that asks the runtime to pause collections for a while) in a normal web app.

  Gif Technologies: how you can influence it
  • What is server GC?

    Server GC gives each core its own heap and collector thread, which raises throughput on a multi-core server at the cost of more memory. The ASP.NET Core project template sets it; a console app defaults to workstation. Workstation GC uses one heap and suits small processes or latency-sensitive desktop work. GCSettings.IsServerGC tells you which one you are in.

Production bugs, and how to tell them

Each one: what you see, why, the fix in code, and the line to say.

• Every statement reprint added another 130 KB

  Container Apps replica log, 02:14:
  Replica statement-worker--7f9c restarted
  Reason: memory limit reached (1 GiB)
  (no exception in the app log)

  Cause

  StatementPdfCache was a static Dictionary whose key was the contract id plus DateTime.UtcNow.Ticks. Every request was a miss, and nothing removed an entry, so every rendered PDF stayed in the dictionary for the life of the process. A static field is a root, so a full collection found every PDF reachable. Each PDF was about 130 KB, over the 85,000-byte limit, so the growth landed on the LOH, which is swept only by a full collection.

  Fix

  Key by contract id, bound the cache to 500 entries, and evict the oldest, all under a lock because one singleton serves every request. The PDF is also written to Blob storage for reprints. A test keeps a WeakReference to a PDF, fills the cache past its limit, forces a full collection and checks the reference is dead. The cache becomes an instance created once in DI, not a static field.

  BeforeBeforeAfterAfter

  1public sealed class StatementService2{3 private readonly PdfRenderer _renderer;4 5 public StatementService(6 PdfRenderer renderer)7 {8 _renderer = renderer;9 }10 11 public byte[] Render(string contractId)12 {13 byte[] pdf = _renderer.Render(contractId);14 StatementPdfCache.Keep(contractId, pdf);15 return pdf;16 }17}

  Before · StatementService.cs · line 1 · 1 of 7The service that renders a statement. It is created per request by the controller.

  BackNext line

  Say it

  A static cache with a timestamp in the key is the worst of both worlds: it never hits and it never stops growing. The GC could not free the PDFs because the static field was a root. We keyed by contract, bounded the cache, and proved with a weak reference that an evicted PDF is collected.

• Every rate import left a dead view behind

  UDC portal, a week after release 1.9:
  memory at 1.2 GB and rising, p95 (the 95th-percentile response time) up 400 ms
  no exception in the app log

  Cause

  RateView subscribed to the static event RateFeed.RateImported in its constructor. The portal created a new view for every request with new RateView(), and nothing unsubscribed. A static event holds a delegate for each subscriber in its invocation list, so the event was a root and every view ever created stayed alive. The rate feed imported every 15 minutes, which made the leak faster without causing an error.

  Fix

  RateView implements IDisposable and unsubscribes in Dispose. The endpoint gets the view from DI as a scoped service, so the scope disposes it at the end of the request. A test makes a view, disposes it, forces a collection and checks a WeakReference is dead.

  RateFeed.csBeforeBeforeAfterAfterAfter

  Diagram · step 1The request creates a RateView, whose constructor subscribes. The event now holds a delegate to it.Show diagram

  Static eventRateImported · a root

  RateView (now)this request

  subscribers: 1

  1public static class RateFeed2{3 public static event4 Action<string>? RateImported;5 6 public static void Import(string rate)7 {8 RateImported?.Invoke(rate);9 }10}

  RateFeed.cs · line 1 · 1 of 5static: one feed for the whole process. The event below lives as long as the app.

  BackNext line

  Say it

  A static event is a root: the publisher's invocation list holds every subscriber. The view subscribed in its constructor and nothing ever unsubscribed, so the GC could not free any view. We made the view disposable and let the request scope dispose it, and a weak-reference test proves it is collected.

Traps

• Don't say"The garbage collector will clean it up eventually."SayOnly if nothing can reach it. A static field, a static event, an unbounded cache, a running timer or a live thread is a root, and the collector is right to keep the object. For a connection or a file handle, the collector does not call Dispose at all; using does.
• Don't say"Memory is high, so I will call GC.Collect."SayA forced full collection pauses the app and frees only what is already unreachable. If memory comes back, the object was unreachable and the collector would have got it; if it does not, the root is still there and you have added a pause. I find the root instead.
• Don't say"A leak means someone forgot to free memory."SayIn .NET the memory is managed, so there is no free to forget. A leak is a reference that should have gone: a cache key that changes every time, an event subscription never removed, a singleton that captures a request object. The question is always "what still points at this?"

Traps

What should you say instead? Tap to check.

• Don't say"The garbage collector will clean it up eventually."

  Say instead

  Only if nothing can reach it. A static field, a static event, an unbounded cache, a running timer or a live thread is a root, and the collector is right to keep the object. For a connection or a file handle, the collector does not call Dispose at all; using does.

• Don't say"Memory is high, so I will call GC.Collect."

  Say instead

  A forced full collection pauses the app and frees only what is already unreachable. If memory comes back, the object was unreachable and the collector would have got it; if it does not, the root is still there and you have added a pause. I find the root instead.

• Don't say"A leak means someone forgot to free memory."

  Say instead

  In .NET the memory is managed, so there is no free to forget. A leak is a reference that should have gone: a cache key that changes every time, an event subscription never removed, a singleton that captures a request object. The question is always "what still points at this?"