dotnet build... stuck!

Introduction

Recently, strange occurrences started happening during project’s cloud pipeline build. Every so often a build would either get stuck hanging at the dotnet build execution or it would throw following error:

dotnet/sdk/6.0.425/Roslyn/Microsoft.CSharp.Core.targets(75,5): 
  error MSB6006: "csc.dll" exited with code 137. 
    [/src/CleverHeap.Demo.DAL/CleverHeap.Demo.DAL.csproj]

Background

For a while, it was undeterministic. On occasion, build ran successfully in an average time of around 3min, but other times process would stuck hanging and timeout after 30mins (duration limit imposed on a pipeline). As we progressed with development, after a while, the build kept stucking all the time or either failing by throwing previously mentioned error.

The problem of troubleshooting such issue was that we couldn’t reproduce it on our local environment. Build always run successfully on our local machines, so everyone’s first premise was on causality with amount of resources. Searching the error trace lead us to the same conclusion:

Exit code 137 typically means that your process is getting killed because it is running out of memory.¹

Solutions

However, digging through known issues and troubleshooting pipeline, led us to Reason 01 which was the case for our project. We also discovered additional group of reasons that might also cause similar problems, therefore we choose to document them here for sake of accessibility…

Reason 01

Building project containing EntityFramework migration scripts causes solution build to stuck

Overtime, your project containing migration scripts can get to a large number of scripts. Those scripts can include mix of EF tool C# code together with different SQL code as well. For this reason code analyzers can consume a lot of time and resources during build. In case your EF code first classes and migration scripts are in separate project (like in my case), then there is no much need for code analyzers for this project. You can choose to disable them by stating following in your .csproj file²:

<Project Sdk="Microsoft.NET.Sdk">
  ...
  <PropertyGroup>
    <EnableNETAnalyzers>False</EnableNETAnalyzers>
    <AnalysisLevel>none</AnalysisLevel>
    <RunAnalyzersDuringBuild>false</RunAnalyzersDuringBuild>
    <RunAnalyzersDuringLiveAnalysis>false</RunAnalyzersDuringLiveAnalysis>
    <RunAnalyzers>false</RunAnalyzers>
  </PropertyGroup>
  ...
</Project>

Once we have disabled running analyzers, our build pipeline had never got stuck again.

Reason 02

In case you are using docker build with your dotnet solution you should check if any of the projects contain Watch statements in .csproj. This can cause long builds that can end up with timeout error as well.

Watcher statements are used to notice changes of certain files, so they contain relative paths to specific files. For example, a path like ..\**\*.env can end up scanning entire root directory and file system of your container in search of *.env files³. In order to prevent it, you can add a condition and pass parameter value to check in your Dockerfile.

<Project Sdk="Microsoft.NET.Sdk">
  ...
  <Watch Include="..\**\*.env" Condition=" '$(IsDockerBuild)' != 'true' " />
  ...
</Project> 

Now in your Dockerfile, you should add /p:IsDockerBuild=true at dotnet restore statement:

...
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
...
RUN dotnet restore "CleverHeap.Demo.Api.csproj" /p:IsDockerBuild=true
...

Reason 03

Lack of RAM for the build pipeline.

In case of a large dotnet solution with way too many projects or source files, you might experience the same problem due to exceeded RAM allocated by the build pipeline. The first thing to do would be to check RAM allocation on your local machine when running docker build or dotnet build for your solution. If the allocated RAM is more than maximum RAM of your build pipeline, then a good suggestion would be to check Roslyn options for your dotnet solution. For more on this, you can check this JetBrains blog⁴.

References

• [Github] DotNet Roslyn issue thread
• [Reddit] DotNet: Build time takes long
• [Github] .NET6 docker build stuck on dotnet restore
• [JetBrains] A Story About .csproj, Large Solutions and Memory Usage