Skip to content

Commit 895f18f

Browse files
crummeldagood
authored andcommitted
Update source-build to use core repos with Arcade (#916)
* Update to latest CoreFX master with Arcade. * Remove outdated CoreFX patch. * Don't think this is needed anymore. * Update build command for CoreFX * "--" no longer needed in arguments. * Add global.json to source-build. * Add Arcade to source-build. * Fix global.json. * Call Arcade init in init-tools * New SDK appears to want this lowercased extension. * Some more tweaks to how Arcade is loaded. * CoreFX has a global.json now. * Allow skipping global.json values and add alternate SDK path. * A couple more properties to pass into CoreFX for Arcade. * Add another CoreFX SDK for us to restore. * Revert "Don't think this is needed anymore." This reverts commit 2bce8d8. * Revert "Revert "Don't think this is needed anymore."" This reverts commit 4b4773f. * Update submodules * Add patches for arcade * update arcade version * update proj files * update standard * patch standard * Fix/reapply patches after merge * add source to restore, add BuildToolsPackageDir to dir.props * revert 554083d * Clean up Arcade init scripts Remove old scripts that don't exist in Arcade anymore. Make eng/common/tools.sh executable. Fix Windows init-tools. Change \ to / for PACKAGES_DIR because the \ was escaping a ", causing mayhem in the inner init-tools.cmd. The CoreFX init-tools call (through Arcade) also uses /. * Add Arcade submodule Starts with WIP branch, adds fixes and repo dependencies. Converts the fork to patches on an official commit. Add Arcade to known good, add prereq for CoreFX. Migrate roslyn-tools to new SDK toolset flow. * Fix ref package resassembly: *, not *.* In certain packages, there are files with no extension. Failing to repack these causes failures when restored. For example, runtime.linux-x64.Microsoft.NETCore.DotNetAppHost/3.0.0-preview-27115-02 is missing runtimes/linux-x64/native/apphost when repacked. * Fix Arcade bool arg passing on Windows * Ensure CoreFX uses our BuildTools Manually set new-style semaphore using BuildToolsVersion.txt. * Configure CoreFX to build properly Disable node reuse. Add --pack. Fix PackagesOutput. Pass Arcade args in Windows-friendly way. For Win, replace "-buildArch" with "/p:ArchGroup". The shell scripts make this translation, but Windows scripts don't. * Disable dotnet/standard node reuse, use Arcade * Upgrade CoreCLR to CoreFX dependency This gets the Index and Range types, fixing GenFacades errors. * Address my own style comments from review * Remove arcade-minimalci-sample This submodule was here to try out Arcade. Now we are really using Arcade, so it's unnecessary. * Remove unused Arcade common scripts We can wait until we actually onboard to Arcade to take these. * Remove CoreCLR PackagesOutput: use Repo API * Standard global.json update (merge duplication) * Clean up patches * Use source-built IL SDK in CoreFX to fix offline The part of CoreFX that restores this package doesn't seem to use the prebuilt directory correctly. Use a source-built SDK to remove the prebuilt and fix the failure. * Update baselines * Revert "--source" passed to echo but not command
1 parent 7220fcc commit 895f18f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+15133
-292
lines changed

.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,6 @@
7575
[submodule "src/aspnet-razor"]
7676
path = src/aspnet-razor
7777
url = https://github.com/aspnet/Razor
78+
[submodule "src/arcade"]
79+
path = src/arcade
80+
url = https://github.com/dotnet/arcade

BuildToolsVersion.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
2.2.0-preview1-03013-01
1+
3.0.0-preview1-03406-03

Directory.Build.props

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Project>
3+
<PropertyGroup>
4+
<ImportNetSdkFromRepoToolset>false</ImportNetSdkFromRepoToolset>
5+
<_SuppressSdkImports>true</_SuppressSdkImports>
6+
</PropertyGroup>
7+
</Project>

Directory.Build.targets

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Project>
3+
</Project>

NuGet.Config renamed to NuGet.config

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,7 @@
55
<clear />
66
<add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
77
<add key="nuget-build" value="https://dotnet.myget.org/F/nuget-build/api/v3/index.json" />
8+
<add key="arcade" value="https://dotnetfeed.blob.core.windows.net/dotnet-tools-internal/index.json" />
9+
<add key="dotnet-core" value="https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json" />
810
</packageSources>
911
</configuration>

build-source-tarball.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ cp $SCRIPT_ROOT/*.props $TARBALL_ROOT/
108108
cp $SCRIPT_ROOT/*.targets $TARBALL_ROOT/
109109
cp $SCRIPT_ROOT/init-tools.msbuild $TARBALL_ROOT/
110110
cp $SCRIPT_ROOT/DotnetCLIVersion.txt $TARBALL_ROOT/
111+
cp $SCRIPT_ROOT/BuildToolsVersion.txt $TARBALL_ROOT/
111112
cp $SCRIPT_ROOT/ProdConFeed.txt $TARBALL_ROOT/
112113
cp $SCRIPT_ROOT/smoke-test* $TARBALL_ROOT/
113114
cp -r $SCRIPT_ROOT/keys $TARBALL_ROOT/

dependencies.props

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,12 @@
2727
<MicrosoftNETSdkRazorPackageVersion>2.1.0</MicrosoftNETSdkRazorPackageVersion>
2828
</PropertyGroup>
2929

30+
<!-- Package versions used as toolsets -->
31+
<PropertyGroup>
32+
<BuildToolsPackage>microsoft.dotnet.buildtools</BuildToolsPackage>
33+
<BuildToolsPackageVersion>$([System.IO.File]::ReadAllText('$(MSBuildThisFileDirectory)BuildToolsVersion.txt').Trim())</BuildToolsPackageVersion>
34+
</PropertyGroup>
35+
3036
<!-- Build stabilization properties as passed in by ProdCon. -->
3137
<PropertyGroup>
3238
<UseStableVersions Condition="'$(UseStableVersions)' == ''">false</UseStableVersions>

dir.props

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,11 @@
137137
<CliInternalBuildVersion>30000001-1</CliInternalBuildVersion>
138138
</PropertyGroup>
139139

140+
<!-- With Arcade, this package version is stored in a property we need to override. -->
141+
<ItemGroup>
142+
<ExtraPackageVersionPropsPackageInfo Include="BuildToolsPackageVersion" Version="$(BuildToolsPackageVersion)" />
143+
</ItemGroup>
144+
140145
<ItemGroup>
141146
<ExtraPackageVersionPropsPackageInfo Include="DotnetCliInternalVersion" Version="3.0.100-$(CliInternalReleaseTag)-$(CliInternalBuildVersion)" />
142147
</ItemGroup>

eng/Version.Details.xml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Dependencies>
3+
<ProductDependencies></ProductDependencies>
4+
<ToolsetDependencies>
5+
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="1.0.0-beta.18531.9">
6+
<Uri>https://github.com/dotnet/arcade</Uri>
7+
<Sha>4ffd1c0020510ffed9e854834df13f0207056012</Sha>
8+
</Dependency>
9+
</ToolsetDependencies>
10+
</Dependencies>

eng/Versions.props

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
3+
<PropertyGroup>
4+
<VersionPrefix>1.0.0</VersionPrefix>
5+
<PreReleaseVersionLabel>prerelease</PreReleaseVersionLabel>
6+
</PropertyGroup>
7+
</Project>

eng/common/tools.ps1

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
# Initialize variables if they aren't already defined
2+
3+
$ci = if (Test-Path variable:ci) { $ci } else { $false }
4+
$configuration = if (Test-Path variable:configuration) { $configuration } else { "Debug" }
5+
$nodereuse = if (Test-Path variable:nodereuse) { $nodereuse } else { !$ci }
6+
$prepareMachine = if (Test-Path variable:prepareMachine) { $prepareMachine } else { $false }
7+
$restore = if (Test-Path variable:restore) { $restore } else { $true }
8+
$verbosity = if (Test-Path variable:verbosity) { $verbosity } else { "minimal" }
9+
$warnaserror = if (Test-Path variable:warnaserror) { $warnaserror } else { $true }
10+
$msbuildEngine = if (Test-Path variable:msbuildEngine) { $msbuildEngine } else { $null }
11+
$useInstalledDotNetCli = if (Test-Path variable:useInstalledDotNetCli) { $useInstalledDotNetCli } else { $true }
12+
13+
set-strictmode -version 2.0
14+
$ErrorActionPreference = "Stop"
15+
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
16+
17+
function Create-Directory([string[]] $path) {
18+
if (!(Test-Path $path)) {
19+
New-Item -path $path -force -itemType "Directory" | Out-Null
20+
}
21+
}
22+
23+
function InitializeDotNetCli([bool]$install) {
24+
# Don't resolve runtime, shared framework, or SDK from other locations to ensure build determinism
25+
$env:DOTNET_MULTILEVEL_LOOKUP=0
26+
27+
# Disable first run since we do not need all ASP.NET packages restored.
28+
$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
29+
30+
# Source Build uses DotNetCoreSdkDir variable
31+
if ($env:DotNetCoreSdkDir -ne $null) {
32+
$env:DOTNET_INSTALL_DIR = $env:DotNetCoreSdkDir
33+
}
34+
35+
# Find the first path on %PATH% that contains the dotnet.exe
36+
if ($useInstalledDotNetCli -and ($env:DOTNET_INSTALL_DIR -eq $null)) {
37+
$env:DOTNET_INSTALL_DIR = ${env:PATH}.Split(';') | where { ($_ -ne "") -and (Test-Path (Join-Path $_ "dotnet.exe")) }
38+
}
39+
40+
$dotnetSdkVersion = $GlobalJson.tools.dotnet
41+
42+
# Use dotnet installation specified in DOTNET_INSTALL_DIR if it contains the required SDK version,
43+
# otherwise install the dotnet CLI and SDK to repo local .dotnet directory to avoid potential permission issues.
44+
if (($env:DOTNET_INSTALL_DIR -ne $null) -and (Test-Path(Join-Path $env:DOTNET_INSTALL_DIR "sdk\$dotnetSdkVersion"))) {
45+
$dotnetRoot = $env:DOTNET_INSTALL_DIR
46+
} else {
47+
$dotnetRoot = Join-Path $RepoRoot ".dotnet"
48+
$env:DOTNET_INSTALL_DIR = $dotnetRoot
49+
50+
if (-not (Test-Path(Join-Path $env:DOTNET_INSTALL_DIR "sdk\$dotnetSdkVersion"))) {
51+
if ($install) {
52+
InstallDotNetSdk $dotnetRoot $dotnetSdkVersion
53+
} else {
54+
Write-Host "Unable to find dotnet with SDK version '$dotnetSdkVersion'" -ForegroundColor Red
55+
ExitWithExitCode 1
56+
}
57+
}
58+
}
59+
60+
return $dotnetRoot
61+
}
62+
63+
function GetDotNetInstallScript([string] $dotnetRoot) {
64+
$installScript = "$dotnetRoot\dotnet-install.ps1"
65+
if (!(Test-Path $installScript)) {
66+
Create-Directory $dotnetRoot
67+
Invoke-WebRequest "https://dot.net/v1/dotnet-install.ps1" -OutFile $installScript
68+
}
69+
70+
return $installScript
71+
}
72+
73+
function InstallDotNetSdk([string] $dotnetRoot, [string] $version) {
74+
$installScript = GetDotNetInstallScript $dotnetRoot
75+
& $installScript -Version $version -InstallDir $dotnetRoot
76+
if ($lastExitCode -ne 0) {
77+
Write-Host "Failed to install dotnet cli (exit code '$lastExitCode')." -ForegroundColor Red
78+
ExitWithExitCode $lastExitCode
79+
}
80+
}
81+
82+
function InitializeVisualStudioBuild {
83+
$vsToolsPath = $env:VS150COMNTOOLS
84+
if ($vsToolsPath -eq $null) {
85+
$vsToolsPath = $env:VS160COMNTOOLS
86+
}
87+
88+
if (($vsToolsPath -ne $null) -and (Test-Path $vsToolsPath)) {
89+
$vsInstallDir = [System.IO.Path]::GetFullPath((Join-Path $vsToolsPath "..\.."))
90+
} else {
91+
$vsInfo = LocateVisualStudio
92+
93+
$vsInstallDir = $vsInfo.installationPath
94+
$vsSdkInstallDir = Join-Path $vsInstallDir "VSSDK\"
95+
$vsVersion = $vsInfo.installationVersion.Split('.')[0] + "0"
96+
97+
Set-Item "env:VS$($vsVersion)COMNTOOLS" (Join-Path $vsInstallDir "Common7\Tools\")
98+
Set-Item "env:VSSDK$($vsVersion)Install" $vsSdkInstallDir
99+
$env:VSSDKInstall = $vsSdkInstallDir
100+
}
101+
102+
return $vsInstallDir
103+
}
104+
105+
function LocateVisualStudio {
106+
$vswhereVersion = $GlobalJson.tools.vswhere
107+
108+
if (!$vsWhereVersion) {
109+
Write-Host "vswhere version must be specified in /global.json." -ForegroundColor Red
110+
ExitWithExitCode 1
111+
}
112+
113+
$toolsRoot = Join-Path $RepoRoot ".tools"
114+
$vsWhereDir = Join-Path $toolsRoot "vswhere\$vswhereVersion"
115+
$vsWhereExe = Join-Path $vsWhereDir "vswhere.exe"
116+
117+
if (!(Test-Path $vsWhereExe)) {
118+
Create-Directory $vsWhereDir
119+
Write-Host "Downloading vswhere"
120+
Invoke-WebRequest "https://github.com/Microsoft/vswhere/releases/download/$vswhereVersion/vswhere.exe" -OutFile $vswhereExe
121+
}
122+
123+
$vsInfo = & $vsWhereExe `
124+
-latest `
125+
-prerelease `
126+
-format json `
127+
-requires Microsoft.Component.MSBuild `
128+
-requires Microsoft.VisualStudio.Component.VSSDK `
129+
-requires Microsoft.VisualStudio.Component.Roslyn.Compiler | ConvertFrom-Json
130+
131+
if ($lastExitCode -ne 0) {
132+
Write-Host "Failed to locate Visual Studio (exit code '$lastExitCode')." -ForegroundColor Red
133+
ExitWithExitCode $lastExitCode
134+
}
135+
136+
# use first matching instance
137+
return $vsInfo[0]
138+
}
139+
140+
function ConfigureTools {
141+
# Include custom tools configuration
142+
$script = Join-Path $EngRoot "configure-toolset.ps1"
143+
144+
if (Test-Path $script) {
145+
. $script
146+
}
147+
}
148+
149+
function InitializeTools() {
150+
ConfigureTools
151+
152+
$tools = $GlobalJson.tools
153+
154+
# Initialize dotnet cli if listed in 'tools'
155+
$dotnetRoot = $null
156+
if ((Get-Member -InputObject $tools -Name "dotnet") -ne $null) {
157+
$dotnetRoot = InitializeDotNetCli -install:$restore
158+
}
159+
160+
if (-not $msbuildEngine) {
161+
# Presence of vswhere.version indicates the repo needs to build using VS msbuild.
162+
if ((Get-Member -InputObject $tools -Name "vswhere") -ne $null) {
163+
$msbuildEngine = "vs"
164+
} elseif ($dotnetRoot -ne $null) {
165+
$msbuildEngine = "dotnet"
166+
} else {
167+
Write-Host "-msbuildEngine must be specified, or /global.json must specify 'tools.dotnet' or 'tools.vswhere'." -ForegroundColor Red
168+
ExitWithExitCode 1
169+
}
170+
}
171+
172+
if ($msbuildEngine -eq "dotnet") {
173+
if (!$dotnetRoot) {
174+
Write-Host "/global.json must specify 'tools.dotnet'." -ForegroundColor Red
175+
ExitWithExitCode 1
176+
}
177+
178+
$script:buildDriver = Join-Path $dotnetRoot "dotnet.exe"
179+
$script:buildArgs = "msbuild"
180+
} elseif ($msbuildEngine -eq "vs") {
181+
$vsInstallDir = InitializeVisualStudioBuild
182+
183+
$script:buildDriver = Join-Path $vsInstallDir "MSBuild\15.0\Bin\msbuild.exe"
184+
$script:buildArgs = ""
185+
} else {
186+
Write-Host "Unexpected value of -msbuildEngine: '$msbuildEngine'." -ForegroundColor Red
187+
ExitWithExitCode 1
188+
}
189+
190+
InitializeToolSet $script:buildDriver $script:buildArgs
191+
InitializeCustomToolset
192+
}
193+
194+
function InitializeToolset([string] $buildDriver, [string]$buildArgs) {
195+
$toolsetVersion = $GlobalJson.'msbuild-sdks'.'Microsoft.DotNet.Arcade.Sdk'
196+
$toolsetLocationFile = Join-Path $ToolsetDir "$toolsetVersion.txt"
197+
198+
if (Test-Path $toolsetLocationFile) {
199+
$path = Get-Content $toolsetLocationFile -TotalCount 1
200+
if (Test-Path $path) {
201+
$script:ToolsetBuildProj = $path
202+
return
203+
}
204+
}
205+
206+
if (-not $restore) {
207+
Write-Host "Toolset version $toolsetVersion has not been restored."
208+
ExitWithExitCode 1
209+
}
210+
211+
$ToolsetRestoreLog = Join-Path $LogDir "ToolsetRestore.binlog"
212+
$proj = Join-Path $ToolsetDir "restore.proj"
213+
214+
'<Project Sdk="Microsoft.DotNet.Arcade.Sdk"/>' | Set-Content $proj
215+
MSBuild $proj /t:__WriteToolsetLocation /clp:None /bl:$ToolsetRestoreLog /p:__ToolsetLocationOutputFile=$toolsetLocationFile
216+
217+
if ($lastExitCode -ne 0) {
218+
Write-Host "Failed to restore toolset (exit code '$lastExitCode'). See log: $ToolsetRestoreLog" -ForegroundColor Red
219+
ExitWithExitCode $lastExitCode
220+
}
221+
222+
$path = Get-Content $toolsetLocationFile -TotalCount 1
223+
if (!(Test-Path $path)) {
224+
throw "Invalid toolset path: $path"
225+
}
226+
227+
$script:ToolsetBuildProj = $path
228+
}
229+
230+
function InitializeCustomToolset {
231+
if (-not $restore) {
232+
return
233+
}
234+
235+
$script = Join-Path $EngRoot "restore-toolset.ps1"
236+
237+
if (Test-Path $script) {
238+
. $script
239+
}
240+
}
241+
242+
function ExitWithExitCode([int] $exitCode) {
243+
if ($ci -and $prepareMachine) {
244+
Stop-Processes
245+
}
246+
exit $exitCode
247+
}
248+
249+
function Stop-Processes() {
250+
Write-Host "Killing running build processes..."
251+
Get-Process -Name "msbuild" -ErrorAction SilentlyContinue | Stop-Process
252+
Get-Process -Name "dotnet" -ErrorAction SilentlyContinue | Stop-Process
253+
Get-Process -Name "vbcscompiler" -ErrorAction SilentlyContinue | Stop-Process
254+
}
255+
256+
function MsBuild() {
257+
$warnaserrorSwitch = if ($warnaserror) { "/warnaserror" } else { "" }
258+
& $buildDriver $buildArgs $warnaserrorSwitch /m /nologo /clp:Summary /v:$verbosity /nr:$nodereuse $args
259+
}
260+
261+
$RepoRoot = Resolve-Path (Join-Path $PSScriptRoot "..\..")
262+
$EngRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
263+
$ArtifactsDir = Join-Path $RepoRoot "artifacts"
264+
$ToolsetDir = Join-Path $ArtifactsDir "toolset"
265+
$LogDir = Join-Path (Join-Path $ArtifactsDir "log") $configuration
266+
$TempDir = Join-Path (Join-Path $ArtifactsDir "tmp") $configuration
267+
$GlobalJson = Get-Content -Raw -Path (Join-Path $RepoRoot "global.json") | ConvertFrom-Json
268+
269+
if ($env:NUGET_PACKAGES -eq $null) {
270+
# Use local cache on CI to ensure deterministic build,
271+
# use global cache in dev builds to avoid cost of downloading packages.
272+
$env:NUGET_PACKAGES = if ($ci) { Join-Path $RepoRoot ".packages" }
273+
else { Join-Path $env:UserProfile ".nuget\packages" }
274+
}
275+
276+
Create-Directory $ToolsetDir
277+
Create-Directory $LogDir
278+
279+
if ($ci) {
280+
Create-Directory $TempDir
281+
$env:TEMP = $TempDir
282+
$env:TMP = $TempDir
283+
}

0 commit comments

Comments
 (0)