Merge Main into Feature/4.0 (#6747)

* Update build templates to handle feature branches (#6744)

* Update build templates

* Update build templates to include all releases/* and feature/*

* Update releases to release

* Update triggers for PR Validation Build

* Add triggers for Code Coverage

* Update version to 4.0 for feature branch (#6743)

* Add missing implementation for datetime relevant arrow type into dataframe (#6675)

* Add missing implementation for datetime relevant arrow type

* Return required usage

* Fix the behavior or column SetName method (#6676)

* Fix the behavior or column SetName method

* Fix stack overflow exception

* Fix merge issues

---------

Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>

* Fix DataFrame to allow to store columns with size more than 2 Gb (#6710)

* Fix error with allocating more than MaxCapacity of Byte Memory Buffer

* Remove Unit test as it consumes too much memory

* Fix issue with increasing buffer capacity over limit when double it size

* avoid empty dataset (#6756)

* Fix dataframe arithmetics for columns having several value buffers (column size is more than 2 Gb) (#6724)

* Fix dataframe arithmetics

* Fix

* Run tests that requires more than 2 Gb of Memory only on 64-bit env (#6758)

* Reduce coupling of Data.Analysis.Tests project (#6759)

* Provide ability to filter dataframe column by null via ElementWise Methods (#6723)

* Provide ability to filter by null value

* Add comments

* Fix code review findings

* Fix incorrect DataFrame min max computation with NULL (#6734)

* Step 1

* Step 2

* Fixed code review findings

* Clean DataFrame meaningless code (#6761)

* Add NameEntityRecognition and Q&A deep learning tasks. (#6760)

* NER

* QA almost done, runtime error

* QA finished

* fixes from PR comments

* fixed build

* build fixes

* perf changes

* made disposable

* fixed not disposing model

* added some disposables to TensorFlow for memory

* build testing

* fixing build

* added missing dispose

* build fixes

* build fixes

* testing macos fix

* fix issue (#6768)

* fixed mac build and minor torch sharp changes (#6776)

* Improve DataFrame Arithmetics implementation (#6763)

* Change methods signature generation

* Change DataFrameColumn Arithmetics

* Change DataFrameColumn Operations

* Fix unit tests

* Fix spaces

* Fix code review findings

* Add QA sweepable estimator in AutoML  (#6781)

* Add QA sweepable

* clean

* Modernized some argument checks that still used string literals for parameter names (#6766)

Co-authored-by: John Doe <john@doe>

* removed deprecated yosemite brew (#6805)

* Add TargetType to Type_convert (#6785)

* Add target Type in convert  type

* Add custom type "DataKind"

* clean

* Add DataKind name space

* clean test

* File-scoped namespaces in files under `Environment` (`Microsoft.ML.Core`) (#6791)

Co-authored-by: Lehonti Ramos <john@doe>

* File-scoped namespaces in files under `EntryPoints` (`Microsoft.ML.Core`) (#6790)

Co-authored-by: Lehonti Ramos <john@doe>

* Fix issue with addIndexColumn in DataFrame.LoadCsv (#6769)

* Fix issue with addIndexColumn in DataFrame.LoadCsv

* Fix tests

* Fix DataFrame.LoadCsv can not load CSV with duplicate column names (#6772)

* File-scoped namespaces in files under `ComponentModel` (`Microsoft.ML.Core`) (#6788)

Co-authored-by: Lehonti Ramos <john@doe>

* File-scoped namespaces in files under `Data` (`Microsoft.ML.Core`) (#6789)

Co-authored-by: Lehonti Ramos <john@doe>

* Fix inconsistent null handling in DataFrame Arithmetics (#6770)

* Fix inconsistent null handling in DataFrame Arithmetics

* Fix Null Count and division by zero issues

* Minor changes to restart build and rerun flaky tests

* File-scoped namespaces in files under `Prediction` (`Microsoft.ML.Core`) (#6792)

Co-authored-by: Lehonti Ramos <john@doe>

* Allow to define CultureInfo for parsing values on reading DataFrame from csv (#6782)

* Use CultureInfo for parsing values in csv file

* Fix merge issues

* Append dataframe rows based on column names (#6808)

* Append dataframe rows based on column names

* Update DataFrame.cs

---------

Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>

* removed codecov token (#6811)

* Fix wrong type conversion on PrimitiveDataFrameColumn (#6834)

* Fix wrong type conversion on PrimitiveDataFrameColumn

* Added tests for #6829

* Fix test

* Add file generated from tt template and fix unit tests

---------

Co-authored-by: Aleksei Smirnov <tlalok@inbox.ru>

* update interactive kernel version (#6836)

* update interactive kernel version

* update

* Update Microsoft.Data.Analysis.Interactive.Tests.csproj

* Add performance benchmarks for dataframe arithmetic operations  (#6827)

* Add performance tests

* Add extra tests

* Fix

* Fix typo

* Fix Divide_Int16 and Divide_Int32_Int16 benchmarks

* Fix

* Change csproj file

* Update BenchmarkDotNetVersion to 0.13.5

* Fix

* Change to 0.13.1 because that is what is latest version in our nuget feeds.

---------

Co-authored-by: Jake Radzikowski <JakeRad@Microsoft.com>

* Improve performance of column cloning inside DataFrame arithmetics (#6814)

* Optimize PrimitiveColumnContainer.Clone method

* Avoid unnecessary type conversion during binary operations

* Remove using

* Fix DataFrameBuffer constructor

* remove uncorrectly added using

* Make DataFrameBuffer Length field protected

* Fix typo

* Use RawSpan

* Simplify tt files for PrimitiveDataFrameColumnAritmetics (#6830)

* First step of tt refactoring

* Step 2

* Step 3

* Addresses #6533 (#6838)

* Initial structure and started fleshing out some sections

* Some corrections and paragraph on DL usages

* Starting fleshing out DL on ML.NET section

* Addresses #6533

* Update dependencies (#6837)

* Update dependencies

* Add reference to NuGet.Packaging.Core

* PrimitiveDataFrameColumn.Clone method crashes when is used with IEnumerable mapIndices argument (#6822)

* Split Test for AppendMany into 4 different tests

* Block init of null validity buffer instead of setting individual bits

* Add unit tests for PrimitiveDataFrameColumn.Clone

* Fixes #6821

* Fix

* Fix bug with AppendMany values to not empty column

* Restart unit tests

* Add more unit tests

* Fix failing unit test

* Fix code review findings

* 6847 incorrectly sets column value (#6849)

* Fix DataFrame incorrectly sets column value for index higher than Buffer.MaxCapacity

* Revert renaming

* Increase performance of arithmetic operations by enhancing calculations on nullable values  (#6846)

* Optimize PrimitiveColumnContainer.Clone method

* Avoid unnecessary type conversion during binary operations

* Remove using

* Fix DataFrameBuffer constructor

* remove uncorrectly added using

* Make DataFrameBuffer Length field protected

* Add performance tests

* Split Test for AppendMany into 4 different tests

* Block init of null validity buffer instead of setting individual bits

* Add unit tests for PrimitiveDataFrameColumn.Clone

* Fixes #6821

* Fix

* Add extra tests

* Fix

* Fix typo

* Fix Divide_Int16 and Divide_Int32_Int16 benchmarks

* Fix

* Avoid using constructor, that copies memory

* First step of tt refactoring

* Step 2

* Step 3

* Move iteration over buffers outside of the PrimitiveDataFrameColumnArithmetic

* Change PrimitiveDataFrameColumnArithmetic

* Fix typo

* Use RawSpan

* Fix bug with AppendMany values to not empty column

* Restart unit tests

* Add more unit tests

* Add GetBitCount method

* Fix failing unit test

* Implementation

* Change unit tests

* Update unit tests

* Refactoring BinaryOperation

* Intermediate changes

* Intermediate results

* Implement Binary Scalar Reverse Operarions

* Add implementation for BinaryIntOperations

* Implement Comparison Operations

* Implement actual calculations for Comparison operations

* Uncomment performance tests

* Remove unintentional code changes

* Add reference to Apache Arrow project license in THIRD-PARTY-NOTICES

* Fix license issues

* Fixes incorrect work of DataFrame with VBufferColumn when number of e… (#6851)

* Fixes incorrect work of DataFrame with VBufferColumn when number of elements is greater than Int.MaxValue

* Fix calculation of max capacity and amount of required buffers

* Fix unit test

* Run test allocating more than 2 Gb of memory on 64bit env only

* Fix StringDataFrameColumn same way as VBufferDataFrameColumn

* Fix wrong amount of buffers created in constructor of StringDataFrameColumn

* Fix code review findings

---------

Co-authored-by: Aleksei Smirnov <tlalok@inbox.ru>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Xiaoyun Zhang <xiaoyuz@microsoft.com>
Co-authored-by: zewditu Hailemariam <36615490+zewditu@users.noreply.github.com>
Co-authored-by: Lehonti Ramos <17771375+Lehonti@users.noreply.github.com>
Co-authored-by: John Doe <john@doe>
Co-authored-by: Raffaello Fraboni <10281615+novelhawk@users.noreply.github.com>
Co-authored-by: R. G. Esteves <rodolfo.g.esteves@intel.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>

Author: 10 people

.vsts-dotnet-ci.yml

@@ -2,6 +2,20 @@
 # ML.NET's PR validation build
 ################################################################################
 
+pr:
+  branches:
+    include:
+    - main
+    - feature/*
+    - release/*
+
+trigger:
+  branches:
+    include:
+    - main
+    - feature/*
+    - release/*
+
 resources:
   containers:
   - container: CentosContainer

Microsoft.ML.sln

@@ -168,6 +168,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML.Tokenizers.Tes
 EndProject
 Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Microsoft.ML.FSharp.Tests", "test\Microsoft.ML.FSharp.Tests\Microsoft.ML.FSharp.Tests.fsproj", "{041CB5CD-5832-413E-A894-D9DBED210B16}"
 EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Data.Analysis.PerformanceTests", "test\Microsoft.Data.Analysis.PerformanceTests\Microsoft.Data.Analysis.PerformanceTests.csproj", "{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -788,6 +790,14 @@ Global
 		{041CB5CD-5832-413E-A894-D9DBED210B16}.Release|Any CPU.Build.0 = Release|Any CPU
 		{041CB5CD-5832-413E-A894-D9DBED210B16}.Release|x64.ActiveCfg = Release|Any CPU
 		{041CB5CD-5832-413E-A894-D9DBED210B16}.Release|x64.Build.0 = Release|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Debug|x64.Build.0 = Debug|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Release|Any CPU.Build.0 = Release|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Release|x64.ActiveCfg = Release|Any CPU
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD}.Release|x64.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -870,6 +880,7 @@ Global
 		{BBC3A950-BD68-45AC-9DBD-A8F4D8847745} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
 		{C3D82402-F207-4F19-8C57-5AF0FBAF9682} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
 		{041CB5CD-5832-413E-A894-D9DBED210B16} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
+		{FB8A8823-CC6C-4C2F-8539-05FBFB7C91CD} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {41165AF1-35BB-4832-A189-73060F82B01D}

THIRD-PARTY-NOTICES.TXT

@@ -85,3 +85,24 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
+
+
+License notice for BitUtility
+------------------------------------------
+
+https://github.com/apache/arrow/blob/main/csharp/src/Apache.Arrow/BitUtility.cs
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

build/.night-build.yml

@@ -15,9 +15,8 @@ schedules:
   branches:
     include:
     - main
-    - releases/1.6.0
-    - features/automl
-    - features/integrationPackage
+    - feature/*
+    - release/*
   always: true
 
 resources:

build/.outer-loop-build.yml

@@ -15,9 +15,8 @@ schedules:
   branches:
     include:
     - main
-    - releases/1.6.0
-    - features/automl
-    - features/integrationPackage
+    - feature/*
+    - release/*
   always: true
 
 

build/ci/job-template.yml

@@ -44,7 +44,6 @@ jobs:
       ${{ if eq(parameters.codeCoverage, 'false') }}:
         hardLink: '/p:CreateHardLinksForCopyLocalIfPossible=True'
       testTargetFramework: '/p:TestTargetFramework=$(_targetFramework)'
-      CODECOV_TOKEN: 03031e35-fe75-4e4c-87ee-e919ae601748
     strategy:
       matrix:
         ${{ if eq(parameters.customMatrixes, '') }}:
@@ -68,7 +67,7 @@ jobs:
     steps:
     # Extra MacOS step required to install OS-specific dependencies
     - ${{ if and(contains(parameters.pool.vmImage, 'macOS'), not(contains(parameters.name, 'cross')))  }}:
-      - script: export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=TRUE && brew update && brew unlink libomp && brew install $(Build.SourcesDirectory)/build/libomp.rb --build-from-source --formula
+      - script: export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=TRUE && brew unlink libomp && brew install $(Build.SourcesDirectory)/build/libomp.rb --build-from-source --formula
         displayName: Install MacOS build dependencies
     # Extra Apple MacOS step required to install OS-specific dependencies
     - ${{ if and(contains(parameters.pool.vmImage, 'macOS'), contains(parameters.name, 'cross'))  }}:

build/codecoverage-ci.yml

@@ -2,6 +2,20 @@
 # ML.NET's Code Coverage validation build
 ################################################################################
 
+pr:
+  branches:
+    include:
+    - main
+    - feature/*
+    - release/*
+
+trigger:
+  branches:
+    include:
+    - main
+    - feature/*
+    - release/*
+
 jobs:
 - template: /build/ci/job-template.yml
   parameters:

build/libomp.rb

@@ -15,7 +15,6 @@ class Libomp < Formula
   end
 
   depends_on "cmake" => :build
-  depends_on macos: :yosemite
 
   def install
     system "cmake", ".", *std_cmake_args

build/vsts-ci.yml

@@ -100,7 +100,7 @@ stages:
     pool:
       vmImage: macOS-12
     steps:
-    - script: export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 && brew update &&  rm '/usr/local/bin/2to3-3.11' && brew unlink libomp && brew install $(Build.SourcesDirectory)/build/libomp.rb --build-from-source --formula
+    - script: export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 &&  rm '/usr/local/bin/2to3-3.11' && brew unlink libomp && brew install $(Build.SourcesDirectory)/build/libomp.rb --build-from-source --formula
       displayName: Install build dependencies
     # Only build native assets to avoid conflicts.
     - script: ./build.sh -projects $(Build.SourcesDirectory)/src/Native/Native.proj -configuration $(BuildConfig) /p:TargetArchitecture=x64 /p:CopyPackageAssets=true

eng/Versions.props

@@ -16,10 +16,12 @@
     <MicrosoftBclAsyncInterfacesVersion>6.0.0</MicrosoftBclAsyncInterfacesVersion>
     <MicrosoftExtensionsVersion>2.1.0</MicrosoftExtensionsVersion>
     <MicrosoftExtensionsDependencyInjectionVersion>6.0.0</MicrosoftExtensionsDependencyInjectionVersion>
-    <SkiaSharpVersion>2.88.3</SkiaSharpVersion>
+    <NuGetVersion>6.7.0</NuGetVersion>
+    <SkiaSharpVersion>2.88.6</SkiaSharpVersion>
     <SystemBuffersVersion>4.5.1</SystemBuffersVersion>
     <SystemCodeDomVersion>4.5.0</SystemCodeDomVersion>
     <SystemCollectionsImmutableVersion>1.5.0</SystemCollectionsImmutableVersion>
+    <SystemConfigurationConfigurationManagerVersion>6.0.1</SystemConfigurationConfigurationManagerVersion>
     <SystemIOFileSystemAccessControl>4.5.0</SystemIOFileSystemAccessControl>
     <SystemMemoryVersion>4.5.5</SystemMemoryVersion>
     <SystemReflectionEmitLightweightVersion>4.3.0</SystemReflectionEmitLightweightVersion>
@@ -30,13 +32,13 @@
     <SystemTextJsonVersion>6.0.1</SystemTextJsonVersion>
     <SystemThreadingChannelsVersion>4.7.1</SystemThreadingChannelsVersion>
     <!-- Other product dependencies -->
-    <ApacheArrowVersion>2.0.0</ApacheArrowVersion>
+    <ApacheArrowVersion>11.0.0</ApacheArrowVersion>
     <GoogleProtobufVersion>3.19.6</GoogleProtobufVersion>
     <LightGBMVersion>2.3.1</LightGBMVersion>
     <MicrosoftCodeAnalysisAnalyzersVersion>3.3.0</MicrosoftCodeAnalysisAnalyzersVersion>
     <MicrosoftCodeAnalysisCSharpVersion>3.9.0</MicrosoftCodeAnalysisCSharpVersion>
-    <MicrosoftDotNetInteractiveFormattingVersion>1.0.0-beta.22504.6</MicrosoftDotNetInteractiveFormattingVersion>
-    <MicrosoftDotNetInteractiveVersion>1.0.0-beta.22504.6</MicrosoftDotNetInteractiveVersion>
+    <MicrosoftDotNetInteractiveFormattingVersion>1.0.0-beta.23461.4</MicrosoftDotNetInteractiveFormattingVersion>
+    <MicrosoftDotNetInteractiveVersion>1.0.0-beta.23461.4</MicrosoftDotNetInteractiveVersion>
     <MicrosoftMLOnnxRuntimeVersion>1.14.0</MicrosoftMLOnnxRuntimeVersion>
     <MlNetMklDepsVersion>0.0.0.12</MlNetMklDepsVersion>
     <!--  
@@ -74,11 +76,11 @@
     <SystemCompositionVersion>1.2.0</SystemCompositionVersion>
     <!-- Test-only Dependencies -->
     <ApprovalTestsVersion>5.4.7</ApprovalTestsVersion>
-    <BenchmarkDotNetVersion>0.12.0</BenchmarkDotNetVersion>
+    <BenchmarkDotNetVersion>0.13.1</BenchmarkDotNetVersion>
     <DotNetRuntime60Version>6.0.9</DotNetRuntime60Version>
     <DotNetRuntime80Version>8.0.0-preview.3.23174.8</DotNetRuntime80Version>
     <FluentAssertionVersion>5.10.2</FluentAssertionVersion>
-    <MicrosoftCodeAnalysisTestingVersion>1.1.2-beta1.22512.1</MicrosoftCodeAnalysisTestingVersion>
+    <MicrosoftCodeAnalysisTestingVersion>1.1.2-beta1.23431.1</MicrosoftCodeAnalysisTestingVersion>
     <MicrosoftDotNetXUnitExtensionsVersion>8.0.0-beta.23265.1</MicrosoftDotNetXUnitExtensionsVersion>
     <MicrosoftExtensionsDependencyModelVersion>2.1.0</MicrosoftExtensionsDependencyModelVersion>
     <MicrosoftExtensionsTestVersion>3.0.1</MicrosoftExtensionsTestVersion>
@@ -87,7 +89,7 @@
     <MicrosoftMLTestDatabasesVersion>0.0.6-test</MicrosoftMLTestDatabasesVersion>
     <MicrosoftMLTestModelsVersion>0.0.7-test</MicrosoftMLTestModelsVersion>
     <SystemDataSqlClientVersion>4.6.1</SystemDataSqlClientVersion>
-    <SystemDataSQLiteCoreVersion>1.0.113</SystemDataSQLiteCoreVersion>
+    <SystemDataSQLiteCoreVersion>1.0.118</SystemDataSQLiteCoreVersion>
     <XunitCombinatorialVersion>1.2.7</XunitCombinatorialVersion>
     <XUnitVersion>2.4.2</XUnitVersion>
     <!-- Opt-out repo features -->