Merge from 'main' to 'sycl-web' (17 commits)

  CONFLICT (content): Merge conflict in clang/lib/Sema/SemaDeclAttr.cpp

Author: Pavel V Chupin

clang/docs/HLSLSupport.rst

@@ -0,0 +1,240 @@
+============
+HLSL Support
+============
+
+.. contents::
+   :local:
+
+Introduction
+============
+
+HLSL Support is under active development in the Clang codebase. This document
+describes the high level goals of the project, the guiding principles, as well
+as some idiosyncrasies of the HLSL language and how we intend to support them in
+Clang.
+
+Project Goals
+=============
+
+The long term goal of this project is to enable Clang to function as a
+replacement for the `DirectXShaderCompiler (DXC)
+<https://github.com/microsoft/DirectXShaderCompiler/>`_ in all its supported
+use cases. Accomplishing that goal will require Clang to be able to process most
+existing HLSL programs with a high degree of source compatibility.
+
+Non-Goals
+---------
+
+HLSL ASTs do not need to be compatible between DXC and Clang. We do not expect
+identical code generation or that features will resemble DXC's implementation or
+architecture. In fact, we explicitly expect to deviate from DXC's implementation
+in key ways.
+
+Guiding Principles
+==================
+
+This document lacks details for architectural decisions that are not yet
+finalized. Our top priorities are quality, maintainability, and flexibility. In
+accordance with community standards we are expecting a high level of test
+coverage, and we will engineer our solutions with long term maintenance in mind.
+We are also working to limit modifications to the Clang C++ code paths and
+share as much functionality as possible.
+
+Architectural Direction
+=======================
+
+HLSL support in Clang is expressed as C++ minus unsupported C and C++ features.
+This is different from how other Clang languages are implemented. Most languages
+in Clang are additive on top of C.
+
+HLSL is not a formally or fully specified language, and while our goals require
+a high level of source compatibility, implementations can vary and we have some
+flexibility to be more or less permissive in some cases. For modern HLSL DXC is
+the reference implementation.
+
+The HLSL effort prioritizes following similar patterns for other languages,
+drivers, runtimes and targets. Specifically, We will maintain separation between
+HSLS-specific code and the rest of Clang as much as possible following patterns
+in use in Clang code today (i.e. ParseHLSL.cpp, SemaHLSL.cpp, CGHLSL*.cpp...).
+We will use inline checks on language options where the code is simple and
+isolated, and prefer HLSL-specific implementation files for any code of
+reasonable complexity.
+
+In places where the HLSL language is in conflict with C and C++, we will seek to
+make minimally invasive changes guarded under the HLSL language options. We will
+seek to make HLSL language support as minimal a maintenance burden as possible.
+
+DXC Driver
+----------
+
+A DXC driver mode will provide command-line compatibility with DXC, supporting
+DXC's options and flags. The DXC driver is HLSL-specific and will create an
+HLSLToolchain which will provide the basis to support targeting both DirectX and
+Vulkan.
+
+Parser
+------
+
+Following the examples of other parser extensions HLSL will add a ParseHLSL.cpp
+file to contain the implementations of HLSL-specific extensions to the Clang
+parser. The HLSL grammar shares most of its structure with C and C++, so we will
+use the existing C/C++ parsing code paths.
+
+Sema
+----
+
+HLSL's Sema implementation will also provide an ``ExternalSemaSource``. In DXC,
+an ``ExternalSemaSource`` is used to provide definitions for HLSL built-in data
+types and built-in templates. Clang is already designed to allow an attached
+``ExternalSemaSource`` to lazily complete data types, which is a **huge**
+performance win for HLSL.
+
+CodeGen
+-------
+
+Like OpenCL, HLSL relies on capturing a lot of information into IR metadata.
+*hand wave* *hand wave* *hand wave* As a design principle here we want our IR to
+be idiomatic Clang IR as much as possible. We will use IR attributes wherever we
+can, and use metadata as sparingly as possible. One example of a difference from
+DXC already implemented in Clang is the use of target triples to communicate
+shader model versions and shader stages.
+
+Our HLSL CodeGen implementation should also have an eye toward generating IR
+that will map directly to targets other than DXIL. While IR itself is generally
+not re-targetable, we want to share the Clang CodeGen implementation for HLSL
+with other GPU graphics targets like SPIR-V and possibly other GPU and even CPU
+targets.
+
+HLSL Language
+=============
+
+The HLSL language is insufficiently documented, and not formally specified.
+Documentation is available on `Microsoft's website
+<https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl>`_.
+The language syntax is similar enough to C and C++ that carefully written C and
+C++ code is valid HLSL. HLSL has some key differences from C & C++ which we will
+need to handle in Clang.
+
+HLSL is not a conforming or valid extension or superset of C or C++. The
+language has key incompatibilities with C and C++, both syntactically and
+semantically.
+
+An Aside on GPU Languages
+-------------------------
+
+Due to HLSL being a GPU targeted language HLSL is a Single Program Multiple Data
+(SPMD) language relying on the implicit parallelism provided by GPU hardware.
+Some language features in HLSL enable programmers to take advantage of the
+parallel nature of GPUs in a hardware abstracted language.
+
+HLSL also prohibits some features of C and C++ which can have catastrophic
+performance or are not widely supportable on GPU hardware or drivers. As an
+example, register spilling is often excessively expensive on GPUs, so HLSL
+requires all functions to be inlined during code generation, and does not
+support a runtime calling convention.
+
+Pointers & References
+---------------------
+
+HLSL does not support referring to values by address. Semantically all variables
+are value-types and behave as such. HLSL disallows the pointer dereference
+operators (unary ``*``, and ``->``), as well as the address of operator (unary
+&). While HLSL disallows pointers and references in the syntax, HLSL does use
+reference types in the AST, and we intend to use pointer decay in the AST in
+the Clang implementation.
+
+HLSL ``this`` Keyword
+---------------------
+
+HLSL does support member functions, and (in HLSL 2021) limited operator
+overloading. With member function support, HLSL also has a ``this`` keyword. The
+``this`` keyword is an example of one of the places where HLSL relies on
+references in the AST, because ``this`` is a reference.
+
+Bitshifts
+---------
+
+In deviation from C, HLSL bitshifts are defined to mask the shift count by the
+size of the type. In DXC, the semantics of LLVM IR were altered to accommodate
+this, in Clang we intend to generate the mask explicitly in the IR. In cases
+where the shift value is constant, this will be constant folded appropriately,
+in other cases we can clean it up in the DXIL target.
+
+Non-short Circuiting Logical Operators
+--------------------------------------
+
+In HLSL 2018 and earlier, HLSL supported logical operators (and the ternary
+operator) on vector types. This behavior required that operators not short
+circuit. The non-short circuiting behavior applies to all data types until HLSL
+2021. In HLSL 2021, logical and ternary operators do not support vector types
+instead builtin functions ``and``, ``or`` and ``select`` are available, and
+operators short circuit matching C behavior.
+
+Precise Qualifier
+-----------------
+
+HLSL has a ``precise`` qualifier that behaves unlike anything else in the C
+language. The support for this qualifier in DXC is buggy, so our bar for
+compatibility is low.
+
+The ``precise`` qualifier applies in the inverse direction from normal
+qualifiers. Rather than signifying that the declaration containing ``precise``
+qualifier be precise, it signifies that the operations contributing to the
+declaration's value be ``precise``. Additionally, ``precise`` is a misnomer:
+values attributed as ``precise`` comply with IEEE-754 floating point semantics,
+and are prevented from optimizations which could decrease *or increase*
+precision.
+
+Differences in Templates
+------------------------
+
+HLSL uses templates to define builtin types and methods, but disallowed
+user-defined templates until HLSL 2021. HLSL also allows omitting empty template
+parameter lists when all template parameters are defaulted. This is an ambiguous
+syntax in C++, but Clang detects the case and issues a diagnostic. This makes
+supporting the case in Clang minimally invasive.
+
+Vector Extensions
+-----------------
+
+HLSL uses the OpenCL vector extensions, and also provides C++-style constructors
+for vectors that are not supported by Clang.
+
+Standard Library
+----------------
+
+HLSL does not support the C or C++ standard libraries. Like OpenCL, HLSL
+describes its own library of built in types, complex data types, and functions.
+
+Unsupported C & C++ Features
+----------------------------
+
+HLSL does not support all features of C and C++. In implementing HLSL in Clang
+use of some C and C++ features will produce diagnostics under HLSL, and others
+will be supported as language extensions. In general, any C or C++ feature that
+can be supported by the DXIL and SPIR-V code generation targets could be treated
+as a clang HLSL extension. Features that cannot be lowered to DXIL or SPIR-V,
+must be diagnosed as errors.
+
+HLSL does not support the following C features:
+
+* Pointers
+* References
+* ``goto`` or labels
+* Variable Length Arrays
+* ``_Complex`` and ``_Imaginary``
+* C Threads or Atomics (or Obj-C blocks)
+* ``union`` types `(in progress for HLSL 202x) <https://github.com/microsoft/DirectXShaderCompiler/pull/4132>`_
+* Most features C11 and later
+
+HLSL does not support the following C++ features:
+
+* RTTI
+* Exceptions
+* Multiple inheritance
+* Access specifiers
+* Anonymous or inline namespaces
+* ``new`` & ``delete`` operators in all of their forms (array, placement, etc)
+* Constructors and destructors
+* Any use of the ``virtual`` keyword
+* Most features C++11 and later

clang/docs/ReleaseNotes.rst

@@ -54,6 +54,20 @@ Major New Features
   There is an analogous ``zero_call_used_regs`` attribute to allow for finer
   control of this feature.
 
+- Clang now supports randomizing structure layout in C. This feature is a
+  compile-time hardening technique, making it more difficult for an attacker to
+  retrieve data from structures. Specify randomization with the
+  ``randomize_layout`` attribute. The corresponding ``no_randomize_layout``
+  attribute can be used to turn the feature off.
+
+  A seed value is required to enable randomization, and is deterministic based
+  on a seed value. Use the ``-frandomize-layout-seed=`` or
+  ``-frandomize-layout-seed-file=`` flags.
+
+  .. note::
+
+      Randomizing structure layout is a C-only feature.
+
 Bug Fixes
 ------------------
 - ``CXXNewExpr::getArraySize()`` previously returned a ``llvm::Optional``

clang/docs/index.rst

@@ -45,6 +45,7 @@ Using Clang as a Compiler
    OpenCLSupport
    OpenMPSupport
    SYCLSupport
+   HLSLSupport
    ThinLTO
    APINotes
    CommandGuide/index

clang/include/clang/AST/Decl.h

@@ -4052,6 +4052,12 @@ class RecordDecl : public TagDecl {
     RecordDeclBits.ParamDestroyedInCallee = V;
   }
 
+  bool isRandomized() const { return RecordDeclBits.IsRandomized; }
+
+  void setIsRandomized(bool V) { RecordDeclBits.IsRandomized = V; }
+
+  void reorderFields(const SmallVectorImpl<Decl *> &Fields);
+
   /// Determines whether this declaration represents the
   /// injected class name.
   ///

clang/include/clang/AST/DeclBase.h

@@ -313,6 +313,7 @@ class alignas(8) Decl {
   friend class ASTReader;
   friend class CXXClassMemberWrapper;
   friend class LinkageComputer;
+  friend class RecordDecl;
   template<typename decl_type> friend class Redeclarable;
 
   /// Access - Used by C++ decls for the access specifier.
@@ -1540,10 +1541,13 @@ class DeclContext {
 
     /// Represents the way this type is passed to a function.
     uint64_t ArgPassingRestrictions : 2;
+
+    /// Indicates whether this struct has had its field layout randomized.
+    uint64_t IsRandomized : 1;
   };
 
   /// Number of non-inherited bits in RecordDeclBitfields.
-  enum { NumRecordDeclBits = 14 };
+  enum { NumRecordDeclBits = 15 };
 
   /// Stores the bits used by OMPDeclareReductionDecl.
   /// If modified NumOMPDeclareReductionDeclBits and the accessor

clang/include/clang/AST/Randstruct.h

@@ -0,0 +1,38 @@
+//===- Randstruct.h - Interfact for structure randomization -------*- C++ -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains the interface for Clang's structure field layout
+// randomization.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_AST_RANDSTRUCT_H
+#define LLVM_CLANG_AST_RANDSTRUCT_H
+
+namespace llvm {
+template <typename T> class ArrayRef;
+template <typename T> class SmallVectorImpl;
+class StringRef;
+} // end namespace llvm
+
+namespace clang {
+
+class ASTContext;
+class Decl;
+class RecordDecl;
+
+namespace randstruct {
+
+bool randomizeStructureLayout(const ASTContext &Context, llvm::StringRef Name,
+                              llvm::ArrayRef<Decl *> Fields,
+                              llvm::SmallVectorImpl<Decl *> &FinalOrdering);
+
+} // namespace randstruct
+} // namespace clang
+
+#endif // LLVM_CLANG_AST_RANDSTRUCT_H

clang/include/clang/Basic/Attr.td

@@ -4859,3 +4859,18 @@ def HLSLNumThreads: InheritableAttr {
   let LangOpts = [HLSL];
   let Documentation = [NumThreadsDocs];
 }
+
+def RandomizeLayout : InheritableAttr {
+  let Spellings = [GCC<"randomize_layout">];
+  let Subjects = SubjectList<[Record]>;
+  let Documentation = [ClangRandomizeLayoutDocs];
+  let LangOpts = [COnly];
+}
+
+def NoRandomizeLayout : InheritableAttr {
+  let Spellings = [GCC<"no_randomize_layout">];
+  let Subjects = SubjectList<[Record]>;
+  let Documentation = [ClangRandomizeLayoutDocs];
+  let LangOpts = [COnly];
+}
+def : MutualExclusions<[RandomizeLayout, NoRandomizeLayout]>;

clang/include/clang/Basic/AttrDocs.td

@@ -7957,3 +7957,35 @@ dictate the thread id. Total number of threads executed is ``X * Y * Z``.
 The full documentation is available here: https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/sm5-attributes-numthreads
   }];
 }
+
+def ClangRandomizeLayoutDocs : Documentation {
+  let Category = DocCatDecl;
+  let Heading = "randomize_layout, no_randomize_layout";
+  let Content = [{
+The attribute ``randomize_layout``, when attached to a C structure, selects it
+for structure layout field randomization; a compile-time hardening technique. A
+"seed" value, is specified via the ``-frandomize-layout-seed=`` command line flag.
+For example:
+
+.. code-block:: bash
+
+  SEED=`od -A n -t x8 -N 32 /dev/urandom | tr -d ' \n'`
+  make ... CFLAGS="-frandomize-layout-seed=$SEED" ...
+
+You can also supply the seed in a file with ``-frandomize-layout-seed-file=``.
+For example:
+
+.. code-block:: bash
+
+  od -A n -t x8 -N 32 /dev/urandom | tr -d ' \n' > /tmp/seed_file.txt
+  make ... CFLAGS="-frandomize-layout-seed-file=/tmp/seed_file.txt" ...
+
+The randomization is deterministic based for a given seed, so the entire
+program should be compiled with the same seed, but keep the seed safe
+otherwise.
+
+The attribute ``no_randomize_layout``, when attached to a C structure,
+instructs the compiler that this structure should not have its field layout
+randomized.
+  }];
+}