[GR-16004] [GR-15444] [GR-15997] Update to FFI 1.11.1 and add documentation.

PullRequest: truffleruby/895

Author: eregon

doc/contributor/ffi.md

@@ -0,0 +1,26 @@
+# FFI
+
+We have our own implementation of FFI, implemented entirely in Ruby + Truffle
+NFI.
+
+Using the FFI gem C extension would be suboptimal because Sulong uses Truffle
+NFI to call native methods. Truffle NFI is implemented with `libffi`. The FFI C
+extension uses its own `libffi` to call native functions. This means we would
+use Truffle NFI's `libffi` to call the C extension `libffi` functions, adding
+overhead on every call to the C extension `libffi` functions.
+
+FFI is also one way to avoid C extensions altogether, so it makes sense to not
+require C extension support for gems using FFI.
+
+## Structure
+
+* `lib/truffle/ffi`: Unchanged Ruby files from the gem.
+* `lib/truffle/truffle/ffi_backend`: our Ruby + Truffle NFI backend for FFI,
+  which replaces the C extension and has the same API.
+* `lib/truffle/ffi.rb` coordinates `require`-ing the above.
+* `spec/ffi`: Specs from the FFI gem.
+* `src/main/ruby/truffleruby/core/truffle/ffi`: Implementation of
+  `Truffle::FFI::Pointer`, which is also used in core and is aliased to
+  `FFI::Pointer` once `"ffi"` is required. `Truffle::FFI::Pointer` should
+  therefore have the same API as `FFI::Pointer` in the FFI gem.
+* `tool/import-ffi.sh`: Imports files from a checkout of the FFI gem.

doc/legal/legal.md

@@ -141,7 +141,7 @@ files.
 
 #### FFI
 
-TruffleRuby includes parts of the FFI gem 1.9.25. The FFI gem is copyright
+TruffleRuby includes parts of the FFI gem 1.11.1. The FFI gem is copyright
 2008-2016, Ruby FFI project contributors, and covered by the three-clause BSD
 licence (see `ffi.txt`).
 

doc/user/security.md

@@ -138,7 +138,9 @@ Consider potential vulnerabilities in your system `libssl`.
 
 ## FFI
 
-We are not aware of any known vulnerabilities.
+Number | Description | Their Mitigation | Test | Our Mitigation
+--- | --- | --- | --- | ---
+CVE-2018-1000201 | A DLL loading issue can be hijacked on Windows when a `Symbol` is used for the library name | Treat Symbols the same as Strings in `ffi_lib` | | Applied the same mitigation, by using a version of FFI which fixed this vulnerability
 
 # Notes on hashing
 

doc/user/truffleruby-additions.md

@@ -77,6 +77,13 @@ and set.
 
 `AtomicReference` is marshalable.
 
+## FFI
+
+TruffleRuby includes a built-in implementation of [Ruby-FFI](https://github.com/ffi/ffi),
+compatible with version 1.11.1 of the `ffi` gem. This should be transparent: you can
+just install the `ffi` gem as normal, and it will use our built-in implementation,
+regardless of the version of the `ffi` gem.
+
 ## Polyglot programming
 
 The `Polyglot` and `Java` modules provides access to the polyglot programming

lib/truffle/ffi.rb

@@ -22,7 +22,6 @@ def self.find_type_size(type)
   end
 
   module Platform
-    CPU = RbConfig::CONFIG['host_cpu']
     # eregon: I would like to use rbconfig/sizeof here, but the linker requires
     # ffi, and the linker is needed to build the rbconfig/sizeof C extension,
     # so we need to break the cycle.
@@ -35,7 +34,6 @@ module Platform
 
 # Require the pure-Ruby Truffle NFI backend
 require_relative 'truffle/ffi_backend/last_error'
-require_relative 'truffle/ffi_backend/data_converter'
 require_relative 'truffle/ffi_backend/type'
 require_relative 'truffle/ffi_backend/struct_layout'
 require_relative 'truffle/ffi_backend/struct'

lib/truffle/truffle/ffi_backend/data_converter.rb renamed to lib/truffle/ffi/data_converter.rb

@@ -26,8 +26,7 @@
 # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
 # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
 # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-#
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.#
 
 module FFI
   # This module is used to extend somes classes and give then a common API.

lib/truffle/ffi/ffi.rb

@@ -29,6 +29,7 @@
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 require 'ffi/platform'
+require 'ffi/data_converter'
 require 'ffi/types'
 require 'ffi/library'
 require 'ffi/errno'

lib/truffle/ffi/library.rb

@@ -126,7 +126,7 @@ def ffi_lib(*names)
               else
                 # TODO better library lookup logic
                 unless libname.start_with?("/") || FFI::Platform.windows?
-                  path = ['/usr/lib/','/usr/local/lib/'].find do |pth|
+                  path = ['/usr/lib/','/usr/local/lib/','/opt/local/lib/'].find do |pth|
                     File.exist?(pth + libname)
                   end
                   if path

lib/truffle/ffi/platform.rb

@@ -54,6 +54,10 @@ module Platform
       RbConfig::CONFIG['host_os'].downcase
     end
 
+    OSVERSION = RbConfig::CONFIG['host_os'].gsub(/[^\d]/, '').to_i
+
+    CPU = RbConfig::CONFIG['host_cpu']
+
     ARCH = case CPU.downcase
     when /amd64|x86_64/
       "x86_64"
@@ -82,7 +86,6 @@ def self.is_os(os)
       OS == os
     end
 
-    NAME = "#{ARCH}-#{OS}"
     IS_GNU = defined?(GNU_LIBC)
     IS_LINUX = is_os("linux")
     IS_MAC = is_os("darwin")
@@ -92,6 +95,11 @@ def self.is_os(os)
     IS_SOLARIS = is_os("solaris")
     IS_WINDOWS = is_os("windows")
     IS_BSD = IS_MAC || IS_FREEBSD || IS_NETBSD || IS_OPENBSD
+
+    # Add the version for known ABI breaks
+    name_version = "12" if IS_FREEBSD && OSVERSION >= 12 # 64-bit inodes
+
+    NAME = "#{ARCH}-#{OS}#{name_version}"
     CONF_DIR = File.join(File.dirname(__FILE__), 'platform', NAME)
 
     public

lib/truffle/ffi/platform/aarch64-freebsd/types.conf

@@ -0,0 +1,128 @@
+rbx.platform.typedef.__int8_t = char
+rbx.platform.typedef.__uint8_t = uchar
+rbx.platform.typedef.__int16_t = short
+rbx.platform.typedef.__uint16_t = ushort
+rbx.platform.typedef.__int32_t = int
+rbx.platform.typedef.__uint32_t = uint
+rbx.platform.typedef.__int64_t = long_long
+rbx.platform.typedef.__uint64_t = ulong_long
+rbx.platform.typedef.__int_least8_t = char
+rbx.platform.typedef.__uint_least8_t = uchar
+rbx.platform.typedef.__int_least16_t = short
+rbx.platform.typedef.__uint_least16_t = ushort
+rbx.platform.typedef.__int_least32_t = int
+rbx.platform.typedef.__uint_least32_t = uint
+rbx.platform.typedef.__int_least64_t = long_long
+rbx.platform.typedef.__uint_least64_t = ulong_long
+rbx.platform.typedef.__int_fast8_t = int
+rbx.platform.typedef.__uint_fast8_t = uint
+rbx.platform.typedef.__int_fast16_t = int
+rbx.platform.typedef.__uint_fast16_t = uint
+rbx.platform.typedef.__int_fast32_t = int
+rbx.platform.typedef.__uint_fast32_t = uint
+rbx.platform.typedef.__int_fast64_t = long_long
+rbx.platform.typedef.__uint_fast64_t = ulong_long
+rbx.platform.typedef.__intptr_t = long
+rbx.platform.typedef.__uintptr_t = ulong
+rbx.platform.typedef.__intmax_t = long_long
+rbx.platform.typedef.__uintmax_t = ulong_long
+rbx.platform.typedef.__register_t = long_long
+rbx.platform.typedef.__vaddr_t = ulong
+rbx.platform.typedef.__paddr_t = ulong
+rbx.platform.typedef.__vsize_t = ulong
+rbx.platform.typedef.__psize_t = ulong
+rbx.platform.typedef.__clock_t = int
+rbx.platform.typedef.__clockid_t = int
+rbx.platform.typedef.__off_t = long_long
+rbx.platform.typedef.__ptrdiff_t = long
+rbx.platform.typedef.__size_t = ulong
+rbx.platform.typedef.__ssize_t = long
+rbx.platform.typedef.__time_t = int
+rbx.platform.typedef.__timer_t = int
+rbx.platform.typedef.__wchar_t = int
+rbx.platform.typedef.__wint_t = int
+rbx.platform.typedef.__rune_t = int
+rbx.platform.typedef.__wctrans_t = pointer
+rbx.platform.typedef.__wctype_t = pointer
+rbx.platform.typedef.__cpuid_t = ulong
+rbx.platform.typedef.__dev_t = int
+rbx.platform.typedef.__fixpt_t = uint
+rbx.platform.typedef.__gid_t = uint
+rbx.platform.typedef.__id_t = uint
+rbx.platform.typedef.__in_addr_t = uint
+rbx.platform.typedef.__in_port_t = ushort
+rbx.platform.typedef.__ino_t = uint
+rbx.platform.typedef.__key_t = long
+rbx.platform.typedef.__mode_t = uint
+rbx.platform.typedef.__nlink_t = uint
+rbx.platform.typedef.__pid_t = int
+rbx.platform.typedef.__rlim_t = ulong_long
+rbx.platform.typedef.__sa_family_t = uchar
+rbx.platform.typedef.__segsz_t = int
+rbx.platform.typedef.__socklen_t = uint
+rbx.platform.typedef.__swblk_t = int
+rbx.platform.typedef.__uid_t = uint
+rbx.platform.typedef.__useconds_t = uint
+rbx.platform.typedef.__suseconds_t = int
+rbx.platform.typedef.u_char = uchar
+rbx.platform.typedef.u_short = ushort
+rbx.platform.typedef.u_int = uint
+rbx.platform.typedef.u_long = ulong
+rbx.platform.typedef.unchar = uchar
+rbx.platform.typedef.ushort = ushort
+rbx.platform.typedef.uint = uint
+rbx.platform.typedef.ulong = ulong
+rbx.platform.typedef.cpuid_t = ulong
+rbx.platform.typedef.register_t = long_long
+rbx.platform.typedef.int8_t = char
+rbx.platform.typedef.uint8_t = uchar
+rbx.platform.typedef.int16_t = short
+rbx.platform.typedef.uint16_t = ushort
+rbx.platform.typedef.int32_t = int
+rbx.platform.typedef.uint32_t = uint
+rbx.platform.typedef.int64_t = long_long
+rbx.platform.typedef.uint64_t = ulong_long
+rbx.platform.typedef.intptr_t = long
+rbx.platform.typedef.uintptr_t = ulong
+rbx.platform.typedef.u_int8_t = uchar
+rbx.platform.typedef.u_int16_t = ushort
+rbx.platform.typedef.u_int32_t = uint
+rbx.platform.typedef.u_int64_t = ulong_long
+rbx.platform.typedef.quad_t = long_long
+rbx.platform.typedef.u_quad_t = ulong_long
+rbx.platform.typedef.qaddr_t = pointer
+rbx.platform.typedef.vaddr_t = ulong
+rbx.platform.typedef.paddr_t = ulong
+rbx.platform.typedef.vsize_t = ulong
+rbx.platform.typedef.psize_t = ulong
+rbx.platform.typedef.caddr_t = string
+rbx.platform.typedef.daddr_t = int
+rbx.platform.typedef.daddr32_t = int
+rbx.platform.typedef.daddr64_t = long_long
+rbx.platform.typedef.dev_t = int
+rbx.platform.typedef.fixpt_t = uint
+rbx.platform.typedef.gid_t = uint
+rbx.platform.typedef.id_t = uint
+rbx.platform.typedef.ino_t = uint
+rbx.platform.typedef.key_t = long
+rbx.platform.typedef.mode_t = uint
+rbx.platform.typedef.nlink_t = uint
+rbx.platform.typedef.pid_t = int
+rbx.platform.typedef.rlim_t = ulong_long
+rbx.platform.typedef.segsz_t = int
+rbx.platform.typedef.swblk_t = int
+rbx.platform.typedef.uid_t = uint
+rbx.platform.typedef.useconds_t = uint
+rbx.platform.typedef.suseconds_t = int
+rbx.platform.typedef.in_addr_t = uint
+rbx.platform.typedef.in_port_t = ushort
+rbx.platform.typedef.sa_family_t = uchar
+rbx.platform.typedef.socklen_t = uint
+rbx.platform.typedef.clock_t = int
+rbx.platform.typedef.clockid_t = int
+rbx.platform.typedef.size_t = ulong
+rbx.platform.typedef.ssize_t = long
+rbx.platform.typedef.time_t = int
+rbx.platform.typedef.timer_t = int
+rbx.platform.typedef.off_t = long_long
+rbx.platform.typedef.__fd_mask = int