improve Pin documentation

Author: RalfJung

src/libcore/marker.rs

@@ -597,7 +597,8 @@ unsafe impl<T: ?Sized> Freeze for &mut T {}
 
 /// Types which can be safely moved after being pinned.
 ///
-/// Since Rust itself has no notion of immovable types, and will consider moves to always be safe,
+/// Since Rust itself has no notion of immovable types, and will consider moves
+/// (e.g. through assignment or [`mem::replace`]) to always be safe,
 /// this trait cannot prevent types from moving by itself.
 ///
 /// Instead it can be used to prevent moves through the type system,
@@ -606,7 +607,12 @@ unsafe impl<T: ?Sized> Freeze for &mut T {}
 /// See the [`pin module`] documentation for more information on pinning.
 ///
 /// Implementing this trait lifts the restrictions of pinning off a type,
-/// which then allows it to move out with functions such as [`replace`].
+/// which then allows it to move out with functions such as [`mem::replace`].
+///
+/// `Unpin` has no consequence at all for non-pinned data. In particular,
+/// [`mem::replace`] will happily move `!Unpin` data. However, you cannot use
+/// [`mem::replace`] on data wrapped inside a [`Pin`], and *that* is what makes
+/// this system work.
 ///
 /// So this, for example, can only be done on types implementing `Unpin`:
 ///
@@ -623,7 +629,7 @@ unsafe impl<T: ?Sized> Freeze for &mut T {}
 ///
 /// This trait is automatically implemented for almost every type.
 ///
-/// [`replace`]: ../../std/mem/fn.replace.html
+/// [`mem::replace`]: ../../std/mem/fn.replace.html
 /// [`Pin`]: ../pin/struct.Pin.html
 /// [`pin module`]: ../../std/pin/index.html
 #[stable(feature = "pin", since = "1.33.0")]

src/libcore/pin.rs

@@ -16,29 +16,30 @@
 //! but doesn't allow moving `T`. The pointer value itself (the `Box`) can still be moved,
 //! but the value behind it cannot.
 //!
-//! Since data can be moved out of `&mut` and `Box` with functions such as [`swap`],
+//! Since data can be moved out of `&mut` and `Box` with functions such as [`mem::swap`],
 //! changing the location of the underlying data, [`Pin`] prohibits accessing the
 //! underlying pointer type (the `&mut` or `Box`) directly, and provides its own set of
 //! APIs for accessing and using the value. [`Pin`] also guarantees that no other
 //! functions will move the pointed-to value. This allows for the creation of
 //! self-references and other special behaviors that are only possible for unmovable
 //! values.
 //!
-//! However, these restrictions are usually not necessary. Many types are always freely
-//! movable. These types implement the [`Unpin`] auto-trait, which nullifies the effect
-//! of [`Pin`]. For `T: Unpin`, `Pin<Box<T>>` and `Box<T>` function identically, as do
-//! `Pin<&mut T>` and `&mut T`.
+//! It is worth reiterating that [`Pin`] does *not* change the fact that the Rust compiler
+//! considers all types movable.  [`mem::swap`] remains callable for any `T`.  Instead, `Pin`
+//! prevents certain *values* (pointed to by pointers wrapped in `Pin`) from being
+//! moved by making it impossible to call methods like [`mem::swap`] on them.
 //!
-//! Note that pinning and `Unpin` only affect the pointed-to type. For example, whether
-//! or not `Box<T>` is `Unpin` has no affect on the behavior of `Pin<Box<T>>`. Similarly,
-//! `Pin<Box<T>>` and `Pin<&mut T>` are always `Unpin` themselves, even though the
-//! `T` underneath them isn't, because the pointers in `Pin<Box<_>>` and `Pin<&mut _>`
-//! are always freely movable, even if the data they point to isn't.
+//! # `Unpin`
 //!
-//! [`Pin`]: struct.Pin.html
-//! [`Unpin`]: ../../std/marker/trait.Unpin.html
-//! [`swap`]: ../../std/mem/fn.swap.html
-//! [`Box`]: ../../std/boxed/struct.Box.html
+//! However, these restrictions are usually not necessary. Many types are always freely
+//! movable, even when pinned. These types implement the [`Unpin`] auto-trait, which
+//! nullifies the effect of [`Pin`]. For `T: Unpin`, `Pin<Box<T>>` and `Box<T>` function
+//! identically, as do `Pin<&mut T>` and `&mut T`.
+//!
+//! Note that pinning and `Unpin` only affect the pointed-to type, not the pointer
+//! type itself that got wrapped in `Pin`. For example, whether or not `Box<T>` is
+//! `Unpin` has no affect on the behavior of `Pin<Box<T>>` (here, `T` is the
+//! pointed-to type).
 //!
 //! # Examples
 //!
@@ -94,6 +95,106 @@
 //! // let new_unmoved = Unmovable::new("world".to_string());
 //! // std::mem::swap(&mut *still_unmoved, &mut *new_unmoved);
 //! ```
+//!
+//! # `Drop` guarantee
+//!
+//! The purpose of pinning is to be able to rely on the placement of some data in memory.
+//! To make this work, not just moving the data is restricted; deallocating or overwriting
+//! it is restricted, too. Concretely, for pinned data you have to maintain the invariant
+//! that *it will not get overwritten or deallocated until `drop` was called*.
+//! ("Overwriting" here refers to other ways of invalidating storage, such as switching
+//! from one enum variant to another.)
+//!
+//! The purpose of this guarantee is to allow data structures that store pointers
+//! to pinned data. For example, in an intrusive doubly-linked list, every element
+//! will have pointers to its predecessor and successor in the list. Every element
+//! will be pinned, because moving the elements around would invalidate the pointers.
+//! Moreover, the `Drop` implemenetation of a linked list element will patch the pointers
+//! of its predecessor and successor to remove itself from the list. Clearly, if an element
+//! could be deallocated or overwritten without calling `drop`, the pointers into it
+//! from its neighbouring elements would become invalid, breaking the data structure.
+//!
+//! Notice that this guarantee does *not* mean that memory does not leak! It is still
+//! completely okay not to ever call `drop` on a pinned element (e.g., you can still
+//! call [`mem::forget`] on a `Pin<Box<T>>`). What you may not do is free or reuse the storage
+//! without calling `drop`.
+//!
+//! # `Drop` implementation
+//!
+//! If your type relies on pinning (for example, because it contains internal
+//! references, or because you are implementing something like the intrusive
+//! doubly-linked list mentioned in the previous section), you have to be careful
+//! when implementing `Drop`: notice that `drop` takes `&mut self`, but this
+//! will be called even if your type was previously pinned! It is as if the
+//! compiler automatically called `get_unchecked_mut`. This can never cause
+//! a problem in safe code because implementing a type that relies on pinning
+//! requires unsafe code, but be aware that deciding to make use of pinning
+//! in your type (for example by implementing some operation on `Pin<&[mut] Self>`)
+//! has consequences for your `Drop` implemenetation as well.
+//!
+//! # Projections and Structural Pinning
+//!
+//! One interesting question arises when considering pinning and "container types" --
+//! types such as `Vec` or `Box` but also `RefCell`; types that serve as wrappers
+//! around other types.  When can such a type have a "projection" operation, an
+//! operation with type `fn(Pin<&[mut] Container<T>>) -> Pin<&[mut] T>`?
+//! This does not just apply to generic container types, even for normal structs
+//! the question arises whether `fn(Pin<&[mut] Struct>) -> Pin<&[mut] Field>`
+//! is an operation that can be soundly added to the API.
+//!
+//! This question is closely related to the question of whether pinning is "structural":
+//! when you have pinned a container, have you pinned its contents? Adding a
+//! projection to the API answers that question with a "yes" by offering pinned access
+//! to the contents.
+//!
+//! In general, as the author of a type you get to decide whether pinning is structural, and
+//! whether projections are provided. However, there are a couple requirements to be
+//! upheld when adding projection operations:
+//!
+//! 1. The container must only be [`Unpin`] if all its fields are `Unpin`. This is the default,
+//!    but `Unpin` is a safe trait, so as the author of the container it is your responsibility
+//!    *not* to add something like `impl<T> Unpin for Container<T>`. (Notice that adding a
+//!    projection operation requires unsafe code, so the fact that `Unpin` is a safe trait
+//!    does not break the principle that you only have to worry about any of this if
+//!    you use `unsafe`.)
+//! 2. The destructor of the container must not move out of its argument. This is the exact
+//!    point that was raised in the [previous section][drop-impl]: `drop` takes `&mut self`,
+//!    but the container (and hence its fields) might have been pinned before.
+//!    You have to guarantee that you do not move a field inside your `Drop` implementation.
+//! 3. Your container type must *not* be `#[repr(packed)]`. Packed structs have their fields
+//!    moved around when they are dropped to properly align them, which is in conflict with
+//!    claiming that the fields are pinned when your struct is.
+//! 4. You must make sure that you uphold the [`Drop` guarantee][drop-guarantee]:
+//!    you must make sure that, once your container is pinned, the memory containing the
+//!    content is not overwritten or deallocated without calling the content's destructors.
+//!    This can be tricky, as witnessed by `VecDeque`: the destructor of `VecDeque` can fail
+//!    to call `drop` on all elements if one of the destructors panics. This violates the
+//!    `Drop` guarantee, because it can lead to elements being deallocated without
+//!    their destructor being called.
+//! 5. You must not offer any other operations that could lead to data being moved out of
+//!    the fields when your type is pinned. This is usually not a concern, but can become
+//!    tricky when interior mutability is involved. For example, imagine `RefCell`
+//!    would have a method `fn get_pin_mut(self: Pin<&mut Self>) -> Pin<&mut T>`.
+//!    This would be catastrophic, because it is possible to move out of a pinned
+//!    `RefCell`: from `x: Pin<&mut RefCell<T>>`, use `let y = x.into_ref().get_ref()` to obtain
+//!    `y: &RefCell<T>`, and from there use `y.borrow_mut().deref_mut()` to obtain `&mut T`
+//!    which can be used with [`mem::swap`].
+//!
+//! On the other hand, if you decide *not* to offer any pinning projections, you
+//! are free to do `impl<T> Unpin for Container<T>`.  In the standard library,
+//! we do this for all pointer types: `Box<T>: Unpin` holds for all `T`.
+//! It makes a lot of sense to do this for pointer types, because moving the `Box<T>`
+//! does not actually move the `T`: the `Box<T>` can be freely movable even if the `T`
+//! is not. In fact, even `Pin<Box<T>>` and `Pin<&mut T>` are always `Unpin` themselves,
+//! for the same reason.
+//!
+//! [`Pin`]: struct.Pin.html
+//! [`Unpin`]: ../../std/marker/trait.Unpin.html
+//! [`mem::swap`]: ../../std/mem/fn.swap.html
+//! [`mem::forget`]: ../../std/mem/fn.forget.html
+//! [`Box`]: ../../std/boxed/struct.Box.html
+//! [drop-impl]: #drop-implementation
+//! [drop-guarantee]: #drop-guarantee
 
 #![stable(feature = "pin", since = "1.33.0")]
 
@@ -170,7 +271,12 @@ where
     P::Target: Unpin,
 {
     /// Construct a new `Pin` around a pointer to some data of a type that
-    /// implements `Unpin`.
+    /// implements [`Unpin`].
+    ///
+    /// Unlike `Pin::new_unchecked`, this method is safe because the pointer
+    /// `P` dereferences to an [`Unpin`] type, which nullifies the pinning guarantees.
+    ///
+    /// [`Unpin`]: ../../std/marker/trait.Unpin.html
     #[stable(feature = "pin", since = "1.33.0")]
     #[inline(always)]
     pub fn new(pointer: P) -> Pin<P> {
@@ -191,15 +297,46 @@ impl<P: Deref> Pin<P> {
     /// not guarantee that the data `P` points to is pinned, constructing a
     /// `Pin<P>` is undefined behavior.
     ///
+    /// By using this method, you are making a promise about the `P::Deref` and
+    /// `P::DerefMut` implementations, if they exist. Most importantly, they
+    /// must not move out of their `self` arguments: `Pin::as_mut` and `Pin::as_ref`
+    /// will call `DerefMut::deref_mut` and `Deref::deref` *on the pinned pointer*
+    /// and expect these methods to uphold the pinning invariants.
+    /// Moreover, by calling this method you promise that the reference `P`
+    /// dereferences to will not be moved out of again; in particular, it
+    /// must not be possible to obtain a `&mut P::Target` and then
+    /// move out of that reference (using, for example [`replace`]).
+    ///
+    /// For example, the following is a *violation* of `Pin`'s safety:
+    /// ```
+    /// use std::mem;
+    /// use std::pin::Pin;
+    ///
+    /// fn foo<T>(mut a: T, b: T) {
+    ///     unsafe { let p = Pin::new_unchecked(&mut a); } // should mean `a` can never move again
+    ///     let a2 = mem::replace(&mut a, b);
+    ///     // the address of `a` changed to `a2`'s stack slot, so `a` got moved even
+    ///     // though we have previously pinned it!
+    /// }
+    /// ```
+    ///
     /// If `pointer` dereferences to an `Unpin` type, `Pin::new` should be used
     /// instead.
+    ///
+    /// [`replace`]: ../../std/mem/fn.replace.html
     #[stable(feature = "pin", since = "1.33.0")]
     #[inline(always)]
     pub unsafe fn new_unchecked(pointer: P) -> Pin<P> {
         Pin { pointer }
     }
 
     /// Gets a pinned shared reference from this pinned pointer.
+    ///
+    /// This is a generic method to go from `&Pin<SmartPointer<T>>` to `Pin<&T>`.
+    /// It is safe because, as part of the contract of `Pin::new_unchecked`,
+    /// the pointee cannot move after `Pin<SmartPointer<T>>` got created.
+    /// "Malicious" implementations of `SmartPointer::Deref` are likewise
+    /// ruled out by the contract of `Pin::new_unchecked`.
     #[stable(feature = "pin", since = "1.33.0")]
     #[inline(always)]
     pub fn as_ref(self: &Pin<P>) -> Pin<&P::Target> {
@@ -209,13 +346,22 @@ impl<P: Deref> Pin<P> {
 
 impl<P: DerefMut> Pin<P> {
     /// Gets a pinned mutable reference from this pinned pointer.
+    ///
+    /// This is a generic method to go from `&mut Pin<SmartPointer<T>>` to `Pin<&mut T>`.
+    /// It is safe because, as part of the contract of `Pin::new_unchecked`,
+    /// the pointee cannot move after `Pin<SmartPointer<T>>` got created.
+    /// "Malicious" implementations of `SmartPointer::DerefMut` are likewise
+    /// ruled out by the contract of `Pin::new_unchecked`.
     #[stable(feature = "pin", since = "1.33.0")]
     #[inline(always)]
     pub fn as_mut(self: &mut Pin<P>) -> Pin<&mut P::Target> {
         unsafe { Pin::new_unchecked(&mut *self.pointer) }
     }
 
-    /// Assign a new value to the memory behind the pinned reference.
+    /// Assigns a new value to the memory behind the pinned reference.
+    ///
+    /// This overwrites pinned data, but that is okay: its destructor gets
+    /// run before being overwritten, so no pinning guarantee is violated.
     #[stable(feature = "pin", since = "1.33.0")]
     #[inline(always)]
     pub fn set(self: &mut Pin<P>, value: P::Target)
@@ -227,17 +373,21 @@ impl<P: DerefMut> Pin<P> {
 }
 
 impl<'a, T: ?Sized> Pin<&'a T> {
-    /// Construct a new pin by mapping the interior value.
+    /// Constructs a new pin by mapping the interior value.
     ///
     /// For example, if you  wanted to get a `Pin` of a field of something,
     /// you could use this to get access to that field in one line of code.
+    /// However, there are several gotchas with these "pinning projections";
+    /// see the [`pin` module] documentation for further details on that topic.
     ///
     /// # Safety
     ///
     /// This function is unsafe. You must guarantee that the data you return
     /// will not move so long as the argument value does not move (for example,
     /// because it is one of the fields of that value), and also that you do
     /// not move out of the argument you receive to the interior function.
+    ///
+    /// [`pin` module]: ../../std/pin/index.html#projections-and-structural-pinning
     #[stable(feature = "pin", since = "1.33.0")]
     pub unsafe fn map_unchecked<U, F>(self: Pin<&'a T>, func: F) -> Pin<&'a U> where
         F: FnOnce(&T) -> &U,
@@ -249,11 +399,21 @@ impl<'a, T: ?Sized> Pin<&'a T> {
 
     /// Gets a shared reference out of a pin.
     ///
+    /// This is safe because it is not possible to move out of a shared reference.
+    /// It may seem like there is an issue here with interior mutability: in fact,
+    /// it *is* possible to move a `T` out of a `&RefCell<T>`. However, this is
+    /// not a problem as long as there does not also exist a `Pin<&T>` pointing
+    /// to the same data, and `RefCell` does not let you create a pinned reference
+    /// to its contents. See the discussion on ["pinning projections"] for further
+    /// details.
+    ///
     /// Note: `Pin` also implements `Deref` to the target, which can be used
     /// to access the inner value. However, `Deref` only provides a reference
     /// that lives for as long as the borrow of the `Pin`, not the lifetime of
     /// the `Pin` itself. This method allows turning the `Pin` into a reference
     /// with the same lifetime as the original `Pin`.
+    ///
+    /// ["pinning projections"]: ../../std/pin/index.html#projections-and-structural-pinning
     #[stable(feature = "pin", since = "1.33.0")]
     #[inline(always)]
     pub fn get_ref(self: Pin<&'a T>) -> &'a T {
@@ -306,13 +466,17 @@ impl<'a, T: ?Sized> Pin<&'a mut T> {
     ///
     /// For example, if you  wanted to get a `Pin` of a field of something,
     /// you could use this to get access to that field in one line of code.
+    /// However, there are several gotchas with these "pinning projections";
+    /// see the [`pin` module] documentation for further details on that topic.
     ///
     /// # Safety
     ///
     /// This function is unsafe. You must guarantee that the data you return
     /// will not move so long as the argument value does not move (for example,
     /// because it is one of the fields of that value), and also that you do
     /// not move out of the argument you receive to the interior function.
+    ///
+    /// [`pin` module]: ../../std/pin/index.html#projections-and-structural-pinning
     #[stable(feature = "pin", since = "1.33.0")]
     pub unsafe fn map_unchecked_mut<U, F>(self: Pin<&'a mut T>, func: F) -> Pin<&'a mut U> where
         F: FnOnce(&mut T) -> &mut U,