Update documentation about AMP communication

Change-Id: I3d7926cb16b613f588ef33944546df394193160e

1 files changed, 184 insertions, 88 deletions

diff --git a/README.md b/README.md
index aa8cf33..e216c9d 100644
--- a/README.md
+++ b/README.md
@@ -1,13 +1,17 @@
 # Linux driver stack for Arm(R) Ethos(TM)-U
 
-The Linux driver stack for Arm(R) Ethos(TM)-U provides an example of how a rich
-operating system like Linux can dispatch inferences to an Arm Cortex(R)-M
-subsystem, consisting of an Arm Cortex-M of choice and an Arm Ethos-U NPU.
+The Linux driver stack for Arm(R) Ethos(TM)-U demonstrates how to setup a
+Asymmetric MultiProcessing (AMP) system, dispatching inferences to an Ethos-U
+subsystem consisting of an Arm Cortex(R)-M and one or multiple Ethos-U NPUs.
 
-## Building
+The communication between Linux and the firmware is based on the Linux APIs
+remoteproc, rpmsg, virtio, reset and mailbox. These APIs are implemented on the
+firmware side by the OpenAMP framework.
+
+# Building
 
 The driver stack comes with a CMake based build system. A toolchain file is
-provided for reference how to cross compile for an Aarch64 based system.
+provided for reference how to cross compile for Aarch64 based systems.
 
 Note that if your host system provides cross compilers and libraries of newer
 versions than what is supported on your target system, you might be required to
@@ -17,78 +21,190 @@ While out of scope for this README, an example
 show what it could look like. Another option is to run a Docker image of an
 appropriate Linux distribution suited to build for your needs.
 
-Building the kernel modules requires a configured Linux kernel source tree. How
-to download and configure the Linux kernel goes beyond the scope of this readme.
-Please refer to the Linux kernel official documentation.
+Building the kernel modules requires a configured Linux kernel source tree.
+Please refer to the Linux kernel official documentation for instructions how to
+configure and build the Linux kernel.
 
 ```
-$ mkdir build
-$ cd build
-$ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain/aarch64-linux-gnu.cmake -DKDIR=<Kernel directory>
-$ make
+$ cmake -B build --toolchain $PWD/cmake/toolchain/aarch64-linux-gnu.cmake -DKDIR=<Kernel directory>
+$ cmake --build build
 ```
 
-## DTB
-
-The kernel driver uses the mailbox APIs as a doorbell mechanism.
+# Kernel drivers
+
+The communication between Linux and the Ethos-U subsystem is based on the
+remoteproc, rpmsg and virtio APIs, which provide a standardized way of
+communicating between Linux and a wide range of Asymmetric MultiProcessing (AMP)
+systems. Because these APIs are native to the Linux kernel, a bare metal
+application would typically use the [OpenAMP](https://www.openampproject.org/)
+framework to provide an implementation for embedded systems.
+
+To setup the AMP communication for a target platform, a user would for most
+platforms need to provide four kernel drivers:
+[remoteproc](https://www.kernel.org/doc/html/latest/staging/remoteproc.html),
+[reset](https://www.kernel.org/doc/html/latest/driver-api/reset.html),
+[mailbox](https://www.kernel.org/doc/html/latest/driver-api/mailbox.html) and
+[rpmsg](https://www.kernel.org/doc/html/latest/staging/rpmsg.html).
+
+## Remoteproc driver
+
+The remoteproc driver is responsible for parsing the firmware binary, allocating
+resources and booting up the firmware. It needs help of a reset driver to
+_assert_ and _deassert_ reset for the Cortex-M CPU, and a mailbox driver to send
+and receive IRQs between the Cortex-M CPU and the host CPU.
+
+The firmware binary contains a _resource table_ that lists resources Linux is
+required to allocate before the firmware is booted up. These resources usually
+include the rpmsg virtio message queues, memory resources and trace buffers.
+
+The [ethosu_remoteproc.c](remoteproc/ethosu_remoteproc.c) is provided for
+reference, but could be replaced by any remoteproc driver.
+
+## Ethos-U rpmsg driver
+
+The [Ethos-U rpmsg driver](kernel/ethosu_driver.c) uses dynamic name resolution.
+This means that when the firmware boots up it calls `rpmsg_create_ept()` to
+create a dynamic endpoint with a name of maximum 32 characters. The service name
+must be unique and is forwarded to Linux in a _name resolution_ message.
+
+On the Linux side the name resolution message is handled by the
+[rpmsg_ns.c](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/rpmsg/rpmsg_ns.c)
+driver, which allocates a new rpmsg device and probes the rpmsg bus for a driver
+ready to serve the firmware. Dynamic name resolution has the advantage that the
+rpmsg probe function is called after the firmware has booted up, and the kernel
+driver will consequently only allocate resources while the firmware is running.
+
+The
+[message handler openamp](https://git.mlplatform.org/ml/ethos-u/ethos-u-core-platform.git/tree/applications/message_handler_openamp)
+creates a new endpoint named `ethos-u-0.0`, which is serviced by the
+[Ethos-U rpmsg driver](kernel/ethosu_driver.c). It is important that firmware
+and rpmsg driver use the same endpoint name, or else the endpoint will not be
+successfully created.
+
+The purpose of the Ethos-U rpmsg driver is to enable user space applications to
+dispatch inferences to the the Ethos-U subsystem. The driver presents a
+[Userspace API (UAPI)](kernel/uapi/ethosu.h), which applications use IOCTL on to
+allocate buffers, create networks and dispatch inferences. The types for the
+rpmsg communication between Linux and the firmware are defined in
+[ethosu_core_rpmsg.h](kernel/ethosu_core_rpmsg.h).
+
+# DTB
+
+The DTB example below is fetched from a Corstone-1000 FVP, which has been
+modified to insert an Ethos-U subsystem in one of the M-class slots. Mailbox
+communication is based on the Arm MHU v2, and the reset is controlled using a IO
+mapped bridge.
+
+## ethosu-rproc
+
+The [Ethos-U remoteproc driver](remoteproc/ethosu_remoteproc.c) is used to load
+and bootup the firmware.
+
+The firmware in this particular example has two memory regions that are
+hard coded into the binary, the _code_ and _shared memory_ segments. These memory
+regions and their sizes are configured by the `reg` entry. Because the _shared
+memory_ region is located in DDR, the `rproc_shared` is required to carve out
+the memory region from the dynamic memory pool.
+
+The `memory-region` entry references the shared memory pool. This is the memory
+region used to allocate DMA buffers for the virtio rpmsg message queues and
+other memory resources.
+
+The `ethosu,dma-ranges` is specific to the Ethos-U remoteproc driver, and is
+used to translate Linux physical addresses into DMA addresses. The reason for
+not using `dma-range` directly is that `dma-ranges` is not fully supported by
+the
+[virtio_ring.c](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/virtio/virtio_ring.c)
+kernel driver. Buffers sent to the firmware will contain Linux physical
+addresses instead of DMA addresses.
+
+The `reset` entry is referencing the driver used to control reset for the
+subsystem.
+
+The `mboxes` entry is referencing the MHU v2 driver used for mailbox
+communication.
+
+## Corstone-1000 DTB
 
 ```
 / {
-  reserved-memory {
-    #address-cells = <2>;
-    #size-cells = <2>;
-    ranges;
-
-    ethosu_msg: ethosu_msg@80000000 {
-      compatible = "shared-dma-pool";
-      reg = <0 0x80000000 0 0x00040000>;
-      no-map;
+    reserved-memory {
+        #address-cells = <1>;
+        #size-cells = <1>;
+        ranges;
+
+        /* Reserved memory region.  */
+        rproc_shared: rproc_shared@84000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x84000000 0x01000000>;
+            no-map;
+        };
+
+        /* Memory region used for DMA buffer allocation */
+        rproc_reserved: rproc_reserved@85000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x85000000 0x08000000>;
+            no-map;
+        };
+    };
+
+    mbox_es0mhu0: mhu@1b000000 {
+        compatible = "arm,mhuv2","arm,primecell";
+        reg = <0x1b000000 0x1000>,
+              <0x1b010000 0x1000>;
+        clocks = <&refclk100mhz>;
+        clock-names = "apb_pclk";
+        interrupts = <0 12 4>;
+        interrupt-names = "mhu_rx";
+        #mbox-cells = <1>;
+        mbox-name = "arm-es0-mhu0";
     };
 
-    ethosu_reserved: ethosu_reserved@80040000 {
-      compatible = "shared-dma-pool";
-      reg = <0 0x80040000 0 0x00040000>;
-      no-map;
+    cs1k_rst_es0: cs1k_reset_es0@1A010310 {
+        compatible = "arm,cs1k_es_rst";
+        #reset-cells = <0>;
+        reg = <0x1A010310 0x4>,
+              <0x1A010314 0x4>;
+        reg-names = "rstreg", "streg";
+    };
+
+    /* - compatible         : "arm,ethosu-rproc"
+     * - reg                : Memory regions reserved for firmware binary
+     * - ethosu,dma-ranges  : Address translation from bus address (Cortex-M)
+     *                        to Linux kernel physical address
+     *      - bus address      (#address-cells)
+     *      - physical address (#address-cells)
+     *      - size             (#size-cells)
+     * - memory-region      : Memory region for allocation of DMA buffers
+     * - mboxes             : Mailbox driver used to raise interrupts on
+     *                        remote CPU
+     * - resets		: Reset driver used to reset remote CPU
+     */
+    ethosu-rproc {
+        compatible = "arm,ethosu-rproc";
+
+        // Address mapping from DMA to physical
+        ethosu,dma-ranges = <0x00000000 0x44000000 0x00200000>,
+                            <0x60000000 0x80000000 0x10000000>;
+
+        reg = <0x44000000 0x00200000>,
+              <0x84000000 0x01000000>;
+        reg-names = "rom", "shared";
+
+        // Memory region to allocate buffers from
+        memory-region = <&rproc_reserved>;
+
+        // Mailbox IRQ communication
+        mboxes = <&mbox_es0mhu0 0>, <&mbox_es0mhu0 0>;
+        mbox-names = "tx", "rx";
+
+        // Reset handler
+        resets = <&cs1k_rst_es0 0>;
     };
-  };
-
-  ethosu_mailbox: mhu@6ca00000 {
-    compatible = "arm,mhu", "arm,primecell";
-    reg = <0x0 0x6ca00000 0x0 0x1000>;
-    interrupts = <0 168 4>;
-    interrupt-names = "npu_rx";
-    #mbox-cells = <1>;
-    clocks = <&soc_refclk100mhz>;
-    clock-names = "apb_pclk";
-  };
-
-  ethosu_mcu {
-    compatible ="arm, ethosu-sgm775-rproc";
-    reg = <0x0 0x500f0000 0x0 0x1000>,
-          <0x0 0x50100000 0x0 0x100000>;
-    reg-names = "bridge", "firmware";
-  };
-
-  ethosu {
-    #address-cells = <2>;
-    #size-cells = <2>;
-
-    compatible = "arm,ethosu";
-    reg = <0 0x80000000 0 0x00010000>,
-          <0 0x80010000 0 0x00010000>;
-    reg-names = "in_queue", "out_queue";
-    memory-region = <&ethosu_reserved>;
-    dma-ranges = <0 0x60000000 0 0x80000000 0 0x20000000>;
-    mboxes= <&ethosu_mailbox 0>, <&ethosu_mailbox 0>;
-    mbox-names = "tx", "rx";
-    ethosu-rproc = <&ethosu_mcu>;
-  };
 };
 ```
 
-# Documentation
-
-## Driver library
+# Driver library
 
 The purpose of the driver library is to provide user friendly C++ APIs for
 dispatching inferences to the Ethos-U kernel driver.
@@ -121,27 +237,7 @@ complete.
 
 ![Driver library](docs/driver_library_sequence.svg "Driver library sequence diagram")
 
-## Ethos-U core interface
-
-The task of the Ethos-U kernel driver is to present a Userspace API (UAPI) to
-user space, and to communicate with the Cortex-M in the Ethos-U subsystem.
-
-The communication with the Ethos-U subsystem is based on message passing in
-shared memory, and the Linux kernel mailbox APIs for triggering IRQs on the
-remote CPU.
-
-The address of the message queues is hard coded in the Cortex-M application, and
-configured in the DTB for the kernel driver.
-
-When the kernel driver allocates dynamic memory for the Ethos-U subsystem it
-must be able to map a physical address to a bus address. The DTB contains a
-`dma-ranges` which define how to remap physical addresses to the Cortex-M
-address space.
-
-Triggering IRQs on a remote CPU requires external hardware support, for example
-the Arm MHU.
-
-### Device and buffer
+## Device and buffer
 
 The device driver creates a device node at `/dev/ethosu<nr>` that a user space
 application can open and issues IOCTL requests to. This is how buffers and
@@ -152,7 +248,7 @@ for reading and/or writing.
 
 ![Create buffer](docs/kernel_buffer.svg "Create buffer")
 
-### Network
+## Network
 
 Creating a network assumes that the device node has already been opened, and
 that a buffer has been allocated and populated with the network model.
@@ -165,7 +261,7 @@ been destructed.
 
 ![Create network](docs/kernel_network.svg "Create network")
 
-### Inference
+## Inference
 
 Creating an inference assumes that a network has already been created, IFM
 buffers have been allocated and populated with data, and OFM buffers have been
@@ -219,7 +315,7 @@ can't be easily moved.
 # Licenses
 
 The kernel drivers are provided under a GPL v2 license. All other software
-componantes are provided under an Apache 2.0 license.
+components are provided under an Apache 2.0 license.
 
 Please see [LICENSE-APACHE-2.0.txt](LICENSE-APACHE-2.0.txt) and
 [LICENSE-GPL-2.0.txt](LICENSE-GPL-2.0.txt) for more information.