The Graphics Execution Manager
	      Part of the Direct Rendering Manager
                  ==============================
		  
		 Keith Packard <keithp@keithp.com>
		   Eric Anholt <eric@anholt.net>
			   2008-5-9

Contents:

 1. GEM Overview
 2. API overview and conventions
 3. Object Creation/Destruction
 4. Reading/writing contents
 5. Mapping objects to userspace
 6. Memory Domains
 7. Execution (Intel specific)
 8. Other misc Intel-specific functions

1. Graphics Execution Manager Overview

Gem is designed to manage graphics memory, control access to the graphics
device execution context and handle the essentially NUMA environment unique
to modern graphics hardware. Gem allows multiple applications to share
graphics device resources without the need to constantly reload the entire
graphics card. Data may be shared between multiple applications with gem
ensuring that the correct memory synchronization occurs.

Graphics data can consume arbitrary amounts of memory, with 3D applications
constructing ever larger sets of textures and vertices. With graphics cards
memory space growing larger every year, and graphics APIs growing more
complex, we can no longer insist that each application save a complete copy
of their graphics state so that the card can be re-initialized from user
space at each context switch. Ensuring that graphics data remains persistent
across context switches allows applications significant new functionality
while also improving performance for existing APIs.

Modern linux desktops include significant 3D rendering as a fundemental
component of the desktop image construction process. 2D and 3D applications
paint their content to offscreen storage and the central 'compositing
manager' constructs the final screen image from those window contents.  This
means that pixel image data from these applications must move within reach
of the compositing manager and used as source operands for screen image
rendering operations.

Gem provides simple mechanisms to manage graphics data and control execution
flow within the linux operating system. Using many existing kernel
subsystems, it does this with a modest amount of code.

2. API Overview and Conventions

All APIs here are defined in terms of ioctls appplied to the DRM file
descriptor. To create and manipulate objects, an application must be
'authorized' using the DRI or DRI2 protocols with the X server. To relax
that, we will need to implement some better access control mechanisms within
the hardware portion of the driver to prevent inappropriate
cross-application data access.

Any DRM driver which does not support GEM will return -ENODEV for all of
these ioctls. Invalid object handles return -EINVAL. Invalid object names
return -ENOENT. Other errors are as documented in the specific API below.

To avoid the need to translate ioctl contents on mixed-size systems (with
32-bit user space running on a 64-bit kernel), the ioctl data structures
contain explicitly sized objects, using 64-bits for all size and pointer
data and 32-bits for identifiers. In addition, the 64-bit objects are all
carefully aligned on 64-bit boundaries. Because of this, all pointers in the
ioctl data structures are passed as uint64_t values. Suitable casts will
be necessary.

One significant operation which is explicitly left out of this API is object
locking. Applications are expected to perform locking of shared objects
outside of the GEM api. This kind of locking is not necessary to safely
manipulate the graphics engine, and with multiple objects interacting in
unknown ways, per-object locking would likely introduce all kinds of
lock-order issues. Punting this to the application seems like the only
sensible plan. Given that DRM already offers a global lock on the hardware,
this doesn't change the current situation.

3. Object Creation and Destruction

Gem provides explicit memory management primitives. System pages are
allocated when the object is created, either as the fundemental storage for
hardware where system memory is used by the graphics processor directly, or
as backing store for graphics-processor resident memory.

Objects are referenced from user space using handles. These are, for all
intents and purposes, equivalent to file descriptors. We could simply use
file descriptors were it not for the small limit (1024) of file descriptors
available to applications, and for the fact that the X server (a rather
significant user of this API) uses 'select' and has a limited maximum file
descriptor for that operation. Given the ability to allocate more file
descriptors, and given the ability to place these 'higher' in the file
descriptor space, we'd love to simply use file descriptors.

Objects may be published with a name so that other applications can access
them. The name remains valid as long as the object exists. Right now, our
DRI APIs use 32-bit integer names, so that's what we expose here

 A. Creation

		struct drm_gem_create {
			/**
			 * Requested size for the object.
			 *
			 * The (page-aligned) allocated size for the object
			 * will be returned.
			 */
			uint64_t size;
			/**
			 * Returned handle for the object.
			 *
			 * Object handles are nonzero.
			 */
			uint32_t handle;
			uint32_t pad;
		};
	
		/* usage */
    		create.size = 16384;
		ret = ioctl (fd, DRM_IOCTL_GEM_CREATE, &create);
		if (ret == 0)
			return create.handle;

	Note that the size is rounded up to a page boundary, and that
	the rounded-up size is returned in 'size'. No name is assigned to
	this object, making it local to this process.

	If insufficient memory is availabe, -ENOMEM will be returned.

 B. Closing

		struct drm_gem_close {
			/** Handle of the object to be closed. */
			uint32_t handle;
			uint32_t pad;
		};
		

		/* usage */
		close.handle = <handle>;
		ret = ioctl (fd, DRM_IOCTL_GEM_CLOSE, &close);

	This call makes the specified handle invalid, and if no other
	applications are using the object, any necessary graphics hardware
	synchronization is performed and the resources used by the object
	released.

 C. Naming

		struct drm_gem_flink {
			/** Handle for the object being named */
			uint32_t handle;
		
			/** Returned global name */
			uint32_t name;
		};
		
		/* usage */
		flink.handle = <handle>;
		ret = ioctl (fd, DRM_IOCTL_GEM_FLINK, &flink);
		if (ret == 0)
			return flink.name;

	Flink creates a name for the object and returns it to the
	application. This name can be used by other applications to gain
	access to the same object.

 D. Opening by name

		struct drm_gem_open {
			/** Name of object being opened */
			uint32_t name;
		
			/** Returned handle for the object */
			uint32_t handle;
			
			/** Returned size of the object */
			uint64_t size;
		};
		
		/* usage */
		open.name = <name>;
		ret = ioctl (fd, DRM_IOCTL_GEM_OPEN, &open);
		if (ret == 0) {
			*sizep = open.size;
			return open.handle;
		}

	Open accesses an existing object and returns a handle for it. If the
	object doesn't exist, -ENOENT is returned. The size of the object is
	also returned. This handle has all the same capabilities as the
	handle used to create the object. In particular, the object is not
	destroyed until all handles are closed.

4. Basic read/write operations

By default, gem objects are not mapped to the applications address space,
getting data in and out of them is done with I/O operations instead. This
allows the data to reside in otherwise unmapped pages, including pages in
video memory on an attached discrete graphics card. In addition, using
explicit I/O operations allows better control over cache contents, as
graphics devices are generally not cache coherent with the CPU, mapping
pages used for graphics into an application address space requires the use
of expensive cache flushing operations. Providing direct control over
graphics data access ensures that data are handled in the most efficient
possible fashion.

 A. Reading

		struct drm_gem_pread {
			/** Handle for the object being read. */
			uint32_t handle;
			uint32_t pad;
			/** Offset into the object to read from */
			uint64_t offset;
			/** Length of data to read */
			uint64_t size;
			/** Pointer to write the data into. */
			uint64_t data_ptr;	/* void * */
		};

	This copies data into the specified object at the specified
	position. Any necessary graphics device synchronization and
	flushing will be done automatically.
		
		struct drm_gem_pwrite {
			/** Handle for the object being written to. */
			uint32_t handle;
			uint32_t pad;
			/** Offset into the object to write to */
			uint64_t offset;
			/** Length of data to write */
			uint64_t size;
			/** Pointer to read the data from. */
			uint64_t data_ptr;	/* void * */
		};
		
	This copies data out of the specified object into the
	waiting user memory. Again, device synchronization will
	be handled by the kernel to ensure user space sees a
	consistent view of the graphics device.

5. Mapping objects to user space

For most objects, reading/writing is the preferred interaction mode.
However, when the CPU is involved in rendering to cover deficiencies in
hardware support for particular operations, the CPU will want to directly
access the relevant objects. 

Because mmap is fairly heavyweight, we allow applications to retain maps to
objects persistently and then update how they're using the memory through a
separate interface. Applications which fail to use this separate interface
may exhibit unpredictable behaviour as memory consistency will not be
preserved.

 A. Mapping

		struct drm_gem_mmap {
			/** Handle for the object being mapped. */
			uint32_t handle;
			uint32_t pad;
			/** Offset in the object to map. */
			uint64_t offset;
			/**
			 * Length of data to map.
			 *
			 * The value will be page-aligned.
			 */
			uint64_t size;
			/** Returned pointer the data was mapped at */
			uint64_t addr_ptr;	/* void * */
		};
		
		/* usage */
		mmap.handle = <handle>;
		mmap.offset = <offset>;
		mmap.size = <size>;
		ret = ioctl (fd, DRM_IOCTL_GEM_MMAP, &mmap);
		if (ret == 0)
			return (void *) (uintptr_t) mmap.addr_ptr;


 B. Unmapping

		munmap (addr, length);

	Nothing strange here, just use the normal munmap syscall.

6. Memory Domains

Graphics devices remain a strong bastion of non cache-coherent memory. As a
result, accessing data through one functional unit will end up loading that
cache with data which then needs to be manually synchronized when that data
is used with another functional unit.

Tracking where data are resident is done by identifying how functional units
deal with caches. Each cache is labeled as a separate memory domain. Then,
each sequence of operations is expected to load data into various read
domains and leave data in at most one write domain. Gem tracks the read and
write memory domains of each object and performs the necessary
synchronization operations when objects move from one domain set to another.

For example, if operation 'A' constructs an image that is immediately used
by operation 'B', then when the read domain for 'B' is not the same as the
write domain for 'A', then the write domain must be flushed, and the read
domain invalidated. If these two operations are both executed in the same
command queue, then the flush operation can go inbetween them in the same
queue, avoiding any kind of CPU-based synchronization and leaving the GPU to
do the work itself.

6.1 Memory Domains (GPU-independent)

 * DRM_GEM_DOMAIN_CPU.

 Objects in this domain are using caches which are connected to the CPU.
 Moving objects from non-CPU domains into the CPU domain can involve waiting
 for the GPU to finish with operations using this object. Moving objects
 from this domain to a GPU domain can involve flushing CPU caches and chipset
 buffers.

6.1 GPU-independent memory domain ioctl

This ioctl is independent of the GPU in use. So far, no use other than
synchronizing objects to the CPU domain have been found; if that turns out
to be generally true, this ioctl may be simplified further.
   
 A. Explicit domain control

		struct drm_gem_set_domain {
			/** Handle for the object */
			uint32_t handle;