summaryrefslogtreecommitdiff
path: root/shared-core
AgeCommit message (Collapse)Author
2008-01-24Fixup modeset ioctl number & typedef usageJesse Barnes
Should be 0x08 rather than 0xa0, and shouldn't use typedefs.
2008-01-24Merge commit 'airlied/i915-ttm-cfu'Eric Anholt
This requires updated Mesa to handle the new relocation format.
2008-01-24Remove broken 'in vblank' accountingJesse Barnes
We need to return an accurate vblank count to the callers of ->get_vblank_counter, and in the Intel case the actual frame count register isn't udpated until the next active line is displayed, so we need to return one more than the frame count register if we're currently in a vblank period. However, none of the various ways of doing this is working yet, so disable the logic for now. This may result in a few missed events, but should fix the hangs some people have seen due to the current code tripping the wraparound logic in drm_update_vblank_count.
2008-01-24i915: fix missing header when copying data from userspaceDave Airlie
2008-01-24i915 make relocs use copy from userDave Airlie
Switch relocs to using copy from user and remove index and pass buffer handles in instead.
2008-01-23Fix thinko in get_vblank_counterJesse Barnes
Should use vtotal not htotal to figure out if we're in a vblank period.
2008-01-23Fix IS_I915G macroJesse Barnes
One to many parantheses...
2008-01-23nouveau: Fix warning in nouveau_mem.cMaarten Maathuis
2008-01-23drm/i915: add support for E7221Dave Airlie
2008-01-22Correct vblank count valueJesse Barnes
The frame count registers don't increment until the start of the next frame, so make sure we return an incremented count if called during the actual vblank period.
2008-01-22i915 irq fixesJesse Barnes
Ack the IRQs correctly (PIPExSTAT first followed by IIR). Don't read vblank counter registers on disabled pipes (might hang otherwise). And deal with flipped pipe/plane mappings if present.
2008-01-22Merge branch 'master' into vblank-rework, including mach64 supportJesse Barnes
Conflicts: linux-core/drmP.h linux-core/drm_drv.c shared-core/i915_drv.h shared-core/i915_irq.c shared-core/mga_irq.c shared-core/radeon_irq.c shared-core/via_irq.c Mostly trivial conflicts. mach64 support from Mathieu Bérard.
2008-01-22Revert "Fix pipe<->plane mapping vs. vblank handling (again)"Dave Airlie
This reverts commit bfc29606e4a818897eebca46a5e23bbe7bc3ce25. This regresses i915 here for me I can't get greater than 0.333 fps with gears
2008-01-21nouveau: don't forget NV80.Stephane Marchesin
2008-01-21nouveau: new card family for old card designs.Stephane Marchesin
2008-01-17Add additional explanation of DRM_BO_FLAG_CACHED_MAPPED before I forget again.Eric Anholt
2008-01-15i915: Add chipset id for Intel Integrated Graphics DeviceZhenyu Wang
This adds new chipset id in drm. Signed-off-by: Zhenyu Wang <zhenyu.z.wang@intel.com>
2008-01-15radeon_ms: use radeon connector type insted of drmJerome Glisse
2008-01-15radeon_ms: cope with lastest drm modesetting changeJerome Glisse
2008-01-15radeon_ms: add rom parsing & adapt codeJerome Glisse
Add rom (only combios for now) parsing and use informations retrieve instead of hardcoded table. Shuffle code around a bit.
2008-01-15Properly propagate the user-space fence flags.Thomas Hellstrom
This avoids a sync flush when user-space has already programmed and MI_FLUSH in the batchbuffer.
2008-01-14nouveau: make mem alloc debug a little more verbose.Stephane Marchesin
2008-01-11nv05: enable ctx/op methods, and ignore patch valid failures.Ben Skeggs
Yes, I'm quite aware "real" nv04 doesn't support this, hopefully the GPU will just ignore those PGRAPH_DEBUG_3 bits on that hw.
2008-01-08nouveau: AGP reset correction - don't touch FW bitStuart Bennett
2008-01-07nv50: more small changesBen Skeggs
2008-01-07nv50: oops, lost some state saving along the way somewhere.Ben Skeggs
xf86-video-nv will now work again after nouveau.
2008-01-07nv50: hook up timer funcs...Ben Skeggs
2008-01-07nv50: abort on chips without ctx ucodeBen Skeggs
2008-01-07nv50: some needed ctx valsBen Skeggs
2008-01-07nv50: some cleanups + small changesBen Skeggs
2008-01-07Nouveau: ppc oops.Stephane Marchesin
2008-01-07Nouveau: move PPC bios copy to firstopen.Stephane Marchesin
2008-01-06nouveau: Add ctx_voodoo for NV86Jeremy Kolb
2008-01-04via: add P4M900 pci id.Xavier Bachelot
bug 12108
2008-01-04drm: move drm_head to drm_minor and fix up usersDave Airlie
2008-01-04[PATCH] nouveau: reset AGP on init for < nv40Stuart Bennett
This is necessary for AGP to work after running bios init scripts on nv3x, and is seen in mmio traces of all cards (nv04-nv4x) I'm not making the equivalent change to nv40_mc.c, as early cards (6200, 6800gt) use the 0x000018XX PBUS and later cards use the 0x000880XX PBUS and I don't know the effects of using the wrong one
2008-01-04[PATCH] nouveau: Fix nv20/30 context loadingStuart Bennett
Don't set the context as valid until it has been loaded
2008-01-03mach64: some more minor cleanupsDave Airlie
2008-01-03mach64: cleanup some of the macro formattingDave Airlie
2008-01-03drm: cleanup DRM_DEBUG() parametersMárton Németh
As DRM_DEBUG macro already prints out the __FUNCTION__ string (see drivers/char/drm/drmP.h), it is not worth doing this again. At some other places the ending "\n" was added. airlied:- I cleaned up a few that this patch missed also
2008-01-03Merge branch 'r500-support'Dave Airlie
2008-01-03remove duplicate pciidsDave Airlie
2007-12-21Rename inappropriately named 'mask' fields to 'proposed_flags' instead.Keith Packard
Flags pending validation were stored in a misleadingly named field, 'mask'. As 'mask' is already used to indicate pieces of a flags field which are changing, it seems better to use a name reflecting the actual purpose of this field. I chose 'proposed_flags' as they may not actually end up in 'flags', and in an case will be modified when they are moved over. This affects the API, but not ABI of the user-mode interface.
2007-12-20radeon_ms: update to follow lastest modesetting changeJerome Glisse
2007-12-19radeon_ms: add sarea & install headerJerome Glisse
2007-12-18Merge branch 'modesetting-airlied' into modesetting-101Dave Airlie
2007-12-18remove output namesDave Airlie
2007-12-18Fixed buildJakob Bornecrantz
2007-12-18Fix and cleanup of HotplugJakob Bornecrantz
2007-12-18Modesetting HotplugJakob Bornecrantz
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-14 10:31:34 +0000
ref='#n639'>639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 
/*
 * Copyright 2012 Red Hat Inc.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
 * to deal in the Software without restriction, including without limitation
 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
 * and/or sell copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 * OTHER DEALINGS IN THE SOFTWARE.
 *
 * Authors: Ben Skeggs
 */

#ifdef HAVE_CONFIG_H
#include <config.h>
#endif

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <assert.h>
#include <errno.h>

#include <xf86drm.h>
#include <xf86atomic.h>
#include "libdrm_lists.h"
#include "nouveau_drm.h"

#include "nouveau.h"
#include "private.h"

struct nouveau_pushbuf_krec {
	struct nouveau_pushbuf_krec *next;
	struct drm_nouveau_gem_pushbuf_bo buffer[NOUVEAU_GEM_MAX_BUFFERS];
	struct drm_nouveau_gem_pushbuf_reloc reloc[NOUVEAU_GEM_MAX_RELOCS];
	struct drm_nouveau_gem_pushbuf_push push[NOUVEAU_GEM_MAX_PUSH];
	int nr_buffer;
	int nr_reloc;
	int nr_push;
	uint64_t vram_used;
	uint64_t gart_used;
};

struct nouveau_pushbuf_priv {
	struct nouveau_pushbuf base;
	struct nouveau_pushbuf_krec *list;
	struct nouveau_pushbuf_krec *krec;
	struct nouveau_list bctx_list;
	struct nouveau_bo *bo;
	uint32_t type;
	uint32_t suffix0;
	uint32_t suffix1;
	uint32_t *ptr;
	uint32_t *bgn;
	int bo_next;
	int bo_nr;
	struct nouveau_bo *bos[];
};

static inline struct nouveau_pushbuf_priv *
nouveau_pushbuf(struct nouveau_pushbuf *push)
{
	return (struct nouveau_pushbuf_priv *)push;
}

static int pushbuf_validate(struct nouveau_pushbuf *, bool);
static int pushbuf_flush(struct nouveau_pushbuf *);

static bool
pushbuf_kref_fits(struct nouveau_pushbuf *push, struct nouveau_bo *bo,
		  uint32_t *domains)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct nouveau_device *dev = push->client->device;
	struct nouveau_bo *kbo;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	int i;

	/* VRAM is the only valid domain.  GART and VRAM|GART buffers
	 * are all accounted to GART, so if this doesn't fit in VRAM
	 * straight up, a flush is needed.
	 */
	if (*domains == NOUVEAU_GEM_DOMAIN_VRAM) {
		if (krec->vram_used + bo->size > dev->vram_limit)
			return false;
		krec->vram_used += bo->size;
		return true;
	}

	/* GART or VRAM|GART buffer.  Account both of these buffer types
	 * to GART only for the moment, which simplifies things.  If the
	 * buffer can fit already, we're done here.
	 */
	if (krec->gart_used + bo->size <= dev->gart_limit) {
		krec->gart_used += bo->size;
		return true;
	}

	/* Ran out of GART space, if it's a VRAM|GART buffer and it'll
	 * fit into available VRAM, turn it into a VRAM buffer
	 */
	if ((*domains & NOUVEAU_GEM_DOMAIN_VRAM) &&
	    krec->vram_used + bo->size <= dev->vram_limit) {
		*domains &= NOUVEAU_GEM_DOMAIN_VRAM;
		krec->vram_used += bo->size;
		return true;
	}

	/* Still couldn't fit the buffer in anywhere, so as a last resort;
	 * scan the buffer list for VRAM|GART buffers and turn them into
	 * VRAM buffers until we have enough space in GART for this one
	 */
	kref = krec->buffer;
	for (i = 0; i < krec->nr_buffer; i++, kref++) {
		if (!(kref->valid_domains & NOUVEAU_GEM_DOMAIN_GART))
			continue;

		kbo = (void *)(unsigned long)kref->user_priv;
		if (!(kref->valid_domains & NOUVEAU_GEM_DOMAIN_VRAM) ||
		    krec->vram_used + kbo->size > dev->vram_limit)
			continue;

		kref->valid_domains &= NOUVEAU_GEM_DOMAIN_VRAM;
		krec->gart_used -= kbo->size;
		krec->vram_used += kbo->size;
		if (krec->gart_used + bo->size <= dev->gart_limit) {
			krec->gart_used += bo->size;
			return true;
		}
	}

	/* Couldn't resolve a placement, need to force a flush */
	return false;
}

static struct drm_nouveau_gem_pushbuf_bo *
pushbuf_kref(struct nouveau_pushbuf *push, struct nouveau_bo *bo,
	     uint32_t flags)
{
	struct nouveau_device *dev = push->client->device;
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct nouveau_pushbuf *fpush;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	uint32_t domains, domains_wr, domains_rd;

	domains = 0;
	if (flags & NOUVEAU_BO_VRAM)
		domains |= NOUVEAU_GEM_DOMAIN_VRAM;
	if (flags & NOUVEAU_BO_GART)
		domains |= NOUVEAU_GEM_DOMAIN_GART;
	domains_wr = domains * !!(flags & NOUVEAU_BO_WR);
	domains_rd = domains * !!(flags & NOUVEAU_BO_RD);

	/* if buffer is referenced on another pushbuf that is owned by the
	 * same client, we need to flush the other pushbuf first to ensure
	 * the correct ordering of commands
	 */
	fpush = cli_push_get(push->client, bo);
	if (fpush && fpush != push)
		pushbuf_flush(fpush);

	kref = cli_kref_get(push->client, bo);
	if (kref) {
		/* possible conflict in memory types - flush and retry */
		if (!(kref->valid_domains & domains))
			return NULL;

		/* VRAM|GART buffer turning into a VRAM buffer.  Make sure
		 * it'll fit in VRAM and force a flush if not.
		 */
		if ((kref->valid_domains  & NOUVEAU_GEM_DOMAIN_GART) &&
		    (            domains == NOUVEAU_GEM_DOMAIN_VRAM)) {
			if (krec->vram_used + bo->size > dev->vram_limit)
				return NULL;
			krec->vram_used += bo->size;
			krec->gart_used -= bo->size;
		}

		kref->valid_domains &= domains;
		kref->write_domains |= domains_wr;
		kref->read_domains  |= domains_rd;
	} else {
		if (krec->nr_buffer == NOUVEAU_GEM_MAX_BUFFERS ||
		    !pushbuf_kref_fits(push, bo, &domains))
			return NULL;

		kref = &krec->buffer[krec->nr_buffer++];
		kref->user_priv = (unsigned long)bo;
		kref->handle = bo->handle;
		kref->valid_domains = domains;
		kref->write_domains = domains_wr;
		kref->read_domains = domains_rd;
		kref->presumed.valid = 1;
		kref->presumed.offset = bo->offset;
		if (bo->flags & NOUVEAU_BO_VRAM)
			kref->presumed.domain = NOUVEAU_GEM_DOMAIN_VRAM;
		else
			kref->presumed.domain = NOUVEAU_GEM_DOMAIN_GART;

		cli_kref_set(push->client, bo, kref, push);
		atomic_inc(&nouveau_bo(bo)->refcnt);
	}

	return kref;
}

static uint32_t
pushbuf_krel(struct nouveau_pushbuf *push, struct nouveau_bo *bo,
	     uint32_t data, uint32_t flags, uint32_t vor, uint32_t tor)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct drm_nouveau_gem_pushbuf_reloc *krel;
	struct drm_nouveau_gem_pushbuf_bo *pkref;
	struct drm_nouveau_gem_pushbuf_bo *bkref;
	uint32_t reloc = data;

	pkref = cli_kref_get(push->client, nvpb->bo);
	bkref = cli_kref_get(push->client, bo);
	krel  = &krec->reloc[krec->nr_reloc++];

	krel->reloc_bo_index = pkref - krec->buffer;
	krel->reloc_bo_offset = (push->cur - nvpb->ptr) * 4;
	krel->bo_index = bkref - krec->buffer;
	krel->flags = 0;
	krel->data = data;
	krel->vor = vor;
	krel->tor = tor;

	if (flags & NOUVEAU_BO_LOW) {
		reloc = (bkref->presumed.offset + data);
		krel->flags |= NOUVEAU_GEM_RELOC_LOW;
	} else
	if (flags & NOUVEAU_BO_HIGH) {
		reloc = (bkref->presumed.offset + data) >> 32;
		krel->flags |= NOUVEAU_GEM_RELOC_HIGH;
	}
	if (flags & NOUVEAU_BO_OR) {
		if (bkref->presumed.domain & NOUVEAU_GEM_DOMAIN_VRAM)
			reloc |= vor;
		else
			reloc |= tor;
		krel->flags |= NOUVEAU_GEM_RELOC_OR;
	}

	return reloc;
}

static void
pushbuf_dump(struct nouveau_pushbuf_krec *krec, int krec_id, int chid)
{
	struct drm_nouveau_gem_pushbuf_reloc *krel;
	struct drm_nouveau_gem_pushbuf_push *kpsh;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	struct nouveau_bo *bo;
	uint32_t *bgn, *end;
	int i;

	err("ch%d: krec %d pushes %d bufs %d relocs %d\n", chid,
	    krec_id, krec->nr_push, krec->nr_buffer, krec->nr_reloc);

	kref = krec->buffer;
	for (i = 0; i < krec->nr_buffer; i++, kref++) {
		err("ch%d: buf %08x %08x %08x %08x %08x\n", chid, i,
		    kref->handle, kref->valid_domains,
		    kref->read_domains, kref->write_domains);
	}

	krel = krec->reloc;
	for (i = 0; i < krec->nr_reloc; i++, krel++) {
		err("ch%d: rel %08x %08x %08x %08x %08x %08x %08x\n",
		    chid, krel->reloc_bo_index, krel->reloc_bo_offset,
		    krel->bo_index, krel->flags, krel->data,
		    krel->vor, krel->tor);
	}

	kpsh = krec->push;
	for (i = 0; i < krec->nr_push; i++, kpsh++) {
		kref = krec->buffer + kpsh->bo_index;
		bo = (void *)(unsigned long)kref->user_priv;
		bgn = (uint32_t *)((char *)bo->map + kpsh->offset);
		end = bgn + (kpsh->length /4);

		err("ch%d: psh %08x %010llx %010llx\n", chid, kpsh->bo_index,
		    (unsigned long long)kpsh->offset,
		    (unsigned long long)(kpsh->offset + kpsh->length));
		while (bgn < end)
			err("\t0x%08x\n", *bgn++);
	}
}

static int
pushbuf_submit(struct nouveau_pushbuf *push, struct nouveau_object *chan)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->list;
	struct nouveau_device *dev = push->client->device;
	struct drm_nouveau_gem_pushbuf_bo_presumed *info;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	struct drm_nouveau_gem_pushbuf req;
	struct nouveau_fifo *fifo = chan->data;
	struct nouveau_bo *bo;
	int krec_id = 0;
	int ret = 0, i;

	if (chan->oclass != NOUVEAU_FIFO_CHANNEL_CLASS)
		return -EINVAL;

	if (push->kick_notify)
		push->kick_notify(push);

	nouveau_pushbuf_data(push, NULL, 0, 0);

	while (krec && krec->nr_push) {
		req.channel = fifo->channel;
		req.nr_buffers = krec->nr_buffer;
		req.buffers = (uint64_t)(unsigned long)krec->buffer;
		req.nr_relocs = krec->nr_reloc;
		req.nr_push = krec->nr_push;
		req.relocs = (uint64_t)(unsigned long)krec->reloc;
		req.push = (uint64_t)(unsigned long)krec->push;
		req.suffix0 = nvpb->suffix0;
		req.suffix1 = nvpb->suffix1;
		req.vram_available = 0; /* for valgrind */
		req.gart_available = 0;

		if (dbg_on(0))
			pushbuf_dump(krec, krec_id++, fifo->channel);

#ifndef SIMULATE
		ret = drmCommandWriteRead(dev->fd, DRM_NOUVEAU_GEM_PUSHBUF,
					  &req, sizeof(req));
		nvpb->suffix0 = req.suffix0;
		nvpb->suffix1 = req.suffix1;
		dev->vram_limit = (req.vram_available * 80) / 100;
		dev->gart_limit = (req.gart_available * 80) / 100;
#else
		if (dbg_on(31))
			ret = -EINVAL;
#endif

		if (ret) {
			err("kernel rejected pushbuf: %s\n", strerror(-ret));
			pushbuf_dump(krec, krec_id++, fifo->channel);
			break;
		}

		kref = krec->buffer;
		for (i = 0; i < krec->nr_buffer; i++, kref++) {
			bo = (void *)(unsigned long)kref->user_priv;

			info = &kref->presumed;
			if (!info->valid) {
				bo->flags &= ~NOUVEAU_BO_APER;
				if (info->domain == NOUVEAU_GEM_DOMAIN_VRAM)
					bo->flags |= NOUVEAU_BO_VRAM;
				else
					bo->flags |= NOUVEAU_BO_GART;
				bo->offset = info->offset;
			}

			if (kref->write_domains)
				nouveau_bo(bo)->access |= NOUVEAU_BO_WR;
			if (kref->read_domains)
				nouveau_bo(bo)->access |= NOUVEAU_BO_RD;
		}

		krec = krec->next;
	}

	return ret;
}

static int
pushbuf_flush(struct nouveau_pushbuf *push)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	struct nouveau_bufctx *bctx, *btmp;
	struct nouveau_bo *bo;
	int ret = 0, i;

	if (push->channel) {
		ret = pushbuf_submit(push, push->channel);
	} else {
		nouveau_pushbuf_data(push, NULL, 0, 0);
		krec->next = malloc(sizeof(*krec));
		nvpb->krec = krec->next;
	}

	kref = krec->buffer;
	for (i = 0; i < krec->nr_buffer; i++, kref++) {
		bo = (void *)(unsigned long)kref->user_priv;
		cli_kref_set(push->client, bo, NULL, NULL);
		if (push->channel)
			nouveau_bo_ref(NULL, &bo);
	}

	krec = nvpb->krec;
	krec->vram_used = 0;
	krec->gart_used = 0;
	krec->nr_buffer = 0;
	krec->nr_reloc = 0;
	krec->nr_push = 0;

	DRMLISTFOREACHENTRYSAFE(bctx, btmp, &nvpb->bctx_list, head) {
		DRMLISTJOIN(&bctx->current, &bctx->pending);
		DRMINITLISTHEAD(&bctx->current);
		DRMLISTDELINIT(&bctx->head);
	}

	return ret;
}

static void
pushbuf_refn_fail(struct nouveau_pushbuf *push, int sref, int srel)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct drm_nouveau_gem_pushbuf_bo *kref;

	kref = krec->buffer + sref;
	while (krec->nr_buffer-- > sref) {
		struct nouveau_bo *bo = (void *)(unsigned long)kref->user_priv;
		cli_kref_set(push->client, bo, NULL, NULL);
		nouveau_bo_ref(NULL, &bo);
		kref++;
	}
	krec->nr_buffer = sref;
	krec->nr_reloc = srel;
}

static int
pushbuf_refn(struct nouveau_pushbuf *push, bool retry,
	     struct nouveau_pushbuf_refn *refs, int nr)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	int sref = krec->nr_buffer;
	int ret = 0, i;

	for (i = 0; i < nr; i++) {
		kref = pushbuf_kref(push, refs[i].bo, refs[i].flags);
		if (!kref) {
			ret = -ENOSPC;
			break;
		}
	}

	if (ret) {
		pushbuf_refn_fail(push, sref, krec->nr_reloc);
		if (retry) {
			pushbuf_flush(push);
			nouveau_pushbuf_space(push, 0, 0, 0);
			return pushbuf_refn(push, false, refs, nr);
		}
	}

	return ret;
}

static int
pushbuf_validate(struct nouveau_pushbuf *push, bool retry)
{
	struct nouveau_pushbuf_priv *nvpb = nouveau_pushbuf(push);
	struct nouveau_pushbuf_krec *krec = nvpb->krec;
	struct drm_nouveau_gem_pushbuf_bo *kref;
	struct nouveau_bufctx *bctx = push->bufctx;
	struct nouveau_bufref *bref;
	int relocs = bctx ? bctx->relocs * 2: 0;
	int sref, srel, ret;

	ret = nouveau_pushbuf_space(push, relocs, relocs, 0);
	if (ret || bctx == NULL)
		return ret;

	sref = krec->nr_buffer;
	srel = krec->nr_reloc;

	DRMLISTDEL(&bctx->head);
	DRMLISTADD(&bctx->head, &nvpb->bctx_list);

	DRMLISTFOREACHENTRY(bref, &bctx->pending, thead) {
		kref = pushbuf_kref(push, bref->bo, bref->flags);
		if (!kref) {
			ret = -ENOSPC;
			break;
		}

		if (bref->packet) {
			pushbuf_krel(push, bref->bo, bref->packet, 0, 0, 0);
			*push->cur++ = 0;
			pushbuf_krel(push, bref->bo, bref->data, bref->flags,
					   bref->vor, bref->tor);
			*push->cur++ = 0;
		}
	}

	DRMLISTJOIN(&bctx->pending, &bctx->current);
	DRMINITLISTHEAD(&bctx->pending);

	if (ret) {
		pushbuf_refn_fail(push, sref, srel);
		if (retry) {
			pushbuf_flush(push);
			return pushbuf_validate(push, false);
		}
	}

	return 0;
}

int
nouveau_pushbuf_new(struct nouveau_client *client, struct nouveau_object *chan,
		    int nr, uint32_t size, bool immediate,
		    struct nouveau_pushbuf **ppush)
{
	struct nouveau_device *dev = client->device;
	struct nouveau_fifo *fifo = chan->data;
	struct nouveau_pushbuf_priv *nvpb;
	struct nouveau_pushbuf *push;
	struct drm_nouveau_gem_pushbuf req = {};
	int ret;

	if (chan->oclass != NOUVEAU_FIFO_CHANNEL_CLASS)
		return -EINVAL;

	/* nop pushbuf call, to get the current "return to main" sequence
	 * we need to append to the pushbuf on early chipsets
	 */
	req.channel = fifo->channel;
	req.nr_push = 0;
	ret = drmCommandWriteRead(dev->fd, DRM_NOUVEAU_GEM_PUSHBUF,
				  &req, sizeof(req));
	if (ret)
		return ret;

	nvpb = calloc(1, sizeof(*nvpb) + nr * sizeof(*nvpb->bos));
	if (!nvpb)
		return -ENOMEM;

#ifndef SIMULATE
	nvpb->suffix0 = req.suffix0;
	nvpb->suffix1 = req.suffix1;
#else
	nvpb->suffix0 = 0xffffffff;
	nvpb->suffix1 = 0xffffffff;
#endif
	nvpb->krec = calloc(1, sizeof(*nvpb->krec));
	nvpb->list = nvpb->krec;
	if (!nvpb->krec) {
		free(nvpb);
		return -ENOMEM;
	}

	push = &nvpb->base;
	push->client = client;
	push->channel = immediate ? chan : NULL;
	push->flags = NOUVEAU_BO_RD;