2024年5月13日发(作者：z41次列车时刻表)

Multi-layeredImage-BasedRendering

SingBingKang

CompaqComputerCorporation

CambridgeResearchLab

OneKendallSquare,Bldg.700

Cambridge,MA02139

sbk@

Abstract

Inthisarticle,wedescribeamulti-layeredimage-based

renderingsystemthatcanusedifferenttypesofinputto

parate

inputthatcanbemanipulatedindependentlyiscalleda

mplementation,thetypesoflayersthatcan

bemanipulatedaretheimage-basedand3-D-basedlay-

putationrequiredforrenderingthenewly-

craftedenvironmentisreducedbyusingcachedcom-

positesnapshotsofthatenvironmentatreferenceposes.

Thesecachedsnapshotsareusedtodirectlygeneratenovel

views,andtheoriginallayersareusedonlywhenneces-

rkeyconceptistheidentificationoftypes

ofholesgeneratedasaresultofpixeltransferfromthe

imal

renderingquality,thealgorithmusedinfillingtheseholes

isspecifictotheholetype(eitherintralayerorinterlayer).

Theideasembodiedinourmulti-layeredIBRsystemare

usefulinaugmentingthecapabilitiesofapplicationsthat

requirefastandgeometricallyconsistentrenderingof3-D

scenessuchasvideoediting.

Keywords:Image-basedrendering,layeredrepresenta-

tion,trilinear/trifocaltensor

Introduction

3-Dscenerenderingisaccomplishedeitherthroughthe

conventionalgraphicspipelineofmanipulatingandpro-

jectinga3-Dmodelorusingtechniquesthatoperateon

ter,alsoknownas

image-basedrendering(IBR),generallyhasthepositive

characteristicsofhavingrenderingspeedsindependentof

scenecomplexity,non-relianceon3-Dgraphicsaccelera-

tors,moderateCPUcosts,andexcellentpotentialforpro-

ficationsofimage-

basedrenderingtechniquesaredescribedindetailin[7].

Theworkdescribedinthisarticlefallsintoacategoryof

image-basedrenderingtechniquescalledgeometrically-

validpixelreprojection,nd

ofmethodusesasmallnumberofimagestogener-

atenovelviewsbasedonfeaturecorrespondencesacross

HuongQuynhDinh

CollegeofComputing

GeorgiaInstituteofTechnology

Atlanta,GA30332-0280

quynh@

theimagesandappliescomputervisiontechniquessuch

asstereo,structurefrommotion,orprojectiverecovery

posedtech-

niqueextendsexistingmethodstoallowthecreationof

newandgeometrically-consistentenvironmentsfromdis-

accomplishedusingalayeredrep-

epresentation,eachlayerisdeemed

tobedifferentifitcomesfromadifferentsource,orits

posecanbeindependentlycontrolledrelativetoallother

layers.

PriorWork

ThefundamentallimitationofpriorIBRmethodsistheir

assumptionthatimagesusedinreprojectionrepresenta

isareasonableassumptioninvisualizingmodelsofasin-

glerigidobjectorscene,butitisinadequateforapplica-

tionssuchasvideoeditingandsynthesis,wherethereare

-

comethislimitationbyintroducingthenotionofmultiple

le

multiplerigidmotionsbyassigningeachonetoaseparate

ulti-layerIBRalgorithm,eachlayer

isreprojectedseparately,andthereprojectedlayersare

combinedtoformthetotalimage.

Layeredvideorepresentationsincludingcorrespon-

denceinformationaredescribedin[19]forvideocod-

r,thisworkdoesnotdescribeageometri-

callycorrectrenderingalgorithmandameansforediting

themodelsorcombiningmodelsfrommultiplesourcesto

,itsimplyfocuses

oncodinganexistingsequenceeffitional

videoeditingsystems,suchastheAvidsystemdescribed

s5,644,364[8]and5,654,737[3],alsoem-

ployalayeredrepresentationbutdonotincludegeometric

informationandcannotsupportgeometrically-valid3-D

rendering.

InBakerandSzeliski’swork[2],theyrefertolayers

asdisconnectedimageregions,eachofwhichhavingdis-

scribea

techniquethat,giventhesegmentedregionsinmultiple

images,

ourwork,wewouldconsidertheresultofthis“multi-

layered”stereorecoverytechniqueasbeingonelayeror

multiplelayers,dependingonwhethertheyaretreatedas

awholeorindependently(respectively)ingeneratingnew

environments.

Inanotherrelatedwork,Shadeetal.[16]describeda

renderingtechniquethatusesinpartalayered-depthim-

age,orLDI,inwhichmultipledepthsmaybeencoded

thisconstantwithineach

notnecessarilysoforournotionofalayer.

TheirtechniqueextendsMcMillanandBishop’spainter

algorithm[11]totransferalldepthlevelswithineach

arison,wecacheapairofviews;newviews

arecomputedusingthesecachedviewsfirstbeforefilling

inpossibleholes.

Organization

Inthisarticle,wedescribethearchitecturalconceptsof

ourmulti-layeredIBRsysteminSection,whichallows

,layers)such

asstills,video,hestageformulti-

layeredIBR,wedescribepixeltransfermechanismfora

uently,Sectiondetails

howthecompositefromthedifferentinputsaregener-

mentalre-

sultsusingimagesofrealandsyntheticobjectsareshown

ideadiscussionofthemeritsand

disadvantagesofourapproachinSection,followedby

concludingremarksinSection.

Wenowdescribetheconceptsforthemulti-layeredIBR

atnotalltheideashavebeenimplemented;

partsthathavebeenimplementedareclearlydelineated

inSection.

Themulti-layeredIBRsystem

Thegeneralarchitectureforourproposedmulti-layered

image-basedrenderingsystemisdepictedintheblock

ourmainparts:input,view

selection,viewpointsynchronization,andnovelviewgen-

uttothesystemisasetofmodelswhich

identifiedthreetypesofmodels:acollectionofstillim-

agesofastaticscene,amattedvideosequencecorre-

spondingtoasinglecoherentrigidbodymotion,anda

retwobasic

controlinputstothesystem:areferenceindexthatspec-

ifieswhichpartofeachmodelshouldberendered,anda

virtualviewthatspecifiesthevirtualcamerawhichwill

beusedtorenderandcompositeallofthemodels.

Thefirstprocessingstepintheblockdiagramisan

intra-layerindexingstage,inwhichthereferenceindices

andvirtualviewsareusedtoselectthecorrectpartof

Layer 1

Reference index

Virtual layer view

Layer 2Intra-layer

Viewpoint

Novel view

indexing

synchronizationgeneration

Virtual composite view

Layer N

Figure1:Architectureformulti-layerimage-basedren-

deringsystem.

mple,foracollectionofimages,one

canchoosearegionwithintheimagesortheimageas

3-Dmodel,onecanchoose

eringavideoclip,

thereferenceindexwouldspecifyadesiredframewhile

thevirtualviewspecifiesachangeinthevirtualcamera

e

desiredpartofeachmodelhasbeenidentified,eachmodel

isrenderedintoasetofpixelvaluesinanoutputbuffer.

Thenovelviewisusuallyaperturbationofthereference

viewpointsofthe“background”layer,whichwedefineas

thelayerthatdominatestheoutputbyvirtueofitsrelative

eferenceviewpointsareusedasglobalref-

erences;hence,afterallthereferenceindiceshavebeen

identified,eachlayeristhentransformedtobecompatible

withthe“background”referenceviewpointsandmerged

achedim-

agesarethenusedtogeneratenovelviewsusinganypixel

transfertechnique.

Apointofclarificasewhere

onlyoneimageisusedandthedepthisknown,thenthere

isonlyone“referenceframe.”Forthecasewhereno

groundtruthisknown,atleasttwoimagesareusedto

createanovelview,assumingthatfullpointcorrespon-

case,these

imagesarethe“referenceframes,”andtheirrespective

viewpoints,“referenceviewpoints.”

Multi-layeredmodelrepresentation

Inthissection,wedescribeinmoredetailthethreetypes

ofmodelscorrespondingtostills,video,and3-Dmod-

andedblockdiagraminFigure2showseach

typealongwithamoredetailedviewoftheprocessing

firstmodeltypeconsistsofasetofstillim-

ages,notnecessarilyinanyparticularorder,describinga

delisoftenusedtodescribeastatic

background,whichformsthebackdropforforegroundob-

jects,

withtheimagesareasetofcorrespondencemaps,one

foreachimage,thatmatchpixelsinthatimagetocorre-

orrespondences