Current Issue Cover
基于网格形变的立体图像内容重组

柴雄力,邵枫,姜求平,蒋刚毅(宁波大学)

摘 要
目的 近年来,随着数字摄影技术的飞速发展,图像增强技术越来越受到重视。图像构图作为图像增强中影响美学的重要因素,一直以来都是研究的热点。为此,从立体图像布局调整出发,本文提出一种基于Delaunay网格形变的立体图像内容重组方法。方法 首先将待重组的一对立体图像记为源图像,将用于重组规则确定的一幅图像记为参考图像。然后对源图像需要调整的目标、特征线和其余区域进行取点操作,建立Delaunay网格。将源图像的左图与参考图像进行模板匹配操作,得到源图像与参考图像在结构布局上的对应关系。最后利用网格形变的特性,移动和缩放目标对象,并对立体图像的深度进行自适应调整。结果 针对目标对象移动、缩放和特征线调整这几方面进行优化,实验结果表明重组后立体图像构图与参考图像一致且深度能自适应调整。与最新方法进行比较,实验结果表明本文方法在目标对象分割精度和图像语义保持方面具有优势。结论 本文根据网格形变相关理论,构建图像质量能量项、布局匹配能量项和视差适应能量项,实现了立体图像的内容重组。与现有需要提取和粘贴目标对象的重组方法不同,本文方法对目标对象的分割精度要求不高,不需要图像修复技术和图像混合技术,重组后的立体图像没有伪影和语义错误出现。用户可以通过参考图像来引导立体图像的布局调整,达到用户所期望的图像增强效果。
关键词
Stereoscopic image recomposition based on mesh deformation

Chai Xiongli,Shao Feng,Jiang Qiuping,Jiang Gangyi()

Abstract
Objective In recent years, with the rapid development of digital photography, image enhancement technique receives more and more attention. Image enhancement aims at improving the visual quality of an image, which can be achieved by tone mapping, denoising, recomposition and so on. In aesthetic evaluation of images, recomposition considers object relationships and geometric structure, which is one of the most influential aesthetic factors. As a research hotspot, image recomposition utilizes some photographic composition rules such as rule of thirds, visual balance, diagonal dominance and object size to capture aesthetically pleasing content. Using image recomposition technology, non-photographic experts can also get photographic images that conform to aesthetics of the images through post-processing. These years have been undergone a tremendous boom in stereoscopic technologies. Various stereoscopic services and applications are now available. This brings great demands for the availability of 3D content. Due to the separation of stereoscopic content production and display, there is a growing demand for stereoscopic image layout adjustment. But the content modi?cation of 3D image is not easy compared to the conventional 2D image modi?cation. It requires additional care due to the additional depth dimension. The misalignment between the left and right image may lead the 3D viewing discomfort and cause eyestrain, and headache. Considering the above factors, starting form stereoscopic image layout adjustment, we propose a stereoscopic image recomposition method based on Delaunay mesh deformation and depth adaptation. Method First, a pair of stereoscopic images to be recomposed is recorded as source images including left image and right image, and a binary image used for rule determination is recorded as a reference image. We use alpha matting to obtain a precise region with opacity value for each object in the left image and calculate the significance of stereo images. Then we detect feature points from the left image and use Delaunay triangulation algorithm to generate the mesh as follow. We employ an edge detection operator, for example Canny operator and utilize corner detection algorithm, such as Harris corner detection, for extracting some feature points in target object. To evenly sample feature points in the feature lines, we use the Hough transform to detect feature lines and select points in the left image. Here we classify the location of the target object and the feature line into three categories: intersect, above separate, below separate. When intersecting with the feature line, the target object moves along with the feature line, and the rest situation can be considered separately. We evenly discretize the left image boundary to use all the points there as part of the feature points and sample the remaining area to gain feature points. Based on feature points, the Delaunay triangulation mesh can be automatically generated. After completing the establishment of the left image meshes, the meshes in the right image are mapped by parallax from the left image meshes and the reference image meshes are also built like the operation of the left image. A template matching operation is performed on the contents of the left image and the reference image to obtain the correspondence relationship between the source image and the reference image in the layout. In the optimization process, we construct energy terms from three aspects: image quality, layout adjustment, and depth adaptation. Finally, based on the characteristics of mesh deformation, the target object is moved and scaled, and the depth of the stereo images is adaptively adjusted. The disparity change ratio of the target object is consistent with the size scaling. Result The paper carries out experimental design from two aspects of single object and multi objects, which proves that this method is applicable to all kinds of objects. The experimental results show that the stereoscopic image after recomposition is consistent with the layout of reference image for the target object movement, scaling and feature line adjustment and that the depth can be adjusted adaptively. We have adjusted the coefficients of different energy terms to prove the new framework proposed in the paper can achieve satisfactory stereoscopic content recomposition. Compared with the latest method, the experimental results show that the optimization method used in this paper has advantages in the segmentation accuracy of the target object and the preservation of image semantics. Conclusion In this paper, the image quality energy term, layout matching energy term and disparity adaptive energy term are constructed based on the theory of mesh deformation, and the content recomposition of the stereoscopic image is achieved according to energy term optimization. Different from the existing recomposition method which needs to extract and paste the target object, this method does not require high accuracy in the segmentation of the target object. There is no need for image inpainting technique and image blending technique. The stereoscopic image after recomposition has no artifacts and semantic errors. The user can guide the layout adjustment of the stereoscopic image by using the reference image to achieve the image enhancement desired by the user. In the future, we can combine mesh deformation technology and cropping technology to further enhance the efficiency and flexibility of stereoscopic image recomposition.
Keywords
QQ在线


订阅号|日报