浏览全部资源
扫码关注微信
Published： 16 April 2024 ，
移动端阅览
陈缘， 赵洋， 张效娟， 刘晓平. 2024. 有限色彩空间下的线稿上色. 中国图象图形学报， 29(04):0978-0988
Chen Yuan， Zhao Yang， Zhang Xiaojuan， Liu Xiaoping. 2024. Sketch colorization with finite color space prior. Journal of Image and Graphics， 29(04):0978-0988
目的
2
线稿上色是由线条构成的黑白线稿草图涂上颜色变为彩色图像的过程，在卡通动画制作和艺术绘画等领域中是非常关键的步骤。全自动线稿上色方法可以减轻绘制过程中烦琐耗时的手工上色的工作量，然而自动理解线稿中的稀疏线条并选取合适的颜色仍较为困难。
方法
2
依据现实场景中特定绘画类型常有固定用色风格偏好这一先验，本文聚焦于有限色彩空间下的线稿自动上色，通过约束色彩空间，不仅可以降低语义理解的难度，还可以避免不合理的用色。具体地，本文提出一种两阶段线稿自动上色方法。在第1阶段，设计一个灰度图生成器，对输入的稀疏线稿补充线条和细节，以生成稠密像素的灰度图像。在第2阶段，首先设计色彩推理模块，从输入的颜色先验中推理得到适合该线稿的色彩子空间，再提出一种多尺度的渐进融合颜色信息的生成网络以逐步生成高质量的彩色图像。
结果
2
实验在3个数据集上与4种线稿自动上色方法进行对比，在上色结果的客观质量对比中，所提方法取得了更高的峰值信噪比（peak signal to noise ratio，PSNR）和结构相似性（structural similarity index measure，SSIM）值以及更低的均方误差；在上色结果的色彩指标对比中，所提方法取得了最高的色彩丰富度分数；在主观评价和用户调查中，所提方法也取得了与人的主观审美感受更一致的结果。此外，消融实验结果也表明了本文所使用的模型结构及色彩空间限制有益于上色性能的提升。
结论
2
实验结果表明，本文提出的有限色彩空间下的线稿自动上色方法可以有效地完成多类线稿的自动上色，并且可以简单地通过调整颜色先验以获得更多样的彩色图像。
Objective
2
In the art field， an exquisite painting typically takes considerable effort from sketch drawing in the early stage to coloring and polishing. With the rise of animation， painting， graphic design， and other related industries， sketch colorization has become one of the most tedious and repetitive processes. Although some computer-aided design tools have appeared in past decades， they still require humans to accomplish colorization operations， and drawing an exquisite painting is difficult for ordinary users. Meanwhile， automatic sketch colorization is still a difficult problem. Therefore， both the academia and industry are in urgent need of convenient and efficient sketch colorization methods. With the development of deep neural networks （DNNs）， DNN-based colorization methods have achieved promising performance in recent years. However， most studies have focused on grayscale image colorization for natural images， which is quite different from sketch colorization. At present， only a few studies have focused on automatic sketch colorization， and they typically require user guidance or are designed for certain types， such as anime characters. However， automatically understanding sparse lines and selecting appropriate colors remain extremely difficult and ill-posed problems. Thus， disharmonious colors frequently appear in recent automatic sketch colorization results， e.g.， red grass and black sun. Therefore， most sketch colorization methods reduce difficulty through user guidance， which can be roughly divided into three types： reference image-based， color hints-based， and text expression-based. Although these semi-automatic methods can provide more reasonable results， they still require inefficient user interaction processes.
Method
2
In practice， we observe the phenomenon that colors used in paintings of a particular style are typically fixed and finite， rather than arbitrary colorization. Therefore， this study focuses on automatic sketch colorization with finite color space prior， which can effectively reduce the difficulty of understanding semantics and avoid undesired colors. In particular， a two-stage multi-scale colorization network is designed. In the first stage， a subnetwork is proposed to generate a dense grayscale image from the input sparse sketch. It adopts a commonly used U-Net structure. The U-Net structure can obtain a large receptive field， and thus， is helpful for understanding high-level semantics. In the second stage， a multi-scale generative adversarial network is designed to colorize the grayscale image in accordance with the input color palette. In this study， we adopt three scales. At the minimum scale， the input color palette prior is used to guide color reasoning， which contains several specified dominant colors. Then， the sketch content features extracted by the content encoder are converted into spatial attention and multiplied by the color features extracted by the color encoder to perceive preliminary color reasoning. The structure of two other scales is similar to the minimum scale. Color guidance information is gradually fused to generate higher-quality and higher-resolution results. Adversarial learning is adopted. Hence， three scales of discriminators that correspond to each generator and a discriminator for grayscale generation are used. Intermediate loss is designed to guide the generation of grayscale image in the first stage. Pixel-wise loss， adversarial loss， and TV loss are used in the second stage. Considering the application scenarios， we construct three datasets， namely， the Regong Tibetan painting dataset， the Thangka （Tibetan scroll painting） elements dataset， and a specific cartoon dataset. Each dataset contains images， color palettes， and corresponding sketches.
Result
2
Our model is implemented with the Tensorflow platform. For the three datasets， all the images with the size of 256 × 256 pixels are divided into training and testing sets with a ratio of 9∶1. The proposed method is compared with the baseline model Pix2Pix and several automatic sketch colorization methods： AdvSegLoss， PaintsChainer， and Style2Paints V4.5. Notably， PaintsChainer and Style2Paints are colorization tools that can be used directly， and they require humans to input sketches manually. Hence， we only retrain Pix2Pix and AdvSegLoss with their official implementations on the same datasets for fair comparison. For quantitative evaluation， peak signal to noise ratio （PSNR）， structural similarity index measure （SSIM）， and mean squared error （MSE） are used to measure the objective quality of the output coloring images. Meanwhile， the colorfulness score is used to measure the color vividness of coloring images. The proposed method nearly achieves the highest PSNR and SSIM values and the lowest MSE values on the three datasets， and it obtains the best colorfulness scores on all datasets. Although quantitative evaluation has shown the effectiveness of the proposed method， comparing colorization results through human subjective feelings is more meaningful. Hence， subjective results and user studies are provided to evaluate the performance of the proposed method. For qualitative evaluation， the proposed method can reproduce harmonious colors and obtain better visual perception than comparison methods. Furthermore， owing to the decoupled color reasoning and fusion modules in our model， the color of the output image can be changed flexibly with different input color priors. In a user study， most users prefer our results on both aspects of color selection and overall image quality. Moreover， we conduct ablation experiments， including training a single-stage model； that is， coloring an image without generating a grayscale image first and using only single-scale in the second stage. The performance of the single-stage model is the poorest， proving that learning automatic colorization directly is an extremely difficult task. The performance of the single-scale model is poorer than that of the multi-scale model， verifying the effectiveness of the multi-scale strategy.
Conclusion
2
Experimental results on three datasets show that the proposed automatic sketch colorization method based on finite color space prior can colorize sketch effectively and easily generate various colorful results by changing color prior.
线稿上色有限色彩空间卡通绘画生成对抗网络
sketch colorizationfinite color spacecartoonpaintinggenerative adversarial network
Cao L Q， Shang Y X， Liu T T， Li Z J and Ma A L. 2019. Novelimage colorization of a local adaptive weighted average filter. Journal of Image and Graphics， 24（8）： 1249-1257
曹丽琴， 商永星， 刘婷婷， 李治江， 马爱龙. 2019. 局部自适应的灰度图像彩色化. 中国图象图形学报， 24（8）： 1249-1257 ［DOI： 10.11834/jig.180608http://dx.doi.org/10.11834/jig.180608］
Cao Y， Tian H and Mok P Y. 2023. Attention-aware anime line drawing colorization//Proceedings of 2023 IEEE International Conference on Multimedia and Expo. Brisbane， Australia： IEEE： 1637-1642 ［DOI： 10.1109/ICME55011.2023.00282http://dx.doi.org/10.1109/ICME55011.2023.00282］
Chen S Y， Su W C， Gao L， Xia S H and Fu H B. 2020. DeepFaceDrawing： deep generation of face images from sketches. ACM Transactions on Graphics， 39（4）： #72 ［DOI： 10.1145/3386569.3392386http://dx.doi.org/10.1145/3386569.3392386］
Cho Y， Lee J， Yang S， Kim J， Park Y， Lee H， Khan M A， Kim D and Choo J. 2023. Guiding users to where to give color hints for efficient interactive sketch colorization via unsupervised region prioritization//Proceedings of 2023 IEEE Winter Conference on Applications of Computer Vision. Waikoloa， USA： IEEE： 1818-1827 ［DOI： 10.1109/WACV56688.2023.00186http://dx.doi.org/10.1109/WACV56688.2023.00186］
Cui J R， Zhong H W， Liu H L and Fu Y L. 2022. Exemplar-based sketch colorization with cross-domain dense semantic correspondence. Mathematics， 10（12）： #1988 ［DOI： 10.3390/math10121988http://dx.doi.org/10.3390/math10121988］
Dou Z， Wang N， Wang S J， Wang Z H and Li H J. 2022. Sketch colorization method with drawing prior. Computer Science， 49（4）： 195-202
窦智， 王宁， 王世杰， 王智慧， 李豪杰. 2022. 结合绘画先验的线稿上色方法. 计算机科学， 49（4）： 195-202 ［DOI： 10.11896/jsjkx.210300140http://dx.doi.org/10.11896/jsjkx.210300140］
Gao C Y， Liu Q， Xu Q， Wang L M， Liu J Z and Zou C Q. 2020. SketchyCOCO： image generation from freehand scene sketches//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： 5173-5182 ［DOI： 10.1109/CVPR42600.2020.00522http://dx.doi.org/10.1109/CVPR42600.2020.00522］
Hasler D and Suesstrunk S E. 2003. Measuring colorfulness in natural images//Proceedings of SPIE Human vision and electronic imaging VIII. Santa Clara， USA： SPIE： 87-95 ［DOI： 10.1117/12.477378http://dx.doi.org/10.1117/12.477378］
Hati Y， Jouet G， Rousseaux F and Duhart C. 2019. PaintsTorch： a user-guided anime line art colorization tool with double generator conditional adversarial network//Proceedings of 2019 European Conference on Visual Media Production. London， UK： ACM： 5 ［DOI： 10.1145/3359998.3369401］
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Hicsonmez S， Samet N， Akbas E and Duygulu P. 2021. Adversarial segmentation loss for sketch colorization//Proceedings of 2021 IEEE International Conference on Image Processing. Anchorage， USA： IEEE： 2403-2407 ［DOI： 10.1109/ICIP42928.2021.9506637http://dx.doi.org/10.1109/ICIP42928.2021.9506637］
Huang J L， Liao J and Kwong S. 2021. Semantic example guided image-to-image translation. IEEE Transactions on Multimedia， 23： 1654-1665 ［DOI： 10.1109/TMM.2020.3001536http://dx.doi.org/10.1109/TMM.2020.3001536］
Huang Z T， Zhao N X and Liao J. 2022. UniColor： a unified framework for multi-modal colorization with Transformer. ACM Transactions on Graphics， 41（6）： #205 ［DOI： 10.1145/3550454.3555471http://dx.doi.org/10.1145/3550454.3555471］
Kim H， Jhoo H Y， Park E and Yoo S. 2019. Tag2Pix： line art colorization using text tag with SECat and changing loss//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 9056-9064 ［DOI： 10.1109/ICCV.2019.00915http://dx.doi.org/10.1109/ICCV.2019.00915］
Li Y H， Chen X J， Wu F and Zha Z J. 2019. LinesToFacePhoto： face photo generation from lines with conditional self-attention generative adversarial networks//Proceedings of the 27th ACM International Conference on Multimedia. Nice， France： ACM： 2323-2331 ［DOI： 10.1145/3343031.3350854http://dx.doi.org/10.1145/3343031.3350854］
Liu G， Chen X and Hu Y Z. 2019. Anime sketch coloring with swish-gated residual U-Net//Proceedings of the 10th International Symposium on Computational Intelligence and Intelligent Systems. Jiujiang， China： Springer： 190-204 ［DOI： 10.1007/978-981-13-6473-0_17http://dx.doi.org/10.1007/978-981-13-6473-0_17］
Liu X T， Wu W L， Li C Z， Li Y F and Wu H S. 2022. Reference-guided structure-aware deep sketch colorization for cartoons. Computational Visual Media， 8： 135-148 ［DOI： 10.1007/s41095-021-0228-6http://dx.doi.org/10.1007/s41095-021-0228-6］
Mao X D， Li Q， Xie H R， Lau R Y K， Wang Z and Smolley S P. 2017. Least squares generative adversarial networks//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice. Italy： IEEE： 2813-2821 ［DOI： 10.1109/ICCV.2017.304http://dx.doi.org/10.1109/ICCV.2017.304］
Weng S C， Sun J M， Li Y， Li S and Shi B X. 2022. CT2： colorization Transformer via color tokens//Proceedings of the 17th European Conference on Computer Vision. Tel Aviv， Israel： Springer： 1-16 ［DOI： 10.1007/978-3-031-20071-7_1http://dx.doi.org/10.1007/978-3-031-20071-7_1］
Yonetsuji T. 2017. PaintsChainer ［EB/OL］. ［2023-04-02］. https://petalica.com/index_zh.htmlhttps://petalica.com/index_zh.html
Yuan M C and Simo-Serra E. 2021. Line art colorization with concatenated spatial attention//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 3941-3945 ［DOI： 10.1109/CVPRW53098.2021.00442http://dx.doi.org/10.1109/CVPRW53098.2021.00442］
Yun J， Lee S， Park M and Choo J. 2023. iColoriT： towards propagating local hints to the right region in interactive colorization by leveraging vision Transformer//Proceedings of 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa， USA： IEEE： 1787-1796 ［DOI： 10.1109/WACV56688.2023.00183http://dx.doi.org/10.1109/WACV56688.2023.00183］
Zhang L M. 2017. SketchKeras ［EB/OL］. ［2023-04-02］. https://github.com/lllyasviel/sketchKerashttps://github.com/lllyasviel/sketchKeras
Zhang L M. 2022. Style2Paints V45 ［EB/OL］. ［2023-04-02］. https://github.com/lllyasviel/style2paints/tree/master/V4.5https://github.com/lllyasviel/style2paints/tree/master/V4.5
Zhang S C and Zhou J T. 2022. Attention-based unsupervised sketch colorization of anime avatar//Proceedings of the 2nd International Conference on Control and Intelligent Robotics. Nanjing， China： ACM： 829-833 ［DOI： 10.1145/3548608.3559316http://dx.doi.org/10.1145/3548608.3559316］
Zhao Y Z， Po L M， Cheung K W， Yu W Y and Rehman Y A U. 2021. SCGAN： saliency map-guided colorization with generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology， 31（8）： 3062-3077 ［DOI： 10.1109/TCSVT.2020.3037688http://dx.doi.org/10.1109/TCSVT.2020.3037688］
Zhu S， Chen Z J and Ye D Y. 2020. Automatic colorization algorithm with anime effect for scene sketches. Pattern Recognition and Artificial Intelligence， 33（8）： 671-680
朱松， 陈昭炯， 叶东毅. 2020. 场景线稿动漫效果的自动上色算法. 模式识别与人工智能， 33（8）： 671-680 ［DOI： 10.16451/j.cnki.issn1003-6059.202008001http://dx.doi.org/10.16451/j.cnki.issn1003-6059.202008001］
Zou C Q， Mo H R， Gao C Y， Du R F and Fu H B. 2019. Language-based colorization of scene sketches. ACM Transactions on Graphics， 38（6）： #233 ［DOI： 10.1145/3355089.3356561http://dx.doi.org/10.1145/3355089.3356561］
相关作者
相关机构
京公网安备11010802024621