摘 要：目的 对城市进展过程中产生的建筑固废进行处理，并将之转换为资源和能源，是极佳的保护环境的经济发展模式。然而人工分拣处理存在着效率慢、污染严重、对人身危害大等问题。目前工业界在探索一种有效的基于机械臂自动抓取的建筑固废自动分拣系统，其中图像分割技术是非常必要的一个环节。但是工业现场的环境因素造成固废对象的颜色严重退化，会影响最终的固废对象分割。本文针对建筑固废图像分割对象难度大的现状，提出一种基于多模态深度神经网络的方法来解决固废对象分割问题。方法 首先, 在颜色退化严重的场景下，把RGB图像和深度图一起作为深度卷积神经网络的输入，利用深度卷积神经网络进行高维特征学习，通过softmax分类器获得每个像素的标签分配概率。其次，基于新的能量函数建立全连接条件随机场，通过最小化能量函数寻找全局最优解来分割图像，从而为每一类固废对象产生一个独立的分割块。最后，利用局部轮廓信息计算深度梯度，实现同一类别的不同实例的固废对象精确分割。结果 在固废图像测试集上,该方法取得了90.02%均像素精度和89.03%均交并比（MIOU）。此外，与目前一些优秀的语义分割算法相比，也表现出了优越性。结论 本文提出的方法能够对每一个固废对象同时进行有效的分割和分类，为建筑垃圾自动分拣系统提供准确的固废对象轮廓和类别信息，从而方便实现机械臂的自动抓取。
Multimodal Deep Neural Network for Construction Waste Object Segmentation
zhangjianhua,chenjiawei,zhangshaobo,guojianshuang,liusheng(College of Computer Science and Technology, Zhejiang University of Technology)
Abstract: Objective Construction waste is no longer useless nowadays. It is an excellent economic development mode to protect the environment by recycling waste generated during construction and converting it into resources and energy. The current situation of construction waste in China has become more and more severe. With the development of urbanization, the old buildings are demolished and rebuilt and especially are replaced by skyscrapers. And once inhabited areas have been gradually transformed into cities with ever expaning sizes of this cities. These cities have high speed development along with serious hidden dangers. Construction waste generated by a large number of construction work sites have become more and more difficult to ignore. Urban construction waste refers to all kinds of construction waste generated during the construction, transformation, decoration, demolition and laying of various buildings and structures and their auxiliary facilities. It mainly includes muck, waste concrete, waste brick, waste pipe, waste wood, etc. According to statistics, the amount of construction building waste in China now accounts for 30% to 40% of the total amount of municipal waste. In the next ten years, China will produce more than 1.5 billion tons of construction waste per year on average. It is estimated that by 2020, the construction waste will reach 2.6 billion tons; by 2030, it will reach 7.3 billion tons. Resource utilization and recycling is are inevitable choices for dealing with construction waste in buildings. To effectively deal with construction waste in buildings, you can start from its characteristics. The construction waste is a mixture of various building materials wastes, which are actually a unutilized resources. In the 1990s, several communities in California first launched a single-stream recycling project, which refered to the mixture of all paper products, plastics, glass, metal, and other waste. It was separated into single item by a sorting system. In the sorting system, waste was mainly processed by a combination of hardware equipment and manpower. The system was not fully automated and relied mainly on human recycling, so it was not efficient. This was a meaningful attempt to let people understand the feasibility of recycling waste. For construction waste, there are many construction wastes, such as waste bricks, waste rock, scrap steel, etc., which can be recycled after being sorted, rejected or crushed. However a system like a single-stream recycling project is not capable to handle a large amount of construction waste. With the development of artificial intelligence technology, the use of intelligent robotic equipment in the field of construction waste recycling can greatly improve the capability, efficiency and safety of recycling. Among them, the robotic arm is the most widely used automated mechanical device in the industrial field. It can quickly grasp objects and can work continuously. The emergence of robotic arms provides a new and efficient solution for the automatic sorting of construction waste in buildings. The use of robotic arms to sort construction waste is a revolutionary innovation for the construction waste treatment industry. For the robotic arm grabbing task, the position information and contour information of the object are indispensable. The application of computer image segmentation algorithms in this scene is undoubtedly very suitable. Through the image segmentation algorithms, the construction waste image can be accurately segmented to obtain the position and contour of each object. Combining robotic arms and image segmentation algorithms to achieve efficient construction waste recovery is worth looking forward to. However, due to the characteristics of industrial sites and construction waste objects, it is very difficult to segment construction waste objects from the obtained construction waste images by the segmentation algorithms. In terms of the difficulty of object segmenting in construction waste image, this paper proposes a construction waste object segmentation method based on multimodal information deep neural network to solve the image segmentation problem, and provides accurate construction waste object contour and category information for the construction waste automatic sorting system. Therefore, it is capable to realize automatic grabbing using the robot arm. Method First of all, in the scenes with severe color degradation, feature learning with RGB images alone does not meet the actual needs. Therefore, it is necessary to train the salient features with depth information. We treat the RGB image and the corresponding depth image as the input of the deep convolutional neural network. The deep convolutional neural network is used to perform high-dimensional feature learning, and the feature maps obtained from the convolutional layers of the last layer are weighted and summed, and then feed as input data of the Softmax classifier, finally we obtain the label allocation probability of each pixel. Based on the probability that each pixel belongs to a category, we construct a multi-label full connected conditional random field. The unary energy term treat each pixel as an independent item, without considering the relationship between the pixels. The binary energy term represents the relationship among pixels. So that similar pixels are divided into the same category, and pixels with large differences between each other are assigned to different categories, which makes the segmented edges smoother. We able to obtain more accurate segmentation results. Therefore, according to the actual situation, We propose an energy function suitable for construction waste objects. The global optimal solution is obtained by minimizing the energy function to segment the object in the image, thereby generating an independent segmentation block for each type of construction waste object. Finally, Fine segmentation of local ambiguous regions is done based on the depth gradient information. The ambiguous area refers to an adhesion areas between construction waste objects that are difficult to distinguish due to degradation of visual characteristics. The depth gradient information is used to obtain the local depth edge map, from which the local ambiguity area is extracted. For the local ambiguity area, the algorithm extracts the effective internal edge to segment the adhesion objects belonging to the same class. Result On the construction waste image test set, our method achieves 90.02% Mean Pixel Accuracy(MPA) and 89.03% Mean Intersection Over Union(MIOU). In addition, compared with some excellent semantic segmentation algorithms, the experimental results show that the proposed method obtain better performance and improve the segmentation accuracy. Conclusion The algorithm proposed in this paper can segment and classify most construction waste object effectively at the same time, and provide accurate contour and classification information of the construction waste object to a construction waste automatic sorting system, so as to facilitate the automatic grasping construction waste by the robotic arm.