Real-time Apple Picking Pattern Recognition for Picking Robot Based on Improved YOLOv5m

YAN Bin, FAN Pan, WANG Meirong, SHI Shuaiqi, LEI Xiaoyan, YANG Fuzeng

Abstract

In order to accurately identify the different fruit targets on apple trees, and automatically distinguish the fruit occluded by different branches, providing visual guidance for the mechanical picking end-effector to actively adjust the pose of apple picking to avoid the shelter of the branches, a real-time recognition method of apple picking pattern based on improved YOLOv5m for picking robot was proposed. Firstly, BottleneckCSP module was designed and improved to BottleneckCSP-B module which was used to replace the BottleneckCSP module in backbone architecture of original YOLOv5m network. The ability of image deep feature extraction of the original BottleneckCSP module was enhanced, and the original YOLOv5m backbone network was lightweight designed and improved. Secondly, SE module was inserted to the proposed improved backbone network, to better extract the features of different apple targets. Thirdly, the bonding fusion mode of feature maps, which were input to the target detection layer of medium size in the original YOLOv5m network, were improved, and the recognition accuracy of apple was improved. Finally, the initial anchor box sizes of the original network were improved, avoiding the misrecognition of apples in farther plant row. The experimental results indicated that the graspable, circuitous-graspable (up-graspable, down-graspable, left-graspable, right-graspable) and ungraspable apples could be identified effectively by using the proposed improved model in the study. The recognition recall, precision, mAP and F1 were 85.9%, 81.0%, 80.7% and 83.4%, respectively. The average recognition time was 0.025s per image. Contrasted with original YOLOv5m, YOLOv3 and EfficientDet-D0 model, the mAP of the proposed improved YOLOv5m model was increased by 5.4 percentage points, 22 percentage points and 20.6 percentage points, respectively on test set. The size of the improved model was 89.59% of original YOLOv5m model. The proposed method can provide technical support for the picking end-effector of robot to pick apples in different poses avoiding the shelter of branches, to reduce the loss of apple picking.

 

Keywords:  apple, picking robot, YOLOv5m, picking pattern recognition, visual guidance, deep learning



Download Full Text:

PDF


References


FAN P, LANG G D, GUO P J, et al. Multi-feature patch-based segmentation technique in the gray-centered RGB color space for improved apple target recognition [J]. Agriculture, 2021, 11(3):273.

FAN P, LANG G D, YAN B, et al. A method of segmenting apples based on gray-centered RGB color space [J]. Remote Sensing, 2021, 13(6):1211.

FAN P, YAN B, WANG M R, et al. Three-finger grasp planning and experimental analysis of picking patterns for robotic apple harvesting [J]. Computers and Electronics in Agriculture, 2021, 188:1 - 19.

FU L, GAO F, WU J, et al. Application of consumer RGB-D cameras for fruit detection and localization in field; a critical review [J]. Computers and Electronics in Agriculture, 2020, 177:1 - 11.

ZHANG Z, IGATHINATHANE C, LI J, et al. Technology progress in mechanical harvest of fresh market apples [J]. Computers and Electronics in Agriculture, 2020, 175:1 - 12.

YAN B, FAN P, LEI X Y, et al. A real-time apple targets detection method for picking robot based on improved YOLOv5 [J]. Remote Sensing, 2021, 13(9):1619.

LIU Mochen, GAO Tiantian, MA Zongxu, et al. Target detection model of corn weeds in field environment based on MSRCR algorithm and YOLOv4 – tiny [J]. Transactions of the Chinese Society for Agricultural Machinery, 2022, 53( 2):246 - 255, 335.(in Chinese)

WANG Qiaohua, GU Wei, CAI Peizhong, et al. Detection method of double side breakage of population cotton seed based on improved YOLO v4 [J]. Transactions of the Chinese Society for Agricultural Machinery, 2022, 53(1):389 - 397. (in Chinese)

ZHANG Hongming, LI Yongheng, ZHOU Lixiang, et al. Multi-objective skeleton extraction method of beef cattle based on improved YOLO v3 [J]. Transactions of the Chinese Society for Agricultural Machinery, 2022, 53(3):285 - 293. (in Chinese)

ZHANG Lu, HUANG Lin, LI Beibei, et al. Fish school counting method based on multi-scale fusion and no anchor YOLO v3 [J]. Transactions of the Chinese Society for Agricultural Machinery, 2021, 52 (Supp.):237 - 244. (in Chinese)

BRESILLA K, PERULLI G D, BOINI A, et al. Single-shot convolution neural networks for real-time fruit detection within the tree [J]. Frontiers in Plant Science, 2019, 10:1- 12.

WU Xing, QI Zeyu, WANG Longjun, et al. Apple detection method based on light-YOLO v3 convolutional neural network [J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(8):17 - 25. (in Chinese)

ZHAO Dean, WU Rendi, LIU Xiaoyang, et al. Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background [J]. Transactions of the CSAE, 2019, 35(3):164 - 173. (in Chinese)

ZHAO Hui, QIAO Yanjun, WANG Hongjun, et al. Apple fruit recognition in complex orchard environment based on improved YOLO v3 [J]. Transactions of the CSAE, 2021, 37(16):127 - 135. (in Chinese)

LU S, CHEN W, ZHANG X, et al. Canopy-attention-YOLOv4-based immature/mature apple fruit detection on dense-foliage tree architectures for early crop load estimation [J]. Computers and Electronics in Agriculture, 2022, 193:1 -

FU L, MAJEED Y, ZHANG X, et al. Faster R - CNN-based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting [J]. Bio systems Engineering, 2020, 197:245 - 256.

GAO F, FU L, ZHANG X, et al. Multi-class fruit-on-plant detection for apple in SNAP system using Faster R – CNN [J]. Computers and Electronics in Agriculture, 2020, 176:1 - 10.

GENE-MOLA J, VILAPLANA V, ROSELL-POLO JR, et al. Multi-modal deep learning for Fuji apple detection using RGB – D cameras and their radiometric capabilities [J]. Computers and Electronics in Agriculture, 2019, 162: 689 - 698.

ZHANG J, KARKEE M, ZHANG Q, et al. Multi-class object detection using Faster R - CNN and estimation of shaking locations for automated shake-and-catch apple harvesting [J]. Computers and Electronics in Agriculture, 2020, 173:1 - 10.

KANG H, CHEN C. Fruit detection, segmentation and 3D visualization of environments in apple orchards [J]. Computers and Electronics in Agriculture, 2020, 171:1 - 10.

WANG Dandan, HE Dongjian. Recognition of apple targets before fruits thinning by robot based on R - FCN deep convolution neural network [J]. Transactions of the CSAE, 2019, 35(3):156 - 163. (in Chinese)


Refbacks

  • There are currently no refbacks.