Teleoperation Visualization System of Orchard Robot Based on Enhancing Telepresence

WANG Yundong, ZHOU Jun, SUN Jingwei, WANG Kai, JIANG Zizhen, ZHANG Zhen

Abstract

Aiming at the problem of lack of presence when the orchard robot used a monocular camera for remote operation, only using two-dimensional video to obtain environmental information lacked presence, a set of orchard environment information visualization system based on the enhanced sense of presence was designed. The system consisted of computing server, cloud server, network camera, LiDAR, embedded development platform, etc. The computing server adopted the T7920 workstation, and deployed the Tensorflow computing framework and the Open3D algorithm library of point cloud on it. After receiving the environmental image and point cloud data forwarded by the cloud server, the computing server enhanced the navigation information of the image, and surface reconstructed the point cloud. The embedded development platform could collect raw data from webcam and LiDAR, and uploaded them to cloud servers. A ZeroMQ-based message transfer program and HTML5 background service were deployed on the cloud server, providing cross-Internet message communication services and mobile teleoperation environment information visualization services. The test results showed that the average extraction time of the extraction model for navigation information deployed on the computing server was 86ms, and the average precision of the navigation line extraction was 16°, which were better than the results of the comparison model. The algorithm of point cloud reconstruction can effectively establish scene contours with an average accuracy of 4.9cm and an average reconstruction time of 24ms. The delay of compressed image transmission and enhancement processing did not exceed 230ms, and the transmission delay of point cloud did not exceed 400ms. The parameters could meet the basic requirements of the remote operation robot for orchard operation. The system significantly enhanced telepresence compared with that only with monocular cameras, which provided an effective reference for the remote operation of the orchard robot.


Keywords: orchard robot, teleoperation, telepresence  augmented reality, 3D reconstruction

 

Download Full Text:

PDF


References


LIU Chengliang, GONG Liang, YUAN Jin, et al. Current status and development trends of agricultural robots [ J ]. Transactions of the Chinese Society for Agricultural Machinery, 2022, 53(7) :1 -22,55. (in Chinese)

WANG Hujing, SUN Bingyu. Development status and expectation of agricultural robot [ J ]. Bulletin of Chinese Academy of Sciences, 2015, 30(6): 803 -809. (in Chinese)

AMAYA A, ARACHCHIGE D D K, GREY J, et al. Evaluation of human-robot teleoperation interfaces for soft robotic manipulators[ С ]// Proceedings of the 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN) , 2021.

ZHAO Xincan, ZHAN Penglei, WU Fei, et al. Target recognition and registration in 3D dynamic scene based on domain knowledge [J ]. Robot, 2018, 40(4) : 534 -539. (in Chinese)

CHEN Yinglong, SONG Fujun, ZHANG Junhao, et al. Telerobotic shared control strategy based on telepresence; a review [J ]. Journal of Zhejiang University (Engineering Science) , 2021 , 55(5) ; 831 -842. (in Chinese)

LI Weihua, GUO Junlong, DING Liang, et al. Review and prospect of ground teleoperation technology for lunar rover[ J ]. Acta Aeronautica et Astronautica Sinica, 2023 ,44( 1 ) ; 163 - 182. (in Chinese)

ZHU Xiaojun, WANG Xueqian, МА Yunxuan, et al. Design and experimental study on the teleoperation system for cable- driven and hyper-redundant space flexible manipulator [J ] . Robot, 2022, 44( 1 ) : 9 - 18. ( in Chinese)

XUE Zhihui, LIU Jinguo. Review of space manipulator control technologies [ J ]. Robot, 2022, 44 ( 1 ) : 107 - 128. (in Chinese)

YANG Xian, CHEN Chuanzhi, HUA Changchun. Stability analysis of teleoperated underwater manipulator system with contact force feedback [J/OL]. Control Theory & Applications; 1-10. [2022 — 06 — 17]. http: //kns. cnki. net/kcms/detail/44. 1240. TP. 2021 1231. 1326. 044. html. (in Chinese)

QIN Liujie, SONG Guangming, MAO Juzheng, et al. Shared control of multi-robot formations based on the eye-hand dual- modal human-rohot interface [J ]. Robot, 2022,44(3) : 343 -351 ,360. (in Chinese)

LIU Chuankai, LI Jian, ZHANG Tianzhu, et al. Key technologies in the teleoperation of the Chang’e —5 lunar sampling [J ]. Scientia Sinical Technologica, 2022, 52(3) : 473 -488. (in Chinese)

N1 Dejing, SONG Aiguo, LI Huijun. Survey on robot teleoperation based on virtual reality J]. Chinese Journal of Scientific Instrument, 2017, 38(10): 2351 -2363. (in Chinese)

YU Junzhi, KONG Shihan, MENG Yan. Methods and technologies for visual perception of underwater environment [J ]. Robot, 2022, 44(2) : 224 -235. (in Chinese)

SUN D, KISELEV A, LIAO Q, et al. A new mixed-reality-based teleoperation system for telepresence and maneuverability enhancement J ]. IEEE Transactions on Human-Machine Systems, 2020, 50( 1 ) ; 55 -67.

OPIYO S, ZHOU J, MWANGI E, et al. A review on teleoperation of mobile ground robots: architecture and situation awareness[ J]. International Journal of Control, Automation and Systems, 2021 , 19(3) : 1384 - 1407.

KUO С Y, HUANG С С, TSAI С H, et al. Development of an immersive SLAM-based VR system for teleoperation of a mobile manipulator in an unknown environment [J ]. Computers in Industry, 2021 , 132; 103502.

XIE B, HAN M, JIN J, et al. A generative model-based predictive display for robotic teleoperation [ С ] //2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021 ; 2407 -2413.


Refbacks

  • There are currently no refbacks.