Estimation of the Movable Position of the Robot based on the Object Recognition and GVG Update

1. Introduction

Recently, research on mobile robots that provide social services to people based on location has been actively conducted in the field of robot core technology.^1-7) As the services that robots can perform have become more intelligent and advanced as a result of these studies, the environment in which service tasks can be performed has also become complex and diversified, and service robots that provide personal services to specific users are also on the rise.⁸⁾ Above all, in most specific environments where mobile robots perform delivery tasks, such as shopping malls, department stores, smart factories, and logistics companies, the location of the target performing commands is fixed, but on the contrary, robot environments that provide services to individuals have a characteristic that the command target is unspecified and the user's location is easily moved. For this reason, the location of the service target that cannot be specified must be designated every time through human intervention, and as the number of handling targets of the service robot increases, the number of times to specify the location will be increased.^9,10)

To solve this problem, the mobile robot estimates the user's location from sensor data that can extract images and distances and provides a service that enables gaze tracking.¹¹⁾ However, the recognition range is extremely limited, and location estimation is performed through a complicated calculation process. Accordingly, in order to provide location-based service (LBS) to users far from mobile robots, a method of estimating the user's location using various IoT sensors has been proposed as an alternative.^12-15)

In this paper, we propose a specific process for estimating and updating the optimal movement destination essential for the LBS of a mobile robot in an indoor environment.

2. Experimental setup and method

The process for estimating the optimal position for the mobile robot to move for the purpose of delivering small household items is as follows (see Fig. 1). First, a GVG having obstacle avoidance is created from a grid map essential for moving the mobile robot to a destination. Then, in order to perform the location-based delivery service, the recognition information and location of the target object are estimated from the RGB-D sensor, and the location of the obstacle is detected in real-time using the LiDAR sensor. By reflecting the estimated object and obstacle information, a workspace for manipulating the object is created and the movable working position is selected by applying the optimal estimation process. Thereafter, the existing node is updated to the optimal location by creating or moving the node by calculating the distance co-relationship with the nearby node based on the rule.

Fig. 1

A process for optimal estimation of a robot’s movable position

2.2 Workspace allocation

In the case of the initial GVG based on the grid map, the node is located in a safe area from the already formed obstacles, but there are restrictions in setting the destination when the robot performs work in a dynamic environment or the local environment is variable. Therefore, it is essential in the study of intelligent mobile robots to consider the surrounding accessible area and the shape of obstacles for the purpose of setting the robot's moving destination.^17-21) The proposed process attempts to estimate the optimal position suitable for performing the LBS by updating the GVG based on the location of objects and obstacles, and the geometric configuration of the robot platform. The RGB-D sensor mounted on the robot, the object is recognized using real-time object recognition (YOLO v4) from the extracted RGB images. Then, the reachable workspace where the manipulator of the mobile robot is estimated from the bounding box of the detected object and the depth image.²²⁾ The reachable workspace is a workspace where objects can be manipulated, and the size is determined according to the specifications of the robot platform, but in this study, it was designed in consideration of the basic specifications of the generally used open manipulator. The method of estimating the optimal working position from the collision area (A_COL), and reachable workspace (A_reach) displayed on the grid map is as follows. In order to estimate the movable position of the robot, A_COL has priority over A_reach, and the final workspace (A_work) is the difference of A_COL with respect to A_reach (see Fig. 2).

Fig. 2

Awork considering collision area ACOL

2.3 Optimal position estimation

A_work is decided on the characteristics of the two-dimensional grid map and various geometrical components of the indoor environment. A movable position is properly estimated in general cases, but an inappropriate aspect may occur in the actual movement of the robot. For instance, A_reach which projects 3D coordinate information in 2D, can be separated into several regions by obstacles such as tables or walls on the grid map, and the straight-line distance from the object is the shortest, but the optimal location for the path planning of the mobile robot cannot be estimated.

Therefore, a hybrid approach of the center of gravity (COG) and the area thinning (AT) method is actually applied to improve the efficiency of estimating the optimal position from the selected A_work. As shown in Fig. 3, the AT method can estimate the optimal location through GVG thinning within an area but requires a workspace of a certain size or more. On the other hand, the COG method has the advantage of estimating the optimal location regardless of the size of the workspace but could be located outside the assigned area.

Fig. 3

The hybrid approach of COG and AT

In addition, when the final candidate area is selected to have a size greater than or equal to a pre-determined size, a plurality of optimal positions is estimated by performing a process of dividing the final candidate area into a pre-determined size before applying the hybrid approach.

2.4 GVG node update

When the optimal position within the workspace is selected, the node update type is determined by comparing the distance with the existing node location on the GVG. When the relative distance between the optimal position and the adjacent node is within a certain interval (e.g. 0.5 m), the node is updated while changing the corresponding node to the optimal position. In particular, when the position of the existing node is closer to the object than the estimated optimal position or is located in the collision area, the node is automatically updated to the adjacent optimal position. However, if the relative distance conditions are not satisfied, a new node must be created, and node generation creates nodes from the parent node if the adjacent node is a leaf node and from that node if it is a branch node (see Fig. 4).

Fig. 4

Rule-based node update: movement and creation

3. Experimental results and discussion

To create nodes in the GVG, a grid map was created based on an indoor space with a total area of 76.86 m² (W9.15 m× H8.40 m), and then thinning was first performed on the non-obstacle occupied grid using a GVG. For the optimal position experiment, ROBOTIS' Turtlebot 3 robot platform was used. The collision size of the robot platform base is 0.22 m, which is the radius of the robot's circumscribed circle around the rotation axis, and the RGB-D sensor Astra S (Orbbec Inc.) is mounted on the upper part of the robot. The maximum recognition distance by the real-time object recognition is set to 2.0 m, and the working radius by the robot's platform is set to 0.8 m. Fig. 5 shows an initial node creation in which GVG was performed on the robot platform and grid map used in the experiment.

Fig. 5

Experimental setup of GVG to create initial nodes on the grid map

The experiment for estimating the optimal position is largely composed of 3 steps: 1. Object recognition-based workspace setting in grid map, 2. optimal position estimation by hybrid approach, and 3. rule-based node update on the grid map. Fig. 6(a) and 6(c) show the results of estimating the appropriate workspace and optimal position (red dot) according to the geometrical configuration of an object (blue dot) by applying the AT method. If the workspace is larger than a certain size, a plurality of optimal positions is estimated to increase the robot's accessibility to the object. Fig. 6(a) and 6(b) show the results of the division of the estimated workspace and the estimation of multiple optimal positions when an object is surrounded by temporary obstacles. If the workspace is within a certain size, one optimal position is estimated within a single area as shown in Fig. 6(c) and 6(d).

Fig. 6

Estimation of optimal position by AT method

Fig. 7 shows the result of estimating the optimal position using the COG method in a situation where the workspace has a size that cannot be thinned out. In addition, when an object is located in an area blocked by an occupant grid, the height of the existence of an obstacle between the object and the robot should be considered. However, we assumed that it is possible to work geometrically with respect to the table height through the robot platform design in this experiment. The workspace is divided as a result, and the accessibility to the object is various, but the optimal position is chosen by COG because the size of the workspace is small. If the workspace is overlapped by the occupied grid, it becomes impossible to estimate the optimal position, so it is necessary to sufficiently consider the size of the workspace.

Fig. 7

Estimation of optimal position by COG method

The selected optimal position using the hybrid approach updates the GVG by moving and creating nodes in consideration of the geometric minimum distance with the existing nodes. Fig. 8 shows the result of the GVG update by moving the node because the optimal position is close to the node of the existing GVG. In this way, the optimal position can be configured as a node of the GVG while avoiding the creation of unnecessary nodes.

Fig. 8

The result of GVG update by node movement

Fig. 9 shows the result of the GVG update by creating additional nodes because there are no nodes within the minimum distance around the optimal position. Node creation is performed only for nodes within the reference distance selected as a standard that does not destroy the shape of the existing GVG.

Fig. 9

The result of node update by node creation

As such, each optimal position changes the update method according to the geometric minimum distance with the node. However, a problem occurs in an object having a plurality of optimal positions as shown in Fig. 10. Since the optimal locations are located close to each other, there is a possibility that nodes may be repeatedly designated as moving or creating targets and there may be a problem that information about the optimal position cannot be included.

Fig. 10

The result of node update for multiple working position

Accordingly, for the moved node, a new node is added without moving it again, and a rule that connects the neighboring moved node and the edge is applied to compensate for the loss of information about the optimal location.

As shown in Fig. 11, for an object having a plurality of optimal positions in the grid map, the movement of each node is first performed, and for the remaining positions, nodes are created. Also, the GVG is updated so that all optimal positions can be included in the GVG. Through this experiment, we verified the validity of the process of estimating the optimal position for the actual robot in order to move and create the node using the recognized object location from the existing meaningless node.

Fig. 11

The result of multiple node updates on the grid map

Through this experiment, we verified the validity of the process of estimating the optimal position for the actual robot to move or update the existing node using object recognition based on the initial meaningless node. However, it was confirmed that other locations could be estimated as the optimal location according to the height of the obstacle because it relied on the plane design based on the grid map.

4. Conclusions

In this study, we propose an optimal position estimation process for providing LBS for mobile robots. After assigning the movable workspace based on object recognition and location, the optimal position on the grid map is estimated using the hybrid approach of the AT and the COG method. We conducted experiments on three cases where the object was surrounded by obstacles, the size of the workspace was small, and the object was located in an area where the occupancy grid was blocked. And then, by updating the node on the GVG from the estimated optimal position, the destination of the mobile robot capable of LBS was processed, and the validity of the proposed process was verified from the experimental results.

However, we know that the proposed approach does not lead to optimized results for all LBS. Since the workspace is estimated based on the size of the recognized object, additional countermeasures for object recognition failure are required, and the optimal position estimation failure due to occlusion by other objects or geometric factors in the assigned workspace should be considered. In addition, additional research is needed to increase the effectiveness of topological maps, such as mapping semantic data about the nodes applicable to object inference.

Author contributions

W. J. Lee; Conceptualization, Data curation, Formal analysis, Investigation, Software, Writing–original draft, Y. W. Choi; Resources, Formal analysis, S. S. Yun; Conceptualization, Funding acquisition, Validation, Writing–review & editing, I. Hwang; Conceptualization, Resources, Supervision, Validation, Writing–review & editing.

References

• H. Choi, J. Park and Y. Moon, 2006, “Ubiquitous sensor network based localization system for public guide robot”, The journal of the Korea Institute of Maritime Information & Communication Sciences, 10(10), 1920-1926.
• H. Ahn, J. Choi, H. Moon, M. Jang, S. Kwak and Y. Lim, 2019, “Social Human-Robot Interaction of Human-Care Service Robots”, ACM/IEEE International Conference on Human-Robot Interaction, 698-699. [https://doi.org/10.1109/HRI.2019.8673282]
• J. Pripfl et al., 2016, “Results of a real world trial with a mobile social service robot for older adults”, ACM/IEEE International Conference on Human-Robot Interaction, 497-498. [https://doi.org/10.1109/HRI.2016.7451824]
• C. Shi, S. Satake, T. Kanda and H. Ishiguro, 2016, “How would store managers employ social robots?”, ACM/IEEE International Conference on Human-Robot Interaction, 519-520. [https://doi.org/10.1109/HRI.2016.7451835]
• D. Portugal, P. Alvito, E. Christodoulou et al., 2019, “A Study on the Deployment of a Service Robot in an Elderly Care Center”, IInternational Journal of Social Robotics 11, 317-341. [https://doi.org/10.1007/s12369-018-0492-5]
• W. Chen, W. Chen, C. He, N. Liu, P. Wu and H. Shi, 2020, “Research on Classification and De-tection System of Common Household Tools for Home Service Robot”, International Conference on System Science and Engineering, 1-5. [https://doi.org/10.1109/ICSSE50014.2020.9219314]
• Y. Liu, D. Chen and S. Zhang, 2018, “Obstacle avoidance method based on the movement trend of dynamic obstacles”, International Conference on Control and Robotics Engineering, 45-50. [https://doi.org/10.1109/ICCRE.2018.8376431]
• E. C. Hwang, 2021, “Artificial Intelligence Service Robot Market Trend”, Proceedings of Korean Society of Computer Information Conference, 111-112.
• C. Y. Sohn, J. Y. Oh and Y. S. Chung, 2020, “Development of Annotation Tool for Topological Map”, Proceedings of Symposium of the Korean Institute of communications and Information Sciences, 753-754.
• X. Qi, W. Wang, L. Guo, M. Li, X. Zhang and R. Wei, 2019, “Building a Plutchik’s Wheel In-spired Affective Model for Social Robots”, Journal of Bionic Engineering 16, 209-221. [https://doi.org/10.1007/s42235-019-0018-3]
• S. Yun and S. Yun, 2019, “Implementation of human-tracking robot system using human position and a pan-tilt module”, Proceeding of the Korea Institute of Information and Communication Engineering, 100-103.
• M. Seong and S. Yun, 2020, “Estimation of indoor location using a beacon trilateration”, Proceedings of Institute of Control, Robotics and Systems, 19-20.
• M. Khalaf-Allah, 2013, “Differential ultra-wideband (DUWB) for accurate indoor position estimation: Basic concept and simulation results”, Saudi International Electronics, Communications and Photonics Conference, 1-4. [https://doi.org/10.1109/SIECPC.2013.6550787]
• H. Yu, J. Park, J. Sim and G. Kim, 2015, “Navigation of the mobile robot and estimation of robot pose using beacon”, Proceeding of the Korean Institute of Electrical Engineers, 75-76.
• A. Oh, 2015, “Smart factory logistics management system using house interior position tracking technology based on bluetooth beacon”, The Journal of the Korean Institute of Information and Communication Engineering, 19(11), 2677-2682. [https://doi.org/10.6109/jkiice.2015.19.11.2677]
• C. Choi, J. Song, W. Chung and M. Kim, 2002, “Topological map building based on thinning and its application to localization”, IEEE/RSJ International Conference on Intelligent Robots and Systems 1, 552-557. [https://doi.org/10.1109/IRDS.2002.1041448]
• H. S. Kim and H. B. Lee, 2020, “Autonomous Exploration and Semantic Mapping of a Mobile Robot Using Efficient Frontier Selection”, Journal of Institute of Control, Robotics and Systems, 26(5), 318-324. [https://doi.org/10.5302/J.ICROS.2020.20.0026]
• H. B. Lee, 2019, “Obstacle Avoidance Algorithm for Cooperative Mobile Robots under Velocity and Curvature Constraints”, Journal of Institute of Control, Robotics and Systems, 25(10), 910-915. [https://doi.org/10.5302/J.ICROS.2019.19.0165]
• D. I. Kim and G. S. Sukhatme, 2014, “Semantic labeling of 3D point clouds with object affordance for robot manipulation”, IEEE International Conference on Robotics and Automation, 5578-5584. [https://doi.org/10.1109/ICRA.2014.6907679]
• D. Hsu, J. C. Latcombe and S. Sorkin, 1999, “Placing a robot manipulator amid obstacles for optimized execution”, IEEE International Symposium on Assembly and Task Planning, 280-285. [https://doi.org/10.1109/ISATP.1999.782972]
• A. H. Khan, S. Li and X. Luo, 2020, “Obstacle Avoidance and Tracking Control of Redundant Robotic Manipulator: An RNN-Based Metaheuristic Approach”, IEEE Transactions on Industrial Informatics, 16(7), 4670-4680. [https://doi.org/10.1109/TII.2019.2941916]
• M. Bjelonic, 2018, YOLO ROS: Real-Time Object Detection for ROS. (https://github.com/leggedrobotics/darknet_ros, )