Enhancing robotic perception through synchronized simulation and physical common-sense reasoning

Abstract

We introduce both conceptual and empirical findings arising from the amalgamation of a robotics cognitive architecture with an embedded physics simulator, aligning with the principles outlined in the intuitive physics literature. The employed robotic cognitive architecture, named CORTEX, leverages a highly efficient distributed working memory known as deep state representation. This working memory inherently encompasses a fundamental ontology, state persistency, geometric and logical relationships among elements, and tools for reading, updating, and reasoning about its contents. Our primary objective is to investigate the hypothesis that the integration of a physics simulator into the architecture streamlines the implementation of various functionalities that would otherwise necessitate extensive coding and debugging efforts. Furthermore, we categorize these enhanced functionalities into broad types based on the nature of the problems they address. These include addressing challenges related to occlusion, model-based perception, self-calibration, scene structural stability, and human activity interpretation. To demonstrate the outcomes of our experiments, we employ CoppeliaSim as the embedded simulator and both a Kinova Gen3 robotic arm and the Open-Manipulator-P as the real-world scenarios. Synchronization is maintained between the simulator and the stream of real events. Depending on the ongoing task, numerous queries are computed, and the results are projected into the working memory. Participating agents can then leverage this information to enhance overall performance.