Abstract
Large Language Models (LLMs) are considered state of the art for many tasks in robotics and AI, but there is also increasing evidence of their critical limitations. They can generate arbitrary responses in new situations, do not support rapid incremental adaptation based on limited examples, and are opaque. Toward addressing these limitations, our architecture leverages the complementary strengths of LLMs and knowledge-based reasoning, enabling an AI agent assisting a human user to use an LLM to provide generic abstract predictions of upcoming tasks. Also, the agent reasons with domain-specific knowledge, any recent history of interactions with the user, and semantic databases to: (a) provide contextual prompts to the LLM; and (b) compute a plan of concrete actions that jointly implements the current task and prepare for the anticipated task, diagnosing unexpected outcomes and replanning as needed. Furthermore, the architecture solicits and uses high-level human feedback based on need and availability to incrementally revise the domain-specific knowledge and interactions with the LLM. We ground and evaluate our architecture’s abilities in the realistic VirtualHome simulation environment, demonstrating a substantial performance improvement compared with just using LLMs or an LLM and logical reasoner.
