Robots have evolved significantly since the days of the Roomba. Nowadays, we see drones making doorstep deliveries, self-driving cars navigating urban roads, robo-dogs assisting first responders, and robots performing acrobatics or enhancing efficiency on factory floors. Despite these advancements, Luca Carlone believes we have only scratched the surface of what robotics can achieve.
Carlone, recently tenured as an associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), heads up the SPARK Lab, where his research team is tackling a vital connection between humans and robots: perception. Focused on both theoretical and experimental studies, Carlone and his students strive to give robots an understanding of their surroundings that mimics human perception. As Carlone often emphasizes, perception transcends mere detection.
Robots have made impressive strides in their ability to detect and recognize objects. However, they still lag behind when it comes to comprehending their environments at a higher cognitive level. Humans intuitively perceive not just the shapes and labels of objects but also their physical properties — how they can be manipulated, how they relate to each other, and to ourselves. This understanding is crucial for robots to interact safely and effectively in homes, workplaces, and other complex settings.
Since joining MIT in 2017, Carlone has spearheaded projects that leverage perception and scene-understanding algorithms for diverse applications. His work includes autonomous underground search-and-rescue vehicles, drones capable of picking up and manipulating objects in real-time, and self-driving cars. These innovations could also benefit household robots that respond to spoken commands and anticipate users’ needs based on contextual understanding.
“Perception is a significant bottleneck in getting robots to assist us effectively in real-world scenarios,” Carlone states. “By integrating cognitive reasoning into robot perception, I am confident they can make substantial contributions.”
Exploring New Frontiers
Carlone hails from Salerno, Italy, situated near the picturesque Amalfi Coast. Growing up as the youngest of three sons, he was influenced by his mother’s career as an elementary school teacher and his father’s work as a history professor and publisher. Perhaps this academic atmosphere inspired the brothers, as all three pursued engineering — the eldest in electronics, the middle in mechanical engineering, and Carlone specializing in robotics, then known as mechatronics.
His journey into robotics began unexpectedly during the latter part of his undergraduate studies at the Polytechnic University of Turin, where he initially focused on control theory. He was captivated by a robotics course in his senior year, which illuminated the fascinating possibilities of programming robots for movement and interaction.
“It was truly love at first sight. Crafting algorithms and utilizing math to create a robot’s ‘brain’ and enable it to navigate its environment is incredibly rewarding,” Carlone shares. “From that moment, I committed to pursuing this as my life’s work.”
He later pursued a dual-degree program between the Polytechnic University of Turin and the Polytechnic University of Milan, obtaining master’s degrees in both mechatronics and automation engineering. This prestigious program, known as the Alta Scuola Politecnica, introduced him to management principles, challenging him to collaborate with peers from different disciplines on product design pitches. One notable project involved creating a touch-free table lamp that responded to hand gestures, broadening his perspective on engineering’s impact on daily life.
“It felt like learning to communicate in various languages,” Carlone reflects. “It taught me the importance of looking beyond the technical confines of engineering to create solutions that resonate with real-world needs.”
Pioneering Research
Carlone continued his journey in Turin, completing his PhD in mechatronics with a thesis focused on “simultaneous localization and mapping” (SLAM), a critical problem in robotics. He innovatively reframed previous approaches, resulting in the development of algorithms that create more accurate maps without needing an initial position, which many SLAM methods traditionally required. His groundbreaking work pushed the boundaries in a field thought to be well-explored.
“SLAM revolves around understanding object geometry and a robot’s navigation among those objects,” Carlone explains. “Now, I’m part of a community questioning the next evolution of SLAM.”
To find answers, he took a postdoc position at Georgia Tech, immersing himself in coding and computer vision. This path was partly inspired by a personal experience: a serious medical complication that severely impacted his vision during his PhD. “For a year, I faced potential vision loss, which underscored the significant role of sight and artificial vision in robotics,” he reveals.
Thanks to excellent medical care, his vision fully recovered, enabling him to forge ahead in his research. At Georgia Tech, Carlone collaborated with his advisor, Frank Dellaert, who pioneered an open-source SLAM library, GTSAM — a resource that helped Carlone appreciate the transformative potential of open-source software in advancing robotics.
“In the past, progress in SLAM was sluggish because teams often worked in isolation with proprietary codes,” Carlone notes. “Then, with open-source initiatives, the landscape changed dramatically, facilitating exponential growth in the field over the past decade.”
Advancing Spatial Intelligence
After his tenure at Georgia Tech, Carlone joined MIT in 2015 as a postdoctoral researcher at the Laboratory for Information and Decision Systems (LIDS). He collaborated with Professor Sertac Karaman on software that enables small drones to navigate with minimal onboard power. A year later, he was promoted to research scientist and became a faculty member in AeroAstro in 2017.
“What drew me to MIT was its commitment to values-driven decision-making,” Carlone states. “The guiding principle is to improve society, a mindset that’s incredibly refreshing.”
Currently, Carlone’s team is pushing the envelope on how robots represent their surroundings, striving to go beyond geometric shapes and semantics. By incorporating deep learning and large language models, they are developing algorithms that allow robots to perceive their environments from a higher-level perspective. In just six years, the lab has released over 60 open-source repositories, utilized globally by researchers and industry practitioners alike. His work contributes to a burgeoning field known as “spatial AI.”
“Think of spatial AI as an enhanced version of SLAM,” he explains. “It’s about empowering robots to perceive and understand the world like humans, making them more practical and relatable.”
This ambitious endeavor holds promise for creating intuitive and interactive robots capable of assisting in everyday life, workplace settings, on the roads, or in dangerous and remote areas. Carlone acknowledges the challenges that lie ahead to achieve a level of perception akin to that of human toddlers.
“I have twin daughters who are just 2 years old, and watching them as they navigate cluttered spaces while handling multiple toys with ease makes it clear how far we still have to go with robot perception,” Carlone notes. “Yet, we have exciting new tools at our disposal, and the future is undeniably bright.”
Photo credit & article inspired by: Massachusetts Institute of Technology
