南方科技大学设计学院助理教授万芳博士于2024年10月7日～9日，参加在美国芝加哥举办的第十六届机器人算法基础研讨会（Workshop on the Algorithmic Foundations of Robotics），介绍团队参加ICRA2024布料操控挑战赛期间开发的布料操控图神经网络算法及其相关研究，万芳博士作为Team Greater Bay的领队参加该项赛事。我们针对布料与外部环境物品物理交互过程的图神经网络建模，仅通过机器人的单次抛甩即可实现对布料目标位置与姿态的操控实现。WAFR是机器人运动规划算法领域的国际高水平会议。本次被接收的论文，也是近20年来首次由来自中国内地高校担任第一作者与通讯作者在该会议上发表研究论文。该论文的第一作者是南方科大-香港大学联合培养的杨林瀚博士，由设计与学习研究实验室助理教授宋超阳博士与香港大学副教授潘佳博士担任共同通讯作者。近期我们受邀参加在阿联酋首都阿布扎比举办的IROS2024研讨会【Workshop on Benchmarking via Competitions in Robotic Grasping and Manipulation】，核心参赛队员、南方科大-香港大学联合培养博士研究生孙浩然也将现场参会，进一步介绍我们的参赛方案与相关研究。与赛事主办方合作，我们也将近期联合向机器人学高水平期刊IEEE Transactions on Robotics提交合作论文。期待在IROS的会场与各位师生的现场交流研讨～
Recently, Prof. WAN Fang from the School of Design at the Southern University of Science and Technology participated in the 16th Workshop on Algorithmic Foundations in Robotics (WAFR), held in Chicago, USA, from Oct 7th to 9th, introducing our research on using Graph Neural Networks to learn goal-conditioned manipuation of fabrics with environment-aware dynamics. WAFR is the world-leading platform on motion planning and robotic algorithms. Prof. WAN led the Team Greater Bay to participate in the ICRA2024 Cloth Manipulation Competition in Japan earlier this year. Next week, by invitation from the competition organizers, we will join the IROS 2024 Workshop on Benchmarking via Competitions in Robotic Grasping and Manipulation to present our solutions during the competition. We will also submit a joint manuscript with the competition organizers and the participating teams on fabric manipulation to the IEEE Transactions on Robotics soon.
– WAFR2024: https://bionicdl.ancorasir.com/?p=1277
– IROS2024 Workshop: https://bionicdl.ancorasir.com/?p=1605
– ICRA2024 Competition: https://cse.usf.edu//~yusun/rgmc/2024.html
One Fling to Goal @ WAFR2024
Prof. WAN Fang recently participated in the 16th Workshop on the Algorithmic Foundations of Robotics (WAFR) in Chicago, USA, to share our latest collaboration with Prof. PAN Jia on Environment-Aware Dynamics for Goal-Conditioned Fabric Manipulation, or One Fling to Goal. Fabric manipulation dynamically is commonly seen in manufacturing and domestic settings. While dynamically manipulating a fabric piece to reach a target state is highly efficient, this task presents considerable challenges due to the varying properties of different fabrics, complex dynamics when interacting with environments, and meeting required goal conditions. To address these challenges, we present One Fling to Goal, an algorithm capable of handling fabric pieces with diverse shapes and physical properties across various scenarios. Our method learns a graph-based dynamics model equipped with environmental awareness. With this dynamics model, we devise a real-time controller to enable high-speed fabric manipulation in one attempt, requiring less than 3 seconds to finish the goal-conditioned task. We experimentally validate our method on a goal-conditioned manipulation task in five diverse scenarios. Our method significantly improves this goal-conditioned task, achieving an average error of 13.2mm in complex scenarios. Our method can be seamlessly transferred to real-world robotic systems and generalized to unseen scenarios in a zero-shot manner.
- For further details regarding this paper and Prof. Wan’s presentation, please refer to this link: https://bionicdl.ancorasir.com/?p=1277.
Presented by Dr. Fang Wan @ Chicago, USA, during WAFR2024. 
In industrial settings, swiftly and accurately moving fabrics to target positions for downstream tasks is a common operation. Workers often employ dynamic flinging motions to enhance efficiency. This observation raises our first challenge: Can we train robots to achieve precise fabric positioning in one fling? 
Previous research often involves robots making multiple attempts to learn the delta dynamics between each trial, continuously optimizing the fling trajectory to reach the target position. However, multiple trials reduce efficiency and delta dynamics are difficult to generalize to complex environments, typically only being effective in flat scenarios. 
Furthermore, fabric properties significantly influence the outcome of the fling. For example, With the same trajectory, more elastic fabrics tend to produce more wrinkles, needs a lighter Fling. This diversity leads to our second challenge: How can we adjust the fling trajectory when fabric properties are unknown? 
Beyond industrial applications, fabric manipulation tasks are commonly-used in daily life, where environments are more complex and often involve interactions between the fabric and surrounding objects. This complexity introduces our third challenge: Can we encode the rich interaction between fabric and environment for robotic manipulation? 
To address these challenges, we propose 1Fling2Goal. Our objective is to manipulate the fabric dynamically into the target state with only one fling in complex environments. We utilize a dual-arm robot setup and an RGB-D camera to capture real-time state of the fabric. Our method is tested across various complex scenarios, as shown on the right, where environmental objects are randomly placed within the workspace, and corresponding fabric target states are defined. 
We begin by training a dynamic model of the fabric where the state is represented as a point cloud. We employ a Graph Neural Network (GNN) to predict the fabric’s point-wise velocities at the next time step, capturing the complex dynamics involved in fabric manipulation. 
To enhance the dynamic model’s performance during interactions with the environment, we integrate environmental information into the node features of our GNN. We term this approach Environment-Aware State Representation. For each node vi, we calculate the distance between the fabric xi and the nearest object point qj in the environment. Later we will show our model more accurately predicts motions by involving fabric-environment contact. 
We collected training data from the PyFlex simulation environment. During data collection, we randomized key parameters related to fabric properties, such as weight and size. Which allows our model to have good generalization capabilities across different fabrics. 
With the dynamic model, we implement a two-stage model-based control strategy to optimize the fling trajectory. In stage one, we acquire the fabric’s current state using the RGB-D camera. 
In this work, we simplify potential trajectory to a fling-then-pull movement, which can be parameterized by a single turning point. To get an initial estimation of the fling trajectory, we randomly sample 50 turning points. 
And then utilize the dynamic model to infer the final state for each sampled trajectory. We select the optimal trajectory whose final state has the minimum distance to the target state. 
Despite this initial estimation, unknown fabric properties may prevent accurate task completion. Therefore, we implement real-time refinement of the fling trajectory. At every ten time steps, we sample 9 delta actions to adjust the current action of the initial estimation. Here delta actions include faster, slower flinging movements, as well as no change to the initial estimation. 
We select the best delta actions based on the dynamic model’s inference. For instance, if we detect that the fabric is highly elastic, we adjust by reducing the fling speed appropriately. 
At each control step, we update the fling trajectory and the fabric’s current state, ensuring that adjustments are responsive to real-time observations and predictions. 
Through this iterative process, the fabric is eventually manipulated to reach the target state. 
We evaluate the performance of the proposed GNN dynamic model through a series of comparative experiments. The results show that incorporating environmental awareness into the fabric representation and collecting data across multiple scenarios significantly improve the model’s accuracy and generalization. The video on the right illustrates our model’s ability to accurately predict the fabric’s motion, even during contact with environmental objects. 
For the goal-conditioned fabric manipulation task, we compared our method with state-of-the-art approaches such as FlingBot and IRP. We also conducted ablation studies on key components of our algorithm, namely environmental awareness and Model predictive control. The experimental results indicate that our method exhibits notable stability and generalization, especially in complex environments. 
Here, we present simulation results across different scenarios, including Flat Scenario without environment awareness, and Blue particles represent the fabric’s state, while red arrows indicate velocity of the fabric’s motion. The green rectangles show the fabric’s target state for visualization purpose; We use the fabric’s point cloud to compute the distance between current state and the target state. 
Platform Scenario, 
Hemisphere Scenario, and 
Pole Scenario with Environment Awareness. 
The proposed two-stage strategy transfers seamlessly to real-world robotic experiments. we randomly place the fabric within the workspace and allow the robot to plan an initial trajectory to fling the fabric to the target position and do real-time adjustment according the RGBD observation. Our method proves to be effective across various settings, including flat surfaces. 
platforms, 
hemispheres, 
poles, and 
stools. We also validate our approach on 
unseen fabrics with novel shapes and 
unseen folded target states. It works well since our dynamic model is generalized and target-free. And the control strategy is applicable to any feasible target. 
In summary, we propose a method that encodes environmental information into the dynamic model of fabric, achieving precise fabric manipulation in one fling. Our approach addresses the challenges of unknown fabric properties and complex environmental interactions. 
There are some limitations. Inaccurate perception due to fling speed and partial observation can sometimes cause the fabric to contact the environment too abruptly, leading to task failure. Additionally, the model’s effectiveness can be limited when interacting with unseen fabric configurations that differ drastically from the training data. Future work will focus on improving the robustness to perception accuracy and extending the model to more diverse task configurations. 
Presented by Dr. Fang Wan @ Chicago, USA, during WAFR2024.
CopGNN @ IROS2024
The theoretical foundation of the algorithm presented at WAFR2024 was initially proposed in Prof. Wan’s Youth Grant under the Natural Science Foundation of China Grant 62206119, awarded in 2021. An earlier development of this work was initially submitted to CoRL. After receiving feedback from the reviewer and reviewing the latest literature from recent research, we organized this work for the ICRA2024 Competition on Fabric Manipulation through Team Greater Bay with Prof. Wan as the Team Lead and various members from Prof. Song Chaoyang @ SUSTech and Prof. Pan Jia @ HKU. We tailored the algorithm specifically for the competition and developed the CopGNN algorithm, providing an end-to-end solution to achieve cloth coverage prediction via Graph Neural Networks. Recently, we were invited to the Poster Session of the IROS2024 Workshop on Benchmarking via Competitions in Robotic Grasping and Manipulation. Feel free to join us in Abu Dhabi, UAE, for further discussion. A key member of Team Greater Bay, Mr. Sun Haoran, will also join IROS for this workshop and present our work during the poster session.
- For further details regarding this paper, please refer to this link: https://bionicdl.ancorasir.com/?p=1605.
A Collaborative Work for IEEE Transactions on Robotics
Together with the organizer of the ICRA2024 Cloth Competition and all participating teams, we recently finished wrapping up a joint manuscript to be submitted to the IEEE Transactions on Robotics. Thanks a lot to the IDLab-AIRO at Ghent University for organizing and the euROBin Project for funding this competition. We will share further information on this work once we’ve finalized the formal submission with the lead author, Victor-Louis De Gusseme, from Ghent University.
