跳转至

Embodied AI

约 367 个字 预计阅读时间 1 分钟

Instructor: Jiayuan Gu

Topics

Topic I: 3D Vision

  • Curve Parameterization, Curvature, Tangent, Normal

  • Surface Parameterization, Curvature, Tangent, Normal, Fundamental Theorem

  • 3D Representation: Polygonal Mesh, Point Cloud, Voxel, Implicit Function

  • 3D Rotation: Rotation Matrix, Euler Angle, Angle-Axis, Quaternion

  • 3D Deep Learning I: 3D Networks, 3D Classification, 3D Segmentation

  • 3D Deep Learning II: 3D Detection, 3D Instance Segmentation

  • 3D Deep Learning III: 3D Representation Learning, 3D Shape Completion and Generation

  • 6D Pose Estimation: ICP, Umeyama's Method, Learning-based Methods

  • Multiview Geometry: Multi-view Stereo, SfM, Camera Calibration

  • Rendering: Rasterization, PBR Rendering, Differentiable Rendering

Topic II: Robotics

  • SE(3) Transformation: Screw, Twist

  • Robot Kinematics: Forward Kinematics, Inverse Kinematics

  • Robot Dynamics: Forward Dynamics, Inverse Dynamics

  • Control: Control Theory, Optimal Control

  • Planning: Motion Planning, Task Planning

  • Physical Simulation

Topic III: Imitation Learning

  • Data Collection: Teleoperation, Kinesthetic Teaching, Retargeting

  • Imitation Learning: ACT, Diffusion Policy, Vision-Language-Action Models

Topic IV: Reinforcement Learning

  • RL Basic I: Markov Decision Process

  • RL Basic II: Q-learning, TD Error, REINFORCE

  • Deep RL I: DQN, SAC, PPO

  • Deep RL II: Model-based RL, Offline RL

Prerequisites

  • Math

  • Linear algebra, calculus, and probability

  • Programming

  • Python is mainly used

  • Knowledge about C/C++ is helpful, but not required

  • Basic knowledge about computer vision and machine learning

Course Evaluation

  • Assignments (50%)

    • problem sets, including both math and programming problems

  • Project (40%), either of two options

    • Robot manipulation challenge (in simulation)

    • Research project

  • Quiz (10%)

    • once or twice, including course attendance

Assignments

  • 4 problem sets

    • Problem 0: linear algebra, optimization, machine learning

    • Problem 1: 3D vision

    • Problem 2: robotics

    • Problem 3: reinforcement learning and imitation learning

  • Late policy

    • 10% is deducted as a late penalty within 3 days, otherwise 20%

    • At most 1 assignment can be submitted late without penalty

Course Project: Robot Manipulation Challenge

  • Teamwork (1-4 people)

  • Grading

    • Report (20%)

      • Following a conference template with intro, related work, method, and results

    • Performance (20%)

      • Take the best of your score and rank

      • E.g., if the success rate is 1% but your rank is 1, you still will get full credit

self-talking

The embodiment hypothesis is the idea that intelligence emerges in the interaction of an agent with an environment and as a result of sensorimotor activity.

From perception to reasoning then interation to the real world.