姗�(j墨)鍣ㄤ汉鐨勫悇椤�(l猫i)鍏疯韩鎿嶄綔浠诲嫏(w霉)澶╃劧灏卞皪(du矛)瑾�(y菙)瑷€鎸囦护鐞嗚В銆佸牬(ch菐ng)鏅劅鐭ュ拰鏅�(sh铆)绌鸿(gu墨)鍔冪瓑鑳藉姏鏈夎憲寰堥珮鐨勮姹�锛岄€欒嚜鐒跺紩鐢冲嚭涓€鍊�(g猫)鍟�(w猫n)椤岋細鑳戒笉鑳藉厖鍒嗗埄鐢ㄥぇ妯″瀷鑳藉姏锛屽皣鍏堕伔绉诲埌姗�(j墨)鍣ㄤ汉闋�(l菒ng)鍩燂紝鐩存帴瑕�(gu墨)鍔冨簳灞ゅ嫊(d貌ng)浣滃簭鍒楀憿锛�  

灏�(du矛)姝�锛孊yteDance Research 鍩轰簬闁�(k膩i)婧愮殑澶氭ā鎱�(t脿i)瑾�(y菙)瑷€瑕栬(ju茅)澶фā鍨� OpenFlamingo 闁�(k膩i)鐧�(f膩)浜嗛枊(k膩i)婧�銆佹槗鐢ㄧ殑 RoboFlamingo 姗�(j墨)鍣ㄤ汉鎿嶄綔妯″瀷锛屽彧鐢ㄥ柈姗�(j墨)灏卞彲浠ヨ〒(x霉n)绶�銆備娇鐢ㄧ啊(ji菐n)鍠�銆佸皯閲忕殑寰(di脿o)灏卞彲浠ユ妸 VLM 璁婃垚 Robotics VLM锛屽緸鑰岄仼鐢ㄤ簬瑾�(y菙)瑷€浜や簰鐨勬(j墨)鍣ㄤ汉鎿嶄綔浠诲嫏(w霉)銆�   OpenFlamingo 鍦ㄦ(j墨)鍣ㄤ汉鎿嶄綔鏁�(sh霉)鎿�(j霉)闆� CALVIN 涓婇€�(j矛n)琛屼簡椹�(y脿n)璀夛紝瀵�(sh铆)椹�(y脿n)绲�(ji茅)鏋滆〃鏄�锛孯oboFlamingo 鍙埄鐢ㄤ簡 1% 鐨勫付瑾�(y菙)瑷€妯�(bi膩o)娉ㄧ殑鏁�(sh霉)鎿�(j霉)鍗冲湪涓€绯诲垪姗�(j墨)鍣ㄤ汉鎿嶄綔浠诲嫏(w霉)涓婂彇寰椾簡 SOTA 鐨勬€ц兘銆傞毃钁� RT-X 鏁�(sh霉)鎿�(j霉)闆嗛枊(k膩i)鏀撅紝閲囩敤闁�(k膩i)婧愭暩(sh霉)鎿�(j霉)闋�(y霉)瑷�(x霉n)绶� RoboFlamingo 骞� finetune 鍒颁笉鍚屾(j墨)鍣ㄤ汉骞宠嚭(t谩i)锛屽皣鏈夊笇鏈涙垚鐐轰竴鍊�(g猫)绨�(ji菐n)鍠湁鏁堢殑姗�(j墨)鍣ㄤ汉澶фā鍨� pipeline銆傝珫鏂囬倓娓�(c猫)瑭︿簡鍚勭ó涓嶅悓 policy head銆佷笉鍚岃〒(x霉n)绶磋寖寮忓拰涓嶅悓 Flamingo 绲�(ji茅)妲�(g貌u)鐨� VLM 鍦� Robotics 浠诲嫏(w霉)涓婂井瑾�(di脿o)鐨勮〃鐝�(xi脿n)锛屽緱鍒颁簡涓€浜涙湁鎰忔€濈殑绲�(ji茅)璜栥€�  

鐮旂┒鑳屾櫙

鍩轰簬瑾�(y菙)瑷€鐨勬(j墨)鍣ㄤ汉鎿嶄綔鏄叿韬櫤鑳介牁(l菒ng)鍩熺殑涓€鍊�(g猫)閲嶈鎳�(y墨ng)鐢�锛屽畠娑夊強鍒板妯℃厠(t脿i)鏁�(sh霉)鎿�(j霉)鐨勭悊瑙ｅ拰铏曠悊锛屽寘鎷瑕�(ju茅)銆佽獮(y菙)瑷€鍜屾帶鍒剁瓑銆傝繎骞翠締(l谩i)锛岃瑕�(ju茅)瑾�(y菙)瑷€鍩虹(ch菙)妯″瀷锛圴LMs锛夊凡缍�(j墨ng)鍦ㄥ鍊�(g猫)闋�(l菒ng)鍩熷彇寰椾簡椤憲鐨勯€�(j矛n)灞�锛屽寘鎷湒鍍忔弿杩�銆佽瑕�(ju茅)鍟�(w猫n)绛斿拰鍦栧儚鐢熸垚绛�銆傜劧鑰�锛屽皣閫欎簺妯″瀷鎳�(y墨ng)鐢ㄤ簬姗�(j墨)鍣ㄤ汉鎿嶄綔浠嶇劧瀛樺湪涓€浜涙寫鎴�(zh脿n)锛屼緥濡傚浣曞皣瑕栬(ju茅)鍜岃獮(y菙)瑷€淇℃伅绲�(ji茅)鍚堣捣渚�(l谩i)锛屽浣曡檿鐞嗘(j墨)鍣ㄤ汉鎿嶄綔鐨勬檪(sh铆)搴忔€х瓑銆�   鐐轰簡瑙ｆ焙閫欎簺鍟�(w猫n)椤�锛孊yteDance Research 鐨勬(j墨)鍣ㄤ汉鐮旂┒鍦�(tu谩n)闅�(du矛)鍒╃敤鐝�(xi脿n)鏈夌殑闁�(k膩i)婧� VLM锛孫penFlamingo锛岃ō(sh猫)瑷�(j矛)浜嗕竴濂楁柊鐨勮瑕�(ju茅)瑾�(y菙)瑷€鎿嶄綔妗嗘灦锛孯oboFlamingo銆傚叾涓� VLM 鍙互閫�(j矛n)琛屽柈姝ヨ瑕�(ju茅)瑾�(y菙)瑷€鐞嗚В锛岃€岄澶栫殑 policy head 妯＄祫琚敤渚�(l谩i)铏曠悊姝峰彶淇℃伅銆傚彧闇€瑕佺啊(ji菐n)鍠殑寰(di脿o)鏂规硶灏辫兘璁� RoboFlamingo 閬╂噳(y墨ng)浜庡熀浜庤獮(y菙)瑷€鐨勬(j墨)鍣ㄤ汉鎿嶄綔浠诲嫏(w霉)銆�   RoboFlamingo 鍦ㄥ熀浜庤獮(y菙)瑷€鐨勬(j墨)鍣ㄤ汉鎿嶄綔鏁�(sh霉)鎿�(j霉)闆� CALVIN 涓婇€�(j矛n)琛屼簡椹�(y脿n)璀�锛屽(sh铆)椹�(y脿n)绲�(ji茅)鏋滆〃鏄庯紝RoboFlamingo 鍙埄鐢ㄤ簡 1% 鐨勫付瑾�(y菙)瑷€妯�(bi膩o)娉ㄧ殑鏁�(sh霉)鎿�(j霉)鍗冲湪涓€绯诲垪姗�(j墨)鍣ㄤ汉鎿嶄綔浠诲嫏(w霉)涓婂彇寰椾簡 SOTA 鐨勬€ц兘锛堝浠诲嫏(w霉)瀛�(xu茅)缈�(x铆)鐨� task sequence 鎴愬姛鐜囩偤 66%锛屽钩鍧囦换鍕�(w霉)瀹屾垚鏁�(sh霉)閲忕偤 4.09锛屽熀绶氭柟娉曠偤 38%锛屽钩鍧囦换鍕�(w霉)瀹屾垚鏁�(sh霉)閲忕偤 3.06锛泎ero-shot 浠诲嫏(w霉)鐨勬垚鍔熺巼鐐� 24%锛屽钩鍧囦换鍕�(w霉)瀹屾垚鏁�(sh霉)閲忕偤 2.48锛屽熀绶氭柟娉曠偤 1%锛屽钩鍧囦换鍕�(w霉)瀹屾垚鏁�(sh霉)閲忔槸 0.67锛�锛屽苟涓旇兘澶犻€氶亷(gu貌)闁�(k膩i)鐠�(hu谩n)鎺у埗瀵�(sh铆)鐝�(xi脿n)瀵�(sh铆)鏅�(sh铆)闊挎噳(y墨ng)锛屽彲浠ラ潏娲婚儴缃插湪杓冧綆鎬ц兘鐨勫钩鑷�(t谩i)涓�銆傞€欎簺绲�(ji茅)鏋滆〃鏄庯紝RoboFlamingo 鏄竴绋湁鏁堢殑姗�(j墨)鍣ㄤ汉鎿嶄綔鏂规硶锛屽彲浠ョ偤鏈締(l谩i)鐨勬(j墨)鍣ㄤ汉鎳�(y墨ng)鐢ㄦ彁渚涙湁鐢ㄧ殑鍙冭€冦€�   鏂规硶

鏈伐浣滃埄鐢ㄥ凡鏈夌殑鍩轰簬鍦栧儚 - 鏂囨湰灏�(du矛)鐨勮瑕�(ju茅)瑾�(y菙)瑷€鍩虹(ch菙)妯″瀷锛岄€氶亷(gu貌)瑷�(x霉n)绶寸鍒扮鐨勬柟寮忕敓鎴愭(j墨)鍣ㄤ汉姣忎竴姝ョ殑 relative action銆傛ā鍨嬬殑涓昏妯″鍖呭惈浜� vision encoder锛宖eature fusion decoder 鍜� policy head 涓夊€�(g猫)妯″銆俈ision encoder 妯″鍏堝皣鐣�(d膩ng)鍓嶈瑕�(ju茅)瑙€娓�(c猫)杓稿叆鍒� ViT 涓�锛屽苟閫氶亷(gu貌) resampler 灏�(du矛) ViT 杓稿嚭鐨� token 閫�(j矛n)琛� down sample銆侳eature fusion decoder 灏� text token 浣滅偤杓稿叆锛屽苟鍦ㄦ瘡鍊�(g猫) layer 涓厛灏� vision encoder 鐨� output 浣滅偤 query 閫�(j矛n)琛� cross attention锛屼箣鍚庨€�(j矛n)琛� self attention 浠ュ畬鎴愯瑕�(ju茅)鑸囪獮(y菙)瑷€鐗瑰緛鐨勮瀺鍚�銆傛渶鍚�锛屽皪(du矛) feature fusion decoder 閫�(j矛n)琛� max pooling 鍚庡皣鍏堕€佸叆 policy head 涓�锛宲olicy head 鏍规摎(j霉) feature fusion decoder 杓稿嚭鐨勭暥(d膩ng)鍓嶅拰姝峰彶 token 搴忓垪鐩存帴杓稿嚭鐣�(d膩ng)鍓嶇殑 7 DoF relative action锛屽寘鎷簡 6-dim 鐨勬(j墨)姊拌噦鏈浣嶅Э鍜� 1-dim 鐨� gripper open/close銆�   鍦ㄨ〒(x霉n)绶撮亷(gu貌)绋嬩腑锛孯oboFlamingo 鍒╃敤闋�(y霉)瑷�(x霉n)绶寸殑 ViT銆丩LM 鍜� Cross Attention 鍙冩暩(sh霉)锛屽苟鍙井瑾�(di脿o) resampler銆乧ross attention 鍜� policy head 鐨勫弮鏁�(sh霉)銆�   瀵�(sh铆)椹�(y脿n)绲�(ji茅)鏋�鏁�(sh霉)鎿�(j霉)闆嗭細  

CALVIN锛圕omposing Actions from Language and Vision锛夋槸涓€鍊�(g猫)闁�(k膩i)婧愮殑妯℃摤鍩烘簴(zh菙n)娓�(c猫)瑭�锛岀敤浜庡(xu茅)缈�(x铆)鍩轰簬瑾�(y菙)瑷€鐨� long-horizon 鎿嶄綔浠诲嫏(w霉)銆傝垏鐝�(xi脿n)鏈夌殑瑕栬(ju茅) - 瑾�(y菙)瑷€浠诲嫏(w霉)鏁�(sh霉)鎿�(j霉)闆嗙浉姣�锛孋ALVIN 鐨勪换鍕�(w霉)鍦ㄥ簭鍒楅暦(zh菐ng)搴︺€佸嫊(d貌ng)浣滅┖闁撳拰瑾�(y菙)瑷€涓婇兘鏇寸偤寰�(f霉)闆�锛屽苟鏀寔闈堟椿鍦版寚瀹�鍌虫劅鍣�杓稿叆銆侰ALVIN 鍒嗙偤 ABCD 鍥涘€�(g猫) split锛屾瘡鍊�(g猫) split 灏�(du矛)鎳�(y墨ng)浜嗕笉鍚岀殑 context 鍜� layout銆�   瀹氶噺鍒嗘瀽锛�  

RoboFlamingo 鍦ㄥ悇瑷�(sh猫)缃拰鎸囨(bi膩o)涓婄殑鎬ц兘鍧囩偤鏈€浣筹紝瑾�(shu艒)鏄庝簡鍏跺叿鏈夊緢寮�(qi谩ng)鐨勬ā浠胯兘鍔�銆佽瑕�(ju茅)娉涘寲鑳藉姏浠ュ強瑾�(y菙)瑷€娉涘寲鑳藉姏銆侳ull 鍜� Lang 琛ㄧず妯″瀷鏄惁浣跨敤鏈厤灏�(du矛)鐨勮瑕�(ju茅)鏁�(sh霉)鎿�(j霉)閫�(j矛n)琛岃〒(x霉n)绶达紙鍗虫矑(m茅i)鏈夎獮(y菙)瑷€閰嶅皪(du矛)鐨勮瑕�(ju茅)鏁�(sh霉)鎿�(j霉)锛�锛汧reeze-emb 鎸囩殑鏄噸绲�(ji茅)铻嶅悎瑙ｇ⒓鍣ㄧ殑宓屽叆灞わ紱Enriched 琛ㄧず浣跨敤 GPT-4 澧炲挤(qi谩ng)鐨勬寚浠�銆�   娑堣瀺瀵�(sh铆)椹�(y脿n)锛�  

涓嶅悓鐨� policy head锛�   瀵�(sh铆)椹�(y脿n)鑰冨療浜嗗洓绋笉鍚岀殑绛栫暐闋儴锛歁LP w/o hist銆丮LP w hist銆丟PT 鍜� LSTM銆傚叾涓紝MLP w/o hist 鐩存帴鏍规摎(j霉)鐣�(d膩ng)鍓嶈娓�(c猫)闋�(y霉)娓�(c猫)姝峰彶锛屽叾鎬ц兘鏈€宸紝MLP w hist 灏囨鍙茶娓�(c猫)鍦� vision encoder 绔€�(j矛n)琛岃瀺鍚堝悗闋�(y霉)娓�(c猫) action锛屾€ц兘鏈夋墍鎻愬崌锛汫PT 鍜� LSTM 鍦� policy head 铏曞垎鍒ラ’寮�銆侀毐寮忓湴缍(h霉)姝峰彶淇℃伅锛屽叾琛ㄧ従(xi脿n)鏈€濂斤紝瑾�(shu艒)鏄庝簡閫氶亷(gu貌) policy head 閫�(j矛n)琛屾鍙蹭俊鎭瀺鍚堢殑鏈夋晥鎬�銆�   瑕栬(ju茅)-瑾�(y菙)瑷€闋�(y霉)瑷�(x霉n)绶寸殑褰遍熆锛�   闋�(y霉)瑷�(x霉n)绶村皪(du矛)浜� RoboFlamingo 鐨勬€ц兘鎻愬崌璧峰埌浜嗛棞(gu膩n)閸典綔鐢�銆傚(sh铆)椹�(y脿n)椤ず锛岄€氶亷(gu貌)闋�(y霉)鍏堝湪澶у瀷瑕栬(ju茅)-瑾�(y菙)瑷€鏁�(sh霉)鎿�(j霉)闆嗕笂閫�(j矛n)琛岃〒(x霉n)绶达紝RoboFlamingo 鍦ㄦ(j墨)鍣ㄤ汉浠诲嫏(w霉)涓〃鐝�(xi脿n)寰楁洿濂�銆�   妯″瀷澶у皬鑸囨€ц兘锛�   闆栫劧閫氬父鏇村ぇ鐨勬ā鍨嬫渻(hu矛)甯朵締(l谩i)鏇村ソ鐨勬€ц兘锛屼絾瀵�(sh铆)椹�(y脿n)绲�(ji茅)鏋滆〃鏄庯紝鍗充娇鏄純灏忕殑妯″瀷锛屼篃鑳藉湪鏌愪簺浠诲嫏(w霉)涓婅垏澶у瀷妯″瀷濯茬編銆�   鎸囦护寰(di脿o)鐨勫奖闊匡細   鎸囦护寰(di脿o)鏄竴鍊�(g猫)寮�(qi谩ng)澶х殑鎶€宸э紝瀵�(sh铆)椹�(y脿n)绲�(ji茅)鏋滆〃鏄�锛屽畠鍙互閫�(j矛n)涓€姝ユ彁楂樻ā鍨嬬殑鎬ц兘銆�

瀹氭€х祼(ji茅)鏋�鐩歌純浜庡熀绶氭柟娉�锛孯oboFlamingo 涓嶄絾瀹屾暣鍩�(zh铆)琛屼簡 5 鍊�(g猫)閫ｇ簩(x霉)鐨勫瓙浠诲嫏(w霉)锛屼笖灏�(du矛)浜庡熀绶氶爜(y猫)鍩�(zh铆)琛屾垚鍔熺殑鍓嶅叐鍊�(g猫)瀛愪换鍕�(w霉)锛孯oboFlamingo 鎵€鐢ㄧ殑姝ユ暩(sh霉)涔熸槑椤洿灏戙€�  

绺界祼(ji茅)鏈伐浣滅偤瑾�(y菙)瑷€浜や簰鐨勬(j墨)鍣ㄤ汉鎿嶄綔绛栫暐鎻愪緵浜嗕竴鍊�(g猫)鏂扮⿴鐨勫熀浜庣従(xi脿n)鏈夐枊(k膩i)婧� VLMs 鐨勬鏋�锛屼娇鐢ㄧ啊(ji菐n)鍠井瑾�(di脿o)灏辫兘瀵�(sh铆)鐝�(xi脿n)鍑鸿壊鐨勬晥鏋�銆俁oboFlamingo 鐐烘(j墨)鍣ㄤ汉鎶€琛�(sh霉)鐮旂┒鑰呮彁渚涗簡涓€鍊�(g猫)寮�(qi谩ng)澶х殑闁�(k膩i)婧愭鏋�锛岃兘澶犳洿瀹规槗鍦扮櫦(f膩)鎻枊(k膩i)婧� VLMs 鐨勬經鑳姐€傚伐浣滀腑璞愬瘜鐨勫(sh铆)椹�(y脿n)绲�(ji茅)鏋滄垨瑷卞彲浠ョ偤姗�(j墨)鍣ㄤ汉鎶€琛�(sh霉)鐨勫(sh铆)闅涙噳(y墨ng)鐢ㄦ彁渚涘璨寸殑缍�(j墨ng)椹�(y脿n)鍜屾暩(sh霉)鎿�(j霉)锛屾湁鍔╀簬鏈締(l谩i)鐨勭爺绌跺拰鎶€琛�(sh霉)鐧�(f膩)灞�銆�