add spot_human_recognition demo_code by heissereal · Pull Request #1387 · jsk-ros-pkg/jsk_demos

heissereal · 2023-04-12T08:45:30Z

spotの360度カメラで、人物を認識したら、そこに近づいてふりむいて声をかけ、「あっちに行きたい」と指を指した方向へと先導したいです。　@mqcmd196 cc. @sktometometo @tkmtnt7000

Miyamichi demo

mqcmd196 · 2023-04-13T08:47:55Z

@heissereal
PRでディレクトリ構成を変更しました．

source ~/spot_ws/devel/setup.bash
roscd jsk_2023_04_human_approach

とすると，作業ディレクトリに移って作業できます．

今までやってくれた作業はあるんだけど，今回BoundingBoxが360度カメラの画像から出せるlaunchがあることがわかったので，
/rect_array_in_panorama_to_boundig_box_array/bbox_array をSubscribeして，そこにgo-posする処理を書いてみましょう．
ヒント: 型は jsk_recognition_msgs/BoundingBoxArray，で位置情報とかが入っています．

手を検出するのは明日やってみます

Add handpose recognition and add Rviz

mqcmd196 · 2023-04-17T07:25:02Z

/skeleton_with_depth/output/skeleton (型はjsk_recognition_msgs/HumanSkeletonArray) に奇跡的にこの中身のtopicが入ることがある．

---
header: 
  seq: 4862
  stamp: 
    secs: 1681715628
    nsecs: 217715354
  frame_id: "hand_color_image_sensor"
human_ids: []
skeletons: 
  - 
    header: 
      seq: 4862
      stamp: 
        secs: 1681715628
        nsecs: 217715354
      frame_id: "hand_color_image_sensor"
    bone_names: 
      - right shoulder->right elbow
    bones: 
      - 
        start_point: 
          x: 2.3581541222927527
          y: 0.3860775946041869
          z: 1.5579999685287476
        end_point: 
          x: 1.8728612915368605
          y: 0.7816122661615572
          z: 1.5609999895095825
  - 
    header: 
      seq: 4862
      stamp: 
        secs: 1681715628
        nsecs: 217715354
      frame_id: "hand_color_image_sensor"
    bone_names: []
    bones: []

重要なのはbone_namesのフィールドに，elbowが含まれていること．このときのbonesのフィールドの start_point end_point の2点を拾ってこれれば，腕の指す方向ベクトルがわかる．

Add rosinstall and human pose launch with depth

mqcmd196 · 2023-04-17T08:54:56Z

今まで僕がSpot体内に入って，認識器のlaunchを上げていたのですが，それをやってもらいます

ssh miyamichi@belka

でSpot体内に入ります．でSpot体内で

tmux

と打って，tmuxを立ち上げます．その中で

source ~/ros/spot_ws/devel/setup.bash
roslaunch jsk_2023_04_human_approach recognition.launch

を立ち上げます．そうすると，360度カメラが見えるようになり，360度カメラの人認識，ハンドカメラによる人の姿勢認識が走ります．
このtmux はターミナルの中でターミナルを立ち上げるツールみたいなもので，手元のsshが切れたとしても体内のターミナルに残り続けます．それに接続するには

tmux a

を打ちます．

mqcmd196 · 2023-04-17T09:14:50Z

体内で上げるlaunch (jsk_2023_04_human_approach/launch/recognition.launch) に

360度カメラでの人の位置認識

<include if="$(arg coral)" file="$(find coral_usb)/launch/edgetpu_panorama_object_detector.launch">
  <arg name="INPUT_IMAGE" value="/dual_fisheye_to_panorama/output"/>
</include>

と

ハンドカメラでの人の姿勢認識

<include if="$(arg coral)" file="$(find coral_usb)/launch/edgetpu_human_pose_estimator.launch">
    <arg name="INPUT_IMAGE" value="/spot/camera/hand_color/image"/>
</include>

があって，どっちかコメントインしたらどっちかコメントアウトしないと動かないです．これも直しておきます

add coral switcher in launch file

heissereal · 2023-04-21T01:59:57Z

transformのところでSegmentation Faultが起こります…

mqcmd196 · 2023-04-21T05:18:56Z

@YUKINA-3252 @tkmtnt7000 @Kanazawanaoaki Could you solve this error?

heissereal · 2023-04-21T10:36:24Z

以下のように表示されて、recognition.launchが立ち上がっていないようですが、どこを直したらいいでしょうか

process[insta360/insta_camera_node-1]: started with pid [7198]
process[split_image-2]: started with pid [7203]
process[dual_fisheye_to_panorama-3]: started with pid [7209]
process[camera_tf_publisher-4]: started with pid [7212]
type is jsk_perception/DualFisheyeToPanorama
process[rect_array_in_panorama_to_bounding_box_array-5]: started with pid [7230]
process[miyamichi_demo_nodelet_manager-6]: started with pid [7234]
process[nodelet-7]: started with pid [7240]
process[coral_input_image_mux-8]: started with pid [7257]
[ INFO] [1682073108.894982065]: Loading nodelet /nodelet of type depth_image_proc/register to manager miyamichi_demo_nodelet_manager with the following remappings:
[ INFO] [1682073108.907839539]: /depth/camera_info -> /spot/camera/hand_depth/camera_info
[ INFO] [1682073108.908416156]: /depth/image_rect -> /spot/camera/hand_depth/image
[ INFO] [1682073108.908768035]: /depth_registered/image_rect -> /spot/camera/hand_depth/depth_registered/image
[ INFO] [1682073108.908928155]: /rgb/camera_info -> /spot/camera/hand_color/camera_info
[ INFO] [1682073108.930265040]: waitForService: Service [/miyamichi_demo_nodelet_manager/load_nodelet] has not been advertised, waiting...
[ INFO] [1682073108.956762570]: Initializing nodelet with 8 worker threads.
process[edgetpu_node_manager-9]: started with pid [7353]
[ INFO] [1682073108.982650093]: waitForService: Service [/miyamichi_demo_nodelet_manager/load_nodelet] is now available.
process[skeleton_with_depth-10]: started with pid [7372]
[ INFO] [1682073109.321729514]: advertising
[ INFO] [1682073109.357527520]: Opening camera with vendor=0x2e1a, product=0x1000, serial="", index=0
unsupported descriptor subtype VS_STILL_IMAGE_FRAME
unsupported descriptor subtype VS_COLORFORMAT
unsupported descriptor subtype VS_COLORFORMAT
attempt to claim already-claimed interface 1
[ WARN] [1682073109.515370035]: Unable to set scanning_mode to 1
[ WARN] [1682073109.515485382]: Unable to set auto_exposure to 2
[ WARN] [1682073109.515679040]: Unable to set auto_exposure_priority to 1
[ WARN] [1682073109.515731994]: Unable to set exposure_absolute to 1
[ WARN] [1682073109.515907667]: Unable to set auto_focus to 1
[ WARN] [1682073109.515963428]: Unable to set focus_absolute to 22074
[ WARN] [1682073109.516130940]: Unable to set iris_absolute to 0
[ WARN] [1682073109.516310468]: Unable to set pantilt to 0, 0
[ INFO] [1682073109.625518942]: using default calibration URL
[ INFO] [1682073109.633654133]: camera calibration URL: file:///home/miyamichi/.ros/camera_info/camera.yaml
[ INFO] [1682073109.633782795]: Unable to open camera calibration file [/home/miyamichi/.ros/camera_info/camera.yaml]
[ WARN] [1682073109.633820344]: Camera calibration file /home/miyamichi/.ros/camera_info/camera.yaml not found.
[ INFO] [1682073109.655388587]: light_compen : false
[ INFO] [1682073109.658198510]: refine_align : false
[ INFO] [1682073109.658240188]: fovd         : 210.000
[ INFO] [1682073109.658255857]: save_unwarped: false
[ INFO] [1682073109.658268669]: mls_map_path : /home/miyamichi/ros/spot_ws/src/jsk_recognition/jsk_perception/config/dual_fisheye_to_panorama/fisheye_stitcher_grid_xd_yd_2000x1000.yml.gz
[ INFO] [1682073109.658284061]: blend_image_height : 1000
[ INFO] [1682073109.658297324]: blend_image_width  : 2000
[ INFO] [1682073109.658309911]: blend_param_p_wid : 30
[ INFO] [1682073109.658326595]: blend_param_p_x1 : 75
[ INFO] [1682073109.658339847]: blend_param_p_x2 : 885
[ INFO] [1682073109.658351456]: blend_param_row_start : 295
[ INFO] [1682073109.658363641]: blend_param_row_end : 660
[ INFO] [1682073109.658374951]: output_image_height : 1000
[ INFO] [1682073109.658384406]: output_image_width  : 2000
[INFO] [1682073110.257067]: starting edgetpu_panorama_object_detector
[INFO] [1682073110.266629]: Using transport raw
[ INFO] [1682073110.305020808]: initialize stitcher w:2000 h:1000
Initializing necessary parameters..
[WARN] [1682073114.430297]: subscribing topic seems not to be pulished. waiting... Error: timeout exceeded while waiting for message on topic ~input_class
[WARN] [1682073114.793678]: [/split_image] subscribes topics only with child subscribers. Set '~always_subscribe' as True to have it subscribe always.
[WARN] [1682073115.021633]: [/skeleton_with_depth] subscribes topics only with child subscribers. Set '~always_subscribe' as True to have it subscribe always.
[ INFO] [1682073122.394820020]: Bond broken, exiting
[nodelet-7] process has finished cleanly
log file: /home/miyamichi/.ros/log/94c61ef0-de72-11ed-9c6f-a8a1590e9101/nodelet-7*.log
^C[skeleton_with_depth-10] killing on exit
[edgetpu_node_manager-9] killing on exit
[coral_input_image_mux-8] killing on exit
[miyamichi_demo_nodelet_manager-6] killing on exit
[rect_array_in_panorama_to_bounding_box_array-5] killing on exit
[camera_tf_publisher-4] killing on exit
[dual_fisheye_to_panorama-3] killing on exit
[insta360/insta_camera_node-1] killing on exit
[split_image-2] killing on exit
[WARN] [1682073130.376487]: subscribing topic seems not to be pulished. waiting... Error: rospy shutdown
[INFO] [1682073130.524221]: Node is successfully initialized
shutting down processing monitor...
... shutting down processing monitor complete
done

heissereal · 2023-04-21T11:07:10Z

上のエラーは、Spotを再起動したら直りました。
cc: @tkmtnt7000

k-okada · 2023-04-21T11:17:44Z

治ったようで良かったです．
メール書きかけていた，より詳しい情報を知る方法，のメモをおくります．
＝＝＝
これは，

[WARN] [1682073115.021633]: [/skeleton_with_depth] subscribes topics only with child subscribers. Set '~always_subscribe' as True to have it subscribe always.
[ INFO] [1682073122.394820020]: Bond broken, exiting
[nodelet-7] process has finished cleanly
log file: /home/miyamichi/.ros/log/94c61ef0-de72-11ed-9c6f-a8a1590e9101/nodelet-7*.log　＜＜立ち上がってもここで止まってしまう
^C[skeleton_with_depth-10] killing on exit　＜＜　ここは自分でC-cした

という理解でいいかな．まだ解決していなければ，解決方法ではないけど，より詳しい状況確認法は以下かな．

ROSCONSOLE_FORMAT=[${severity}] [${time}]: [${node}] ${message} roslaunch --screen jsk_2023_04_human_approach recognition.launch としてより多くのエラーメッセージを知ることが出来ます．
(roscd jsk_perception; export GIT_PAGER=cat; git branch -a; git diff; git rev-parse HEAD; cd -)　として，プログラムのソースのバージョンがどいうものかわかります．
別のターミナルで rostopic echo /skeleton_with_depth/output/skeleton としておく．

tkmtnt7000 · 2023-05-17T12:44:47Z

デバッグメモ:
手先のUSBハブに

目玉デバイス
D435
Respeaker

を全て繋いでrealsenseのノードを立ち上げようとするとd435との接続が切れる
目玉デバイスもしくはRespeakerを外すと正常にrealsenseのノードをたちあげられる
3つ繋ぐと電源供給が足りなかった可能性が考えられる

heissereal and others added 6 commits April 12, 2023 17:28

add spot_human_recognition demo_code

e18c589

modified

4ae1855

modified

a94c487

rework

6b5fbda

update recognition launch. expected launch in spot PC

88cc6b9

Merge pull request #1 from mqcmd196/miyamichi_demo

25061a9

Miyamichi demo

mqcmd196 and others added 3 commits April 14, 2023 11:09

detect handpose

21ceaec

add rviz

dfc2352

Merge pull request #2 from mqcmd196/miyamichi_demo

346a989

Add handpose recognition and add Rviz

sktometometo mentioned this pull request Apr 17, 2023

Support panorama detection k-okada/jsk_robot#99

Open

mqcmd196 and others added 3 commits April 17, 2023 16:50

update rosinstall

bdfa475

add human pose launch with depth

d13a9ce

Merge pull request #3 from mqcmd196/miyamichi_demo

cbcabfc

Add rosinstall and human pose launch with depth

heissereal and others added 8 commits April 18, 2023 11:15

recognition bones

97c6dea

recognition bones

0445db6

Merge branch 'master' of github.com:heissereal/jsk_demos

55e8577

modified

04a45a5

add coral switcher in launch file

1efa5ac

Merge pull request #4 from mqcmd196/miyamichi_demo

268c180

add coral switcher in launch file

modified

c534333

Merge branch 'master' of github.com:heissereal/jsk_demos

833cfdf

heissereal added 2 commits May 15, 2023 20:53

modified

cb534d7

modified

50a7468

human_recognition

5600b97

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add spot_human_recognition demo_code#1387

add spot_human_recognition demo_code#1387
heissereal wants to merge 23 commits intojsk-ros-pkg:masterfrom
heissereal:master

heissereal commented Apr 12, 2023

Uh oh!

mqcmd196 commented Apr 13, 2023

Uh oh!

mqcmd196 commented Apr 17, 2023 •

edited

Loading

Uh oh!

mqcmd196 commented Apr 17, 2023

Uh oh!

mqcmd196 commented Apr 17, 2023

Uh oh!

heissereal commented Apr 21, 2023

Uh oh!

mqcmd196 commented Apr 21, 2023

Uh oh!

heissereal commented Apr 21, 2023

Uh oh!

heissereal commented Apr 21, 2023

Uh oh!

k-okada commented Apr 21, 2023

Uh oh!

tkmtnt7000 commented May 17, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

heissereal commented Apr 12, 2023

Uh oh!

mqcmd196 commented Apr 13, 2023

Uh oh!

mqcmd196 commented Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mqcmd196 commented Apr 17, 2023

Uh oh!

mqcmd196 commented Apr 17, 2023

Uh oh!

heissereal commented Apr 21, 2023

Uh oh!

mqcmd196 commented Apr 21, 2023

Uh oh!

heissereal commented Apr 21, 2023

Uh oh!

heissereal commented Apr 21, 2023

Uh oh!

k-okada commented Apr 21, 2023

Uh oh!

tkmtnt7000 commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mqcmd196 commented Apr 17, 2023 •

edited

Loading

tkmtnt7000 commented May 17, 2023 •

edited

Loading