JRDB Leaderboard

Individual Action Submissions

Name

mAP

JRDB-Act

5.365%

Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, Hamid Rezatofighi JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection in CVPR2022

Details

Name	JRDB-Act
Submission Date	2021-12-01 11:34:48+00:00
Abstract	The availability of large-scale video action understanding datasets has facilitated advances in the interpretation of visual scenes containing people. However, learning to recognise human actions and their social interactions in an unconstrained real-world environment comprising numerous people, with potentially highly unbalanced and long-tailed distributed action labels from a stream of sensory data captured from a mobile robot platform remains a significant challenge, not least owing to the lack of a reflective large-scale dataset. In this paper, we introduce JRDB-Act, as an extension of the existing JRDB, which is captured by a social mobile manipulator and reflects a real distribution of human daily-life actions in a university campus environment. JRDB-Act has been densely annotated with atomic actions, comprises over 2.8M action labels, constituting a large-scale spatio-temporal action detection dataset. Each human bounding box is labeled with one pose-based action label and multiple (optional) interaction-based action labels. Moreover JRDB-Act provides social group annotation, conducive to the task of grouping individuals based on their interactions in the scene to infer their social activities (common activities in each social group). Each annotated label in JRDB-Act is tagged with the annotators' confidence level which contributes to the development of reliable evaluation strategies. In order to demonstrate how one can effectively utilise such annotations, we develop an end-to-end trainable pipeline to learn and infer these tasks, i.e. individual action and social group detection.
Publication title	JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection	Publication authors:	Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, Hamid Rezatofighi
Publication venue and year	CVPR2022	Publication URL:	https://openaccess.thecvf.com/content/CVPR2022/papers/Ehsanpour_JRDB-Act_A_Large-Scale_Dataset_for_S
Code Language	N/A	Hardware	N/A
Code Website	N/A	Code URL	N/A

Visualisation

Name: huang-intersection-2019-01-22_0, Video Generating currently not available!

Best 3 By mAP ↓

Worst 3 By mAP ↓

Per-sequence Results

Sequence	mAP	Category_1	Category_2	Category_3	Category_4	Category_5	Category_6	Category_7	Category_8	Category_9	Category_10	Category_11	Category_12	Category_13	Category_14	Category_15	Category_16	Category_17	Category_18	Category_19	Category_20	Category_21	Category_22	Category_23	Category_24
cubberly-auditorium-2019-04-22_1	0.108	0.386	0.34	0.209	0.0	0.0	0.0	0.366	0.0	0.0	-	-	0.0	-	0.0	-	-	-	0.0	-	-	-	-	-	-
discovery-walk-2019-02-28_0	0.163	0.463	0.256	0.142	-	-	-	0.182	-	-	0.1	0.0	-	-	-	-	-	-	0.0	-	-	-	-	-	-
discovery-walk-2019-02-28_1	0.101	0.433	0.273	0.158	-	0.0	0.0	0.222	0.0	-	0.021	0.0	-	-	-	-	-	-	0.0	-	0.0	-	-	-	-
food-trucks-2019-02-12_0	0.13	0.447	0.429	0.444	0.053	0.0	0.004	0.121	0.0	-	0.167	-	0.0	-	-	0.0	-	0.029	-	-	-	-	-	-	0.0
gates-ai-lab-2019-04-17_0	0.1	0.27	0.366	0.32	0.003	0.0	0.0	0.333	0.0	0.0	-	-	0.0	-	-	-	-	0.006	-	-	-	-	-	0.0	0.0
gates-basement-elevators-2019-01-17_0	0.115	0.31	0.249	0.161	0.14	0.0	0.0	0.401	0.0	-	-	0.0	-	-	0.0	-	-	-	0.0	-	-	-	-	-	-
gates-foyer-2019-01-17_0	0.107	0.305	0.417	0.517	0.001	0.0	0.0	0.147	0.0	-	-	-	0.0	-	0.0	-	-	0.0	0.0	-	0.0	-	-	-	-
gates-to-clark-2019-02-28_0	0.211	0.388	0.014	0.7	-	0.0	0.0	0.501	-	-	0.083	-	-	-	-	-	-	-	0.0	-	-	-	-	-	-
hewlett-class-2019-01-23_0	0.14	0.421	0.576	0.176	-	0.0	0.0	0.356	0.0	-	-	-	0.0	-	0.0	-	-	0.013	-	-	-	-	-	0.0	-
hewlett-class-2019-01-23_1	0.15	0.584	0.575	0.104	0.001	0.0	0.0	0.087	0.0	-	-	-	-	-	0.0	-	-	-	-	-	-	-	-	-	-
huang-2-2019-01-25_1	0.143	0.405	0.285	0.342	0.253	0.0	0.0	0.143	0.0	0.0	-	-	0.0	-	-	-	-	-	-	-	-	-	-	-	-
huang-intersection-2019-01-22_0	0.271	0.499	0.599	0.405	-	-	-	0.278	-	-	0.12	-	-	-	-	-	-	-	-	-	-	0.0	-	0.0	-
indoor-coupa-cafe-2019-02-06_0	0.059	0.09	0.251	0.227	0.136	0.003	0.003	0.177	0.0	0.0	-	-	0.0	0.0	0.0	0.0	0.0	-	-	0.0	-	-	-	-	-
lomita-serra-intersection-2019-01-30_0	0.169	0.443	0.507	0.012	-	0.0	0.0	0.11	-	-	0.196	-	-	-	-	-	-	0.25	-	-	-	-	0.0	-	-
meyer-green-2019-03-16_1	0.122	0.297	0.084	0.465	0.069	0.0	0.0	0.413	0.0	-	0.011	-	0.0	-	-	-	-	0.0	-	-	-	-	-	-	-
nvidia-aud-2019-01-25_0	0.095	0.346	0.255	0.25	0.136	0.0	0.0	0.15	0.0	-	-	0.0	0.0	-	0.0	-	-	0.0	-	-	-	-	-	-	-
nvidia-aud-2019-04-18_1	0.212	0.298	0.625	0.586	0.384	0.356	0.144	0.576	0.0	-	-	0.0	0.0	0.0	0.0	-	0.0	-	0.0	-	-	-	-	-	-
nvidia-aud-2019-04-18_2	0.07	0.478	0.111	0.229	0.085	0.0	0.0	0.011	-	-	-	-	0.0	-	0.0	-	-	0.0	0.0	-	-	-	-	0.0	0.0
outdoor-coupa-cafe-2019-02-06_0	0.077	0.218	0.117	0.194	0.077	-	0.0	0.341	0.0	0.0	0.051	-	0.0	0.0	0.0	0.0	-	-	-	-	-	-	-	-	-
quarry-road-2019-02-28_0	0.209	0.487	-	0.436	-	0.0	0.0	0.412	0.0	-	-	-	-	-	-	-	-	0.333	-	-	0.0	-	-	-	-
serra-street-2019-01-30_0	0.14	0.234	0.002	0.416	-	-	-	0.112	0.0	-	0.219	-	-	-	-	-	-	-	-	-	-	-	-	-	0.0
stlc-111-2019-04-19_1	0.212	0.438	0.305	0.293	-	-	0.0	0.448	-	-	-	-	0.0	-	0.0	-	-	-	-	-	-	-	-	-	-
stlc-111-2019-04-19_2	0.149	0.408	0.186	0.392	-	0.0	0.0	0.359	0.0	-	-	0.0	-	-	0.0	-	-	-	-	-	-	-	-	-	-
tressider-2019-03-16_2	0.112	0.424	0.538	0.035	-	0.0	0.0	0.125	-	-	-	0.0	0.0	-	-	-	-	-	0.0	-	-	-	0.0	-	-
tressider-2019-04-26_0	0.045	0.127	0.184	0.25	0.113	0.004	0.004	0.065	0.0	0.0	-	-	0.0	0.0	0.0	0.0	0.0	0.015	-	0.0	-	-	-	0.0	-
tressider-2019-04-26_1	0.071	0.228	0.455	0.167	0.0	0.0	0.0	0.076	0.0	0.0	-	-	-	0.0	-	-	-	0.002	-	-	0.0	-	-	0.0	-
tressider-2019-04-26_3	0.042	0.267	0.226	0.153	0.055	0.0	0.0	0.077	0.0	0.0	-	0.0	0.0	0.0	0.0	0.0	-	0.022	0.0	-	0.0	-	-	0.0	0.0
overall	0.054	0.32	0.336	0.252	0.069	0.003	0.003	0.15	0.0	0.0	0.149	0.0	0.0	0.0	0.0	0.0	0.0	0.006	0.0	0.0	0.0	0.0	0.0	0.0	0.0

I3D-SA-GAT

4.908%

Mahsa Ehsanpour, Alireza Abedin, Fatemeh Saleh, Javen Shi, Ian Reid, Hamid Rezatofighi Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos in ECCV2020

Details

Name	I3D-SA-GAT
Submission Date	2021-11-25 14:53:30+00:00
Abstract	We use the Inflated 3D ConvNet (I3D) as the backbone to capture the spatio-temporal context of an input video clip. we conduct ROIAlign to project the coordinates on the frame’s feature map and slice out the corresponding features for each individual’s bounding box. we adopt self-attention mechanism to directly learn the interactions between any two feature positions of an individual’s feature representation and accordingly leverage this information to refine each individual’s feature map. We then adopt GAT which consumes the individuals’ feature map obtained from the self-attention component, encodes inter-node relations, and generates an updated representation for each individual. Finally, action classification is performed on the obtained individuals' feature representation.
Publication title	Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos	Publication authors:	Mahsa Ehsanpour, Alireza Abedin, Fatemeh Saleh, Javen Shi, Ian Reid, Hamid Rezatofighi
Publication venue and year	ECCV2020	Publication URL:	https://arxiv.org/pdf/2007.02632.pdf
Code Language	N/A	Hardware	N/A
Code Website	N/A	Code URL	N/A

Visualisation

Name: huang-intersection-2019-01-22_0, Video Generating currently not available!

Best 3 By mAP ↓

Worst 3 By mAP ↓

Per-sequence Results

Sequence	mAP	Category_1	Category_2	Category_4	Category_5	Category_6	Category_8	Category_9	Category_10	Category_11	Category_12	Category_13	Category_14	Category_15	Category_16	Category_17	Category_18	Category_19	Category_20	Category_21	Category_22	Category_23	Category_24
cubberly-auditorium-2019-04-22_1	0.058	0.375	0.324	0.0	0.0	0.0	0.0	0.0	-	-	0.0	-	0.0	-	-	-	0.0	-	-	-	-	-	-
discovery-walk-2019-02-28_0	0.153	0.472	0.335	-	-	-	-	-	0.262	0.0	-	-	-	-	-	-	0.0	-	-	-	-	-	-
discovery-walk-2019-02-28_1	0.047	0.434	0.076	-	0.0	0.0	0.0	-	0.011	0.0	-	-	-	-	-	-	0.0	-	0.0	-	-	-	-
food-trucks-2019-02-12_0	0.09	0.345	0.502	0.111	0.0	0.0	0.0	-	0.167	-	0.048	-	-	0.0	-	0.0	-	-	-	-	-	-	0.0
gates-ai-lab-2019-04-17_0	0.043	0.169	0.371	0.016	0.0	0.0	0.0	0.0	-	-	0.0	-	-	-	-	0.0	-	-	-	-	-	0.0	0.0
gates-basement-elevators-2019-01-17_0	0.127	0.551	0.354	0.313	0.0	0.0	0.0	-	-	0.176	-	-	0.0	-	-	-	0.0	-	-	-	-	-	-
gates-foyer-2019-01-17_0	0.075	0.351	0.367	0.0	0.0	0.0	0.0	-	-	-	0.259	-	0.0	-	-	0.0	0.0	-	0.0	-	-	-	-
gates-to-clark-2019-02-28_0	0.106	0.412	0.017	-	0.0	0.0	-	-	0.417	-	-	-	-	-	-	-	0.0	-	-	-	-	-	-
hewlett-class-2019-01-23_0	0.085	0.28	0.564	-	0.0	0.0	0.0	-	-	-	0.093	-	0.0	-	-	0.0	-	-	-	-	-	0.0	-
hewlett-class-2019-01-23_1	0.093	0.52	0.313	0.0	0.0	0.0	0.0	-	-	-	-	-	0.0	-	-	-	-	-	-	-	-	-	-
huang-2-2019-01-25_1	0.092	0.296	0.214	0.407	0.0	0.0	0.0	0.0	-	-	0.0	-	-	-	-	-	-	-	-	-	-	-	-
huang-intersection-2019-01-22_0	0.173	0.405	0.477	-	-	-	-	-	0.195	-	-	-	-	-	-	-	-	-	-	0.136	-	0.0	-
indoor-coupa-cafe-2019-02-06_0	0.037	0.048	0.248	0.133	0.0	0.0	0.0	0.0	-	-	0.127	0.0	0.0	0.0	0.0	-	-	0.0	-	-	-	-	-
lomita-serra-intersection-2019-01-30_0	0.161	0.419	0.734	-	0.0	0.0	-	-	0.293	-	-	-	-	-	-	0.0	-	-	-	-	0.0	-	-
meyer-green-2019-03-16_1	0.036	0.206	0.05	0.051	0.0	0.0	0.0	-	0.061	-	0.033	-	-	-	-	0.0	-	-	-	-	-	-	-
nvidia-aud-2019-01-25_0	0.052	0.202	0.264	0.152	0.0	0.0	0.0	-	-	0.0	0.0	-	0.0	-	-	0.0	-	-	-	-	-	-	-
nvidia-aud-2019-04-18_1	0.085	0.312	0.643	0.242	0.0	0.0	0.0	-	-	0.0	0.0	0.0	0.0	-	0.0	-	0.0	-	-	-	-	-	-
nvidia-aud-2019-04-18_2	0.082	0.581	0.197	0.284	0.0	0.0	-	-	-	-	0.0	-	0.0	-	-	0.0	0.0	-	-	-	-	0.0	0.0
outdoor-coupa-cafe-2019-02-06_0	0.04	0.143	0.143	0.066	-	0.0	0.0	0.0	0.018	-	0.154	0.0	0.0	0.0	-	-	-	-	-	-	-	-	-
quarry-road-2019-02-28_0	0.044	0.352	-	-	0.0	0.0	0.0	-	-	-	-	-	-	-	-	0.0	-	-	0.0	-	-	-	-
serra-street-2019-01-30_0	0.062	0.173	0.004	-	-	-	0.0	-	0.258	-	-	-	-	-	-	-	-	-	-	-	-	-	0.0
stlc-111-2019-04-19_1	0.111	0.44	0.304	-	-	0.0	-	-	-	-	0.031	-	0.0	-	-	-	-	-	-	-	-	-	-
stlc-111-2019-04-19_2	0.092	0.487	0.339	-	0.0	0.0	0.0	-	-	0.0	-	-	0.0	-	-	-	-	-	-	-	-	-	-
tressider-2019-03-16_2	0.09	0.312	0.334	-	0.0	0.0	-	-	-	0.0	0.074	-	-	-	-	-	0.0	-	-	-	0.182	-	-
tressider-2019-04-26_0	0.024	0.044	0.209	0.16	0.0	0.0	0.0	0.0	-	-	0.002	0.0	0.0	0.0	0.0	0.0	-	0.0	-	-	-	0.0	-
tressider-2019-04-26_1	0.051	0.117	0.544	0.002	0.0	0.0	0.0	0.0	-	-	-	0.0	-	-	-	0.0	-	-	0.0	-	-	0.0	-
tressider-2019-04-26_3	0.03	0.197	0.234	0.077	0.0	0.0	0.0	0.0	-	0.0	0.072	0.0	0.0	0.0	-	0.0	0.0	-	0.0	-	-	0.0	0.0
overall	0.049	0.255	0.374	0.103	0.0	0.0	0.0	0.0	0.19	0.127	0.02	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.027	0.082	0.0	0.0

Symbol	Description
Individual Image	Method uses individual images from each camera
Stitched Image	Method uses stitched images combined from the individual cameras
Pointcloud	Method uses 3D pointcloud data
Online Tracking	Method does frame-by-frame processing with no lookahead
Offline Tracking	Method does not do in-order frame processing
Public Detections	Method uses publicly available detections
Private Detections	Method uses its own private detections

Leaderboard

Instructions

Individual Action Submissions

Additional Information Used

Evaluation Measures[1]

Reference