You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Train, test and compare multiple MSA models in a unified framework.
Supports 3 MSA datasets: MOSI, MOSEI, and CH-SIMS.
1. Get Started
1.1 Use Python API
Run pip install -r requirements.txt in your python virtual environment.
pip install -r requirements.txt
Open the run1 file and right-click to run it.
2. Experimental Parameters and Results
2.1 Partial Parameters
Hyper-parameter
CMU-MOSI
CMU-MOSEI
CH-SIMS
Batch Size
8
64
32
Learning Rate
1e-4
1e-4
1e-4
Epochs
50
50
50
2.2 Results
2.2.1 CMU-MOSI Results
Models
Acc-2 (↑)
F1-Score (↑)
Acc-5 (↑)
Acc-7 (↑)
MAE (↓)
Corr (↑)
Aligned/Unaligned
LF-DNN
77.55/78.86
77.47/78.85
38.00
34.21
0.975
0.651
Unaligned
TFN
78.77/80.13
78.72/80.16
41.30
36.39
0.899
0.681
Unaligned
LMF
77.60/78.66
77.61/78.73
37.85
34.06
0.949
0.665
Unaligned
MULT
78.81/80.49
78.64/80.39
42.76
36.59
0.909
0.691
Unaligned
MISA
81.97/83.69
81.92/83.70
48.79
43.25
0.770
0.778
Unaligned
SELF-MM
82.75/84.76
82.63/84.71
53.74
46.21
0.720
0.789
Unaligned
CENet
82.07/83.64
82.06/83.68
47.47
41.45
0.780
0.767
Unaligned
MMIM
82.41/84.25
82.31/84.21
51.90
45.09
0.734
0.776
Unaligned
TFR-NET
76.24/77.39
76.25/77.47
42.42
37.80
0.967
0.641
Unaligned
ALMT
82.61/84.55
82.51/84.52
51.60
46.21
0.735
0.783
Unaligned
DMAE (OURS)
85.59/87.50
85.63/87.55
58.08
48.03
0.682
0.820
Unaligned
EF-LSTM
71.02/62.85
58.99/48.51
41.36
41.36
0.840
0.669
Aligned
MFN
76.92/78.15
76.89/78.19
41.45
35.71
0.940
0.661
Aligned
GRAPH-MFN
76.04/77.49
75.88/77.42
35.96
32.17
0.998
0.649
Aligned
TETN
81.78/83.54
81.70/83.52
51.80
45.14
0.733
0.791
Aligned
MCTN
77.89/79.12
77.93/79.22
34.26
32.46
0.992
0.667
Aligned
MAG-BERT
81.83/83.74
81.76/83.74
52.24
45.82
0.731
0.781
Aligned
MMF
77.50/78.76
77.42/78.74
40.72
36.25
0.928
0.654
Aligned
DMAE (OURS)
84.72/87.96
84.64/87.98
56.77
46.29
0.697
0.815
Aligned
2.2.2 CMU-MOSEI Results
Models
Acc-2 (↑)
F1-Score (↑)
Acc-5 (↑)
Acc-7 (↑)
MAE (↓)
Corr (↑)
Aligned/Unaligned
LF-DNN
80.10/82.47
80.38/82.18
54.23
52.82
0.554
0.736
Unaligned
TFN
77.71/82.38
78.39/82.38
53.57
52.13
0.567
0.725
Unaligned
LMF
80.50/84.09
80.99/84.04
54.29
52.80
0.560
0.736
Unaligned
MULT
79.42/84.20
80.06/84.20
55.28
53.72
0.550
0.741
Unaligned
MISA
81.53/84.59
81.92/84.50
54.04
52.39
0.545
0.748
Unaligned
SELF-MM
78.06/83.27
78.84/83.37
55.32
53.90
0.531
0.764
Unaligned
CENet
82.36/85.54
82.67/85.40
55.83
54.15
0.532
0.769
Unaligned
MMIM
79.38/82.94
79.90/82.87
53.85
52.37
0.569
0.727
Unaligned
TFR-NET
-/-
-/-
-/-
-/-
-/-
-/-
Unaligned
ALMT
81.22/85.44
81.75/85.42
53.94
52.16
0.546
0.767
Unaligned
DMAE (OURS)
84.93/86.86
84.95/86.86
56.23
54.52
0.509
0.771
Unaligned
EF-LSTM
80.10/81.59
80.37/81.40
51.74
50.63
0.593
0.692
Aligned
MFN
79.62/83.52
80.26/83.56
52.95
51.62
0.571
0.722
Aligned
GRAPH-MFN
79.80/83.68
80.44/83.72
52.59
51.40
0.566
0.730
Aligned
TETN
81.58/85.04
82.02/84.98
55.99
54.14
0.541
0.761
Aligned
MCTN
-/-
-/-
-/-
-/-
-/-
-/-
Aligned
MAG-BERT
80.37/84.55
80.99/84.58
55.43
53.72
0.541
0.762
Aligned
MMF
80.52/83.33
80.96/83.26
51.59
50.70
0.579
0.725
Aligned
DMAE (OURS)
84.77/86.51
85.28/86.57
56.60
55.32
0.509
0.763
Aligned
2.2.3 CH-SIMS Results
Models
Acc-2 (↑)
F1-Score (↑)
Acc-3 (↑)
Acc-5 (↑)
MAE (↓)
Corr (↑)
Aligned/Unaligned
LF-DNN
76.15
76.21
65.43
40.63
0.444
0.559
Unaligned
TFN
78.34
78.30
66.01
39.75
0.431
0.579
Unaligned
LMF
76.66
76.62
65.21
38.37
0.441
0.582
Unaligned
MULT
76.81
76.84
65.87
38.80
0.444
0.573
Unaligned
MLF-DNN
76.07
76.54
66.30
43.18
0.414
0.613
Unaligned
SELF-MM
78.99
78.86
64.19
41.28
0.426
0.588
Unaligned
CENet
68.64
57.2
51.72
21.59
0.593
0.370
Unaligned
EF-LSTM
78.48
78.38
66.89
37.12
0.447
0.600
Unaligned
MFN
78.56
78.37
66.08
39.75
0.435
0.578
Unaligned
GRAPH-MFN
76.66
76.94
65.87
39.83
0.433
0.579
Unaligned
TETN
77.17
76.82
63.60
39.83
0.446
0.558
Unaligned
MTEN
76.00
76.32
64.04
42.67
0.432
0.582
Unaligned
MLMF
78.26
78.46
68.05
38.80
0.419
0.630
Unaligned
ALMT
78.12
78.32
64.04
36.40
0.437
0.571
Unaligned
MFM
73.45
74.01
59.59
32.38
0.484
0.504
Unaligned
MISA
77.61
77.45
64.33
40.41
0.440
0.568
Unaligned
MAG-BERT
72.14
63.52
56.97
27.57
0.539
0.271
Unaligned
MMIM
74.98
75.15
61.42
38.07
0.476
0.497
Unaligned
TFR-NET
69.37
56.82
51.64
21.23
0.584
0.094
Unaligned
DMAE (OURS)
79.87
80.68
66.08
41.14
0.423
0.589
Unaligned
3. Citation
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotations of Modality
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
M-SENA: An Integrated Platform for Multimodal Sentiment Analysis
Please cite our paper if you find our work useful for your research:
@inproceedings{yu2020ch,
title={CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality},
author={Yu, Wenmeng and Xu, Hua and Meng, Fanyang and Zhu, Yilin and Ma, Yixiao and Wu, Jiele and Zou, Jiyun and Yang, Kaicheng},
booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
pages={3718--3727},
year={2020}
}
@inproceedings{yu2021learning,
title={Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis},
author={Yu, Wenmeng and Xu, Hua and Yuan, Ziqi and Wu, Jiele},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={35},
number={12},
pages={10790--10797},
year={2021}
}