We-Math

Does Your Large Multimodal Model
Achieve Human-like Mathematical Reasoning?

¹Beijing University of Posts and Telecommunications, ²Wechat, Tencent Inc., ³Huazhong University of Science and Technology, ⁴Beijing Institute of Technology

*Equal contribution
†Corresponding author

Introduction

Inspired by human-like mathematical reasoning, we introduce We-Math, the first benchmark specifically designed to explore the problem-solving principles beyond the end-to-end performance.

We meticulously collect and categorize 6.5K visual math problems, spanning 67 hierarchical knowledge concepts and 5 layers of knowledge granularity. We firstly decompose composite problems into sub-problems according to the required knowledge concepts and introduce a novel four-dimensional metric, namely Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM) to hierarchically assess inherent issues in LMMs' reasoning process.

With We-Math, we conduct a thorough evaluation of existing LMMs in visual mathematical reasoning and reveal a negative correlation between solving step and problem-specific performance. We confirm the IK issue of LMMs can be effectively improved via knowledge augmentation strategy. More notably, the primary challenge of GPT-4o has significantly transitioned from IK to IG, establishing it as the first LMM advancing towards the knowledge generalization stage. In contrast, other LMMs exhibit a marked inclination towards Rote Memorization they correctly solve composite problems involving multiple knowledge concepts, yet fail in answering sub-problems.

We anticipate that We-Math will open new pathways for advancements in visual mathematical reasoning for LMMs.

Overview of LMMs' performances on We-Math. Figures from left to right illustrates the (1) accuracy of different LMMs on various problem-solving steps, (2) the performance in different visual mathematics categories and (3) the result in knowledge based reasoning evaluation.

Overview

Different from existing benchmarks, We-Math is constructed around textbook knowledge units, decomposing composite problem solutions into sub-problems based on the knowledge concepts.
(1) Hierarchical Knowledge Structure. Strictly adheres to the knowledge presented in mathematics textbooks, featuring a rigorous hierarchical and multi-category architecture. It ensures the independence of knowledge concepts within the same level, while establishing logical relationships among concepts at different hierarchical levels.
(2) Knowledge based Reasoning Evaluation. To explore how LMMs solve problems. Drawing upon that humans tackle problems incrementally by leveraging fundamental knowledge concepts, we break down complex mathematical problems into more manageable sub105 problems. Furthermore, we employ diverse measurement dimensions for meticulous evaluations.
(3) Knowledge Concept Augmentation. To alleviate the inherent issues during the problem-solving process, we heuristically introduce descriptions for 67 knowledge concepts from Wikipedia and textbooks, thereby providing essential knowledge support for the reasoning processes of LMMs.

The pipeline of knowledge-based data decomposition (an example of a two-step problem in We-Math).

The pipeline of knowledge-based data decomposition (an example of a three-step problem in We-Math).

Metric for Reasoning Evaluation

Based on the decomposed multi-step problems, we further reveal the inherent issues of LMMs in problem-solving process. We feed both the M one-step sub-problems and the original problem into LMMs, and classifying the responses into four categories
1. Insufficient Knowledge (IK): Part of one-step problems contain errors, and the multi-step problem is wrong. It is reasonable because model's insufficient grasp of single knowledge concept may lead to errors in multi-step problem.
2. Inadequate Generalization (IG): One-Step problems are all correct, but the multi-step problem is incorrect. This is also considered reasonable. While LMMs are capable of understanding individual knowledge concepts, they may struggle to generalize that knowledge to solve composite problems.
3. Complete Mastery (CM): One-Step problems are all correct, and multi-step problem is also answered correctly. This result demonstrates that the model's results are both reliable and accurate.
4. Rote Memorization (RM): One-Step problems contain errors, but the multi-step problem is answered correctly, which contradicts human logical thinking. If a model can solve composite multi-step problems but fails to answer the one-step problems needed in the process, it raises doubts about the model's reliability.

An example of the four-dimensional metrics for evaluating a two-step problem, using both loose and strict settings.

Diagram illustrating strict metric in three-step problem.

Diagram illustrating loose metric in three-step problem.

Leaderboard on We-Math (testmini)

Accuracy scores on the testmini subset (1,740 examples) of We-Math.

#	Model	Source	Date	Avg(Strict)	IK(Strict)	IG(Strict)	CM(Strict)	RM(Strict)	Avg(Loose)	IK(Loose)	IG(Loose)	CM(Loose)	RM(Loose)
1	GPT-4o 🥇	Link	2024-05	42.86%	31.24% (164)	15.24% (80)	35.24% (185)	34.16% (96)	60.57%	31.24% (164)	15.24% (80)	52.95% (278)	1.07% (3)
2	GPT-4V 🥈	Link	2024-04	31.05%	39.81% (209)	14.48% (76)	23.81% (125)	47.92% (115)	51.43%	39.81% (209)	14.48% (76)	44.19% (232)	3.33% (8)
3	Gemini-1.5-pro 🥉	Link	2024-05	26.38%	42.86% (225)	11.24% (59)	20.76% (109)	54.77% (132)	46.00%	42.86% (225)	11.24% (59)	40.38% (212)	12.03% (29)
4	LLaVA-NeXT-110B	Link	2024-05	19.24%	50.29% (264)	14.48% (76)	12.00% (63)	65.95% (122)	37.90%	50.29% (264)	14.48% (76)	30.67% (161)	12.97% (24)
5	InternVL-Chat-V1.5	Link	2024-04	14.95%	56.19% (295)	13.90% (73)	8.00% (42)	73.25% (115)	32.67%	56.19% (295)	13.90% (73)	25.71% (135)	14.01% (22)
6	GLM-4V-9B	Link	2024-06	14.86%	52.95% (278)	9.52% (50)	10.10% (53)	73.10% (144)	35.05%	52.95% (278)	9.52% (50)	30.29% (159)	19.29% (38)
7	LLaVA-NeXT-72B	Link	2024-05	13.43%	58.86% (309)	7.05% (37)	9.90% (52)	70.95% (127)	31.52%	58.86% (309)	7.05% (37)	28.00% (147)	17.88% (32)
8	InternLM-XComposer2-VL-7B	Link	2024-04	12.67%	56.38% (296)	10.48% (55)	7.43% (39)	77.59% (135)	30.95%	56.38% (296)	10.48% (55)	25.71% (135)	22.41% (39)
9	LongVA	Link	2024-06	11.52%	61.14% (321)	8.95% (47)	7.05% (37)	76.43% (120)	27.71%	61.14% (321)	8.95% (47)	23.24% (122)	22.29% (35)
10	Phi3-Vision-4.2B	Link	2024-05	10.57%	58.86% (309)	8.95% (47)	6.10% (32)	81.07% (137)	29.81%	58.86% (309)	8.95% (47)	25.33% (133)	21.30% (36)
11	Qwen-VL-Max	Link	2024-01	10.48%	65.14% (342)	7.62% (40)	6.67% (35)	75.52% (108)	25.52%	65.14% (342)	7.62% (40)	21.71% (114)	20.28% (29)
12	MiniCPM-LLaMA3-V 2.5	Link	2024-05	9.52%	60.19% (316)	9.14% (48)	4.95% (26)	83.85% (135)	28.00%	60.19% (316)	9.14% (48)	23.43% (123)	23.60% (38)
13	G-LLaVA-13B	Link	2024-03	6.48%	64.19% (337)	4.57% (24)	4.19% (22)	86.59% (142)	22.29%	64.19% (337)	4.57% (24)	20.00% (105)	35.98% (59)
14	DeepSeek-VL-7B	Link	2024-03	6.29%	69.14% (363)	4.57% (24)	4.00% (21)	84.78% (117)	20.95%	69.14% (363)	4.57% (24)	18.67% (98)	28.99% (40)
15	DeepSeek-VL-1.3B	Link	2024-03	5.90%	71.05% (373)	2.67% (14)	4.57% (24)	82.61% (114)	21.52%	71.05% (373)	2.67% (14)	20.19% (106)	23.19% (32)
16	LLaVA-1.6-13B	Link	2024-03	5.24%	69.14% (363)	3.24% (17)	3.62% (19)	86.90% (126)	22.00%	69.14% (363)	3.24% (17)	20.38% (107)	26.21% (38)
17	LLaVA-1.6-7B	Link	2024-03	3.33%	78.29% (411)	2.48% (13)	2.10% (11)	89.11% (90)	13.81%	78.29% (411)	2.48% (13)	12.57% (66)	34.65% (35)

#	Model	Source	Date	S1	S2	S3	UCU(Mem)	AL(Mem)	CPF(PF)	UPF(PF)	CSF(SF)	USF(SF)	BTF(TMF)	CCF(TMF)	Dir(PD)	Pos(PD)	RoM(PD)	CCP(PD)
1	GPT-4o 🥇	Link	2024-05	73.33%	57.22%	46.06%	87.10%	45.79%	76.74%	70.99%	82.21%	65.66%	58.10%	70.00%	93.10%	80.43%	58.79%	70.00%
2	GPT-4V 🥈	Link	2024-04	65.51%	49.17%	38.18%	82.54%	38.42%	70.67%	60.22%	76.58%	56.32%	57.76%	63.33%	79.29%	57.48%	47.80%	63.33%
3	Gemini-1.5-pro 🥉	Link	2024-05	56.13%	51.39%	33.94%	50.99%	31.23%	61.75%	45.03%	69.95%	57.54%	39.24%	60.00%	68.81%	54.13%	40.66%	60.00%
4	LLaVA-NeXT-110B	Link	2024-05	53.74%	36.94%	31.52%	39.48%	57.72%	59.48%	53.06%	52.25%	50.22%	54.09%	40.00%	54.76%	55.86%	40.11%	40.00%
5	InternVL-Chat-V1.5	Link	2024-04	49.38%	30.56%	28.48%	43.95%	29.82%	52.23%	52.06%	44.19%	48.15%	47.05%	36.67%	65.71%	50.47%	36.54%	36.67%
6	GLM-4V-9B	Link	2024-06	47.33%	37.22%	38.18%	53.37%	37.02%	51.32%	46.52%	50.60%	38.22%	44.09%	53.33%	40.95%	49.27%	36.81%	53.33%
7	LLaVA-NeXT-72B	Link	2024-05	42.88%	35.56%	30.91%	31.65%	25.26%	43.25%	42.39%	46.14%	41.76%	44.22%	36.67%	44.29%	38.93%	32.97%	36.67%
8	InternLM-XComposer2-VL-7B	Link	2024-04	47.00%	33.06%	33.33%	31.25%	46.49%	47.70%	42.57%	51.44%	43.87%	41.13%	40.00%	65.48%	53.87%	55.22%	40.00%
9	LongVA	Link	2024-06	43.54%	30.56%	28.48%	24.50%	39.82%	45.09%	40.75%	51.85%	42.49%	45.60%	20.00%	44.52%	40.74%	47.53%	20.00%
10	Phi3-Vision-4.2B	Link	2024-05	42.14%	34.17%	27.88%	28.67%	15.96%	47.23%	38.83%	49.99%	44.41%	28.76%	50.00%	48.57%	49.19%	26.37%	50.00%
11	Qwen-VL-Max	Link	2024-01	40.82%	30.28%	20.61%	19.35%	25.26%	39.82%	41.44%	43.64%	48.02%	43.82%	26.67%	41.43%	35.09%	40.66%	26.67%
12	MiniCPM-LLaMA3-V 2.5	Link	2024-05	39.75%	31.11%	29.70%	28.57%	37.02%	40.81%	39.82%	40.97%	38.61%	31.96%	43.33%	40.95%	42.70%	43.96%	43.33%
13	G-LLaVA-13B	Link	2024-03	32.43%	30.56%	32.73%	33.33%	29.12%	32.04%	37.88%	19.57%	33.51%	37.12%	40.00%	31.19%	33.21%	25.55%	40.00%
14	DeepSeek-VL-7B	Link	2024-03	32.59%	26.67%	25.45%	16.57%	35.09%	27.27%	38.01%	24.18%	38.65%	50.02%	23.33%	24.52%	41.01%	51.65%	23.33%
15	DeepSeek-VL-1.3B	Link	2024-03	31.44%	27.78%	23.03%	27.78%	23.86%	22.76%	36.92%	30.36%	34.18%	44.46%	33.33%	48.10%	41.77%	37.09%	33.33%
16	LLaVA-1.6-13B	Link	2024-03	29.38%	25.28%	32.73%	21.73%	23.16%	23.37%	34.72%	25.26%	26.36%	37.52%	30.00%	26.90%	28.87%	37.09%	30.00%
17	LLaVA-1.6-7B	Link	2024-03	22.96%	20.83%	15.76%	18.45%	20.53%	16.92%	29.63%	15.57%	18.60%	42.67%	26.67%	17.62%	43.31%	28.85%	26.67%

🚨 To submit your results to the leaderboard, please send to this email with your result json files.

Results on Existing Foundation Models

The visualization of different LMMs' performances on each category.

The performance of different LMMs on four-dimensional metrics under strict metric.

The performance of different LMMs on four-dimensional metrics under loose metric.

The Leaderboard of different LMMs under the strict and loose metric (average score %). ~ represents an approximate estimate of the total parameters nums in LMMs.

Knowledge Card

The description of the knowledge concept "Angles and Length".

The description of the knowledge concept "Correspondence of Coordinates and Positions".

The description of the knowledge concept "Basic Transformations of Figures".

The description of the knowledge concept "Cutting and Combining of Figures".

The description of the knowledge concepts "Direction" and "Position".

The description of the knowledge concept "Route Map".

The description of the knowledge concept "Understanding and Conversion of Units".

The description of the knowledge concept "Calculation of Solid Figures".

The description of the knowledge concept "Calculation of Plane Figures".

The description of the knowledge concept "Understanding of Plane Figures".

The description of the knowledge concept "Understanding of Solid Figures".

BibTeX

@misc{qiao2024wemathdoeslargemultimodal, title={We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?}, author={Runqi Qiao and Qiuna Tan and Guanting Dong and Minhui Wu and Chong Sun and Xiaoshuai Song and Zhuoma GongQue and Shanglin Lei and Zhe Wei and Miaoxuan Zhang and Runfeng Qiao and Yifan Zhang and Xiao Zong and Yida Xu and Muxi Diao and Zhimin Bao and Chen Li and Honggang Zhang}, year={2024}, eprint={2407.01284}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2407.01284}, }