From de9217976c1e69f3353b68a06105afadd76aeb33 Mon Sep 17 00:00:00 2001 From: troyyyyy Date: Wed, 27 Mar 2024 21:57:33 +0800 Subject: [PATCH] [MNT] add memory usage comparison in HWF --- docs/Examples/HWF.rst | 41 ++++++++++++++++++++++++++++++++++++++++- examples/hwf/README.md | 30 +++++++++++++++++++++++++++++- examples/hwf/hwf.ipynb | 19 ++++++------------- 3 files changed, 75 insertions(+), 15 deletions(-) diff --git a/docs/Examples/HWF.rst b/docs/Examples/HWF.rst index cb0cb3e..9ab0c54 100644 --- a/docs/Examples/HWF.rst +++ b/docs/Examples/HWF.rst @@ -422,10 +422,44 @@ Log: abl - INFO - Test start: abl - INFO - Evaluation ended, hwf/character_accuracy: 0.997 hwf/reasoning_accuracy: 0.986 +Environment +----------- + +For all experiments, we used a single linux server. Details on the specifications are listed in the table below. + +.. raw:: html + + + + + + + + + + + + + + + + + + + + +
CPUGPUMemoryOS
2 * Xeon Platinum 8358, 32 Cores, 2.6 GHz Base FrequencyA100 80GB512GBUbuntu 20.04
+ Performance ----------- -We present the results of ABL as follows, which include the reasoning accuracy (for different equation lengths in the HWF dataset), and the training time (to achieve the accuracy using all equation lengths). These results are compared with the following methods: +We present the results of ABL as follows, which include the reasoning accuracy (for different equation lengths in the HWF dataset), training time (to achieve the accuracy using all equation lengths), and average memory usage (using all equation lengths). These results are compared with the following methods: - `NGS `_: A neural-symbolic framework that uses a grammar model and a back-search algorithm to improve its computing process; @@ -448,6 +482,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( Reasoning Accuracy
(for different equation lengths) Training Time (s)
(to achieve the Acc. using all lengths) + Average Memory Usage (MB)
(using all lengths) 1 @@ -466,6 +501,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( 5.2 98.4 426.2 + 3705 DeepProbLog @@ -475,6 +511,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( timeout timeout timeout + 4315 DeepStochLog @@ -484,6 +521,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( timeout timeout timeout + 4355 ABL @@ -493,6 +531,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( 97.2 98.6 77.3 + 3074 diff --git a/examples/hwf/README.md b/examples/hwf/README.md index 7412ce0..fc15d1a 100644 --- a/examples/hwf/README.md +++ b/examples/hwf/README.md @@ -46,9 +46,32 @@ optional arguments: ``` +## Environment + +For all experiments, we used a single linux server. Details on the specifications are listed in the table below. + + + + + + + + + + + + + + + + + + +
CPUGPUMemoryOS
2 * Xeon Platinum 8358, 32 Cores, 2.6 GHz Base FrequencyA100 80GB512GBUbuntu 20.04
+ ## Performance -We present the results of ABL as follows, which include the reasoning accuracy (for different equation lengths in the HWF dataset), and the training time (to achieve the accuracy using all equation lengths). These results are compared with the following methods: +We present the results of ABL as follows, which include the reasoning accuracy (for different equation lengths in the HWF dataset), training time (to achieve the accuracy using all equation lengths), and average memory usage (using all equation lengths). These results are compared with the following methods: - [**NGS**](https://github.com/liqing-ustc/NGS): A neural-symbolic framework that uses a grammar model and a back-search algorithm to improve its computing process; - [**DeepProbLog**](https://github.com/ML-KULeuven/deepproblog/tree/master): An extension of ProbLog by introducing neural predicates in Probabilistic Logic Programming; @@ -60,6 +83,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( Reasoning Accuracy
(for different equation lengths) Training Time (s)
(to achieve the Acc. using all lengths) + Average Memory Usage (MB)
(using all lengths) 1 @@ -78,6 +102,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( 5.2 98.4 426.2 + 3705 DeepProbLog @@ -87,6 +112,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( timeout timeout timeout + 4315 DeepStochLog @@ -96,6 +122,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( timeout timeout timeout + 4355 ABL @@ -105,6 +132,7 @@ We present the results of ABL as follows, which include the reasoning accuracy ( 97.2 99.2 77.3 + 3074 diff --git a/examples/hwf/hwf.ipynb b/examples/hwf/hwf.ipynb index f67cc9d..be1a280 100644 --- a/examples/hwf/hwf.ipynb +++ b/examples/hwf/hwf.ipynb @@ -434,7 +434,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We present the results of ABL as follows, which include the reasoning accuracy (for different equation lengths in the HWF dataset), and the training time (to achieve the accuracy using all equation lengths). These results are compared with the following methods:\n", + "We present the results of ABL as follows, which include the reasoning accuracy (for different equation lengths in the HWF dataset), training time (to achieve the accuracy using all equation lengths), and average memory usage (using all equation lengths). These results are compared with the following methods:\n", "\n", "- [**NGS**](https://github.com/liqing-ustc/NGS): A neural-symbolic framework that uses a grammar model and a back-search algorithm to improve its computing process;\n", "\n", @@ -447,19 +447,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", "\n", "\n", " \n", " \n", " \n", " \n", + " \n", " \n", " \n", " \n", @@ -478,6 +472,7 @@ " \n", " \n", " \n", + " \n", " \n", " \n", " \n", @@ -487,6 +482,7 @@ " \n", " \n", " \n", + " \n", " \n", " \n", " \n", @@ -496,6 +492,7 @@ " \n", " \n", " \n", + " \n", " \n", " \n", " \n", @@ -505,16 +502,12 @@ " \n", " \n", " \n", + " \n", " \n", "\n", "
Reasoning Accuracy
(for different equation lengths)		Training Time (s)
(to achieve the Acc. using all lengths)		Average Memory Usage (MB)
(using all lengths)
15.298.4426.23705
DeepProbLogtimeouttimeouttimeout4315
DeepStochLogtimeouttimeouttimeout4355
ABL97.299.277.33074
\n", "

* timeout: need more than 1 hour to execute

" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] } ], "metadata": {