Repository navigation
High memory usage #832
Description
Activity
Could you post specific memory consumption and approx time time to when analysis completes. Generally you don't have to wait until it completes as it is a background task that collects information in stages. 4GB consumption may happen, eventually memory is released.
#450 was about 30GB+ or runaway consuming all memory, so it is not a duplicate. Number of tasks is limited so process should not be consuming all CPU, limit is ~40% or so.
If we can clone projects for investigation it would be helpful too.
Thanks.
It was strange enough for me.
-
I work normally with WinPython. As I wrote above the analysis task do not stop (also over night). At the end VSCode just crashed. All the time the GUI was not usable at all.
-
Today I installed Anaconda3, and then tried again, and VSCode works with Python as usual, just these messages in output Python pane still here:
Traceback (most recent call last): File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 1489, in <module> state.initial_import(sys.argv[2]) File "C:\Users\zkr\.vscode\extensions\ms-python.python-2019.3.6139\languageServer.0.2.31\scrape_module.py", line 872, in initial_import mod = __import__(self.module_name) File "c:\3rd\anaconda3\lib\site-packages\scipy\__init__.py", line 62, in <module> from numpy import show_config as show_numpy_config File "c:\3rd\anaconda3\lib\site-packages\numpy\__init__.py", line 140, in <module> from . import _distributor_init File "c:\3rd\anaconda3\lib\site-packages\numpy\_distributor_init.py", line 34, in <module> from . import _mklinit ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.Looks like the server can not import some DLL related to numpy now.
When you look at my report above (with WinPython), there was some DLL related to cvxpy.Could it be some ABI issue ?
Because I noted yesterday, that I updated my Visual Studio Build Tools 2017 and decided reinstall cvxpy package (actually rebuilding related DLLs)pip install -U cvxpyAnd after restarting VSCode (yesterday with WinPython) there was not messages with cvxpy, but many related to another packages like "DLL load failed"
-
Some further information:
I did have consumption at 22GB+, with CPU usage at around 15-25%.
It happens when I open any python file. This is after 30 seconds after opening a simple python file, which just imports pandas.
It was analyzing a few files indicated at the bottom of the toolbar:
After a few minutes it has stabilized to this:
I believe the issue is when I have virtual environments folder within the workspace, it causes a huge spike in RAM usage by analyzing over 7000+ files.
Version: 1.32.3 (user setup)
Commit: a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4
Date: 2019-03-14T23:43:35.476Z
Electron: 3.1.6
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 10.0.17134After analysis, I think memory consumption
is caused when the python plugin creats the python library index by reading the information into memory in real time. (The memory consumed by the Python library index is for the IntelliSense, I think)Further,
memory consumption and approx time time to when analysis completesdepends on the size of the extra library installed for Python. (In other words, the bigger the Python library, the greater the memory occupies, and the more time required)
For anaconda installed by default, memory consumption is about 6GB and approx time time to when analysis completes is about 15 mins.I can debug existing code before it's done, but I can't use IntelliSense when writing code.
Thanks！
RAM consumption for me hit 7.7GB before running out of memory.
Amount of modules remaining to analyze continuously rises as wellalgo99 The DLL importing thing was technically a problem in the old version, but just wasn't printed. The currently released version does print it, but we merged a PR which when released will re-hide it (see #823).
@suiahaw The language server doesn't keep an on-disk version of the analysis yet (and never has, apart from scraped information from compiled libraries that cannot be directly parsed and analyzed). That is #472.
Sivaharan Rajkumar (@dilzeem),
import pandasmeans the LS will analyze pandas and its transitive dependencies (numpy etc), so there is a lot going on at first. Taking a couple of minutes for that is not unexpected, but after that you should have many completions available (and many which we simply did not have at all in the previous language server). Just want to set expectations around expected behavior versus actual bugs. We want to track down bugs like memory leaks and analysis not completing, and we will make improvements in the future around caching and type stubs to speed up analysis of common packages, but its also the case that having some initial cost to analysis is to be expected when importing complex packages.algo99 The DLL importing thing was technically a problem in the old version, but just wasn't printed. The currently released version does print it, but we merged a PR which when released will re-hide it (see #823).
@suiahaw The language server doesn't keep an on-disk version of the analysis yet (and never has, apart from scraped information from compiled libraries that cannot be directly parsed and analyzed). That is #472.
If no
an on-disk version of the analysisexists, this means that whenever I write a Python program using vscode, I would have to suffer extra memory and time overhead. This will make me unable to use vscode as my main development tool.an on-disk version of the analysisis necessary. On the other hand, the extra disk overhead brought by the analysis can be solved by database compression and periodic reminding users to clean up.Sivaharan Rajkumar (@dilzeem) - 6GB peak is not unusual at the moment considering that used to be 30GB+ in #450, 20 appears excessive though.
Mikhail Arkhipov (@MikhailArkhipov), Graham Wheeler (@gramster) thanks for the quick response.
Okay, it was surprising as I didn't have any issues like this until the March Python Extension update. So it was unexpected behavior.
Okay I will see if I can recreate the 25GB+ scenario tomorrow, and give you an update. The project I was using had a lot of modules, mainly scikit-learn, pandas, dash and gensim.
The computer actually crashes when I opened two similar projects in different windows.
- changed the title
[-]Huge CPU and Memory consumption caused by design problems[/-][+]High memory usage[/+]on Mar 27, 2019 I'm not sure if this is an old bug, since I'm also seeing this behavior only after the most recent update.
Reacted by Ciprian Tomoiaga, 578123043 and JoelinoI started facing this issue in the last week. My laptop has 8GB RAM and the OS is Ubuntu 16.04. After opening a python file, the RAM usage shoots up until it is completely eaten. The only way I am able to use VSCode is by disabling the extension.
I tried disablingpython.jediEnabledand tried removing it from.vscodefolder and reinstalling it. However, the issue still persists.Just an update:
Just importing 4 libraries. Initially got this with analyzing in background at the bottom which is frozen with 9709 items left:
After it has been analyzing for approx 30 mins it is still 20+ GB Ram Usage. Also the number of items left has increased to 10479.
After 40 mins of analyzing VScode crashed.
Just an update:
Just importing 4 libraries
Analyzing in background at the bottom is frozen ad 9709 items left.
It has been analyzing for approx 30 mins and still 20+ GB Ram Usage. Also the number of items left has increased to 10479.I will keep it running, and will see it if reduces, and at approx how long it took.
After experimenting, I think
memory consumption and approx time time to when analysis completesdepends on the size of the extra library installed for Python , instead of that how many libraries be imported.See details： #832 (comment)
83 remaining items
That is not normal, that is a bug. We have been working for many weeks on performance improvements, including memory usage. At the latest revision, analyzing tensorflow on my machine only uses 400 MB, as opposed to the multiple gigabytes it used to take. If you have a project which reproduces it, we'd love to test it and figure out why that would occur.
Also, try switching up to the beta release, which has many more improvements not yet in stable.
"python.analysis.downloadChannel": "beta"
Where am I supposed to put the setting? It isn't recognized in ctrl+,
In your user
settings.json, which you can find by opening your command palette and searching for "JSON".#1133 and #1134 should have improved the case. #1134 is a significant change so version was bumped to 0.3. We just published it to daily and we will let it bake for 2-3 days. If nothing goes wrong, then we'll push it to beta and then to stable.
I am going to close this issue. Please open separate cases for specific sets of libraries or repro steps, like #1157.
Thanks!
I'm having again high memory usage problems on 0.3.20.
These seems to be a leak somewhere: it finishes analyzing fine using about 1gb.
Then over time it will increase to 10gb+.Should I open a new issue?
Yes, please.
Jake Bailey (@jakebailey) I've opened issue #1298.
Running my python code in the terminal and editing it in vscode. After aborting the program and even closing the corresponding vscode window, `Microsoft.Python.LanguageServer' fluctuates between 500 MB to 1.2 GB on my memory. I'm using version 0.3.39 on mac btw.
Masoud Hoveidar (@hoveidar) - this consumption is not unusual if you use large libraries, like, say, use tensoflow. Also, this number is not necessary memory that is held. Managed code like C# or Java releases memory when there is memory pressure in the OS, not immediately, like C++. Thus, if machine has 8GB RAM with 2GB free it may show smaller consumption number than machine with 64GB with 32GB free since runtime simply does not see reasons to release the memory pool back to OS.
This thread is closed so please report specific problems by opening new issues. Each case is different and may need separate investigation by different developers.
Reacted by Masoud HoveidarNot sure why you guys defending this so hard. Its obviously broken!
Frequently it gets stuck scanning for dependencies. I have to restart my cheap laptop after max 30 minutes as it almost freezes.
I thought this is related to the fact that this an old cheap machine. However, now I am at work with a newer PC. I just had to set myself to technical issue and restart the machine as it was about to freeze completely. At least the session was longer the 30 minutes before it happened.
Still, its undeniable that this is related to the python language server.
Reacted by Christian Hess, miffyrcee, Ben Ruijl, Cheng Chen, Hayk Martiros, contang0 and take5vThis isn't a defense or a statement that no issues exist; we're asking that this 4 month old closed issue about a different bug (but with potentially similar symptoms) not be used as the place to talk about a new issue. The analysis not completing may or may not have something to do with memory usage, but in any case is not going to have the same fix as what has been made here.
Note that #1298 is ongoing about a recent memory issue, with some progress made, but the discussion of that issue should happen there, not here. If you can reproduce your issue of the analysis never completing, please do open another issue so we can look into it. Thanks.
Reacted by Sumeet Ranka and drrmmngReacted by Xianglong Wang, Federico Caselli, Tony Benoy and AndersI love using VSCode with Python extension but I am having lately the same issues presented here. VSCode with Python extension cosumes massive amount of memory rendering the system unusable! It is consuming 5GB of RAM! The consumption is in some cases even higher than that. I am working on a laptop with only 8GB of RAM. I am using Windows 10, latest version of Python 3.8, VSCode is updated to latest version (March 2020). I mean this is ridiculous! Any solution yet!
Couple things. First, I'd suggest opening separate issue rather that posting in a closed thread. Each issue is different, there is not much of a common ground - i.e. there is no 'bug' that explains memory footprint in all cased.
Second, it depends what libraries do you use. Some libraries are very large, literally thousands of modules. Multiple data science libraries yield tens of thousands of files. As with other typeless languages, figuring out what particular function might be returning in Python, or what type some variable might have becomes quite expensive. Without type annotations or stubs it literally means walking though every code line in a library looking for what particular code branch might be returning or what members it might be adding to a class.
- locked as resolved and limited conversation to collaborators
on Mar 14, 2020
@suiahaw commented on Tue Mar 26 2019
Issue Type: Bug
Some questions about the python plugin encountered when building python library indexes
I strongly hope that the python plugin does not
read the information into memory in real timewhen creating the python library index, butinstead saves the index filein order to speed up the time and reduce memory overhead.The python library is really too big. Sometimes I have to wait a few minutes for writing a small amount of code.
Extension version: 2019.3.6139
VS Code version: Code 1.32.3 (a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4, 2019-03-14T23:43:35.476Z)
OS version: Windows_NT x64 10.0.17763
System Info
checker_imaging: disabled_off
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
native_gpu_memory_buffers: disabled_software
rasterization: enabled
surface_synchronization: enabled_on
video_decode: enabled
webgl: enabled
webgl2: enabled
algo99 commented on Wed Mar 27 2019
I can confirm, on Windows 10.
After update (yesterday) of Python Language Server and/or vscode_python extension VSCode not more usable. See the disk I/O and memory consumption:
The extension proceeds after VSCode start:
and this process does not stop.
I tried several times to restart VSCode, but it does not help.
algo99 commented on Wed Mar 27 2019
Just noted in Python output pane:
Sivaharan Rajkumar (@dilzeem) commented on Wed Mar 27 2019
I am also getting the same issue as well on Windows 10.
It only happens once I start debugging some code.
It renders VScode unusable.
@suiahaw commented on Wed Mar 27 2019
It only happens when the python plugin loads the python library, which requires a lot of CPU and memory. And these information that is loaded into memory is estimated to be used for IntelliCode.
Jinbiao Yang (@ray306) commented on Wed Mar 27 2019
The same problem happens today. I think it's a bug of new Python plugin just released yesterday (March 26).
Sivaharan Rajkumar (@dilzeem) commented on Wed Mar 27 2019
Possibly a duplicate of of this in the python language server:
#450
@suiahaw commented on Wed Mar 27 2019
I also think that the issues area of the python plugin has a large part of similar content related to such a bug. It seems that this bug is very disturbing programming and needs to be solved urgently. ;-)