Repository navigation
Failure analyzing decorator module, high CPU/RAM usage #1401
Description
Activity
Could you provide
- Amount of RAM used (few hundred meg is not unusual)
- List of packages you use beside system (i.e. tensorflow, numpy, plotly?)
- How long CPU spike may last (LS is heavily multi-threaded so using multiple cores is normal unless it lasts for a long time).
I can't easily reproduce, but an hour ago I edited some code and it seems to re-analyze on every code change, then the entire cpu was unresponsive and I had to power down, so it's more than a few hundred megs.
Now by just opening the project and waiting for awhile, it reaches 6GB ram consumption.
I think spacy and selenium are the heaviest dependencies I have (no tensorflow).
Another thing that may be of interest is that I have two projects opened sometimes, and one depends on the other (
pip install -e).I think the cpu is less of a worry than ram is (makes the mouse unusable).
If they depend on each other and you open them separately, then you'll have two language servers running watching both such that changing one causes the other to be reanalyzed. If that's happening often, then this may be the module reloading memory usage issue I'm currently working on.
Reacted by Isaac VirshupNow for example it's analyzing >5 minutes, and the computer intermittently freezes.
Jake Bailey (@jakebailey) I think this is related, as it's aggravated when I'm working on the two projects at the same time, but it's not explaining the very long analysis even when there's no code change.
I'm also using ipython with autoreload in case it matters.
Can you retest things with the latest version? 0.3.46 contains a memory fix to module reloading (and others), which appears to have restored things to as they were a few weeks ago.
There's definitely an improvement, but still not as it was >2 month ago.
Still any code change in the dependency project causes a spike in RAM (I measured 2GB now), and high cpu.
This is tolerable, but I did manage to reach a state where again analysis never ends, but was not able to repro/measure ram, so will update once it happens again.
Initial analysis ends pretty fast.
Then, sadly after a few edits in the dependency project, analysis starts again and is stuck on the same number (e.g. 1000 items) infinitely. Ram is rising to 5.5g (+ 11g in virtual memory) and the computer becomes unresponsive.
I'd appreciate it if you enable trace logging and upload a copy of the language server logs when this happens. See here for some info: https://github.com/microsoft/python-language-server/blob/master/TROUBLESHOOTING.md#filing-an-issue
okay (I did some small redactions for privacy) https://gist.github.com/uriva/a0f243c8b15a03c264afc6a2ae27a936
Reproduction is just to do several changes across different files in the dependency project. Then it goes into infinite analysis in the other one. You can see that some lines are repeating thousands of times.
This run never ends actually (>>10 minutes), the pc is "usable", but working very hard the entire time (as I said before, sometimes it is even worse, and the pc becomes unresponsive).
This looks similar to #1426, actually. I see
decoratoris failing. In the same environment, can you open a file that's justimport decorator, and see if you see an exception come up when doing something likedecorator.to trigger completion on the module?Seems to be working fine when writing
decorator.suggestions come up normally.2 remaining items
We just built a version which includes some more logging, v0.3.56. I'd appreciate it if you could set the following in your config and reload to trigger a download, then see if you can grab one of the stack traces for those exceptions.
"python.analysis.downloadChannel": "daily"
- changed the title
[-]CPU/RAM spikes[/-][+]Failure analyzing decorator module, high CPU/RAM usage[/+]on Aug 14, 2019 Here is my log after adding :
"python.analysis.downloadChannel": "daily"
log fileThe relevent bits:
Analysis of decorator(Library) failed. System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Python.Analysis.Analyzer.Symbols.FunctionEvaluator.Walk(ReturnStatement node) in E:\A\_work\3\s\src\Analysis\Ast\Impl\Analyzer\Symbols\FunctionEvaluator.cs:line 122 at Microsoft.Python.Parsing.Ast.ReturnStatement.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\ReturnStatement.cs:line 34 at Microsoft.Python.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\SuiteStatement.cs:line 41 at Microsoft.Python.Analysis.Analyzer.Symbols.FunctionEvaluator.Evaluate() in E:\A\_work\3\s\src\Analysis\Ast\Impl\Analyzer\Symbols\FunctionEvaluator.cs:line 64 at Microsoft.Python.Analysis.Analyzer.Symbols.ModuleSymbolTable.Evaluate(ScopeStatement target) in E:\A\_work\3\s\src\Analysis\Ast\Impl\Analyzer\Symbols\ModuleSymbolTable.cs:line 87 at Microsoft.Python.Analysis.Analyzer.ModuleWalker.Walk(FunctionDefinition node) in E:\A\_work\3\s\src\Analysis\Ast\Impl\Analyzer\ModuleWalker.cs:line 189 at Microsoft.Python.Parsing.Ast.FunctionDefinition.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\FunctionDefinition.cs:line 197 at Microsoft.Python.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\SuiteStatement.cs:line 41 at Microsoft.Python.Parsing.Ast.IfStatementTest.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\IfStatementTest.cs:line 41 at Microsoft.Python.Parsing.Ast.IfStatement.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\IfStatement.cs:line 45 at Microsoft.Python.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\SuiteStatement.cs:line 41 at Microsoft.Python.Parsing.Ast.PythonAst.Walk(PythonWalker walker) in E:\A\_work\3\s\src\Parsing\Impl\Ast\PythonAst.cs:line 81 at Microsoft.Python.Analysis.Analyzer.PythonAnalyzerSession.AnalyzeEntry(IDependencyChainNode`1 node, PythonAnalyzerEntry entry, IPythonModule module, PythonAst ast, Int32 version) in E:\A\_work\3\s\src\Analysis\Ast\Impl\Analyzer\PythonAnalyzerSession.cs:line 326 at Microsoft.Python.Analysis.Analyzer.PythonAnalyzerSession.Analyze(IDependencyChainNode`1 node, AsyncCountdownEvent ace, Stopwatch stopWatch) in E:\A\_work\3\s\src\Analysis\Ast\Impl\Analyzer\PythonAnalyzerSession.cs:line 264Cameron Trando (@CTrando) - most probably
FunctionDefinition.Nameis null or_function.DeclaringTypeis null. Neither guaranteed to exist.Hi! Can you give an estimate when this issue will be fixed? Have similar issue on MS language server 0.3.47.0 it consumes ~6GB of ram (all available at that moment). The problem was caused by simply opening of https://github.com/marcospereirampj/python-keycloak repo clone
Konstantin Pozdniakov (@cspwizard) - this issue is specifcally about decorators and exception above. Your case may or may not be the same. Please open separate issue. 0.3.59 is now published, restart VSC to update. Thanks.
Exception is fixed in #1461 - 0.3.61
Hi,
I have
"python.jediEnabled": falsein mysettings.json.Whenever I open a project, I see "analyzing in background" for a bit, then it stops. Usually it's 1200 targets or something similar.
Then from time to time I have a spike of cpu (200%) and ram usage from the process, and the code is again "analyzed in background".
This is coming from
.vscode/extensions/ms-python.python-2019.6.24221/languageServer.0.3.43/Microsoft.Python.LanguageServer.Over the last month the cpu/ram issues were on and off. At first they were making the computer stuck (I had to restart) and other times it went away after awhile. I saw some issues were opened and the closed, but over the last two weeks it again became hard to work with vscode because of this, so probably the problem isn't completely fixed.