This repository was archived by the owner on Apr 14, 2022. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 14, 2022. It is now read-only.
Recursive or looping symlinks can cause language server to repeatedly analyze files #181
Description
Activity
Yeah, I might have solution
Reacted by Sandor, Chuck Chen, FIN-ACK, vc1 and Abdul MohamedMost probably no longer applies, so for verification with #432
If we create correct PathResolverSnapshot, there should be no problem.
Tested using this specific example, the new LS does not loop infinitely. There are some other symlink oddities (long import hints, loss of import completion once a symlink is imported), but I'll file separate issues for those.
Reopening, there may be some other looping issues.
I actually still experiment this issue with the latest VS Code version (1.32.3).
Reacted by Olivier ClouxFrom my side, I seem to have problems in looping too (characteristic of the "Analyzing in background, X items left"). But in my side, it's not the file/directory/workspace structure, but more the virtual environment. With a simple file (no import, one function), and using a conda env from another project I got absolutely no problem.
But when I import certain packages (I noticed sklearn poses problems), LS tries to analyses this and get in trouble. But for example, numpy (also "complex" package) poses no problem at all.STRANGER YET, when going with a blank conda env, with only numpy and sklearn, the same import leads to no problem at all. So it's linked to packages in environment, but there also seems to be a third party factor.
Problematic case:
Environment
- VS code 1.32.3,
- Python 3.7.3 (with anaconda 3)
- Using conda env
- On Debian 10
[edit] and extension version : 2019.3.6352 [/edit]
Description of the problem
As many others, with certain workspaces, the "Analyzing in background, X items left" appears, grows to a certain point, sometimes decreases and grows again, until being stuck indefinitely. Sucks up CPU and memory usage.
"Simple file" in question:
from sklearn.model_selection import KFold def main(): print("This python is not buggy") return True if __name__ == "__main__": main()Content of the problematic conda env (sorry it's not minimal, I didn't find yet a minimal case)
_tflow_select 2.3.0 mkl absl-py 0.7.0 py36_0 alembic 1.0.6 py36_0 apscheduler 3.5.3 py36_1000 conda-forge asn1crypto 0.24.0 py36_0 astor 0.7.1 py36_0 astroid 2.1.0 py36_0 autopep8 1.4.3 py36_0 backcall 0.1.0 py36_0 beautifulsoup4 4.7.1 py36_1 blas 1.0 mkl bleach 3.1.0 py36_0 c-ares 1.15.0 h7b6447c_1 ca-certificates 2018.12.5 0 certifi 2018.11.29 py36_0 cffi 1.11.5 py36he75722e_1 chardet 3.0.4 py36_1 click 7.0 py36_0 cloudpickle 0.6.1 py36_0 cryptography 2.4.2 py36h1ba5d50_0 cycler 0.10.0 py36_0 cytoolz 0.9.0.1 py36h14c3975_1 dask-core 1.0.0 py36_0 dbus 1.13.6 h746ee38_0 decorator 4.3.0 py36_0 entrypoints 0.3 py36_0 expat 2.2.6 he6710b0_0 flask 1.0.2 py36_1 flask-migrate 2.2.1 py36_1000 conda-forge flask-sqlalchemy 2.3.2 py36_0 fontconfig 2.13.0 h9420a91_0 freetype 2.9.1 h8a8886c_1 gast 0.2.2 py36_0 glib 2.56.2 hd408876_0 gmp 6.1.2 h6c8ec71_1 grpcio 1.16.1 py36hf8bcb03_1 gst-plugins-base 1.14.0 hbbd80ab_1 gstreamer 1.14.0 hb453b48_1 h5py 2.9.0 py36h7918eee_0 hdf5 1.10.4 hb1b8bf9_0 icu 58.2 h9c2bf20_1 idna 2.8 py36_0 imageio 2.4.1 py36_0 intel-openmp 2019.1 144 ipykernel 5.1.0 py36h39e3cac_0 ipython 7.2.0 py36h39e3cac_0 ipython_genutils 0.2.0 py36_0 ipywidgets 7.4.2 py36_0 isort 4.3.4 py36_0 itsdangerous 1.1.0 py36_0 jedi 0.13.2 py36_0 jinja2 2.10 py36_0 jpeg 9b h024ee3a_2 jsonschema 2.6.0 py36_0 jupyter 1.0.0 py36_7 jupyter_client 5.2.4 py36_0 jupyter_console 6.0.0 py36_0 jupyter_core 4.4.0 py36_0 keras-applications 1.0.6 py36_0 keras-preprocessing 1.0.5 py36_0 kiwisolver 1.0.1 py36hf484d3e_0 lazy-object-proxy 1.3.1 py36h14c3975_2 libedit 3.1.20181209 hc058e9b_0 libffi 3.2.1 hd88cf55_4 libgcc 7.2.0 h69d50b8_2 libgcc-ng 8.2.0 hdf63c60_1 libgfortran-ng 7.3.0 hdf63c60_0 libpng 1.6.36 hbc83047_0 libprotobuf 3.6.1 hd408876_0 libsodium 1.0.16 h1bed415_0 libstdcxx-ng 8.2.0 hdf63c60_1 libtiff 4.0.10 h2733197_1001 libuuid 1.0.3 h1bed415_2 libxcb 1.13 h1bed415_1 libxml2 2.9.9 he19cac6_0 mako 1.0.7 pypi_0 pypi markdown 3.0.1 py36_0 markupsafe 1.1.0 py36h7b6447c_0 matplotlib 3.0.2 py36h5429711_0 mccabe 0.6.1 py36_1 mistune 0.8.4 py36h7b6447c_0 mkl 2019.1 144 mkl_fft 1.0.10 py36ha843d7b_0 mkl_random 1.0.2 py36hd81dba3_0 nbconvert 5.3.1 py36_0 nbformat 4.4.0 py36_0 ncurses 6.1 he6710b0_1 networkx 2.2 py36_1 notebook 5.7.4 py36_0 numpy 1.15.4 py36h7e9f1db_0 numpy-base 1.15.4 py36hde5b4d6_0 olefile 0.46 py36_0 openssl 1.1.1a h7b6447c_0 pandas 0.23.4 py36h04863e7_0 pandoc 2.2.3.2 0 pandocfilters 1.4.2 py36_1 parso 0.3.1 py36_0 pcre 8.42 h439df22_0 pexpect 4.6.0 py36_0 pickleshare 0.7.5 py36_0 pillow 5.4.1 py36h34e0f95_0 pip 18.0 pypi_0 pypi prometheus_client 0.5.0 py36_0 prompt_toolkit 2.0.7 py36_0 protobuf 3.6.1 py36he6710b0_0 ptyprocess 0.6.0 py36_0 pycodestyle 2.4.0 py36_0 pycparser 2.19 py36_0 pygments 2.3.1 py36_0 pylint 2.2.2 py36_0 pyopenssl 18.0.0 py36_0 pyparsing 2.3.1 py36_0 pyqt 5.9.2 py36h05f1152_2 pysocks 1.6.8 py36_0 python 3.6.8 h0371630_0 python-dateutil 2.7.5 py36_0 python-editor 1.0.3 py36_0 pytz 2018.9 py36_0 pywavelets 1.0.1 py36hdd07704_0 pyyaml 3.13 py36h14c3975_0 pyzmq 17.1.2 py36h14c3975_0 qt 5.9.7 h5867ecd_1 qtconsole 4.4.3 py36_0 readline 7.0 h7b6447c_5 requests 2.21.0 py36_0 scikit-image 0.14.1 py36he6710b0_0 scikit-learn 0.20.2 py36hd81dba3_0 scipy 1.1.0 py36h7c811a0_2 send2trash 1.5.0 py36_0 setuptools 40.6.3 py36_0 sip 4.19.8 py36hf484d3e_0 six 1.12.0 py36_0 soupsieve 1.7.1 py36_0 sqlalchemy 1.2.16 py36h7b6447c_0 sqlite 3.26.0 h7b6447c_0 tensorboard 1.12.2 py36he6710b0_0 tensorflow 1.12.0 mkl_py36h69b6ba0_0 tensorflow-base 1.12.0 mkl_py36h3c3e929_0 termcolor 1.1.0 py36_1 terminado 0.8.1 py36_1 testpath 0.4.2 py36_0 tk 8.6.8 hbc83047_0 toolz 0.9.0 py36_0 tornado 5.1.1 py36h7b6447c_0 traitlets 4.3.2 py36_0 typed-ast 1.1.0 py36h14c3975_0 tzlocal 1.5.1 py36_0 urllib3 1.24.1 py36_0 wcwidth 0.1.7 py36_0 webencodings 0.5.1 py36_1 werkzeug 0.14.1 py36_0 wheel 0.32.3 py36_0 widgetsnbextension 3.4.2 py36_0 wrapt 1.11.0 py36h7b6447c_0 xz 5.2.4 h14c3975_4 yaml 0.1.7 had09818_2 zeromq 4.2.5 hf484d3e_1 zlib 1.2.11 h7b6447c_3Extract of the python output
Analysis of IPython.core.magics.history(Library) queued Analysis of scipy.optimize._slsqp(Compiled) completed in 47.8488 ms. Analysis of sklearn.externals.joblib.pool(Library) queued Analysis of _locale(CompiledBuiltin) queued Analysis of sre_compile(Library) queued Analysis of distutils.command.install(Stub) completed in 48.1344 ms. Analysis of nbformat.v2.nbbase(Library) queued Analysis of IPython.core.magics.extension(Library) queued Analysis of IPython.utils.terminal(Library) queued Analysis of scipy.optimize.moduleTNC(Compiled) completed in 39.3443 ms. Analysis of nbformat.v1.convert(Library) queued Analysis of IPython.utils.dir2(Library) queued Analysis of numpy.testing.noseclasses(Library) queued Analysis of email.parser(Library) queued Analysis of IPython.utils.process(Library) queued Analysis of nbformat.v1.nbbase(Library) queued Analysis of nbformat.v2.nbpy(Library) queued Analysis of prompt_toolkit.filters.app(Library) queued Analysis of notebook.services.contents.fileio(Library) queued Analysis of sre_parse(Stub) queued Analysis of prompt_toolkit.win32_types(Library) queued Analysis of pygments.formatters.latex(Library) queued Analysis of prompt_toolkit.renderer(Library) queued Analysis of tornado.process(Library) queued Analysis of asyncio.transports(Stub) queued Analysis of asyncio.streams(Library) queued Analysis of nbformat.v1.nbjson(Library) queued Analysis of sklearn.externals.joblib.externals.loky.process_executor(Library) queued Analysis of dummy_threading(Library) queued Analysis of sre_parse(Library) queued Analysis of IPython.lib.display(Library) queued Analysis of prompt_toolkit.eventloop.context(Library) queued Import: ipython_genutils.ipstruct /storage/anaconda3/envs/facesearch/lib/python3.6/site-packages/ipython_genutils/ipstruct.py Analysis of distutils.log(Stub) completed in 74.345 ms. Analysis of ipython_genutils.ipstruct(Library) queued Analysis of prompt_toolkit.enums(Library) queued Analysis of IPython.core.magics.execution(Library) queued Analysis of IPython.core.page(Library) queued Analysis of nbformat.v3(Library) queued .... ... Analysis of _heapq(Compiled) completed in 16.2774 ms. Analysis of scipy.interpolate._bsplines(Library) completed in 17.8444 ms. Analysis of colorsys(Library) completed in 0.6466 ms. Analysis of IPython.core.inputtransformer2(Library) queued Analysis of unittest(Library) completed in 31.8242 ms. Analysis of numpy.version(Library) completed in 0.1255 ms. Analysis of sklearn.decomposition._online_lda(Compiled) completed in 0.3446 ms. Analysis of scipy.interpolate.interpnd(Compiled) completed in 1.2661 ms. Analysis of scipy.interpolate._ppoly(Compiled) completed in 9.8247 ms. Analysis of numpy.core._multiarray_tests(Compiled) completed in 0.9455 ms. Analysis of scipy.interpolate.dfitpack(Compiled) completed in 0.3231 ms. Analysis of multiprocessing(Library) completed in 0.1187 ms. Analysis of sklearn.externals.joblib.externals.loky.backend._win_wait(Library) queued Analysis of ipykernel.kernelapp(Library) queued Analysis of multiprocessing.pool(Stub) completed in 20.9803 ms. Analysis of multiprocessing.process(Stub) completed in 0.305 ms. Analysis of numpy.testing._private(Library) completed in 17.188 ms. Analysis of _multiprocessing(Compiled) completed in 10.1877 ms. Analysis of _hashlib(Compiled) completed in 0.4461 ms. Analysis of _sha3(Compiled) completed in 15.738 ms. Analysis of pycparser(Library) completed in 0.6777 ms. Analysis of sklearn.externals.joblib.externals.loky.reusable_executor(Library) queued Analysis of IPython.core.completer(Library) queued Analysis of IPython.core.extensions(Library) queued Analysis of _blake2(Compiled) completed in 17.729 ms. Analysis of _sha512(Compiled) completed in 27.4286 ms. Analysis of _sha256(Compiled) completed in 1.6649 ms. Analysis of _md5(Compiled) completed in 3.1989 ms.SO, some files are queued, while some are completed in a matter of ms.
I guess that is the problem, some recursivity in specific packages. Maybe two (or more) of the packages in the env are referencing each other, creating a loop. Point is, with a conda env having only numpy and sklearn, this issue does not appear.Sorry if I mis-followed the guidelines...
Please file a new issue; this one is closed and is probably not related.
Regarding "only numpy and sklearn", there's a lot behind those libraries which may be affected by things relating to my explanation in #832 (comment).
Original issue can be found here along with the details microsoft/vscode-python#2613
I commented previously on #2297 but I found that symlinks were still causing problems after I grabbed a recent build today.
Environment data
Actual behavior
The Python language server gets caught in a super deep nest of recursive symlinks. The repro case given completes quickly (there's nothing to analyze) but still demonstrates the problem, I think.
If there is actual code to parse/analyze it pretty much goes on forever. "Analyzing workspace, #### items remaining" goes down and then quickly shoots up, and this repeats over and over.
I can workaround this by adding the offending symlinks/folder structures to a VSCode exclusion list.
Expected behavior
The Python language server should be able to handle these structures in some reasonable way.
Steps to reproduce:
import folder.lib.folder/symlink/folder/symlink/folder structure.Logs
Output for
Pythonin theOutputpanel (View→Output, change the drop-down the upper-right of theOutputpanel toPython)Output from
Consoleunder theDeveloper Toolspanel (toggle Developer Tools on underHelp)Symlinks to a common folder (e.g. two projects which have a symlink to the same folder) also get re-analyzed but that's usually okay as analysis is pretty fast in general.