Ask Your Question

API scraping process will be time-consuming for large project including many packages.

asked 2020-04-12 18:55:57 -0600

hongyi-zhao's avatar

updated 2020-04-13 09:50:37 -0600


I have a virtualenv which including 200+ packages and I find that each time when starting the wing for this project the API scraping process will be time-sonsuming -- approximately several minutes. And during this period, I can do nothing but waiting for its completion due to the very slow response speed of cursor.

The cached database is rather huge, see the following:

werner@ubuntu-01:~/.cache/wingpro7$ du -BM cache/db/analysis-store-21.sqllite
534M    cache/db/analysis-store-21.sqllite

Are there some optimization methods for this situation?


edit retag flag offensive close merge delete


Are you updating the packages between runs? Wing should be analyzing your code once and then using the cached information. The size of the cached database by itself does not indicate that there's a problem. Could you submit a bug report from Wing's Help menu with the log?

Wingware Support's avatar Wingware Support  ( 2020-04-13 09:32:27 -0600 )edit

I've just switched the virtualenv used by that project from pip to conda, at the same time, I use same virtualenv's name. Maybe this is the reason.

After I restarts wing, it seems that things are not so bad. Let me keep in inspecting it in the future and feedback if necessary.


hongyi-zhao's avatar hongyi-zhao  ( 2020-04-13 22:37:14 -0600 )edit

2 Answers

Sort by ยป oldest newest most voted

answered 2020-04-14 08:32:48 -0600

updated 2020-04-14 08:33:42 -0600

Based on the comments above, I think when you switched it reanalyzed everything because the files are in different locations, but it caches the results so should not be slow after that is complete, even after restarting Wing.

edit flag offensive delete link more

answered 2020-04-14 01:55:36 -0600

hongyi-zhao's avatar

I want to know whether wing can always ensure there is no redundancy/duplicates in the cache database?


edit flag offensive delete link more


Wing writes info about each file it analyzes to the database -- most everything is keyed off of the filename so info should only be written once.

Wingware Support's avatar Wingware Support  ( 2020-04-14 08:05:47 -0600 )edit

Do you mean using the files' absolute path names as the key? If not, there maybe files located in different folders with the same name that are not keyed uniquely.


hongyi-zhao's avatar hongyi-zhao  ( 2020-04-14 08:11:34 -0600 )edit

All filenames are absolute. I consider the reducing the time spent analyzing to be a higher priority than reducing the size of the database file. A large file does not indicate that there are problems.

Wingware Support's avatar Wingware Support  ( 2020-04-14 08:15:33 -0600 )edit

Another question: for symlinks, should use realpath / physical path. By saying All filenames are absolute, do you mean the same meaning as my above note?


hongyi-zhao's avatar hongyi-zhao  ( 2020-04-14 19:27:07 -0600 )edit

Either symlink paths or real paths should work.

Wingware Support's avatar Wingware Support  ( 2020-04-15 08:53:05 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2020-04-12 18:55:57 -0600

Seen: 787 times

Last updated: Apr 14 '20