API scraping process will be time-consuming for large project including many packages.

analysis
retag add tags

Hi,

I have a virtualenv which including 200+ packages and I find that each time when starting the wing for this project the API scraping process will be time-sonsuming -- approximately several minutes. And during this period, I can do nothing but waiting for its completion due to the very slow response speed of cursor.

The cached database is rather huge, see the following:

werner@ubuntu-01:~/.cache/wingpro7$ du -BM cache/db/analysis-store-21.sqllite
534M    cache/db/analysis-store-21.sqllite

Are there some optimization methods for this situation?

Regards

547

hongyi-zhao

asked 2020-04-12 18:55:57 -0600

4.2k

Wingware Support

updated 2020-04-13 09:50:37 -0600

edit flag offensive 0 remove flag close merge delete

Comments

Are you updating the packages between runs? Wing should be analyzing your code once and then using the cached information. The size of the cached database by itself does not indicate that there's a problem. Could you submit a bug report from Wing's Help menu with the log?

Wingware Support (2020-04-13 09:32:27 -0600) edit

I've just switched the virtualenv used by that project from pip to conda, at the same time, I use same virtualenv's name. Maybe this is the reason.

After I restarts wing, it seems that things are not so bad. Let me keep in inspecting it in the future and feedback if necessary.

Regards

hongyi-zhao (2020-04-13 22:37:14 -0600) edit

add a comment see more comments

2 Answers

Sort by

oldest

newest

most voted

I want to know whether wing can always ensure there is no redundancy/duplicates in the cache database?

Regards

547

hongyi-zhao

answered 2020-04-14 01:55:36 -0600

edit flag offensive 0 remove flag delete link

Comments

Wing writes info about each file it analyzes to the database -- most everything is keyed off of the filename so info should only be written once.

Wingware Support (2020-04-14 08:05:47 -0600) edit

Do you mean using the files' absolute path names as the key? If not, there maybe files located in different folders with the same name that are not keyed uniquely.

Regards

hongyi-zhao (2020-04-14 08:11:34 -0600) edit

All filenames are absolute. I consider the reducing the time spent analyzing to be a higher priority than reducing the size of the database file. A large file does not indicate that there are problems.

Wingware Support (2020-04-14 08:15:33 -0600) edit

Another question: for symlinks, should use realpath / physical path. By saying All filenames are absolute, do you mean the same meaning as my above note?

Regards

hongyi-zhao (2020-04-14 19:27:07 -0600) edit

Either symlink paths or real paths should work.

Wingware Support (2020-04-15 08:53:05 -0600) edit

add a comment see more comments

Based on the comments above, I think when you switched it reanalyzed everything because the files are in different locations, but it caches the results so should not be slow after that is complete, even after restarting Wing.

4.2k

Wingware Support

answered 2020-04-14 08:32:48 -0600, updated 2020-04-14 08:33:42 -0600

edit flag offensive 0 remove flag delete link

Comments

add a comment see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss.

Add Answer

API scraping process will be time-consuming for large project including many packages. edit

Comments

2 Answers

Comments

Comments