First time here? Check out the FAQ!
0

API scraping process will be time-consuming for large project including many packages.

Hi,

I have a virtualenv which including 200+ packages and I find that each time when starting the wing for this project the API scraping process will be time-sonsuming -- approximately several minutes. And during this period, I can do nothing but waiting for its completion due to the very slow response speed of cursor.

The cached database is rather huge, see the following:

werner@ubuntu-01:~/.cache/wingpro7$ du -BM cache/db/analysis-store-21.sqllite
534M    cache/db/analysis-store-21.sqllite

Are there some optimization methods for this situation?

Regards

hongyi-zhao's avatar
497
hongyi-zhao
asked 2020-04-12 18:55:57 -0500
Wingware Support's avatar
4k
Wingware Support
updated 2020-04-13 09:50:37 -0500
edit flag offensive 0 remove flag close merge delete

Comments

Are you updating the packages between runs? Wing should be analyzing your code once and then using the cached information. The size of the cached database by itself does not indicate that there's a problem. Could you submit a bug report from Wing's Help menu with the log?

Wingware Support's avatar Wingware Support (2020-04-13 09:32:27 -0500) edit

I've just switched the virtualenv used by that project from pip to conda, at the same time, I use same virtualenv's name. Maybe this is the reason.

After I restarts wing, it seems that things are not so bad. Let me keep in inspecting it in the future and feedback if necessary.

Regards

hongyi-zhao's avatar hongyi-zhao (2020-04-13 22:37:14 -0500) edit
add a comment see more comments

2 Answers

0

I want to know whether wing can always ensure there is no redundancy/duplicates in the cache database?

Regards

hongyi-zhao's avatar
497
hongyi-zhao
answered 2020-04-14 01:55:36 -0500
edit flag offensive 0 remove flag delete link

Comments

Wing writes info about each file it analyzes to the database -- most everything is keyed off of the filename so info should only be written once.

Wingware Support's avatar Wingware Support (2020-04-14 08:05:47 -0500) edit

Do you mean using the files' absolute path names as the key? If not, there maybe files located in different folders with the same name that are not keyed uniquely.

Regards

hongyi-zhao's avatar hongyi-zhao (2020-04-14 08:11:34 -0500) edit

All filenames are absolute. I consider the reducing the time spent analyzing to be a higher priority than reducing the size of the database file. A large file does not indicate that there are problems.

Wingware Support's avatar Wingware Support (2020-04-14 08:15:33 -0500) edit

Another question: for symlinks, should use realpath / physical path. By saying All filenames are absolute, do you mean the same meaning as my above note?

Regards

hongyi-zhao's avatar hongyi-zhao (2020-04-14 19:27:07 -0500) edit

Either symlink paths or real paths should work.

Wingware Support's avatar Wingware Support (2020-04-15 08:53:05 -0500) edit
add a comment see more comments
0

Based on the comments above, I think when you switched it reanalyzed everything because the files are in different locations, but it caches the results so should not be slow after that is complete, even after restarting Wing.

Wingware Support's avatar
4k
Wingware Support
answered 2020-04-14 08:32:48 -0500, updated 2020-04-14 08:33:42 -0500
edit flag offensive 0 remove flag delete link

Comments

add a comment see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss.

Add Answer