Ask Your Question
0

API scraping process will be time-consuming for large project including many packages.

asked 2020-04-12 18:55:57 -0500

hongyi-zhao gravatar image

updated 2020-04-13 09:50:37 -0500

Hi,

I have a virtualenv which including 200+ packages and I find that each time when starting the wing for this project the API scraping process will be time-sonsuming -- approximately several minutes. And during this period, I can do nothing but waiting for its completion due to the very slow response speed of cursor.

The cached database is rather huge, see the following:

werner@ubuntu-01:~/.cache/wingpro7$ du -BM cache/db/analysis-store-21.sqllite
534M    cache/db/analysis-store-21.sqllite

Are there some optimization methods for this situation?

Regards

edit retag flag offensive close merge delete

Comments

Are you updating the packages between runs? Wing should be analyzing your code once and then using the cached information. The size of the cached database by itself does not indicate that there's a problem. Could you submit a bug report from Wing's Help menu with the log?

Wingware Support gravatar imageWingware Support ( 2020-04-13 09:32:27 -0500 )edit

I've just switched the virtualenv used by that project from pip to conda, at the same time, I use same virtualenv's name. Maybe this is the reason.

After I restarts wing, it seems that things are not so bad. Let me keep in inspecting it in the future and feedback if necessary.

Regards

hongyi-zhao gravatar imagehongyi-zhao ( 2020-04-13 22:37:14 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted
0

answered 2020-04-14 08:32:48 -0500

updated 2020-04-14 08:33:42 -0500

Based on the comments above, I think when you switched it reanalyzed everything because the files are in different locations, but it caches the results so should not be slow after that is complete, even after restarting Wing.

edit flag offensive delete link more
0

answered 2020-04-14 01:55:36 -0500

hongyi-zhao gravatar image

I want to know whether wing can always ensure there is no redundancy/duplicates in the cache database?

Regards

edit flag offensive delete link more

Comments

Wing writes info about each file it analyzes to the database -- most everything is keyed off of the filename so info should only be written once.

Wingware Support gravatar imageWingware Support ( 2020-04-14 08:05:47 -0500 )edit

Do you mean using the files' absolute path names as the key? If not, there maybe files located in different folders with the same name that are not keyed uniquely.

Regards

hongyi-zhao gravatar imagehongyi-zhao ( 2020-04-14 08:11:34 -0500 )edit

All filenames are absolute. I consider the reducing the time spent analyzing to be a higher priority than reducing the size of the database file. A large file does not indicate that there are problems.

Wingware Support gravatar imageWingware Support ( 2020-04-14 08:15:33 -0500 )edit

Another question: for symlinks, should use realpath / physical path. By saying All filenames are absolute, do you mean the same meaning as my above note?

Regards

hongyi-zhao gravatar imagehongyi-zhao ( 2020-04-14 19:27:07 -0500 )edit

Either symlink paths or real paths should work.

Wingware Support gravatar imageWingware Support ( 2020-04-15 08:53:05 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2020-04-12 18:55:57 -0500

Seen: 346 times

Last updated: Apr 14