Ask Your Question
0

Let wingide do the API scraping process/analysing in parallel mode.

asked 2020-03-26 08:56:45 -0500

hongyi-zhao gravatar image

updated 2020-03-26 21:44:06 -0500

Hi,

For a virtualenv which have many packages installed, the API scraping process/analysing will be time-consuming. Say the following for my case,

$ pip list | wc
    163     329    7630

In this case, the windide will take long time for analyzing the packages belong to the virtualenv, so I want to know whether can we let wingide utilize the multiple cores of the host for this job, i.e., done parallelly.

Regards

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

answered 2020-03-26 19:32:00 -0500

Source analysis is currently asynchronous within Wing's process, so just one core. We're moving towards moving that to separate processes so it can use more cores, but this isn't available yet. Part of the process, scraping extension module APIs by loading them does already happen in separate processes and should end up on multiple cores. But a lot of the work is analyzing Python code and not extension modules.

edit flag offensive delete link more

Comments

Anything, if possible, should be done using more cores, IMO.

hongyi-zhao gravatar imagehongyi-zhao ( 2020-03-26 20:08:29 -0500 )edit

Yes, although there will be a preference to control how many cores are used because people won't necessarily want Wing loading them all down if they are also working on other things.

Wingware Support gravatar imageWingware Support ( 2020-03-26 21:38:41 -0500 )edit

Agree with you.

hongyi-zhao gravatar imagehongyi-zhao ( 2020-03-30 23:47:47 -0500 )edit
0

answered 2020-03-26 09:35:37 -0500

Wing doesn't currently have a package manager so I'm a bit unclear on what you're asking, or perhaps this question is off-topic since it's about pip and not Wing.

In any case, I think if you run pip it just uses one core. The only things I could find about this by searching seem to indicate that is still how it works.

We do hope to add a package manger to Wing eventually, and will be sure we consider that this can be slow.

edit flag offensive delete link more

Comments

Sorry for my phrasing, I've changed the title for this question. In fact, I mean the API scraping process/analyzing done by wingide when a new virtualenv added into wingide.

hongyi-zhao gravatar imagehongyi-zhao ( 2020-03-26 19:19:59 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2020-03-26 08:56:45 -0500

Seen: 42 times

Last updated: Mar 26