Repository navigation
Closed
Description
Problem
When multiple JSONRPC refresh requests arrive concurrently, PET serializes them behind a global REFRESH_LOCK mutex. Each queued caller waits for the lock, then runs a full environment scan from scratch — even if the previous caller just completed an identical scan milliseconds ago.
From investigating:
- over half of all failing sessions have all 3 managers (pipenv + poetry + pyenv) failing together, meaning 3 concurrent
refreshcalls are the norm, not the exception - Each refresh acquires
REFRESH_LOCK, runs 12+ locators in parallel threads, scans PATH, global virtualenvs, and workspaces — then the next queued caller repeats all of it - Total wall time = N × single refresh duration instead of 1×
Current Behavior
Client sends: refresh(id=1), refresh(id=2), refresh(id=3) [concurrent]
Thread 1: acquires REFRESH_LOCK → runs full scan (5-15s) → sends reply(id=1) → drops lock
Thread 2: acquires REFRESH_LOCK → runs full scan (5-15s) → sends reply(id=2) → drops lock
Thread 3: acquires REFRESH_LOCK → runs full scan (5-15s) → sends reply(id=3) → drops lock
Total wall time: 15-45s
Relevant code in crates/pet/src/jsonrpc.rs:
lazy_static! {
static ref REFRESH_LOCK: Arc<Mutex<()>> = Arc::new(Mutex::new(()));
}
pub fn handle_refresh(context: Arc<Context>, id: u32, params: Value) {
// ...
thread::spawn(move || {
// Ensure we can have only one refresh at a time.
let lock = REFRESH_LOCK.lock().expect("REFRESH_LOCK mutex poisoned");
// ... full scan runs here ...
send_reply(id, Some(RefreshResult::new(summary.total)));
// ... telemetry + missing envs ...
drop(lock);
});
}Desired Behavior
When a refresh is already in progress, subsequent callers with compatible parameters should join the in-flight refresh rather than queue a separate one. All waiters receive the same result when the single refresh completes.
Client sends: refresh(id=1), refresh(id=2), refresh(id=3) [concurrent]
Thread 1: acquires REFRESH_LOCK → runs full scan (5-15s)
Thread 2: sees refresh in progress → waits for result
Thread 3: sees refresh in progress → waits for result
scan completes → sends reply(id=1), reply(id=2), reply(id=3)
Total wall time: 5-15s (1× instead of 3×)
Metadata
Metadata
Assignees
Labels
No labels
Activity
karthiknadig commentedon Apr 3, 2026
Investigation notes and a concrete proposal.
The current bottleneck is in
crates/pet/src/jsonrpc.rs: eachrefreshrequest spawns a thread, then takesREFRESH_LOCKaround the full refresh path, including locator reconfiguration, discovery, reply, and telemetry. That means concurrent identical refreshes are serialized and each reruns a full scan.Proposed implementation
Replace the coarse global refresh mutex with a small refresh coordinator that tracks one active refresh plus the request ids waiting on it.
Compatibility should be conservative:
RefreshOptionsProposed flow:
refreshparams.config_generationfromContext.The worker should:
environment/managernotifications onceRefreshResultreply to every waiting request idIncompatible refreshes can remain serialized behind the coordinator for the first version. That still fixes the reported N x duplicate scan cost for identical concurrent requests.
Why include configuration generation
During investigation I found a separate correctness issue:
configureandrefreshcan both reconfigure shared locator instances concurrently, and some locators retain mutable config internally. I filed that separately as #385.That race should not block issue #383 entirely, but I do think #383 should be generation-aware from the start. Otherwise a post-
configurerefresh could incorrectly join an in-flight scan that was started under an older config.Recommendation on sequencing
I would not make #385 a hard prerequisite for #383.
I would do this:
configureandrefreshThat ordering gets the performance win without baking in obviously incorrect request joining semantics.
fix: deduplicate concurrent refresh requests (Fixes #383)
fix: formalize refresh locator state (#388)
fix: harden refresh generation guards (Fixes #383)
fix: deduplicate concurrent refresh requests (microsoft#386)