Conversation
|
Got basic timeseries with linestack working. I've also got some code snippets for interpolating to display heatmap with non-uniformly sampled timeseries data. I should be able to have this fully working with time series very soon :D Kooha-2025-12-27-03-50-45.mp4 |
|
Got heatmap to display timeseries working. It should also work with non-uniformly sampled data by interpolating, need to test. Also need to implementing switching between heatmap and line representations, need to delete the graphic when switching. Kooha-2025-12-27-17-47-16.mp4 |
|
So timeseries can be represented with arrays of one of the following shapes (let's ignore x-axis values for now). If we have: l: number of timeseries We can have the following shapes: Extended to n-dimensional arrays (for example, trajectories projected onto principal components?). If each non-timeseries dim is I don't think we can auto-detect if
Scatters can be similar to some cases of nd-lines 🤔 , but x values would be directly specified and the current index is parametric (example with time indicating position in a low dim space). This would actually be true for lines as well if representing trajectories. So for nd-line maybe we have two versions, parametric (y and z are not functions of x, but x, y, z are a function of some other dim) and non-parametric (simple timeseries lines where y and z are functions of x). |
|
I made a more generalist
where: It can map arrays of these dims to a line, line collection, line stack, scatter, or list of scatters (similar to multi-line). I think this is a much more elegant way to deal with things, and Example if we have data that is I think we can also use this for heatmaps and interpolation. Use the reference units to determine a uniform x-range for the current display window, and we can interpolate using EDIT: I think that the |
|
For positions graphics, I should actually do |
|
Some more ideas: Allow any 2-3 dims to be used as the graphic dimensions and specify the slider dims. This would also allow using named dims (such as those used in xarray). We interpret the given order of the EDIT: A limitation of the above is that a user can't collapse multiple "graphic/display dimensions" into "final graphic/display dimensions" if they're hard-coded this like. So something like: An example for images would be collapsing |
|
We can use I was thinking of what's the best way to show a scatter for each keypoint, and I think I should make a |
|
ok I think stuff is working ndpositions-2026-01-29_22.55.54.mp4 |
|
I think I need to make a |
|
A set of imgui UIs that allow controlling some aspects of the "nd graphics" could be useful, such as:
|
Stuff I should finish before implementing the orchestrator:
Things that make me "uncomfortable" that I need to settle:dim shapesDim shape for nd-positions is
Do we just document this well, that the When using nd-positions data in conjunction with nd-image data, we'd have something like this: Where The |
|
Working on "implement mapping from a slider reference index with units (such as time) to array index.", which requires proper implementation of |
|
Window funcs working for ndp_windows-2026-02-01_03.39.52.mp4 |
|
Edit: Fixed
Need to separate |
* add simplejpeg to notebook deps * Update guide.rst * Update guide.rst * Update README.md
|
Idea! I think we can allow global indices to be shared between NDWidgets so then we can have multiple windows. Edit: this works! |
|
This also contains the replaceable buffers PR #974 which also needs a review but those bit of code are simpler and they're in the other parts of the codebase outside the ndwidget subpackage the following modules/classes are ready for review: base classes
General thoughts on NDPositionsProcessor and NDPositions aren't ready for review, but I had a few comments on those and would like your thoughts (should I do ND Timeseries subclass and the "_each" arts) random questions:
I think maybe any other naming things |
There was a problem hiding this comment.
I now realize we can probably auto-gen NDWSubplot.add_nd_graphic methods similar to Subplot.add_graphic methods
| self._timeseries = timeseries | ||
| # TODO: I think this is messy af, NDTimeseriesSubclass??? | ||
| if self._timeseries: | ||
| # makes some assumptions about positional data that apply only to timeseries representations | ||
| # probably don't want to maintain aspect | ||
| self._subplot.camera.maintain_aspect = False | ||
|
|
||
| # auto x range modes make no sense for non-timeseries data | ||
| self.x_range_mode = x_range_mode | ||
|
|
||
| if linear_selector: | ||
| self._linear_selector = LinearSelector( | ||
| 0, limits=(-np.inf, np.inf), edge_color="cyan" | ||
| ) | ||
| self._linear_selector.add_event_handler( | ||
| self._linear_selector_handler, "selection" | ||
| ) | ||
| self._subplot.add_graphic(self._linear_selector) | ||
| else: | ||
| self._linear_selector = None | ||
| else: | ||
| self._linear_selector = None |
There was a problem hiding this comment.
I think we should have an NDTimeseries subclass of NDPositions which has the timeseries specific stuff, i.e. x_range_mode linear_selector, etc., instead of messy if-else logic. 🤔
| # TODO: I think this is messy af, NDTimeseriesSubclass??? | ||
| # x range of the data | ||
| xr = data_slice[0, 0, 0], data_slice[0, -1, 0] | ||
| if self.x_range_mode is not None: | ||
| self.graphic._plot_area.x_range = xr | ||
|
|
||
| # if the update_from_view is polling, this prevents it from being called by setting the new last xrange | ||
| # in theory, but this doesn't seem to fully work yet, not a big deal right now can check later | ||
| self._last_x_range[:] = self.graphic._plot_area.x_range | ||
|
|
||
| if self._linear_selector is not None: | ||
| with pause_events(self._linear_selector): # we don't want the linear selector change to update the indices | ||
| self._linear_selector.limits = xr | ||
| # linear selector acts on `p` dim | ||
| self._linear_selector.selection = indices[ | ||
| self.processor.spatial_dims[1] | ||
| ] | ||
|
|
There was a problem hiding this comment.
similar to other comment, I think we should have an NDTimeseries subclass of NDPositions 🤔
| @property | ||
| def x_range_mode(self) -> Literal["fixed", "auto"] | None: | ||
| """x-range using a fixed window from the display window, or by polling the camera (auto)""" | ||
| return self._x_range_mode | ||
|
|
||
| @x_range_mode.setter | ||
| def x_range_mode(self, mode: Literal[None, "fixed", "auto"]): | ||
| if self._x_range_mode == "auto": | ||
| # old mode was auto | ||
| self._subplot.remove_animation(self._update_from_view_range) | ||
|
|
||
| if mode == "auto": | ||
| self._subplot.add_animations(self._update_from_view_range) | ||
|
|
||
| self._x_range_mode = mode | ||
|
|
||
| def _update_from_view_range(self): | ||
| if self._graphic is None: | ||
| return | ||
|
|
||
| xr = self._subplot.x_range | ||
|
|
||
| # the floating point error near zero gets nasty here | ||
| if np.allclose(xr, self._last_x_range, atol=1e-14): | ||
| return | ||
|
|
||
| last_width = abs(self._last_x_range[1] - self._last_x_range[0]) | ||
| self._last_x_range[:] = xr | ||
|
|
||
| new_width = abs(xr[1] - xr[0]) | ||
| new_index = (xr[0] + xr[1]) / 2 | ||
|
|
||
| if (new_index == self._ref_index[self.processor.spatial_dims[1]]) and ( | ||
| last_width == new_width | ||
| ): | ||
| return | ||
|
|
||
| self.processor.display_window = new_width | ||
| # set the `p` dim on the global index vector | ||
| self._ref_index[self.processor.spatial_dims[1]] = new_index |
There was a problem hiding this comment.
similar to other comment, I think we should have an NDTimeseries subclass of NDPositions 🤔
| slider_dim_transforms=None, | ||
| name: str = None, | ||
| ): | ||
| """ |
There was a problem hiding this comment.
What do you think about having the detailed docstring of the arguments only in the corresponding NDGraphic, ex: in NDImage, and not in the NDImageProcessor since people will usually look at the NDImage arguments (via the auto subplot.add_nd_<...> described in the PR comment)
| colors: ( | ||
| Sequence[str] | np.ndarray | Callable[[slice, np.ndarray], np.ndarray] | ||
| ) = None, | ||
| # TODO: cleanup how this cmap stuff works, require a cmap to be set per-graphic | ||
| # before allowing cmaps_transform, validate that stuff makes sense etc. | ||
| cmap: str = None, # across the line/scatter collection | ||
| cmap_each: Sequence[str] = None, # for each individual line/scatter | ||
| cmap_transform_each: np.ndarray = None, # for each individual line/scatter | ||
| markers: np.ndarray = None, # across the scatter collection, shape [l,] | ||
| markers_each: Sequence[str] = None, # for each individual scatter, shape [l, p] | ||
| sizes: np.ndarray = None, # across the scatter collection, shape [l,] | ||
| sizes_each: Sequence[float] = None, # for each individual scatter, shape [l, p] | ||
| thickness: np.ndarray = None, # for each line, shape [l,] |
There was a problem hiding this comment.
What do you think of these args for the other graphic features. l is the "stack" dimension (i.e. number of lines or scatters in a collection or stack), p is the datapoints dim.
Also I think I will change colors to this:
colors: np.ndarray, # of shape [l,], specify color for each line/scatter
colors_each: np.ndarray, # of shape [l, p], specify color for every datapoint of every line/scatter| # decorator to block re-entrant set_value methods | ||
| # useful when creating complex, circular, bidirectional event graphs | ||
| def set_value_wrapper(self: GraphicFeature, graphic_or_key, value): | ||
| def set_value_wrapper(self: GraphicFeature, graphic_or_key, value, **kwargs): |
There was a problem hiding this comment.
the kwarg is used by the LinearRegionSelectionFeature
|
|
||
| @block_reentrance | ||
| def set_value(self, selector, value: Sequence[float]): | ||
| def set_value(self, selector, value: Sequence[float], *, change: str = "full"): |
There was a problem hiding this comment.
For the new HistogramLUTTool, even logic was much easier if we know whether both vmin, vmax changed or only one of them.
| ] | ||
|
|
||
| def __init__(self, data, isolated_buffer: bool = True): | ||
| def __init__(self, data): |
There was a problem hiding this comment.
an isolated buffer is always created now. There really isn't a usecase for non-isolated buffer IMO, user should use OOC rendering if the data is huge. In NDWidget the final array copy is also very fast anyways because we're always only ever looking at a slice of large data.
There was a problem hiding this comment.
This is a complete rewrite, the git diff viewer is useless lol
|
After speaking with Edoardo I now see a clear use case for a slider dim inverse transforms and the |

Internals
RangeContinuous
Just a dataclass that stores the start, stop, and step of a reference range. A reference range is usually in scientific units, such as seconds/ms for time. Depth is another example of a reference range. A reference range just specifies the min and max of a dimension that will be used as a slider dimension. The imgui sliders use the reference range to determine the min, max, and step size of the UI elements (sliders, step-one button, etc.).
RangeDiscreteis mostly a placeholder for now, not fully implemented yet, waiting for a proper usecase. Genes might be an example? Session index for multi-session data could be another potential usecase.Users must define a reference range for every dimension they want to use as a slider dimension.
TODO: Would be useful to have an automatic reference range that's generated for a slider dimension when it's not specified. For example, sometimes you just want to dump a bunch of calcium arrays and look at them, and they're already in the same time-space. Could have a simple auto-reference range mode that just sets the min and max for eacher slider dim based on the current arrays in the
NDWidget.ReferenceIndex
This manages the reference index for an
NDWidget. It's the only place where indices should be set.TODO: pushing/popping reference ranges.
NDProcessor
This manages n-dimensional data of any type. Subclasses must implement a
get()method which takes a dict mapping dim names -> index for that dim. If specified, window functions are applied on slider dimensions._get_slider_dims_indexertakes the indices dict and outputs a dict of dim_names -> slice._apply_window_functions()uses these slices and applies the window functions. The final output after window functions has the same number of dims as the data, but with 1 element in any dimensions that were sliced (i.e. all slider dims, spatial dims remain untouched). In the end, theget()method must return an array that can be directly mapped to graphic data.Key properties in an
NDProcessor:data- usually an xarray, but subclasses can manage data that is of any type. The NDProcessorNDPP_Pandasfor positional data stored in a pandas dataframe is an example of this.shape,ndim,dims,spatial_dims,slider_dims- self explanatory, dims are always namedwindow_funcs- window functions, dict {dim_name: (func, window_size)}. A window function must takeaxisandkeep_dimsas kwargs.slider_dim_transforms- dict of functions for each slider dim that maps a reference range value to the local array index value. If a user doesn't provide a transform for a dim (for example, they just want to look at many arrays which are already in the same time-space, such as calcium movies) the it just uses an identify mapping.NDPositionsProcessorandNDImageProcessorhave logic specific to those types of data.NDPositionsProcessorhas an additionaldisplay_windowproperties which acts on thedatapointsdimension. Thedatapointsdim is also both a slider dim and spatial dim, but it is excluded forwindow_funcs.datapoints_window_funcis a separate property onNDPositionsProcessorthat applies window funcs for that dimension since it's a bit different from other slider dims.NDGraphic
A n-dimensional graphical representation. It has an
NDProcessorandfpl.Graphicthat it manages.ReferenceIndexsets the indices on anNDGraphic, which then callget()on itsNDProcessorto retrieve the new graphic data, and then sets it on the graphic. It has property aliases for the various processor properties, so that users can set window funcs, slider dim transforms, etc.NDPositionsandNDImageare twoNDGraphicsubclasses that have specific code for positional and image specific representations.NDPositionsalso manages a linear selector for timeseries representations, and allows swapping between scatter, line stack, line collection, and heatmap representations. It also manages an "auto x range" mode for timeseries representationsNDImagealso manages theHistogramLUTTooland swapping betweenImageGraphicandImageVolumeGraphic. It also sets the camera and controller when these are swapped.NDWSubplot
The main entry point for users to add
NDGraphicsto a subplot in anNDWidget. It mainly hasadd_nd<...>methods for users to provide data and kwargs to addNDGraphics.Need to cleanup this subclass so the args and kwargs for each add method are properly specified and documented.
NDWidget
Just a very simple class that has a
fpl.Figure, an instance ofReferenceIndex, and theNDWSubsplots.Very simplified diagram:
s1, ... snare slider dims,d1, d2, ...are spatial dims.User API
The user mainly interacts via the NDW subplots and
NDGraphics. For example with images:Positional data:
Main things left todo
NDProcessor.get()returns empty arraysReferenceIndexReferenceIndexto also ignore it.xarraydep.pandasis anxarraydependency.fastplotlib.ndwidget?get()ReferenceIndex->ReferenceIndices, it manages multiple indicesindices->ref_indices, keep constructor and properties symmetric like the rest of fplimplements #951