Skip to content

Disappointing Converter behavior #197

Open

Description

@hinerm

I am starting with this 16-bit image:
image

If I convert it with Image > Type > 8-bit in Fiji I get:
image

If I denoise this image I get this result (which I'm happy with):
image

But if I try to convert it with e.g. result = ops.unary("convert.int8").input(img).apply() I get.. a 32-bit image??
image

convert.uint8 is 8-bit, at least..
image

Smoothing the uint-8 version requires bumping up the smoothing strength significantly and it still doesn't look great:
image

So, how I account for this behavior is:

  • In the ImageJ type conversion, values are scaled proportionately to the new range
  • In the SJ Ops type conversion, values are directly translated. This can result in type overflow (note the split tails in both the int8 and uint8 conversions) which fundamentally changes the behavior of subsequent image processing.
  • These byte casts are perhaps not working as intended?

@gselzer @ctrueden Is there rationale for not scaling the converters to the target data range?

Activity

  1. added this to the 1.0.0 milestone on Mar 12, 2024
  2. ctrueden commented on Mar 12, 2024

    @ctrueden
    Member

    @hinerm The first problem is due to limitations of the original ImageJ. Image → Type → 8-bit converts to uint8, not int8. When you run the convert.int8 op, you end up with a pixel type not supported by the original ImageJ, so it defaults to coercing it into 32-bit floats. We could potentially improve this for certain special cases, like int8 becoming ByteProcessor with a calibration offset so that it behaves as signed, although there are probably wrinkles here to consider. ImageJ is not really designed to support int8, fundamentally, so some maths may be broken in this circumstance, I'm not sure.

    As for convert.uint8, another reason it looks bad is due to type casting versus clamping. There were big conversations I had with Alison Walter years ago about how this behavior should work, and IIRC we ended up settling on something that let you choose either behavior? But it's been a long time, and I'd need to revisit it to be sure. Certainly the raw type casting is much faster to execute, since you don't need to perform two numeric tests per pixel. But I agree that raw type casting is probably the wrong thing, expectations wise, the majority of the time. And neither raw type casting nor clamping will scale the data to the new data type. That should be a different op, in my opinion—it's not just a type conversion anymore at that point. IIRC, ImageJ Ops has a convert.scale but the design of it wasn't super great, so we should probably rediscuss how we want these sorts of operations to work.

  3. changed the title [-]Disappointing Converter performance[/-] [+]Disappointing Converter behavior[/+] on Mar 12, 2024
  4. gselzer commented on Mar 12, 2024

    @gselzer
    Member

    That should be a different op, in my opinion—it's not just a type conversion anymore at that point. IIRC, ImageJ Ops has a convert.scale but the design of it wasn't super great, so we should probably rediscuss how we want these sorts of operations to work.

    Yeah, I do agree that the best way forward is to make separate Ops for scale vs cast vs clamp, because this is how it worked in ImageJ Ops. I'd also note that NumPy's main mechanism for converting data types, numpy.ndarray.astype, also casts without scaling:

    >>> np.array([10000000], dtype=np.int32).astype(np.int8)
    array([-128], dtype=int8)
    >>> np.array([10000001], dtype=np.int32).astype(np.int8)
    array([-127], dtype=int8)
    >>> np.array([10000002], dtype=np.int32).astype(np.int8)
    array([-126], dtype=int8)
  5. hinerm commented on Mar 12, 2024

    @hinerm
    MemberAuthor

    I was looking around and OpenCV does have a set of normalize methods that we could wrap to Ops.

    Caveats:

    • They have optional parameters which I'm pretty sure wouldn't be captured
    • Conversion to Mats is limited and requires lots of copying.
  6. hinerm commented on Mar 12, 2024

    @hinerm
    MemberAuthor

    Current plan:

    • Add normalize ops, e.g. see those available in OpenCV. Written as a computer that takes an I input and O output and uses the O type min/max for normalizing.
    • Rename current type convert ops to be called cast
    • Possibly refactor current convert.clip Op that accepts a min and max, and make all params I. As a Computer<T extends Type> and/or Function<T extends Comparable>, renamed to math.clip
  7. self-assigned this
    on Mar 12, 2024
  8. ctrueden commented on Mar 14, 2024

    @ctrueden
    Member

    One other thing that occurred to me about the needed clip ops: a most general version could have separate T min, T max, T belowMinValue, T aboveMaxValue arguments, in case you want to do things like NaN out everything out of range. Even more general would be Predicate<T> boundsTest, T outOfBoundsValue or even multiple of those... but it starts to get ridiculous of course.

  9. hinerm commented on Apr 1, 2024

    @hinerm
    MemberAuthor

    It turns out there is a suitable op: image.normalize.

    It works for my use case. I do find it a little unwieldy. I'd like to see overriding signatures variations/optionals, e.g.:
    -[ ] No input sourceMin/sourceMax, instead using stats.min and stats.max to get the display min/max values
    -[ ] No targetMin/targetMin, instead having a signature that accepts an O and uses the getMinValue/getMaxValue of that type.
    -[ ] Could also just omit the O and pull it from the output in the case of a computer

    None of these are absolutely required, just sugar that I think would be nice for users

  10. modified the milestones: 1.0.0, unscheduled on Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions