How can I avoid rounding off values in C?

You can't avoid rounding. I repeat: you cannot avoid it. It is not possible.

There are only a certain number of bits in a float.

Here's the decimal version of the problem if we pretend that floats are in decimal instead of binary:
When I write if(x + y + z == 1) does 1/3 + 1/3 + 1/3 == 1 ? No, not according to the computer, because 1/3 is 0.3333333 with only a certain number of digits and it adds up to 0.9999999, not 1.

It is quite common to just add some "tolerance range" as a quick workaround. Instead of if(x + y + z == 1) you might write if(x + y + z > 0.9999 && x + y + z < 1.0001) and this is often good enough, especially for computer games. How many 9's and 0's do you need? Well, just write a few and then play the game and see if it feels right.

For some applications this might not be suitable, and then you have to invent a completely different way to do whatever you are trying to do. For example, you might store all your numbers as rational numbers (numerator and denominator, a.k.a. fractions) instead of floating-point. Rational numbers can be calculated exactly - if the numerator and denominator don't overflow. C doesn't have rational numbers built in, so you would need to write your own library functions like struct rational, void rational_add(struct rational *a, struct rational *b) and so on.

There are two issues here, and the answer to neither of them is that you want to try to somehow avoid doing any rounding. In fact, you're probably going to need to do some well-chosen rounding.

The first issue is that no finite-precision floating-point representation can represent every fraction exactly. It's especially true that it's impossible to represent irrational numbers like sqrt(233) exactly.

You tried to test your program on a triangle with sides 8, 13, and the square root of 233. Mathematically, that's a perfect right triangle, but it's impossible to ask your program to test that right triangle, because you literally can't say "the square root of 233" when you ask it. You certainly can't say 155.26. You can try 15.26434, but that's inaccurate, as is 15.264337522473748, or 15.2643375224737480252559487. No finite representation is ever going to be perfectly accurate.

And then the second problem is that the inherent imprecision in the representation of most fractions means that you're rarely going to find that, say, x*x + y*y is exactly equal to z*z. This is an example of a comparison for floating-point equality that you probably don't want to try to make. (You will often hear it said that you should "never compare floating-point numbers for equality", which is not too bad as a rule of thumb, but there's a lot more to be said.)

You tried updating your comparison code to

if(x*x + y*y - z*z > 0.999 && x*x + y*y - z*z < 1 || … )

but that's not quite right, either. If the quantities x*x + y*y and z*z are almost equal, their difference will be close to 0, although it could end up on either side. So what you were trying to do is more like

if(x*x + y*y - z*z > -.001 && x*x + y*y - z*z < 0.001 || … )

and this might actually work, to a point. You could simplify it (avoid the repeated subexpression) by writing

if(fabs(x*x + y*y - z*z) < 0.001 || … )

Using a fixed accuracy threshold like 0.001 like this doesn't work particularly well, however. Some better, relative approaches can be found in this question's other answers, or in question 14.5 in the C FAQ list.

Also, as another rule of thumb, you should almost never use type float. You should almost always use type double, which has roughly twice the precision, and will tend to give you far fewer headaches.

To compare floating point values, you should compare equality with a given tolerance. Unless the scale of the numbers are known, it is important to apply the tolerance to a "normalized" expression of the difference between the numbers.

For example, fabs(a-b) < tolerance only gives the desired result if a and b are close to 1. A better alternative is fabs(a/b)-1.0, however, that assumes that b is not 0 or very close to 0. In this case, I assume that the operands are not zero (or close to zero), and thus a solution is as follows:

#include <stdbool.h>
#include <math.h>

bool IsRightTriangle( float x, float y, float  z)
 {
    float tolerance = 0.0001;
    int result;
    
    return (fabs((x*x + y*y) / (z*z) - 1.0) < tolerance ||
        fabs((x*x + z*z) / (y*y) - 1.0) < tolerance ||
        fabs((y*y + z*z) / (x*x) - 1.0) < tolerance);
}