Beside the notes given in another answer (using is/2 instead of =/2 and the typos in the name of the terms t->tree and e->empty), you should consider using an accumulator to allow the prolog system to do tail recursion:
rbt_count_nodes(T,N):-
rbt_count_nodes(T, 0, N).
rbt_count_nodes(nil,N, N):-!.
rbt_count_nodes(tree(_,L,R),C,N):-
succ(C, C1),
rbt_count_nodes(L,C1,NL),
rbt_count_nodes(R,NL,N).
Procedure rbt_count_nodes/2 just calls rbt_count_nodes/3 with zero as the second argument (accumulator). Upon each recursion step, this accumulator gets incremented by one and does recursion. The base case just unifies the accumulator with your output allowing the system to do tail recursion and economize on stack space.
[edit]
As per comments, to make it truly tail recursive (that is, all recursive calls are tail calls) you can add a new clause to rbt_count_nodes/3 (and modify another):
rbt_count_nodes(nil,N, N):-!.
rbt_count_nodes(tree(Node,tree(LNode, LL, LR),R),C,N):-
rbt_count_nodes(tree(Node, LL, tree(LNode, LR, R)), C, N).
rbt_count_nodes(tree(_,nil,R),C,N):-
succ(C, C1),
rbt_count_nodes(R,C1, N).
With this approach, the first clause deal with an empty node, the second one deals with the left side of the tree which just "moves" the branch to the right hand side of the current node, and the third clause just computes the count of nodes.