silent payments: add sending support - #64
Conversation
Any particular reason to use |
I was considering some beneficial aspects of |
I see. Well I admit the |
|
Is there any reason to not upstream algos like coingrinder to |
Well, when I started work on the first coin-selection lib, there was no such thing as BDK.
I would argue that bitcoin-coin-selction in some ways is better reviewed than bdk-coin-select. Murch has been kind enough to review my BnB, SRD and Coin-Grinder implementations over the years (and other areas as well).
For sure, BDK is a big project and popular framework (in bitcoin land anyway), especially if you just want results fast. I've worked for a few startups in the past that used BDK, and for them it made sense because they just wanted to get stuff off the ground, quick.
For sure. I'm not super familiar with bdk-tx nor bdk-sp.. Understandably keeping within the bdk ecosystem may have some ease of use benefits. Although, an open source project like this could use more specialized tooling than one might use if running a startup. |
And I guess lastly I'd add that I upstream improvements back to bitcoin-core instead, which there have been a few here and there. |
967e3c8 to
9fd92f7
Compare
|
Pushed an update, folded into the two commits:
|
| } | ||
|
|
||
| fn cs_feerate(fee_rate: FeeRate) -> bdk_coin_select::FeeRate { | ||
| bdk_coin_select::FeeRate::from_sat_per_wu(fee_rate.to_sat_per_kwu() as f32 / 1000.0) |
There was a problem hiding this comment.
What version of bitcoin are they on? Is this not the bitcoin::FeeRate type?
Not your fault, but this reads horrible and looks prone to a fee-high footgun.
There was a problem hiding this comment.
They are on 0.32. I've also noticed they have been slow to use Bitcoin types. In someways that has been a frustration for me (using bitcoin types) since things like Amount don't perform well compared to native types. I've spent a lot of time in optimizations so that the boundary points are Bitcoin types but internally sometimes revert to more primitive types when performance matters. The other annoying thing is bdk select has no bench-marking or fuzz testing either.
There was a problem hiding this comment.
If I'm understanding this right, it looks like they use their own separate FeeRate (not a rust-bitcoin type): https://github.com/bitcoindevkit/coin-select/blob/master/src/feerate.rs
9fd92f7 to
ae5409c
Compare
| let spendable: Vec<&SpendableCoin> = coins.iter().collect(); | ||
| let candidates: Vec<Candidate> = spendable | ||
| .iter() | ||
| .map(|c| Candidate::new_tr_keyspend(c.coin.value.to_sat())) |
There was a problem hiding this comment.
From what I've read about Silent payments, the receiver is always a taproot output. However, I believe the sender can use any output type to send, right? Why then is every candidate output using tr_keyspending (assuming that tr is Tap Root here). In effect, I think you are telling the selection algo that all inputs are taproot inputs, but I don't think this is necessarily correct.
There was a problem hiding this comment.
That is true in general but for this wallet the coins being spent here are exclusively silent payment outputs received by scanning, so every coin in this wallet is new_tr_keyspend, but if the wallet later holds other input types the candidate weight would need to vary per input.
There was a problem hiding this comment.
I see. Good to know, that does make the spend case in this wallet easier to handle.
There was a problem hiding this comment.
I would also verify that this is default taproot sighash size vs non-default. It's only a byte difference, but if the wallet is for some reason creating non-default, that could add up in the case where lots of inputs are used.
| bdk_coin_select::FeeRate::from_sat_per_wu(fee_rate.to_sat_per_kwu() as f32 / 1000.0) | ||
| } | ||
|
|
||
| fn output_weight(script_len: usize) -> u64 { |
There was a problem hiding this comment.
For clarity and type safety, you might consider returning a Weight type here: https://github.com/rust-bitcoin/rust-bitcoin/blob/master/units/src/weight.rs. A u64 usually indicates you're returning something measured in vB (virtual bytes), although you are multiplying by 4, so I think this is actually a Weight Unit type.
|
|
||
| fn output_weight(script_len: usize) -> u64 { | ||
| // 8-byte value, 1-byte length prefix, then the script, times 4 weight units per byte. | ||
| ((8 + 1 + script_len) * 4) as u64 |
There was a problem hiding this comment.
as looks ok here, but generally it can be sketchy to use as since you may truncate bytes and the program would silently continue.
There was a problem hiding this comment.
Thanks. Weight::from_wu_usize will naturally resolve this.
There was a problem hiding this comment.
from_vb is what you actually want I believe (as commented bellow)
| fee: TargetFee::from_feerate(feerate), | ||
| outputs: TargetOutputs::fund_outputs([(recipient_weight, amount.to_sat())]), | ||
| }; | ||
| let change_policy = ChangePolicy::min_value(DrainWeights::TR_KEYSPEND, change_dust.to_sat()); |
There was a problem hiding this comment.
I'm confused by the use of change_dust here. Generally, the ChangePolicy (see also cost_of_change in bitcoin core) is the cost to create a change output. The reason this matters is that the bnb solution can be exceeded by at most cost_of_change before the solution needs to worry about what to do with the excess. If a solution is found that is greater than the target and less than target + cost to produce a change output, then the solution is considered acceptable since there is not enough leftover to worry creating a change output. If target + cost of change is exceeded, then a different method can be used, like SRD. Instead, you are using change_dust and I'm wondering if this was intentional?
There was a problem hiding this comment.
Got it, my intention was just to avoid creating dust change outputs but it looks like I can use ChangePolicy::min_value_and_waste
There was a problem hiding this comment.
I can use ChangePolicy::min_value_and_waste
here is what the README.md says about that min_value_and_waste.
// We use a change policy that introduces a change output if doing so reduces
// the "waste" (i.e. adding change doesn't increase the fees we'd pay if we factor in the cost to spend the output later on).
That does sound like the correct behavior, since the ultimate goal of coin-selection is to reduce the waste metric. Murch wrote a nice blog post about this: https://murch.one/posts/waste-metric/.
BTW, the waste metric to work "properly" needs to know the correct long_term_fee_rate. Currently, it looks like you have that hard coded as: const LONG_TERM_FEERATE_SAT_PER_VB: f32 = 1.0;. So, that will give you wildly inaccurate results in some cases because the algorithm will most likely always think it's in a high fee rate environment when in fact it may not be. Hopefully that all makes sense.
There was a problem hiding this comment.
That does makes sense. Setting aside an implementation / integration of a fee estimator for a follow up, perhaps it then makes sense to do long_term_fee_rate = current_fee_rate so the waste metric is neutral rather than an assumed future.
There was a problem hiding this comment.
Yeah if it's not handy to find long_term_fee_rate now, some note to followup would be great. In the very least maybe in the commit message.
There was a problem hiding this comment.
Any yes, long_term_fee_rate = current_fee_rate would at least be better than hardcoded 1.
There was a problem hiding this comment.
It would probably be best to just use a selection process that doesn't require knowledge of long_term_fee_rate. If the goal is just to get something out there that works, SRD (single random draw) would be simple, fast and doesn't require knowledge of cost_of_change nor long_term_fee_rate. Then at some later point BnB and/or coin-grinder could be added.
| Recipient::SilentPayment(sp) => sp_output_script(&derived, sp)?, | ||
| }; | ||
| let mut output = vec![TxOut { | ||
| value: amount, |
There was a problem hiding this comment.
If im not mistaken, you are creating a change output that is the size of the transaction target amount. The function build_transaction takes a variable amount, and I don't see that amount be changed, then it's used to create a change output. This should either be a changeless transaction in which case there is no change output, or the change should be the excess of what was found by the selection algorithm.
There was a problem hiding this comment.
s/be changed/is changed
There was a problem hiding this comment.
That's the recipient output, not the change. The first output pays the recipient, so it's amount:
let mut output = vec![TxOut {
value: amount,
script_pubkey: recipient_script,
}];
The change is the next output, with value change_value:
if let Some(value) = change_value {
output.push(TxOut {
value,
script_pubkey: sp_output_script(&derived, change_address)?,
});
}
change_value is the drain value from the selector:
let change_value = drain.is_some().then(|| Amount::from_sat(drain.value));
which comes from selector.drain:
let drain = selector.drain(target, change_policy);
So when drain is None, change_value is None and no change output is added, which is the changeless case. When drain is Some, the change output value is drain.value, the leftover from selection, not amount.
There was a problem hiding this comment.
That's the recipient output, not the change. The first output pays the recipient, so it's amount:
Ah got it. Thanks, that makes sense.
There was a problem hiding this comment.
And yes, that all tracks with me as well. You'll have a changeless solution, which has only the amount sent to the recipient. Then in the case of a change, there will be a second output that goes back to sender. The address that goes back to sender looks like what's provided here sp_output_script(&derived, sp)sp_output_script(&derived, change_address)? and the recipient goes to the scriptpub key here address.script_pubkey()?
There was a problem hiding this comment.
Yes. The recipient is address.script_pubkey() for a plain address, or sp_output_script(&derived, sp) for a silent payment one.
ae5409c to
71cfe92
Compare
|
|
||
| fn output_weight(script_len: usize) -> Weight { | ||
| // 8-byte value, 1-byte length prefix, then the script, times 4 weight units per byte. | ||
| Weight::from_wu_usize((8 + 1 + script_len) * 4) |
There was a problem hiding this comment.
I would probably not depend on from_wu_usize mostly because it is not available in units https://github.com/rust-bitcoin/rust-bitcoin/blob/master/units/src/weight.rs. Hopefully units 1.0 is around the corner.
Anyway, assuming this is only for 32 and 64 bit architectures, id probably do something like
let vb = u64::try_from(8 + 1 + script_len).unwrap();
Weight::from_vb_unchecked(vb)There should probably be some policy that says we don't guarantee no panic paths if someone is using an arch with more than 64 bits.
|
Were you able to test that this works? I tried broadcasting a transaction and the CLI returns, but looking the logs (in It would be nice to have some transaction validation prior to broadcast so we know the broadcast code is buggy. Something like |
|
Also probably good for people to know, you can get signet sats here. (please send them back once we figure this out) |
The testing I did prior was pushing a transaction through
Good call, for right now I will add a check that verifies each signature against the output it spends before broadcast. |
|
What I'm finding is that Faucet payment into the wallet, received and scanned, block 309134: silently dropped on the immediate disconnect and then propagated once the connection was held open, block 309194: same but with a hack-y "fix" I was testing, propagated and mined in block 309195: |
What was the fix? I think we can do something like send a ping and wait for the pong. This is done in private broadcast to my knowledge. |
I'm holding the connection open for a few seconds after sending so the peer can process and relay: Seems ping-pong might accomplish the same thing but in a more signal-driven way. Two larger improvements that are not immediately pertinent, but are worth tracking separately 1) announcing the transaction before sending it or 2) relaying over the connection the node already keeps open. I can do a short write up in #51 |
|
1/ is a nice-to-have but technically not required and I don't think would help in this problem 2/ was deliberately avoided. I plan on adding a Socks5 proxy so we can connect to peers over Tor, however we will likely want to do IBD without Tor, so the current design is flexible to broadcast transactions privately without slowing down IBD. |
|
Makes sense on both. I started thinking along the lines of "how long do I keep the socket open." I implemented the ping-pong approach you described in #66 . Tested on signet, the spend propagated and confirmed in block 309302: |
|
Can be rebased on #66 |
Spending a received output needs its private key, the spend secret plus the tweak recorded when the coin was scanned. Store the spend secret, normalised to the even Y form the receiver derives against, so the spend secret plus the stored tweak reconstructs each coin key. Add build_transaction, which selects coins, builds the recipient and change outputs, and signs each input as a taproot key spend. The recipient may be a silent payment address or a normal bitcoin address. The transaction is locked to the current chain tip to discourage fee sniping. Coins chosen for a spend are reserved so a later spend cannot reuse them. Coin selection uses bdk_coin_select. The transaction is assembled and signed with the bitcoin crate.
Add a sendToAddress RPC and matching CLI command that take a recipient address, an amount, and a fee rate, build the transaction, broadcast it, and return the txid. The recipient may be a silent payment address or a normal bitcoin address. Keep the spend secret at startup when the keys file carries one so the running wallet can sign. A file holding only the public spend key stays watch only and the RPC reports that it cannot sign. The broadcast worker releases the reserved coins when delivery fails, so a transaction that never reached a peer does not lock its inputs until the node restarts.
71cfe92 to
b2c35e4
Compare
|
For reference:
|
|
Yes, also tested it: https://mempool.space/signet/tx/002ea86e3a01d394f040c4dbf58ddd0f525cbe1257559d6aeb85cc1e3c2a6b34 Will send signet sats back. :) |
rustaceanrob
left a comment
There was a problem hiding this comment.
I would like to continue moving things along. As a follow up we can:
- Implement RBF, this a big one because locked coins can be annoying
- Make the coin selection more robust.
- Add roundtrip tests with Bitcoin Core to make sure we're all square
Great work @pzafonte!
It doesn't look like that's the case. Also the LONG_TERM_FEE_RATE is still hard coded as |
|
Oh shoot, that push didn't go through. I pushed up on the same branch. |
Adds a sendToAddress RPC and CLI command that take a silent payment address, an amount, and a fee rate, then build, sign, and broadcast the transaction.
Notes:
bdk_coin_select. It runs branch and bound to minimize the fee and falls back to a largest first pass when no solution is found.