Remote backup failing on ssh

[Doosty]

Hi, i love your project i have been using it for local backups however when i try remote it fails. This is what i did:

• i added the hostnames to /etc/hosts pointing to local ip because initially it was resolving the ip of my public cloud vps machine
• i used --nbd-ip just in case
• i have ssh key authentication configured for the remote root user so passwordless ssh connection works

The backup command still fails. Do i need to do anything more?

myuser in nixos-desktop-01 in /tmp/tmp.gfTYuVn0fk
❯ cat /etc/hosts
127.0.0.1 localhost
::1 localhost
127.0.0.2 nixos-desktop-01.mydomain.com nixos-desktop-01
10.10.50.94 nixos-desktop-02.mydomain.com nixos-desktop-02

myuser in nixos-desktop-01 in /tmp/tmp.gfTYuVn0fk
❯ doggo nixos-desktop-02.mydomain.com
NAME                                 	TYPE	CLASS	TTL	ADDRESS    	NAMESERVER
nixos-desktop-02.mydomain.com.	A   	IN   	0s 	10.10.50.94	127.0.0.53:53	

myuser in nixos-desktop-01 in /tmp/tmp.gfTYuVn0fk
❯ ssh root@nixos-desktop-02.mydomain.com
Last login: Fri Jan 10 12:20:49 2025 from 10.10.70.90

root in nixos-desktop-02 in ~
❯ logout
Connection to nixos-desktop-02.mydomain.com closed.

myuser in nixos-desktop-01 in /tmp/tmp.gfTYuVn0fk took 3s
❯ virtnbdbackup --nbd-ip 10.10.50.94 -U qemu+ssh://root@10.10.50.94/system --ssh-user root -d win7-02 -o /tmp/tmp.gfTYuVn0fk/win7-02
[2025-01-10 12:25:16] INFO lib common - printVersion [main]:  Version: 2.18 Arguments: /nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin/virtnbdbackup --nbd-ip 10.10.50.94 -U qemu+ssh://root@10.10.50.94/system --ssh-user root -d win7-02 -o /tmp/tmp.gfTYuVn0fk/win7-02
[2025-01-10 12:25:16] INFO root .virtnbdbackup-wrapped - main [main]:  Backup level: [copy]
[2025-01-10 12:25:16] INFO virt client - _connect [main]:  Connected to remote host: [nixos-desktop-02.mydomain.com]
[2025-01-10 12:25:16] INFO root .virtnbdbackup-wrapped - main [main]:  Libvirt library version: [10010000]
[2025-01-10 12:25:16] INFO root .virtnbdbackup-wrapped - main [main]:  NBD library version: [1.20.2]
[2025-01-10 12:25:16] INFO root .virtnbdbackup-wrapped - main [main]:  Backup will save [1] attached disks.
[2025-01-10 12:25:16] INFO root .virtnbdbackup-wrapped - main [main]:  Concurrent backup processes: [1]
[2025-01-10 12:25:16] INFO root checkpoint - create [main]:  Loading checkpoints from: [/tmp/tmp.gfTYuVn0fk/win7-02/win7-02.cpt]
[2025-01-10 12:25:16] INFO root checkpoint - redefine [main]:  Loading checkpoint list from: [/tmp/tmp.gfTYuVn0fk/win7-02/checkpoints]
[2025-01-10 12:25:16] INFO root checkpoint - create [main]:  Checkpoint handling.
[2025-01-10 12:25:16] INFO ssh client - connect [main]:  Connecting remote system [nixos-desktop-02.mydomain.com] via ssh, username: [root]
[2025-01-10 12:25:16] INFO paramiko.transport transport - _log [Thread-1]:  Connected (version 2.0, client OpenSSH_9.9)
[2025-01-10 12:25:17] WARNING lib common - sshSession [main]:  Failed to setup SSH connection: [could not get keys from ssh-agent]
[2025-01-10 12:25:17] ERROR root .virtnbdbackup-wrapped - main [main]:  Remote backup detected but ssh session setup failed

[abbbi]

Failed to setup SSH connection: [could not get keys from ssh-agent]

the ssh key that is used for root@10.10.50.94 seems to have a password set but no ssh-agent is running to get the password from
Seems the wrong key is used?

what looks strange to me is the:

root .virtnbdbackup-wrapped

that class is not part of the project itself, i wonder where it comes from?

Also: you can try the verbose option "-v" to show the paramiko output and get more information what ssh key is used
for authentication. The debug option should help to analyse this in detail.

[Doosty]

Ah the -v shows that it is still contacting my public cloud vps even after adding an entry to /etc/hosts and using the --nbd-ip flag. To clarify the example mydomain.com resolves through public dns servers as the ip address to my public cloud vps server. At the same time i also add that same mydomain.com as the networking domain name on my local devices. So when local devices resolve that name they get the public ip address. It doesnt seem to matter that the fully qualified domain name looks like this my-local-device.mydomain.com, it still resolves to the public ip. I guess ill need to remove this domain name from my local devices or configure my local dns server to resolve that hostname locally.

I do not use passwords on ssh keys, i do not use ssh-agent and the -wrapped thing is not relevant, it is just how nixos builds software (wraps it into a script that exports the proper PATH with the specific runtime deps that the app needs).

Thanks for the help, ill mess with the dns some other time, local backups are sufficient for now.

myuser in nixos-desktop-01 in /tmp/tmp.LOFlzaYces
❯ virtnbdbackup --nbd-ip 10.10.50.94 -v -U qemu+ssh://root@10.10.50.94/system --ssh-user root -d win7-02 -o /tmp/tmp.gfTYuVn0fk/win7-02
[2025-01-10 20:33:49] INFO lib common - printVersion [main]:  Version: 2.18 Arguments: /nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin/virtnbdbackup --nbd-ip 10.10.50.94 -v -U qemu+ssh://root@10.10.50.94/system --ssh-user root -d win7-02 -o /tmp/tmp.gfTYuVn0fk/win7-02
[2025-01-10 20:33:49] INFO root .virtnbdbackup-wrapped - main [main]:  Backup level: [copy]
[2025-01-10 20:33:49] DEBUG output target - create [main]:  Create: /tmp/tmp.gfTYuVn0fk/win7-02/checkpoints
[2025-01-10 20:33:49] DEBUG virt client - _connect [main]:  Libvirt URI: [qemu+ssh://root@10.10.50.94/system]
[2025-01-10 20:33:49] INFO virt client - _connect [main]:  Connected to remote host: [nixos-desktop-02.myuserrah.com]
[2025-01-10 20:33:49] INFO root .virtnbdbackup-wrapped - main [main]:  Libvirt library version: [10010000]
[2025-01-10 20:33:49] INFO root .virtnbdbackup-wrapped - main [main]:  NBD library version: [1.20.2]
[2025-01-10 20:33:49] DEBUG virt client - getDomainDisks [main]:  Disk [sda]: file notation
[2025-01-10 20:33:49] DEBUG virt client - getDomainDisks [main]:  Device list: [DomainDisk(target='sda', format='qcow2', filename='win7-02.qcow2', path='/var/lib/libvirt/images/win7-02.qcow2', backingstores=[])]
[2025-01-10 20:33:49] DEBUG virt client - getDomainInfo [main]:  No setting [loader] found: 'NoneType' object has no attribute 'text'
[2025-01-10 20:33:49] DEBUG virt client - getDomainInfo [main]:  No setting [nvram] found: 'NoneType' object has no attribute 'text'
[2025-01-10 20:33:49] DEBUG virt client - getDomainInfo [main]:  No setting [kernel] found: 'NoneType' object has no attribute 'text'
[2025-01-10 20:33:49] DEBUG virt client - getDomainInfo [main]:  No setting [initrd] found: 'NoneType' object has no attribute 'text'
[2025-01-10 20:33:49] DEBUG virt client - getDomainInfo [main]:  Domain Info: [{}]
[2025-01-10 20:33:49] INFO root .virtnbdbackup-wrapped - main [main]:  Backup will save [1] attached disks.
[2025-01-10 20:33:49] INFO root .virtnbdbackup-wrapped - main [main]:  Concurrent backup processes: [1]
[2025-01-10 20:33:49] INFO root checkpoint - create [main]:  Loading checkpoints from: [/tmp/tmp.gfTYuVn0fk/win7-02/win7-02.cpt]
[2025-01-10 20:33:49] INFO root checkpoint - redefine [main]:  Loading checkpoint list from: [/tmp/tmp.gfTYuVn0fk/win7-02/checkpoints]
[2025-01-10 20:33:49] INFO root checkpoint - create [main]:  Checkpoint handling.
[2025-01-10 20:33:49] DEBUG root checkpoint - create [main]:  Checkpoint info: [{'name': 'virtnbdbackup.0', 'parent': '', 'file': '/tmp/tmp.gfTYuVn0fk/win7-02/win7-02.cpt'}].
[2025-01-10 20:33:49] INFO ssh client - connect [main]:  Connecting remote system [nixos-desktop-02.myuserrah.com] via ssh, username: [root]
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  starting thread (client mode): 0x8c7eca70
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Local version/idstring: SSH-2.0-paramiko_3.5.0
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Remote version/idstring: SSH-2.0-OpenSSH_9.9
[2025-01-10 20:33:49] INFO paramiko.transport transport - _log [Thread-1]:  Connected (version 2.0, client OpenSSH_9.9)
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  === Key exchange possibilities ===
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, diffie-hellman-group-exchange-sha256, ext-info-s, kex-strict-s-v00@openssh.com
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  server key: rsa-sha2-512, rsa-sha2-256, ssh-ed25519
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  client encrypt: chacha20-poly1305@openssh.com, aes256-gcm@openssh.com, aes128-gcm@openssh.com, aes256-ctr, aes192-ctr, aes128-ctr
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  server encrypt: chacha20-poly1305@openssh.com, aes256-gcm@openssh.com, aes128-gcm@openssh.com, aes256-ctr, aes192-ctr, aes128-ctr
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  client mac: hmac-sha2-512-etm@openssh.com, hmac-sha2-256-etm@openssh.com, umac-128-etm@openssh.com
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  server mac: hmac-sha2-512-etm@openssh.com, hmac-sha2-256-etm@openssh.com, umac-128-etm@openssh.com
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  client compress: none, zlib@openssh.com
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  server compress: none, zlib@openssh.com
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  client lang: <none>
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  server lang: <none>
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  kex follows: False
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  === Key exchange agreements ===
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Strict kex mode: True
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Kex: curve25519-sha256@libssh.org
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  HostKey: ssh-ed25519
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Cipher: aes128-ctr
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  MAC: hmac-sha2-256-etm@openssh.com
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Compression: none
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  === End of kex handshake ===
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Resetting outbound seqno after NEWKEYS due to strict mode
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Switch to new keys ...
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Resetting inbound seqno after NEWKEYS due to strict mode
[2025-01-10 20:33:49] DEBUG paramiko.transport transport - _log [Thread-1]:  Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'}
[2025-01-10 20:33:49] WARNING lib common - sshSession [main]:  Failed to setup SSH connection: [Host key for server 'nixos-desktop-02.myuserrah.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIB/fJG0Bm8aaGd8EENFz5z/gqEZt9bSfwe4nVIAbgc7e', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIA06JlibpVlnwQYPs3zFVfXnmfile1XmPT+nsPgtgmrb']
[2025-01-10 20:33:49] ERROR root .virtnbdbackup-wrapped - main [main]:  Remote backup detected but ssh session setup failed
# COMMENT: the 'AAAAC3N....gc7e' is the ssh pub key of my public server, not the 10.10.50.94 device.

myuser in nixos-desktop-01 in /tmp/tmp.LOFlzaYces
❯ realpath $(which virtnbdbackup)
/nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin/virtnbdbackup

myuser in nixos-desktop-01 in /tmp/tmp.LOFlzaYces
❯ tree -L 3 /nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18
dr-xr-xr-x    - root root  1 jan  1970 /nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18
dr-xr-xr-x    - root root  1 jan  1970 ├── bin
.r-xr-xr-x  18k root root  1 jan  1970 │   ├── .virtnbdbackup-wrapped
.r-xr-xr-x 9,4k root root  1 jan  1970 │   ├── .virtnbdmap-wrapped
.r-xr-xr-x  11k root root  1 jan  1970 │   ├── .virtnbdrestore-wrapped
.r-xr-xr-x 4,8k root root  1 jan  1970 │   ├── virtnbd-nbdkit-plugin
.r-xr-xr-x 1,2k root root  1 jan  1970 │   ├── virtnbdbackup
.r-xr-xr-x 1,2k root root  1 jan  1970 │   ├── virtnbdmap
.r-xr-xr-x 1,2k root root  1 jan  1970 │   └── virtnbdrestore
dr-xr-xr-x    - root root  1 jan  1970 ├── lib
dr-xr-xr-x    - root root  1 jan  1970 │   └── python3.12
dr-xr-xr-x    - root root  1 jan  1970 │       └── site-packages
dr-xr-xr-x    - root root  1 jan  1970 └── nix-support
.r--r--r--  612 root root  1 jan  1970     └── propagated-build-inputs

myuser in nixos-desktop-01 in /tmp/tmp.LOFlzaYces
❯ cat /nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin/virtnbdbackup
#! /nix/store/5mh7kaj2fyv8mk4sfq1brwxgc02884wi-bash-5.2p37/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/hid1iv9yjq4xqi3gnb7pfa1gsrajm25g-libnbd-1.20.2/bin'':'/':'}
PATH='/nix/store/hid1iv9yjq4xqi3gnb7pfa1gsrajm25g-libnbd-1.20.2/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/8nfid9v93yqq6m3m564mzfxsya74sf90-python3.12-tqdm-4.66.5/bin'':'/':'}
PATH='/nix/store/8nfid9v93yqq6m3m564mzfxsya74sf90-python3.12-tqdm-4.66.5/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin'':'/':'}
PATH='/nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/c9m6yd8fg1flz2j5r4bif1ib5j20a0cy-python3-3.12.8/bin'':'/':'}
PATH='/nix/store/c9m6yd8fg1flz2j5r4bif1ib5j20a0cy-python3-3.12.8/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/j3ml2xbg6kv8fimf0mp281c6qgis92j8-virtnbdbackup-2.18/bin/.virtnbdbackup-wrapped"  "$@"