<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 25, 2021 at 1:20 PM Dylan Reid <<a href="mailto:dgreid@chromium.org">dgreid@chromium.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, Feb 23, 2021 at 8:20 AM Michael S. Tsirkin <<a href="mailto:mst@redhat.com" target="_blank">mst@redhat.com</a>> wrote:<br>
><br>
> Cc: Raphael<br>
><br>
> On Fri, Feb 19, 2021 at 04:04:34PM +0000, Alex Bennée wrote:<br>
> > Hi,<br>
> ><br>
> > I finally got a chance to get down into the guts of vhost-user while<br>
> > attempting to port my original C RPMB daemon to Rust using the<br>
> > vhost-user-backend and related crates. I ended up with this hang during<br>
> > negotiation:<br>
> ><br>
> > startup<br>
> ><br>
> > vhost_user_write req:1 flags:0x1<br>
> > vhost_user_read_start<br>
> > vhost_user_read req:1 flags:0x5<br>
> > vhost_user_backend_init: we got 170000000<br>
> > vhost_user_write req:15 flags:0x1<br>
> > vhost_user_read_start<br>
> > vhost_user_read req:15 flags:0x5<br>
> > vhost_user_set_protocol_features: 2008<br>
> > vhost_user_write req:16 flags:0x1<br>
> > vhost_user_write req:3 flags:0x1<br>
> > vhost_user_write req:1 flags:0x1<br>
> > vhost_user_read_start<br>
> > vhost_user_read req:1 flags:0x5<br>
> > vhost_user_write req:13 flags:0x1<br>
> ><br>
> > kernel initialises device<br>
> ><br>
> > virtio_rpmb virtio1: init done!<br>
> > vhost_user_write req:13 flags:0x1<br>
> > vhost_dev_set_features: 130000000<br>
> > vhost_user_set_features: 130000000<br>
> > vhost_user_write req:2 flags:0x1<br>
> > vhost_user_write req:5 flags:0x9<br>
> > vhost_user_read_start<br>
> ><br>
> > The proximate cause is the vhost crate handling:<br>
> ><br>
> > MasterReq::SET_MEM_TABLE => {<br>
> > let res = self.set_mem_table(&hdr, size, &buf, rfds);<br>
> > self.send_ack_message(&hdr, res)?;<br>
> > }<br>
> ><br>
> > which gates on the replay_ack_enabled flag:<br>
> ><br>
> > fn send_ack_message(<br>
> > &mut self,<br>
> > req: &VhostUserMsgHeader<MasterReq>,<br>
> > res: Result<()>,<br>
> > ) -> Result<()> {<br>
> > if dbg!(self.reply_ack_enabled) {<br>
> > let hdr = self.new_reply_header::<VhostUserU64>(req, 0)?;<br>
> > let val = match res {<br>
> > Ok(_) => 0,<br>
> > Err(_) => 1,<br>
> > };<br>
> > let msg = VhostUserU64::new(val);<br>
> > self.main_sock.send_message(&hdr, &msg, None)?;<br>
> > }<br>
> > Ok(())<br>
> > }<br>
> ><br>
> > which is only set when we have all the appropriate acknowledged flags:<br>
> ><br>
> > fn update_reply_ack_flag(&mut self) {<br>
> > let vflag = VhostUserVirtioFeatures::PROTOCOL_FEATURES.bits();<br>
> > let pflag = VhostUserProtocolFeatures::REPLY_ACK;<br>
> > if (self.virtio_features & vflag) != 0<br>
> > && (self.acked_virtio_features & vflag) != 0<br>
> > && self.protocol_features.contains(pflag)<br>
> > && (self.acked_protocol_features & pflag.bits()) != 0<br>
> > {<br>
> > self.reply_ack_enabled = true;<br>
> > } else {<br>
> > self.reply_ack_enabled = false;<br>
> > }<br>
> > }<br>
> ><br>
> > which from above you can see QEMU helpfully dropped those bits in the<br>
> > reply. It does however work in the C/libvhost version:<br>
> ><br>
> > virtio_rpmb virtio1: init done!<br>
> > vhost_user_write req:13 flags:0x1<br>
> > vhost_dev_set_features: 130000000<br>
> > vhost_user_set_features: 130000000<br>
> > vhost_user_write req:2 flags:0x1<br>
> > vhost_user_write req:37 flags:0x9<br>
> > vhost_user_read_start<br>
> > vhost_user_read req:37 flags:0x5<br>
> > vhost_user_write req:8 flags:0x1<br>
> > vhost_user_write req:10 flags:0x1<br>
> > vhost_user_write req:9 flags:0x1<br>
> > vhost_user_write req:12 flags:0x1<br>
> > vhost_user_write req:13 flags:0x1<br>
> ><br>
> > albeit with a slightly different message sequence<br>
> > (VHOST_USER_ADD_MEM_REG instead of VHOST_USER_SET_MEM_TABLE). Reading<br>
> > the C code you can see why:<br>
> ><br>
> > need_reply = vmsg.flags & VHOST_USER_NEED_REPLY_MASK;<br>
> ><br>
> > reply_requested = vu_process_message(dev, &vmsg);<br>
> > if (!reply_requested && need_reply) {<br>
> > vmsg_set_reply_u64(&vmsg, 0);<br>
> > reply_requested = 1;<br>
> > }<br>
> ><br>
> > So regardless of what may have been negotiated it will always reply with<br>
> > something if the master requested it do so. This points us at the<br>
> > specification which reads:<br>
> ><br>
> > - Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for<br>
> > details.<br>
> ><br>
> > which says in VHOST_USER_PROTOCOL_F_REPLY_ACK that this bit should only<br>
> > be honoured when the feature has been negotiated. Which brings us to a<br>
> > series of questions:<br>
> ><br>
> > - Should QEMU have preserved VhostUserVirtioFeatures::PROTOCOL_FEATURES<br>
> > when doing the eventual VHOST_USER_SET_FEATURES reply?<br>
><br>
> Hmm looks like a bug indeed ... Anyone wants to look<br>
> into fixing that? Marc-André?<br>
<br>
chirantan and keiichi will be implementing vhost-user-vitio-fs on<br>
Chrome OS, maybe one of you two can take a look?<br>
<br></blockquote><div><br></div><div>Yeah, our team is working on vhost-user virtiofs. I think <a class="gmail_plusreply" id="plusReplyChip-2" href="mailto:woodychow@chromium.org" tabindex="-1">+Woody Chow</a> will probably be able to look into this issue.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
><br>
><br>
><br>
> > - Is <a href="http://vhost.rs" rel="noreferrer" target="_blank">vhost.rs</a> being to strict or libvhost-user too lax in interpreting<br>
> > the negotiated features before processing the ``need_reply`` [Bit 3]<br>
> > field of the messages?<br>
> ><br>
> > - are VHOST_USER_SET_MEM_TABLE to VHOST_USER_SET_INFLIGHT_FD included<br>
> > in the "list of the ones that do" require replies or do they only<br>
> > reply when REPLY_ACK has been negotiated as the ambiguous "seealso::"<br>
> > box out seems to imply?<br>
> ><br>
> > Currently I have some hacks in:<br>
> ><br>
> > <a href="https://github.com/stsquad/vhost/tree/my-hacks" rel="noreferrer" target="_blank">https://github.com/stsquad/vhost/tree/my-hacks</a><br>
> ><br>
> > which gets my daemon booting up to the point we actually need to do a<br>
> > transaction. However I won't submit a PR until I've worked out exactly<br>
> > where the problems are.<br>
> ><br>
> > --<br>
> > Alex Bennée<br>
><br>
><br>
> _______________________________________________<br>
> Rust-vmm mailing list<br>
> <a href="mailto:Rust-vmm@lists.opendev.org" target="_blank">Rust-vmm@lists.opendev.org</a><br>
> <a href="http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm" rel="noreferrer" target="_blank">http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm</a><br>
</blockquote></div></div>