[Rust-VMM] CI presentation

[Bradford, Robert]

On Thu, 2019-03-07 at 16:05 +0000, Florescu, Andreea wrote:
> Hey Rob,
> 
> Thanks for putting together this presentation.
> 
> I have some comments in terms of supported platforms:
> - Firecracker also has AMD in the roadmap so ideally we would like
> the crates to be tested on AMD as well.

Sure, the system is flexible enough to have a mixture of fixed agents
or agents that created on demand as the VMs are created.

> - I am slightly against using nested virtualization for the CI. Maybe
> it is appropriate for some crates, but it is not one a catch-it-all
> solutions. I might be biased here, but we should probably test on
> bare metal instances so we can be as close as possible to a
> production setup. Also, I have some concerns around performance
> testing on nested-virt. Amazon can offer some bare metal instances
> for the rust-vmm CI.
> 

That sounds great. One of the reasons for using the nested
virtualisation, which also partly answers your question below, is that
we wanted to not have any artifacts from the build remaining. As such
we created an entirely new VM every time to build from scratch. Further
testing on nested was also an interesting area to ensure testing as the
majority of development of the project happened with NEMU as the L1
hypervisor. By using nesting we got coverage of L2 testing.



> I am not very familiar with Jenkins and I have some questions:
> 1. Does Jenkins take care of a clean environment for each run (either
> PRs or merges)? Do we have to implement our own methods to insure
> that? In the Firecracker CI we are always starting a new instance to
> run the tests. On that instance we do not have anything pre-
> configured. Every dependency lies in a container and that's how we
> make sure we are always testing all PRs in the same manner.
> 2. How easy is it to have smoke testing? I would put `cargo build --
> release` as a smoke test as we shouldn't want to run all tests in
> case the build doesn't work.

1.) If your build scripts don't make system wide changes then you don't
need to take any particular steps to ensure that the builds are clean
(it's always new checkout and build directory.) That can be mitigated
by us by either creating a new instance every time, running all the
tests inside a filesystem namespace, or just by taking care to avoid
contamination in the build system

2.) It's possible to fail early and cause the whole pipeline to abort
if the most simplest step fails.

> 
> Also, what are your thoughts around coverage? I didn't find any CI
> tool that can enforce coverage doesn't drop on PRs when the crate
> requires access to /dev/kvm. Coveralls[1] doesn't seem to have
> support for Rust. Codecov[2] might be a good candidate, but I failed
> to understand how to make it run on a bare-metal instance. To be
> fair, I also didn't spend a lot of time trying to understand it.
> 

I've not looked into coverage testing for Rust sadly.

> Thanks,
> Andreea  
> 
> [1] https://coveralls.io/
> [2] https://codecov.io/
> ________________________________________
> From: Bradford, Robert <robert.bradford at intel.com>
> Sent: Thursday, March 7, 2019 12:56 PM
> To: rust-vmm at lists.opendev.org
> Subject: [Rust-VMM] CI presentation
> 
> Hi folks,
> 
> Unfortunately we ran out of time in the meeting yesterday to cover
> the
> presentation of the NEMU CI that we had planned. I've attached it
> here
> and so perhaps we can discuss it via email instead.
> 
> Of particular note is the last slide where we're (Intel) are
> proposing
> to setup a Jenkins CI PoC and do the work on the current crates to
> get
> them running through there.
> 
> Let me know what you think of this idea.
> 
> Rob
> 
> 
> 
> Amazon Development Center (Romania) S.R.L. registered office: 27A Sf.
> Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania.
> Registered in Romania. Registration number J22/2621/2005.
>