Hi folks, Unfortunately we ran out of time in the meeting yesterday to cover the presentation of the NEMU CI that we had planned. I've attached it here and so perhaps we can discuss it via email instead. Of particular note is the last slide where we're (Intel) are proposing to setup a Jenkins CI PoC and do the work on the current crates to get them running through there. Let me know what you think of this idea. Rob
Hey Rob, Thanks for putting together this presentation. I have some comments in terms of supported platforms: - Firecracker also has AMD in the roadmap so ideally we would like the crates to be tested on AMD as well. - I am slightly against using nested virtualization for the CI. Maybe it is appropriate for some crates, but it is not one a catch-it-all solutions. I might be biased here, but we should probably test on bare metal instances so we can be as close as possible to a production setup. Also, I have some concerns around performance testing on nested-virt. Amazon can offer some bare metal instances for the rust-vmm CI. I am not very familiar with Jenkins and I have some questions: 1. Does Jenkins take care of a clean environment for each run (either PRs or merges)? Do we have to implement our own methods to insure that? In the Firecracker CI we are always starting a new instance to run the tests. On that instance we do not have anything pre-configured. Every dependency lies in a container and that's how we make sure we are always testing all PRs in the same manner. 2. How easy is it to have smoke testing? I would put `cargo build --release` as a smoke test as we shouldn't want to run all tests in case the build doesn't work. Also, what are your thoughts around coverage? I didn't find any CI tool that can enforce coverage doesn't drop on PRs when the crate requires access to /dev/kvm. Coveralls[1] doesn't seem to have support for Rust. Codecov[2] might be a good candidate, but I failed to understand how to make it run on a bare-metal instance. To be fair, I also didn't spend a lot of time trying to understand it. Thanks, Andreea [1] https://coveralls.io/ [2] https://codecov.io/ ________________________________________ From: Bradford, Robert <robert.bradford@intel.com> Sent: Thursday, March 7, 2019 12:56 PM To: rust-vmm@lists.opendev.org Subject: [Rust-VMM] CI presentation Hi folks, Unfortunately we ran out of time in the meeting yesterday to cover the presentation of the NEMU CI that we had planned. I've attached it here and so perhaps we can discuss it via email instead. Of particular note is the last slide where we're (Intel) are proposing to setup a Jenkins CI PoC and do the work on the current crates to get them running through there. Let me know what you think of this idea. Rob Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
Hi Andreea, On Thu, Mar 07, 2019 at 04:05:04PM +0000, Florescu, Andreea wrote:
Hey Rob,
Thanks for putting together this presentation.
I have some comments in terms of supported platforms: - Firecracker also has AMD in the roadmap so ideally we would like the crates to be tested on AMD as well. - I am slightly against using nested virtualization for the CI. Maybe it is appropriate for some crates, but it is not one a catch-it-all solutions. I might be biased here, but we should probably test on bare metal instances so we can be as close as possible to a production setup. Also, I have some concerns around performance testing on nested-virt. Amazon can offer some bare metal instances for the rust-vmm CI.
We've been using nested because it's the cheapest option. We use bare metal on Packet for aarch64. Having access to 100% bare metal is ideal and better, I agree.
I am not very familiar with Jenkins and I have some questions: 1. Does Jenkins take care of a clean environment for each run (either PRs or merges)? Do we have to implement our own methods to insure that? In the Firecracker CI we are always starting a new instance to run the tests. On that instance we do not have anything pre-configured. Every dependency lies in a container and that's how we make sure we are always testing all PRs in the same manner. 2. How easy is it to have smoke testing? I would put `cargo build --release` as a smoke test as we shouldn't want to run all tests in case the build doesn't work.
Also, what are your thoughts around coverage? I didn't find any CI tool that can enforce coverage doesn't drop on PRs when the crate requires access to /dev/kvm. Coveralls[1] doesn't seem to have support for Rust. Codecov[2] might be a good candidate, but I failed to understand how to make it run on a bare-metal instance. To be fair, I also didn't spend a lot of time trying to understand it.
https://github.com/xd009642/tarpaulin is one tool that seems to be used for rust coverage. Cheers, Samuel. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
"Tarpaulin only supports x86_64 processors running Linux. " <- This makes me sad :( ________________________________________ From: Samuel Ortiz <samuel.ortiz@intel.com> Sent: Thursday, March 7, 2019 6:24 PM To: Florescu, Andreea Cc: Bradford, Robert; rust-vmm@lists.opendev.org Subject: Re: [Rust-VMM] CI presentation Hi Andreea, On Thu, Mar 07, 2019 at 04:05:04PM +0000, Florescu, Andreea wrote:
Hey Rob,
Thanks for putting together this presentation.
I have some comments in terms of supported platforms: - Firecracker also has AMD in the roadmap so ideally we would like the crates to be tested on AMD as well. - I am slightly against using nested virtualization for the CI. Maybe it is appropriate for some crates, but it is not one a catch-it-all solutions. I might be biased here, but we should probably test on bare metal instances so we can be as close as possible to a production setup. Also, I have some concerns around performance testing on nested-virt. Amazon can offer some bare metal instances for the rust-vmm CI.
We've been using nested because it's the cheapest option. We use bare metal on Packet for aarch64. Having access to 100% bare metal is ideal and better, I agree.
I am not very familiar with Jenkins and I have some questions: 1. Does Jenkins take care of a clean environment for each run (either PRs or merges)? Do we have to implement our own methods to insure that? In the Firecracker CI we are always starting a new instance to run the tests. On that instance we do not have anything pre-configured. Every dependency lies in a container and that's how we make sure we are always testing all PRs in the same manner. 2. How easy is it to have smoke testing? I would put `cargo build --release` as a smoke test as we shouldn't want to run all tests in case the build doesn't work.
Also, what are your thoughts around coverage? I didn't find any CI tool that can enforce coverage doesn't drop on PRs when the crate requires access to /dev/kvm. Coveralls[1] doesn't seem to have support for Rust. Codecov[2] might be a good candidate, but I failed to understand how to make it run on a bare-metal instance. To be fair, I also didn't spend a lot of time trying to understand it.
https://github.com/xd009642/tarpaulin is one tool that seems to be used for rust coverage. Cheers, Samuel. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
On Thu, 2019-03-07 at 16:05 +0000, Florescu, Andreea wrote:
Hey Rob,
Thanks for putting together this presentation.
I have some comments in terms of supported platforms: - Firecracker also has AMD in the roadmap so ideally we would like the crates to be tested on AMD as well.
Sure, the system is flexible enough to have a mixture of fixed agents or agents that created on demand as the VMs are created.
- I am slightly against using nested virtualization for the CI. Maybe it is appropriate for some crates, but it is not one a catch-it-all solutions. I might be biased here, but we should probably test on bare metal instances so we can be as close as possible to a production setup. Also, I have some concerns around performance testing on nested-virt. Amazon can offer some bare metal instances for the rust-vmm CI.
That sounds great. One of the reasons for using the nested virtualisation, which also partly answers your question below, is that we wanted to not have any artifacts from the build remaining. As such we created an entirely new VM every time to build from scratch. Further testing on nested was also an interesting area to ensure testing as the majority of development of the project happened with NEMU as the L1 hypervisor. By using nesting we got coverage of L2 testing.
I am not very familiar with Jenkins and I have some questions: 1. Does Jenkins take care of a clean environment for each run (either PRs or merges)? Do we have to implement our own methods to insure that? In the Firecracker CI we are always starting a new instance to run the tests. On that instance we do not have anything pre- configured. Every dependency lies in a container and that's how we make sure we are always testing all PRs in the same manner. 2. How easy is it to have smoke testing? I would put `cargo build -- release` as a smoke test as we shouldn't want to run all tests in case the build doesn't work.
1.) If your build scripts don't make system wide changes then you don't need to take any particular steps to ensure that the builds are clean (it's always new checkout and build directory.) That can be mitigated by us by either creating a new instance every time, running all the tests inside a filesystem namespace, or just by taking care to avoid contamination in the build system 2.) It's possible to fail early and cause the whole pipeline to abort if the most simplest step fails.
Also, what are your thoughts around coverage? I didn't find any CI tool that can enforce coverage doesn't drop on PRs when the crate requires access to /dev/kvm. Coveralls[1] doesn't seem to have support for Rust. Codecov[2] might be a good candidate, but I failed to understand how to make it run on a bare-metal instance. To be fair, I also didn't spend a lot of time trying to understand it.
I've not looked into coverage testing for Rust sadly.
Thanks, Andreea
[1] https://coveralls.io/ [2] https://codecov.io/ ________________________________________ From: Bradford, Robert <robert.bradford@intel.com> Sent: Thursday, March 7, 2019 12:56 PM To: rust-vmm@lists.opendev.org Subject: [Rust-VMM] CI presentation
Hi folks,
Unfortunately we ran out of time in the meeting yesterday to cover the presentation of the NEMU CI that we had planned. I've attached it here and so perhaps we can discuss it via email instead.
Of particular note is the last slide where we're (Intel) are proposing to setup a Jenkins CI PoC and do the work on the current crates to get them running through there.
Let me know what you think of this idea.
Rob
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
On 11 Mar 2019, at 15:02, Bradford, Robert <robert.bradford@intel.com> wrote:
On Thu, 2019-03-07 at 16:05 +0000, Florescu, Andreea wrote:
Hey Rob,
Thanks for putting together this presentation.
I have some comments in terms of supported platforms: - Firecracker also has AMD in the roadmap so ideally we would like the crates to be tested on AMD as well.
Sure, the system is flexible enough to have a mixture of fixed agents or agents that created on demand as the VMs are created.
- I am slightly against using nested virtualization for the CI. Maybe it is appropriate for some crates, but it is not one a catch-it-all solutions. I might be biased here, but we should probably test on bare metal instances so we can be as close as possible to a production setup. Also, I have some concerns around performance testing on nested-virt. Amazon can offer some bare metal instances for the rust-vmm CI.
That sounds great. One of the reasons for using the nested virtualisation, which also partly answers your question below, is that we wanted to not have any artifacts from the build remaining. As such we created an entirely new VM every time to build from scratch. Further testing on nested was also an interesting area to ensure testing as the majority of development of the project happened with NEMU as the L1 hypervisor. By using nesting we got coverage of L2 testing.
I am not very familiar with Jenkins and I have some questions: 1. Does Jenkins take care of a clean environment for each run (either PRs or merges)? Do we have to implement our own methods to insure that? In the Firecracker CI we are always starting a new instance to run the tests. On that instance we do not have anything pre- configured. Every dependency lies in a container and that's how we make sure we are always testing all PRs in the same manner. 2. How easy is it to have smoke testing? I would put `cargo build -- release` as a smoke test as we shouldn't want to run all tests in case the build doesn't work.
1.) If your build scripts don't make system wide changes then you don't need to take any particular steps to ensure that the builds are clean (it's always new checkout and build directory.) That can be mitigated by us by either creating a new instance every time, running all the tests inside a filesystem namespace, or just by taking care to avoid contamination in the build system
You can also use the “Delete workspace before build starts” option under the project’s configuration “Build environment” settings.
2.) It's possible to fail early and cause the whole pipeline to abort if the most simplest step fails.
Yes. This is typically the case for normal build rules. You can also do more complicated chained setups if you want to reuse build artefacts from one project into another. But I don’t think it is necessary at that stage.
Also, what are your thoughts around coverage? I didn't find any CI tool that can enforce coverage doesn't drop on PRs when the crate requires access to /dev/kvm. Coveralls[1] doesn't seem to have support for Rust. Codecov[2] might be a good candidate, but I failed to understand how to make it run on a bare-metal instance. To be fair, I also didn't spend a lot of time trying to understand it.
I've not looked into coverage testing for Rust sadly.
Thanks, Andreea
[1] https://coveralls.io/ [2] https://codecov.io/ ________________________________________ From: Bradford, Robert <robert.bradford@intel.com> Sent: Thursday, March 7, 2019 12:56 PM To: rust-vmm@lists.opendev.org Subject: [Rust-VMM] CI presentation
Hi folks,
Unfortunately we ran out of time in the meeting yesterday to cover the presentation of the NEMU CI that we had planned. I've attached it here and so perhaps we can discuss it via email instead.
Of particular note is the last slide where we're (Intel) are proposing to setup a Jenkins CI PoC and do the work on the current crates to get them running through there.
Let me know what you think of this idea.
Rob
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________ Rust-vmm mailing list Rust-vmm@lists.opendev.org http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm
participants (4)
-
Bradford, Robert
-
Christophe de Dinechin
-
Florescu, Andreea
-
Samuel Ortiz