|
Re: Update on Patch and mirroring KernelCI code
Daniel Wagner <daniel.wagner@...>
Hi Don,
On 01/23/2017 01:39 PM, Don Brown wrote: I like the change to the patch and I'll incorporate it tonight.Great. We are already mirroring the KernelCI Backend and Frontend and I hadI agree. There needs to be some strict rules for applying patches. I would prefer to work with the KernelCI upstream to incorporate ourOkay, understood. What about introducing a default configuration that clones for upstream but it could be overwritten as config option? Thanks, Daniel
|
|
|
|
Re: Update on Patch and mirroring KernelCI code
Don Brown <don.f.brown@...>
Hi Daniel, I like the change to the patch and I'll incorporate it tonight. We are already mirroring the KernelCI Backend and Frontend and I had almost the exact same thought this weekend that we already need to modify the code to get ours working. I don't mind doing this in the short term. However, this can be a slippery slope since we could easily get to the point where we've essentially forked their code. I would prefer to work with the KernelCI upstream to incorporate our modifications into their code for the long-term. Don Brown. PMP don.f.brown@... Mobile: (317) 560-0513 Here's to Life, Linux and the Pursuit of Happiness
On Mon, Jan 23, 2017 at 7:00 AM, <cip-dev-request@...> wrote: Send cip-dev mailing list submissions to
|
|
|
|
Re: [PATCH] Use local copy of kernel-ci if present
Daniel Wagner <daniel.wagner@...>
Hi Don,
Over the weekend I had an idea to improve it even more: GIT_SRC="https://github.com/kernelci/kernelci-backend-config.git" if [ -d /vagrant/kernelci-backend-config ]; then GIT_SRC=/vagrant/kernelci-backend-config fi git clone $GIT_SRC kernelci-backend If there is a repo available on /vagrant we clone from it. I think it would be even better just to copy the working directory instead of cloning. That would allow to hack on the files without committing all the time. So this would change to: if [ -d /vagrant/kernelci-backend-config ]; then cp -r /vagrant/kernelci-backend-config kernelci-backend else git clone https://github.com/kernelci/kernelci-backend-config.git kernelci-backend fi What do you think about this? And another idea I had: we should create a mirror of kernelci sources clone from the mirror. I am pretty sure soon we have some patches which need to be around to get our setup running which aren't available in the upstream repository. For example I had to do this here: --- a/roles/install-deps/tasks/install-mongodb.yml +++ b/roles/install-deps/tasks/install-mongodb.yml @@ -2,7 +2,7 @@ - name: Add MongoDB apt key (Ubuntu) apt_key: id=7F0CEB10 - keyserver=hkp://keyserver.ubuntu.com + keyserver=hkp://keyserver.ubuntu.com:80 when: ansible_lsb.id == "Ubuntu" tags: - install @@ -11,7 +11,7 @@ - name: Add MongoDB apt key (Debian) apt_key: id=EA312927 - keyserver=hkp://keyserver.ubuntu.com + keyserver=hkp://keyserver.ubuntu.com:80 when: ansible_lsb.id == "Debian" tags: - install Thanks, Daniel
|
|
|
|
CIP update 08 WK 03
Agustin Benito Bethencourt <agustin.benito@...>
Hi,
this is the report from the activities that Codethink is driving on and or involved. This week is not that we worked more but that we completed more tasks. ++ Meetings * Members meeting on Monday January 16th ** ELC booth/sponsorship logistics as main topic ** Report about testing and kernel maintenance efforts. * Next Members meeting on Monday January 23rd ** Ben Hutchings attending to provide update about the kernel maintenance effort ++ Kernel maintenance * 4.4 CIP kernel updated to the latest LTS. * Backported patches being merged. * Kernel maintenance policies under review. The goal is to arrive to ELC with the policies finished and the backported patches merged. Some additional tasks might be done meanwhile. ++ Testing: Board @desk - Single Developer * KernelCI Virtual Machine Setup & Configuration Howto published[1] ** You can follow other tasks related with the beta release in the ticket #10 * Improvements in the Kernel CI & LAVA v2 configuration. ** LAVA v2 Web Server now sees the devices that are available in the application #4 ** LAVA v2 tests now complete #5 ** QEMU health check are executing and does not get a KVM kernel module related error any more. #8 #9 ** KernelCI VM now boots in Debian Jessie #12 * Beta release initially set for February 4th #10 ** The plan at this point is to release right before ELC. ++ Other topics * Robert Marshall has is no longer part of the team. Christos Karamitsos has substituted him. Chris is also based in Manchester. He will be focused in the issues we have in the kernelci front and back-en related with DDBB. ** Thanks Robert for your effort in pushing the development further. Welcome Christos. Please introduce yourself to the audience (when you recover from that flu). * Ben H. and Don Brown has now edit access to the public wiki. If you have a Linux Foundation ID, please review if you have them too. If not, please let us know through this list. #11 * Agustin (and Yoshi) will attend to FOSDEM. What about you? ++ Coming activities * Investigate DDBB related issues #2 and #3 * Consolidate the KernelCI & LAVA v2 servers into one VM #12 * Add documentation #10 * Preparation of the slides for the Open Source Leadership Summit (Agustin and Nori). #2 https://gitlab.com/cip-project/testing/issues/2 #3 https://gitlab.com/cip-project/testing/issues/3 #4 https://gitlab.com/cip-project/testing/issues/4 #5 https://gitlab.com/cip-project/testing/issues/5 #8 https://gitlab.com/cip-project/testing/issues/8 #9 https://gitlab.com/cip-project/testing/issues/9 #10 https://gitlab.com/cip-project/testing/issues/10 #11 https://gitlab.com/cip-project/testing/issues/11 #12 https://gitlab.com/cip-project/testing/issues/12 [1] https://wiki.linuxfoundation.org/civilinfrastructureplatform/testinglavav2vmsetup Best Regards -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito@codethink.co.uk
|
|
|
|
Re: [PATCH] Use local copy of kernel-ci if present
Daniel Wagner <daniel.wagner@...>
On 01/19/2017 04:14 PM, Wolfgang Mauerer wrote:
Am 19/01/2017 um 15:57 schrieb Daniel Wagner:Good point. I lost myself in the depth of cloudy thoughts.This is done by cloning all the remote repos to the project mainI've had to read the last sentence for a couple of times before
|
|
|
|
Re: [PATCH] Use local copy of kernel-ci if present
Wolfgang Mauerer <wm@...>
Am 19/01/2017 um 15:57 schrieb Daniel Wagner:
In order to develop or debug the setup we should allow to a developerI've had to read the last sentence for a couple of times before I could make sense of it. Perhaps something along the lines of 'The integration scripts test if the repos are present in the shared folder. They are used as source for "git clone" in this case.' the patch as such seems most helpful to me -- struggling with modifications that do not render any effect because sources are pulled from a differnet source have bitten me quite a few times when working with kernelci. Thanks, Wolfgang
|
|
|
|
[PATCH] Use local copy of kernel-ci if present
Daniel Wagner <daniel.wagner@...>
In order to develop or debug the setup we should allow to a developer
to use local copy of the sources. This is done by cloning all the remote repos to the project main folder. This folder is shared with the VM. The intergration-scripts are testing if those repos are in the shared folder than git clone uses this as source. To simplify the developer setup a simple setup-dev-env.sh script is added. Note the current path should be the top folder of the project. Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com> --- Hi, This helped me to quite a bit to get things tested in a more simpler way. What do you think? cheers, daniel integration-scripts/install_backend.sh | 8 +++++++- integration-scripts/install_build_script.sh | 9 ++++++++- integration-scripts/install_frontend.sh | 8 +++++++- scripts/setup-dev-env.sh | 11 +++++++++++ 4 files changed, 33 insertions(+), 3 deletions(-) create mode 100644 scripts/setup-dev-env.sh diff --git a/integration-scripts/install_backend.sh b/integration-scripts/install_backend.sh index 72a974033a3c..157e7184e349 100755 --- a/integration-scripts/install_backend.sh +++ b/integration-scripts/install_backend.sh @@ -4,7 +4,13 @@ # Install kernelci backend cd $HOME && mkdir git-repos && cd git-repos -git clone https://github.com/kernelci/kernelci-backend-config.git kernelci-backend + +GIT_SRC="https://github.com/kernelci/kernelci-backend-config.git" +if [ -d /vagrant/kernelci-backend-config ]; then + GIT_SRC=/vagrant/kernelci-backend-config +fi +git clone $GIT_SRC kernelci-backend + cp /vagrant/config/secrets-backend.yml kernelci-backend/secrets.yml # Fixme: Don't let ansible try to create the file in the first place. diff --git a/integration-scripts/install_build_script.sh b/integration-scripts/install_build_script.sh index 322619317af6..879aaed01792 100755 --- a/integration-scripts/install_build_script.sh +++ b/integration-scripts/install_build_script.sh @@ -3,7 +3,14 @@ # Copyright (C) 2016, Siemens AG, Wolfgang Mauerer <wolfgang.mauerer@siemens.com> # SPDX-License-Identifier: Apache-2.0 -cd $HOME && git clone https://github.com/kernelci/kernelci-build.git +cd $HOME + +GIT_SRC="https://github.com/kernelci/kernelci-build.git" +if [ -d /vagrant/kernelci-build ]; then + GIT_SRC=/vagrant/kernelci-build +fi +git clone $GIT_SRC + cd kernelci-build MASTER_KEY=`cat $HOME/backend-admin-token.txt` diff --git a/integration-scripts/install_frontend.sh b/integration-scripts/install_frontend.sh index 48ab91ab83ce..251ef89d28f3 100755 --- a/integration-scripts/install_frontend.sh +++ b/integration-scripts/install_frontend.sh @@ -4,7 +4,13 @@ # Install kernelci frontend cd $HOME/git-repos -git clone https://github.com/kernelci/kernelci-frontend-config.git kernelci-frontend + +GIT_SRC="https://github.com/kernelci/kernelci-frontend-config.git" +if [ -d /vagrant/kernelci-frontend-config ]; then + GIT_SRC=/vagrant/kernelci-frontend-config +fi +git clone $GIT_SRC kernelci-frontend + sed -i kernelci-frontend/roles/install-app/tasks/main.yml \ -e 's/kernelci\/kernelci-frontend.git/siemens\/kernelci-frontend.git/' diff --git a/scripts/setup-dev-env.sh b/scripts/setup-dev-env.sh new file mode 100644 index 000000000000..59c13b233065 --- /dev/null +++ b/scripts/setup-dev-env.sh @@ -0,0 +1,11 @@ +#!/bin/sh + +if [ ! -f "Vagrantfile" ]; then + echo "script is supposed to be run from the top folder where" + echo "the Vagrantfile is." + exit 1 +fi + +git clone https://github.com/kernelci/kernelci-backend-config.git +git clone https://github.com/kernelci/kernelci-build.git +git clone https://github.com/kernelci/kernelci-frontend-config.git -- 2.9.3
|
|
|
|
Re: kernelci setup fails
Don Brown <don.f.brown@...>
Hi Everyone, I'm reporting on 2 issues today: 1. Re: kernelci setup fails 2. Re: Kernel-CI Frontend broken links to build.log et al. ----- #1 I have fixed the issue of the Kernel-CI Virtual Machine failing to boot on the 'python3-setproctitle' and various other libraries. You can read about it here: https://gitlab.com/cip-project/testing/issues/12 The TL;DR is that the kernelci_backend was broken between 12-Dec-2016 and the current version. I found that by using the 12-Dec-2016 commit, the backend booted up just fine. So, we are back to where we were at the end of December where we can create builds and view them on the Web Frontend, but the links to the build.log, kernel.config, kernel image and dtbs are still broken ----- #2 I have tracked down the general area where the links to the build.log, kernel.config, kernel image and dtbs are broken. I just need someone who knows the code to help fix it. You can read about it here: https://gitlab.com/cip-project/testing/issues/2 Sorry, there is no TL;DR for this one. Sincerely, Don Brown. PMP don.f.brown@... Mobile: (317) 560-0513 Here's to Life, Linux and the Pursuit of Happiness
On Tue, Jan 17, 2017 at 12:14 PM, <cip-dev-request@...> wrote: Send cip-dev mailing list submissions to
|
|
|
|
Re: kernelci setup fails
Agustin Benito Bethencourt <agustin.benito@...>
Hi,
On 17/01/17 17:32, Daniel Wagner wrote: Hi Don,until then... We have discussed what to do. There are several options: 1 Make Python 3.5.2 work on trusty. 2 Move on to Ubuntu 16.04 3 Move to an in-between version. 4 Try to move it into Debian so we can go for a single VM for the whole kernelci tolling. Don is describing in the gitlab ticket his early tries of each option to determine which way to go. Although #4 is the desired one, it seems the most difficult one as well, so we decided to focus on the initial three. We want to have the system working first and then we will figure out how we can simplify the deployment. Best Regards -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito@codethink.co.uk
|
|
|
|
Re: kernelci setup fails
Daniel Wagner <daniel.wagner@...>
On 01/17/2017 03:43 PM, Daniel Wagner wrote:
http://packages.ubuntu.com/search?keywords=python3-setproctitleI gave it a try, no luck though: ==> default: TASK [install-deps : Install MongoDB] ****************************************** ==> default: failed: [kernel-ci-backend] (item=[u'mongodb-org', u'libsnappy1']) => {"failed": true, "item": ["mongodb-org", "libsnappy1"], "msg": "No package matching 'libsnappy1' is available"} So I am stuck at the same level as Don is. Anyway, I was wondering how to be able to have local copies of the kernelci code and use that one as source instead of the hard coded git urls. It seems rather hard to work on the setup if you can't change it :) What about having an option which mounts the local kernelci repos into the virtual machine? Obviously, the kernelci ansible script would also need to be modified to allow overwrites. Thanks, Daniel
|
|
|
|
Re: kernelci setup fails
Daniel Wagner <daniel.wagner@...>
Hi Don,
On 01/17/2017 05:04 PM, Don Brown wrote: I have the same problem and worked on it yesterday from several angles.I'll create a gitlab account soon, so that I can collaborate via the ticket system. Thanks. Daniel
|
|
|
|
Re: kernelci setup fails
Don Brown <don.f.brown@...>
Hi Daniel, I have the same problem and worked on it yesterday from several angles. My notes are logged in Issue #12 in the GitLab repo. The link is: Don Brown. PMP don.f.brown@... Mobile: (317) 560-0513 Here's to Life, Linux and the Pursuit of Happiness
On Tue, Jan 17, 2017 at 7:00 AM, <cip-dev-request@...> wrote: Send cip-dev mailing list submissions to
|
|
|
|
Re: kernelci setup fails
Daniel Wagner <daniel.wagner@...>
Hi Agustin,
On 01/17/2017 03:05 PM, Agustin Benito Bethencourt wrote: maybe this helps?Indeed the second bug report in the ticket is the problem I see too. http://packages.ubuntu.com/search?keywords=python3-setproctitle xenial (16.04LTS) (python): A setproctitle implementation for Python (Python 3) [universe] 1.1.8-1build2: amd64 arm64 armhf i386 powerpc ppc64el s390x yakkety (16.10) (python): A setproctitle implementation for Python (Python 3) 1.1.8-1ubuntu2: amd64 arm64 armhf i386 powerpc ppc64el s390x zesty (python): Setproctitle implementation for Python 3 1.1.10-1: amd64 arm64 armhf i386 powerpc ppc64el s390x So this package is just not part of the trusty version. Is there a specific reason to stick to trusty? Thanks, Daniel
|
|
|
|
Re: kernelci setup fails
Agustin Benito Bethencourt <agustin.benito@...>
Hi Daniel
On 17/01/17 09:49, Daniel Wagner wrote: Hi,maybe this helps? https://gitlab.com/cip-project/testing/issues/12 If this is your case, Don is on it. He talked to one of the kernelci developers, Milo Casagrande about it. -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito@codethink.co.uk
|
|
|
|
kernelci setup fails
Daniel Wagner <wagi@...>
Hi,
running 'vagrant up' from a clean setup consistently fails with this: ==> default: TASK [setup] ******************************************************************* ==> default: ok: [kernel-ci-backend] ==> default: ==> default: TASK [common : Tweak kernel configuration for performance] ********************* ==> default: changed: [kernel-ci-backend] ==> default: ==> default: TASK [install-deps : Install OS dependencies] ********************************** ==> default: failed: [kernel-ci-backend] (item=[u'build-essential', u'git', u'lsb-release', u'python-apt', u'python-pip', u'python-pycurl', u'python-virtualenv', u'python2.7-dev', u'sysfsutils', u'python3', u'python3-yaml', u'python3-setproctitle', u'python3-zmq']) => {"failed": true, "item": ["build-essential", "git", "lsb-release", "python-apt", "python-pip", "python-pycurl", "python-virtualenv", "python2.7-dev", "sysfsutils", "python3", "python3-yaml", "python3-setproctitle", "python3-zmq"], "msg": "No package matching 'python3-setproctitle' is available"} A quick search for setproctitle tells me it is installed via kernelci-backend-config/roles/install-deps/tasks/main.yml: - python3-setproctitle But ubuntu doesn't seem to know this package: vagrant@vagrant-ubuntu-trusty-64:~$ apt-cache search setproctitle libbsd-dev - utility functions from BSD systems - development files python-setproctitle - A setproctitle implementation for Python python-setproctitle-dbg - A setproctitle implementation for Python I haven't spend time to figure out what this package is doing or how it used. So my quick question how to fix this? cheers, daniel
|
|
|
|
Re: [PATCH 4.4-cip 0/6] Extend user-space ASLR range
Agustin Benito Bethencourt <agustin.benito@...>
Hi,
On 16/01/17 10:35, Agustin Benito Bethencourt wrote: Hi,since there is no further feedback, I assume you will merge the patches, isn't is? -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito@codethink.co.uk
|
|
|
|
Re: [PATCH 4.4-cip 0/6] Extend user-space ASLR range
Agustin Benito Bethencourt <agustin.benito@...>
Hi,
On 08/12/16 23:56, Ben Hutchings wrote: This is a backport of changes in 4.5 to extend the range of Addresssince there is no further feedback, I assume you me merge the patches, isn't is? Hopefully in a couple or three more weeks we can start testing it with kernelci tooling. -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito@codethink.co.uk
|
|
|
|
Re: [PATCH 4.4-cip 0/6] Extend user-space ASLR range
Ben Hutchings <ben.hutchings@...>
On Tue, 2017-01-03 at 15:56 -0800, Kees Cook wrote:
On Mon, Dec 19, 2016 at 2:52 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:[...]On 2016-12-09 13:20, Jan Kiszka wrote: Thanks. My backports are identical aside from the placement of someHi!Did you try to discuss the back-port topic with the KSPP folks or otherI've chatted with Elena over this last week, and she talked to Kees who changes in Kconfig files, which shouldn't make a functional difference. Ben. -- Ben Hutchings Software Developer, Codethink Ltd.
|
|
|
|
Re: CIP update 07 WK 01
Robert Marshall <robert.marshall@...>
Apologies, the following item was missed out of the report:
toggle quoted messageShow quoted text
* build artifacts - we're still unable to retrieve these and get 404 errors on selecting the link: https://gitlab.com/cip-project/testing/issues/2 Robert Marshall <robert.marshall@codethink.co.uk> writes:
* Virtual machine creation for the testing
|
|
|
|
CIP update 07 WK 01
Robert Marshall <robert.marshall@...>
* Virtual machine creation for the testing
We've created 2 vagrant instances one for kernelci and the other for lava (lava is Debian based and kernelci is ubuntu based) work is in progress on merging these to make developer setup simpler and reduce the memory requirements. https://gitlab.com/cip-project/testing/issues/7 * There's a github branch to allow the host to view the kernelci vm webserver https://github.com/RobertAJMarshall/kernelci-frontend-config/tree/external-http this will be merged with the gitlab repository when https://gitlab.com/cip-project/testing/issues/3 has been resolved. * The DeviceDictionary error https://gitlab.com/cip-project/testing/issues/5 is still outstanding * Devices have been added to lava VM but are not yet running https://gitlab.com/cip-project/testing/issues/9 * the lava gitlab and the vagrant instance now has been updated with health check tests and extra documentation. * The QEMU health check results in a KVM module not found failure https://gitlab.com/cip-project/testing/issues/8 * Discussions in progress on adding a 'howto' The developers have a document enumerating the steps involved in getting the testing environment up and running, this document will be streamlined and published to the wiki. Meanwhile, we will add the WIP content to the repository so you can follow it. * Minor additions to the wiki has been made, specially in the kernel maintenance page. Robert
|
|
|