Hi,

On 08/11/16 12:08, Agustin Benito Bethencourt wrote:

Hi,

I am writing a description for the wiki and this topic came to my mind.
It is just a detail...

I would like to propose that we say "maintenance" instead of "support"
when referring to the activities we will do in the open within the CIP
group.

Support has a specific meaning in the commercial world. It is referred
to a "service". In order to gain traction in the industry, the confusion
between maintenance and support plays against us.

In the Open Source world, both words are used indistinctly so I do not
expect any issue by using only (mostly) maintenance.

So we would say Super Long Term Maintenance (SLTM).

In the past Members Meeting, last Monday, this idea was discussed. Finally the decision taken was to stick to the (S)LTS term, that is, including the word support, instead of turning it into Maintenance.

A shared argument was to follow the well known kernel nomenclature.

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

I had an issue with my e-mail client. Apologies for sending the mail twice.

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

On 07/11/16 17:03, Agustin Benito Bethencourt wrote:

Hi,

if you have any update not included in this mail, feel free to add it. I
do not intend to provide a full weekly report but to inform about the
activity I am aware of.

++ Meetings

* Members meeting on Monday Nov 31st

++ Kernel maintenance

* Week 44 is Plumbers week so Ben H., like many kernel hackers, are there.
* Link to CIP 4.4.27 kernel:
https://git.kernel.org/cgit/linux/kernel/git/bwh/linux-cip.git

++ 4.4 LSK: evaluation of backported features

* In order to evaluate the features that has been already backported by
Linaro to 4.4 kernel, that might be interested to CIP, please check the
following links:
** List of backported features and other general information:
https://wiki.linaro.org/LSK
** Backported features description: https://wiki.linaro.org/lsk/features
** Link to the LSK-4.4 repo:
https://git.linaro.org/kernel/linux-linaro-stable.git/log/?h=linux-linaro-lsk-v4.4


++ Setting up kernelci to test CIP kernel

* No major progress this week.

++ Other topics

* A white paper is being written to develop the CIP charter.

Correction: the goal of the white paper is to provide an overview/status of CIP today.

Once
Members have something coherent that looks like a draft, I will publish
it in the public wiki to receive your feedback.

Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

I am writing a description for the wiki and this topic came to my mind. It is just a detail...

I would like to propose that we say "maintenance" instead of "support" when referring to the activities we will do in the open within the CIP group.

Support has a specific meaning in the commercial world. It is referred to a "service". In order to gain traction in the industry, the confusion between maintenance and support plays against us.

In the Open Source world, both words are used indistinctly so I do not expect any issue by using only (mostly) maintenance.

So we would say Super Long Term Maintenance (SLTM).

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

if you have any update not included in this mail, feel free to add it. I do not intend to provide a full weekly report but to inform about the activity I am aware of.

++ Meetings

* Members meeting on Monday Nov 31st

++ Kernel maintenance

* Week 44 is Plumbers week so Ben H., like many kernel hackers, are there.
* Link to CIP 4.4.27 kernel: https://git.kernel.org/cgit/linux/kernel/git/bwh/linux-cip.git

++ 4.4 LSK: evaluation of backported features

* In order to evaluate the features that has been already backported by Linaro to 4.4 kernel, that might be interested to CIP, please check the following links:
** List of backported features and other general information: https://wiki.linaro.org/LSK
** Backported features description: https://wiki.linaro.org/lsk/features
** Link to the LSK-4.4 repo: https://git.linaro.org/kernel/linux-linaro-stable.git/log/?h=linux-linaro-lsk-v4.4

++ Setting up kernelci to test CIP kernel

* No major progress this week.

++ Other topics

* A white paper is being written to develop the CIP charter. Once Members have something coherent that looks like a draft, I will publish it in the public wiki to receive your feedback.

Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

On 31/10/16 15:33, Agustin Benito Bethencourt wrote:

Dear CIP friends,

At ELCE Thursday Oct 13th, the CIP group had a f2f meeting. After having
lunch together, Board members met for about an hour. After that the rest
of the members joined and discussed the following topics for over an hour:

The ELCE 2016 video of the CIP talk has been published: https://www.youtube.com/watch?v=FcL2rc2o8-4&index=123&list=PLbzoR-pLrL6pRFP6SOywVJWdEHlmQE51q

I added it to the events page of the wiki: https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipconferences

++ 4.4 kernel repository

* During the event was announced that the first CIP kernel will be 4.4.
At the meeting we clarified where to have the repository for that kernel.
** In the past, we reached the consensus to stay as close to the kernel
community as possible. Following this approach, there is a consensus at
CIP on using kernel.org as the working place for the kernel. The Linux
Foundation fully supports this decision.

* The general idea is to have the working repositories where it make
sense and mirror them all in a central place.

* CIP will use by default the Linux kernel tools/process to
submit/review patches.
** We are currently discussing the general maintainership policies.

++ Other tools. Repositories mirrors

* As mirroring tool we will use Gitlab (as a service for now).
** As a key advantage it was stated that this tool can be set behind the
firewall which for big corporations can make a difference in getting
internal traction towards working in the open. Feature wise, Gitlab has
what CIP needs.

* Gitlab will be use for additional required software beyond the kernel.
** When the account is set up, we will announce it.

++ CIP platforms

* The goal of the discussion was to find a common ground we all can
share, assuming each member will focus most of their energy on those
platforms used in production.

* CIP basic requirements for selecting a platform:
** At least one ARM and one Intel board
** At least one low cost development board for each architecture.
** No matter what CIP chooses, the list can be increased when consensus
is reached.
** If any SoC joins CIP, we will obviously strongly consider adding
their boards to the list.

* The boards selected at this point are:
** Beaglebone Black (TI Sitara 335x). This is a low cost board we
reached consensus upon.
** Altera Cyclone V. CIP will figure out some details about how to
handle off-tree code from it.

* The discussion around other boards is still ongoing.

++ Delivery (release) model

* When do we plan to release the platform? How?
** Agustin Benito Bethencourt (Codethink) will present a proposal to CIP
about this to start the discussion.

++ Whitepaper

* Since CIP was announced, in April 2016, the Group has reached
consensus in several fundamental topics. For this reason, we agreed to
write a Whitepaper to describe the current status and considerations of
the CIP project.

* Current main consensus points are summarised in the CIP slides
presented at various events. The idea is to move from there to a document.

* CIP group will start working on this topic as soon as possible.

++ 2017 CIP roadmap

* In order to define 2017 activity, CIP decided to pick up some key
dates an plan milestones around them.
** Consensus was reached around ELC, LinuxCon Japan and ELCE, based on
the experience from 2016.

Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Interesting to read:

https://lwn.net/Articles/705220/
(subscriptions only until next week)

In a nutshell: Folks are unhappy with the selection and review process
of patches that go into stable kernels. Various measures were proposed
like adding more tags, including "Fixes:", working more test-based,
reverting broken fixes instead of fixing up on top. The latter was the
least common denominator the community agreed upon for now.

Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Dear CIP friends,

At ELCE Thursday Oct 13th, the CIP group had a f2f meeting. After having lunch together, Board members met for about an hour. After that the rest of the members joined and discussed the following topics for over an hour:

++ 4.4 kernel repository

* During the event was announced that the first CIP kernel will be 4.4. At the meeting we clarified where to have the repository for that kernel.
** In the past, we reached the consensus to stay as close to the kernel community as possible. Following this approach, there is a consensus at CIP on using kernel.org as the working place for the kernel. The Linux Foundation fully supports this decision.

* The general idea is to have the working repositories where it make sense and mirror them all in a central place.

* CIP will use by default the Linux kernel tools/process to submit/review patches.
** We are currently discussing the general maintainership policies.

++ Other tools. Repositories mirrors

* As mirroring tool we will use Gitlab (as a service for now).
** As a key advantage it was stated that this tool can be set behind the firewall which for big corporations can make a difference in getting internal traction towards working in the open. Feature wise, Gitlab has what CIP needs.

* Gitlab will be use for additional required software beyond the kernel.
** When the account is set up, we will announce it.

++ CIP platforms

* The goal of the discussion was to find a common ground we all can share, assuming each member will focus most of their energy on those platforms used in production.

* CIP basic requirements for selecting a platform:
** At least one ARM and one Intel board
** At least one low cost development board for each architecture.
** No matter what CIP chooses, the list can be increased when consensus is reached.
** If any SoC joins CIP, we will obviously strongly consider adding their boards to the list.

* The boards selected at this point are:
** Beaglebone Black (TI Sitara 335x). This is a low cost board we reached consensus upon.
** Altera Cyclone V. CIP will figure out some details about how to handle off-tree code from it.

* The discussion around other boards is still ongoing.

++ Delivery (release) model

* When do we plan to release the platform? How?
** Agustin Benito Bethencourt (Codethink) will present a proposal to CIP about this to start the discussion.

++ Whitepaper

* Since CIP was announced, in April 2016, the Group has reached consensus in several fundamental topics. For this reason, we agreed to
write a Whitepaper to describe the current status and considerations of the CIP project.

* Current main consensus points are summarised in the CIP slides presented at various events. The idea is to move from there to a document.

* CIP group will start working on this topic as soon as possible.

++ 2017 CIP roadmap

* In order to define 2017 activity, CIP decided to pick up some key dates an plan milestones around them.
** Consensus was reached around ELC, LinuxCon Japan and ELCE, based on the experience from 2016.

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

in preparation to start reporting about the testing efforts, I have added some content to the wiki to provide context. Please let me know if there is anything wrong or that might be improved.

Link: https://wiki.linuxfoundation.org/civilinfrastructureplatform/start

Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

(2016/09/27 17:29), Agustin Benito Bethencourt wrote:

Hi,

On 22/09/16 14:59, Agustin Benito Bethencourt wrote:

Hi,

at CIP we need to have a clear view of what "Support" means in the
context of the Super Long Term Support kernel.

++ What kind of support will CIP provide? To whom?

CIP will support its members and their developers, not system
administrators or end users. With the current number of members, there
should not be a need for a 'first line' of support between them and the
CIP core developers, though that may change if membership grows
significantly.

We (our company members) agree.

Commercial Linux based distributions like RHEL promise that a subset of
the kernel module API and ABI remains stable within a major release, so
that many out-of-tree modules can be used without needing to update the
module source or binaries along with the kernel. Some IHVs rely on this
to distribute driver modules in binary form.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a single
kernel ABI for IHVs to target anyway

Correction:

Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that is difficult and risky - CIP users set
their own kernel configurations, so there will not be a single kernel
ABI for IHVs to target anyway

* CIP users are responsible for binary releases of both the kernel and
out-of-tree modules, so can ensure that they are properly synchronised.

* The criteria for backporting bug fixes will presumably be based on
'stable-kernel-rules.txt'. However, In CIP context, it is recommended
to be more precise than that.

We think CIP doesn't need to kepp the kernel API/ABI.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

Hi,

(2016/09/22 21:56), Agustin Benito Bethencourt wrote:

Hi,

the below considerations were discussed during the Members meeting.
There is no conclusion yet about how to proceed.

++ Security fixes

+++ Out-of-tree drivers

The embedded systems that CIP will be used in will also often require
out-of-tree drivers and will sometimes include other changes of unknown
quality to their kernel. It needs to be made clear to members that
these modifications are unsupported, and that when they want the CIP
core team to address a bug found in such a modified kernel they must
first demonstrate that it exists in the CIP source release.

I agree on that. CIP don't need to care about out-of-tree drivers.

+++ Security updates

Commercial Linux based distributions like RHEL/SLE typically has
security updates available for the most serious issues (such as
privilege escalation) within a few days of them being publicly known, if
not on the first day.

CIP may be able to achieve this with its source releases but its members
may take much longer to release and deploy binary updates, maybe due to
valid concerns about the risk of regression or limited opportunities to
deploy updates. In the worst case, they may use CIP as an
advertising/compliance point but rarely bother to update. For this
reason Ben H. thinks it's important to draw on the work of the Kernel
Self-Protection Project to add mitigations against common types of
vulnerability. That work is slowly filtering into mainline and at least
some of it could be backported.

I think it is important to prepare for zero-day attacks, and the Kernel
Self-Protection Project would help it. If it could be backported,
doing it is preferred. But does it make the long-term maintenance difficult?

+++ Request to members from maintainer

During the meeting Ben requested CIP members to provide him some
guidelines or policies currently followed to choose security patches.
This information will hopefully provide some light that help maintainers
to define some basic policies for choosing security fixes. This policies
need to be tested over time.

Due to the length of the maintenance period, it is unlikely that the
same person/team maintain the kernel for the entire life cycle so the
main policies at least need to be left written.

Basically, I expect CIP supports all CVEs which are related to supported
features by CIP. But human resources are limited, so CIP members might
need to prioritize them depending on their impact.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

Hi,

(2016/09/22 18:42), Agustin Benito Bethencourt wrote:

Hi,

a second consideration discussed during the meeting. Feel free to
provide your opinions or considerations about them.

++ How should the kernel be released?

+++ Binaries or sources

While commercial distributions like RHEL/SLE include a small number of
supported kernel configurations in binary form, the CIP kernel should be
primarily released as source, with the configuration controlled by its
users.

Basically, releasing the kernel as source is sufficient. But if we
release it as binary only for the specific reference board, users may be happy.

+++ Kernel features

The project needs to decide which architectures, drivers and other
kernel features are to be supported by the core team and its releases.
This could be documented purely as human-readable text or in a
machine-readable form so that the kernel build process can warn when
building an unsupported configuration. It may also be sensible to
specify the supported toolchain versions.

To reduce the maintenance effort, we should limit drivers and features
to be supported (`support' in this context means tracking security/critical
bug fixes, backporting requested patches, and so on).

One idea is to express the supported features in the form of kernel
config. It is machine-readable, and it's not so difficult to covert
it to human-readable. But I'm not sure if it actually work well because
there are non-configurable features.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

Hi,

I'm sorry for the late response.

(2016/09/22 18:31), Agustin Benito Bethencourt wrote:

Hi,

during the Technical Committee Meeting, last Monday, Ben Hutchings
brought to the attention of the participants several topics to consider.
I would like to bring them here. This is the first one

++ When do CIP should pick up a kernel?

+++ Maintainability effort

New major versions of commercial Linux Distributions are released at 3-4
year intervals, so that typically only 4 versions need to be supported
at one time. Given that CIP's support period is meant to be even
longer, it won?t be sustainable to extend every 'long term' branch, but
only takes on a new branch every 2-4 years.

Assuming we release new products in every 2 years, 2-3-year release
cycle would be feasible.

Maintaining multiple branches is hard work, but its effort would be
decreased after 5 years from the release.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

+++ Backport effort

The longer the intervals between new CIP branches, the greater need
there will be for CIP or individual members to backport new hardware
support (which carries its own risks).

+++ Trade-off

This trade-off is perhaps the most difficult issue to decide.


Best Regards

On 2016-09-22 14:59, Agustin Benito Bethencourt wrote:

Hi,

at CIP we need to have a clear view of what "Support" means in the
context of the Super Long Term Support kernel.

++ What kind of support will CIP provide? To whom?

CIP will support its members and their developers, not system
administrators or end users. With the current number of members, there
should not be a need for a 'first line' of support between them and the
CIP core developers, though that may change if membership grows
significantly.

Commercial Linux based distributions like RHEL promise that a subset of
the kernel module API and ABI remains stable within a major release, so
that many out-of-tree modules can be used without needing to update the
module source or binaries along with the kernel. Some IHVs rely on this
to distribute driver modules in binary form.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a single
kernel ABI for IHVs to target anyway

* CIP users are responsible for binary releases of both the kernel and
out-of-tree modules, so can ensure that they are properly synchronised.

* The criteria for backporting bug fixes will presumably be based on
'stable-kernel-rules.txt'. However, In CIP context, it is recommended
to be more precise than that.

Nit: it's stable_kernel_rules.txt. We should start this way and
augment/adjust rules after discussions as we find potential conflicts.

Sounds reasonable to me in general.

Jan

--
Siemens AG
Corporate Technology
Research & Technology Center
CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
Otto-Hahn-Ring 6
81739 Muenchen
Tel.: +49 89 636-634006
Fax: +49 89 636-33045
mailto:jan.kiszka@siemens.com

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard
Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief
Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina
Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin
and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB
12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

On 2016-09-22 11:42, Agustin Benito Bethencourt wrote:

Hi,

a second consideration discussed during the meeting. Feel free to
provide your opinions or considerations about them.

++ How should the kernel be released?

+++ Binaries or sources

While commercial distributions like RHEL/SLE include a small number of
supported kernel configurations in binary form, the CIP kernel should be
primarily released as source, with the configuration controlled by its
users.

Ack.

+++ Kernel features

The project needs to decide which architectures, drivers and other
kernel features are to be supported by the core team and its releases.
This could be documented purely as human-readable text or in a
machine-readable form so that the kernel build process can warn when
building an unsupported configuration. It may also be sensible to
specify the supported toolchain versions.

Not sure how to read these threads as I didn't manage to follow their
evolution: This aspect of support constraints is still in scope? I think
it will be important in order to keep backporting and testing efforts
realistic. The enterprise distros are doing the same.

Regards,
Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Hi,

At LinuxCon EU, during the CIP talk it was announced that the first CIP kernel will be 4.4

Please find the slides of all the 2016 talks about CIP in our wiki[1].

[1] https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipconferences
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

the slides[1] from the talk "Long-Term Maintenance, or how to (mis-)manage embedded systems for 10+ years" by Jan Luebbe are available.

I assume by next week we will have the video.

[1] http://events.linuxfoundation.org/sites/events/files/slides/PRE-trunk-ELCE-LongTermMaintenance.pdf
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

On Fri, 2016-10-07 at 07:34 +0000, 小口琢夫 / KOGUCHI，TAKUO wrote:

Hi Agustin,
Let me understand what you wrote.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a
single kernel ABI for IHVs to target anyway

Correction:

Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that is difficult and risky - CIP users set
their own kernel configurations, so there will not be a single kernel ABI for IHVs to
target anyway

Is the correction correct?

The first version is what I wrote, and the correction says something I
did not mean to say.

I thought you wrote;
* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not (change the kernel module API and/or ABI )is difficult and risky - CIP
Am I wrong?

No, you understand me rightly.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.

Hi Koguchi-san,

I think you are right. It's my fault, sorry.

Daniel

Hi Agustin,
Let me understand what you wrote.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a
single kernel ABI for IHVs to target anyway

Correction:

Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that is difficult and risky - CIP users set
their own kernel configurations, so there will not be a single kernel ABI for IHVs to
target anyway

Is the correction correct?
I thought you wrote;
* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not (change the kernel module API and/or ABI )is difficult and risky - CIP
Am I wrong?

Takuo Koguchi