Hi,

my name is Urs Gleim. I am with Siemens Corporate Technology. I have a history in embedded systems development with all the big proprietary embedded operating systems out there. I worked in projects in the domains automotive (mainly infotainment), industry automation, healthcare and others. Today I am more in the managing role pushing the shift to open source software and (embedded) Linux in particular in the company (lessons learnt from the work with other OS's :-) We run the internal "Corporate Competence Center Embedded Linux" and support businesses units migrating to and maintaining Linux for their products.

The experiences of the last years showed that the way each and every company is building their own customized Linux versions, tool chains, and test infrastructure could be done much more efficiently and more sustainable. Even inside each company there are several solutions for different products.

We have different product cycles and requirements compared to consumer products, data centers, for example. So what every company in the area of systems used in rail, traffic control, healthcare, power generation/transmission, etc. does is: i. adapting Linux for their needs (in terms of build environment, test, features, ...), ii. maintaining these solutions for years.

Furthermore, having standard configurations would lead to a common platform as a basis for others building upon those platform (libraries, network protocols, tools). At the end this is an ecosystem involving and bringing together different communities/projects, suppliers, and the companies creating products. In other domains there are efforts going in the same direction (e.g., AGL, Genivi). So we believe we can do much more by joining forces in the operating system area and Linux will be established even more in the industrial world. One of the strongest pain points today is the effort we spend for (super) long-term maintenance, for each product, in each company. So we start addressing this by creating a  harmonized build and test infrastructure and agreeing on common super long-term maintained kernel versions and patch cycles.

I am looking forward working with you.

Best regards,
Urs

Hi,

On 16.09.2016 12:30, Agustin Benito Bethencourt wrote:

It might be better to provide some typical development and maintenance
schedule/process
to explain why CIP is important for civil infrastructure systems.

Agree.

There is another aspect that many do not consider when approaching the
maintenance problem which is risk.

We are very use to nowadays to evaluate maintenance costs. But
maintenance might involve very high risks in certain environments and
tackle them might require a heavy investment. This might heavily
affect the maintenance policies.

let me try to sketch one example for typical rail automation products:
- development time of a new system: 3-5 years
- customer specific adaptions: 2-4 years
- initial safety certifications / authorization: 1 year
- safety certifications / authorization for follow-up releases: 3-6 months
(depending on amount of changes)
- lifetime today typically 25 years, for some systems up to 50 years

Especially the time (and money) required for the certification for new
releases explains why in general there are no frequent updates.
If new releases are necessary, e.g. because of security reasons the
above mentioned efforts can be drastically reduced by only changing
minor parts (efforts of course vary on type of system and safety level).
Less changes mean less effort. Switching a kernel version is not
considered to be a small change. That's why these products typically
stick to a kernel version for long time which has been tested
extensively in the system environment.

We have similar situations in other domains. Healthcare, for example.
The product lifetimes are shorter, typically 10-15 years. Still there
are certifications (like FDA) which have to be done on system level
which lead to the same problems.

In addition to this there is a general trend that safety critical
functions are more and more implemented in software. This means that the
number of products having those certification need will further increase.

I would also like to state one important point: the super-long-term
maintenance is not a crazy idea by some industry people. It is done
already. Only that it is done behind closed doors by privately
maintained patch-sets. And this is what we would like to change.

best regards,
Urs

On 2016-09-24 12:52, Gleim, Urs wrote:

just a remark: we have some proxy restrictions in the company which
result in the fact that IRC is not really practically usable. So our
main communication channel would be the mailing list I am afraid.

This is a problem for many companies, actually, but I hope we can still encourage people to participate in IRC, mainly because there are so many established FOSS projects and contributors actively using IRC. It's old-school, but it has been working very well for several decades now. Using the newer services (Slack, Google Groups etc) may seem to achieve similar functionality but they don't achieve the same direct connection to other communities.

Codethink's default recommendation to get IRC in the situation you have at Siemens would be to setup a web-based service. We've had success with Shout [1] and its fork The Lounge [2]. Basically these allow people to participate in IRC directly from a web-browser. Normally people run the server on a tiny/cheap cloud machine, eg AWS. Both relatively easy to setup, either by members directly, or maybe by Linux Foundation?

Alternatively we'd be happy to create accounts on our existing Shout instance [3] if that's of interest.

br
Paul

[1] https://github.com/erming/shout
[2] https://thelounge.github.io
[3] https://chat.codethink.co.uk

Hi,

On 22/09/16 14:59, Agustin Benito Bethencourt wrote:

Hi,

at CIP we need to have a clear view of what "Support" means in the
context of the Super Long Term Support kernel.

++ What kind of support will CIP provide? To whom?

CIP will support its members and their developers, not system
administrators or end users. With the current number of members, there
should not be a need for a 'first line' of support between them and the
CIP core developers, though that may change if membership grows
significantly.

Commercial Linux based distributions like RHEL promise that a subset of
the kernel module API and ABI remains stable within a major release, so
that many out-of-tree modules can be used without needing to update the
module source or binaries along with the kernel. Some IHVs rely on this
to distribute driver modules in binary form.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a single
kernel ABI for IHVs to target anyway

Correction:

Upstream fixes frequently change the kernel module API and/or ABI and backporting them in a way that is difficult and risky - CIP users set their own kernel configurations, so there will not be a single kernel ABI for IHVs to target anyway

* CIP users are responsible for binary releases of both the kernel and
out-of-tree modules, so can ensure that they are properly synchronised.

* The criteria for backporting bug fixes will presumably be based on
'stable-kernel-rules.txt'. However, In CIP context, it is recommended
to be more precise than that.

Best Regards

Best regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi Agustin,

Sorry for the late reply.

+++ Request to members from maintainer

During the meeting Ben requested CIP members to provide him some
guidelines or policies currently followed to choose security patches.
This information will hopefully provide some light that help maintainers
to define some basic policies for choosing security fixes. This policies
need to be tested over time.

Due to the length of the maintenance period, it is unlikely that the
same person/team maintain the kernel for the entire life cycle so the
main policies at least need to be left written.

I think that one possible guideline for backporting security patches to the CIP kernel
is to look at kernel CVEs [1]. A table (probably only accessible by members)
indicating whether a CIP kernel is vulnerable or not to a CVE would be of great help.

Another source of bug/security fixes will be the PREEMPT RT patch. We will probably
need to check whether bugs found in new versions are properly backported to our
CIP kernels.
# I see that Ben is already doing this! [2]

Finally, we may need to decide how to disclosure bugs that we may find ourselves
while testing the CIP kernels. I guess we will need to make sure that all members are
ready for the disclosure.

Best regards
Daniel

[1] http://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33
[2] https://lkml.org/lkml/2016/9/29/473

Hi,

CIP ha set up a public wiki. It has very little content at this point and there is no policy yet to add editors. We are sorting things out.

https://wiki.linuxfoundation.org/civilinfrastructureplatform/start

Please take a look and let me know what can we do to improve it. we will include the link in tomorrow's presentation at LinuxCon EU.

Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Just a remark, the presentation is tomorrow, 10/6

--
Urs Gleim | Siemens AG | Corporate Technology

Hi Agustin,

Here are some links for CIP presentations. Probably the following information is also a candidate contents for CIP wiki.

* Towards Sustainable Systems with the Civil Infrastructure Platform (LinuxCon NA 2016, August 2016)
URL: http://events.linuxfoundation.org/sites/events/files/slides/CIP-LinuxConNA.pdf

* Introducing the Civil Infrastructure Platform Project (LinuxCon Japan, July 2016)
URL: http://events.linuxfoundation.jp/sites/events/files/slides/Intriducing_the_CIP-LCJ2016.pdf

* Introducing the Civil Infrastructure Platform Project (ELC 2016, April 2016)
Note: This is the first official presentation by the CIP
URL: http://events.linuxfoundation.org/sites/events/files/slides/Introducing%20the%20Civil%20Infrastructure%20Platform.pdf

-----------------
The following presentation was made before the CIP launch, but it might be a good reference to know how and why we start the CIP.

* Applying the Linux to the Civil Infrastrucure (LinuxCon Japan 2015, June 2015)
URL: http://elinux.org/images/7/74/Applying_Linux_to_the_Civil_Infrastructure-LinuxCon_Japan_2015.pdf

* Applying Linux to the Social Infrastructure BoF (ELC 2015, Apr 2015)
URL: http://events.linuxfoundation.org/sites/events/files/slides/ELC_2015_SIP.pdf

* Using Embedded Linux for Infrastructure Systems (ELC Europe, October 2014)
URL: http://events.linuxfoundation.jp/sites/events/files/slides/ELCE2014-Using%20Embedded%20Linux%20for%20Infrastructure%20Systems.pdf
-----------------

Best regards,
Yoshi

Hi,

I think it is already there.

On 5 October 2016 20:58:09 CEST, KOBAYASHI Yoshitake <yoshitake.kobayashi@toshiba.co.jp> wrote:

Hi Agustin,

Here are some links for CIP presentations. Probably the following
information is also a candidate contents for CIP wiki.

* Towards Sustainable Systems with the Civil Infrastructure Platform
(LinuxCon NA 2016, August 2016)
URL:
http://events.linuxfoundation.org/sites/events/files/slides/CIP-LinuxConNA.pdf

* Introducing the Civil Infrastructure Platform Project (LinuxCon
Japan, July 2016)
URL:
http://events.linuxfoundation.jp/sites/events/files/slides/Intriducing_the_CIP-LCJ2016.pdf

* Introducing the Civil Infrastructure Platform Project (ELC 2016,
April 2016)
Note: This is the first official presentation by the CIP
URL:
http://events.linuxfoundation.org/sites/events/files/slides/Introducing%20the%20Civil%20Infrastructure%20Platform.pdf

-----------------
The following presentation was made before the CIP launch, but it might
be a good reference to know how and why we start the CIP.

* Applying the Linux to the Civil Infrastrucure (LinuxCon Japan 2015,
June 2015)
URL:
http://elinux.org/images/7/74/Applying_Linux_to_the_Civil_Infrastructure-LinuxCon_Japan_2015.pdf

* Applying Linux to the Social Infrastructure BoF (ELC 2015, Apr 2015)
URL:
http://events.linuxfoundation.org/sites/events/files/slides/ELC_2015_SIP.pdf

* Using Embedded Linux for Infrastructure Systems (ELC Europe, October
2014)
URL:
http://events.linuxfoundation.jp/sites/events/files/slides/ELCE2014-Using%20Embedded%20Linux%20for%20Infrastructure%20Systems.pdf
-----------------

Best regards,
Yoshi


On 2016/10/05 2:50, Agustin Benito Bethencourt wrote:

Hi,

CIP ha set up a public wiki. It has very little content at this point

and there is no policy yet to add editors. We are sorting things out.

https://wiki.linuxfoundation.org/civilinfrastructureplatform/start

Please take a look and let me know what can we do to improve it. we

will include the link in tomorrow's presentation at LinuxCon EU.

Best Regards

Sent from mobile. Please excuse my brevity.
--
Agustín Benito Bethencourt
@toscalix

Hi Agustin,
Let me understand what you wrote.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a
single kernel ABI for IHVs to target anyway

Correction:

Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that is difficult and risky - CIP users set
their own kernel configurations, so there will not be a single kernel ABI for IHVs to
target anyway

Is the correction correct?
I thought you wrote;
* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not (change the kernel module API and/or ABI )is difficult and risky - CIP
Am I wrong?

Takuo Koguchi

Hi Koguchi-san,

I think you are right. It's my fault, sorry.

Daniel

On Fri, 2016-10-07 at 07:34 +0000, 小口琢夫 / KOGUCHI，TAKUO wrote:

Hi Agustin,
Let me understand what you wrote.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a
single kernel ABI for IHVs to target anyway

Correction:

Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that is difficult and risky - CIP users set
their own kernel configurations, so there will not be a single kernel ABI for IHVs to
target anyway

Is the correction correct?

The first version is what I wrote, and the correction says something I
did not mean to say.

I thought you wrote;
* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not (change the kernel module API and/or ABI )is difficult and risky - CIP
Am I wrong?

No, you understand me rightly.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.

Hi,

the slides[1] from the talk "Long-Term Maintenance, or how to (mis-)manage embedded systems for 10+ years" by Jan Luebbe are available.

I assume by next week we will have the video.

[1] http://events.linuxfoundation.org/sites/events/files/slides/PRE-trunk-ELCE-LongTermMaintenance.pdf
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi,

At LinuxCon EU, during the CIP talk it was announced that the first CIP kernel will be 4.4

Please find the slides of all the 2016 talks about CIP in our wiki[1].

[1] https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipconferences
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

On 2016-09-22 11:42, Agustin Benito Bethencourt wrote:

Hi,

a second consideration discussed during the meeting. Feel free to
provide your opinions or considerations about them.

++ How should the kernel be released?

+++ Binaries or sources

While commercial distributions like RHEL/SLE include a small number of
supported kernel configurations in binary form, the CIP kernel should be
primarily released as source, with the configuration controlled by its
users.

Ack.

+++ Kernel features

The project needs to decide which architectures, drivers and other
kernel features are to be supported by the core team and its releases.
This could be documented purely as human-readable text or in a
machine-readable form so that the kernel build process can warn when
building an unsupported configuration. It may also be sensible to
specify the supported toolchain versions.

Not sure how to read these threads as I didn't manage to follow their
evolution: This aspect of support constraints is still in scope? I think
it will be important in order to keep backporting and testing efforts
realistic. The enterprise distros are doing the same.

Regards,
Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

On 2016-09-22 14:59, Agustin Benito Bethencourt wrote:

Hi,

at CIP we need to have a clear view of what "Support" means in the
context of the Super Long Term Support kernel.

++ What kind of support will CIP provide? To whom?

CIP will support its members and their developers, not system
administrators or end users. With the current number of members, there
should not be a need for a 'first line' of support between them and the
CIP core developers, though that may change if membership grows
significantly.

Commercial Linux based distributions like RHEL promise that a subset of
the kernel module API and ABI remains stable within a major release, so
that many out-of-tree modules can be used without needing to update the
module source or binaries along with the kernel. Some IHVs rely on this
to distribute driver modules in binary form.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and
backporting them in a way that does not is difficult and risky - CIP
users set their own kernel configurations, so there will not be a single
kernel ABI for IHVs to target anyway

* CIP users are responsible for binary releases of both the kernel and
out-of-tree modules, so can ensure that they are properly synchronised.

* The criteria for backporting bug fixes will presumably be based on
'stable-kernel-rules.txt'. However, In CIP context, it is recommended
to be more precise than that.

Nit: it's stable_kernel_rules.txt. We should start this way and
augment/adjust rules after discussions as we find potential conflicts.

Sounds reasonable to me in general.

Jan

--
Siemens AG
Corporate Technology
Research & Technology Center
CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
Otto-Hahn-Ring 6
81739 Muenchen
Tel.: +49 89 636-634006
Fax: +49 89 636-33045
mailto:jan.kiszka@siemens.com

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard
Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief
Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina
Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin
and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB
12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

Hi,

I'm sorry for the late response.

(2016/09/22 18:31), Agustin Benito Bethencourt wrote:

Hi,

during the Technical Committee Meeting, last Monday, Ben Hutchings
brought to the attention of the participants several topics to consider.
I would like to bring them here. This is the first one

++ When do CIP should pick up a kernel?

+++ Maintainability effort

New major versions of commercial Linux Distributions are released at 3-4
year intervals, so that typically only 4 versions need to be supported
at one time. Given that CIP's support period is meant to be even
longer, it won?t be sustainable to extend every 'long term' branch, but
only takes on a new branch every 2-4 years.

Assuming we release new products in every 2 years, 2-3-year release
cycle would be feasible.

Maintaining multiple branches is hard work, but its effort would be
decreased after 5 years from the release.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

+++ Backport effort

The longer the intervals between new CIP branches, the greater need
there will be for CIP or individual members to backport new hardware
support (which carries its own risks).

+++ Trade-off

This trade-off is perhaps the most difficult issue to decide.


Best Regards

Hi,

(2016/09/22 18:42), Agustin Benito Bethencourt wrote:

Hi,

a second consideration discussed during the meeting. Feel free to
provide your opinions or considerations about them.

++ How should the kernel be released?

+++ Binaries or sources

While commercial distributions like RHEL/SLE include a small number of
supported kernel configurations in binary form, the CIP kernel should be
primarily released as source, with the configuration controlled by its
users.

Basically, releasing the kernel as source is sufficient. But if we
release it as binary only for the specific reference board, users may be happy.

+++ Kernel features

The project needs to decide which architectures, drivers and other
kernel features are to be supported by the core team and its releases.
This could be documented purely as human-readable text or in a
machine-readable form so that the kernel build process can warn when
building an unsupported configuration. It may also be sensible to
specify the supported toolchain versions.

To reduce the maintenance effort, we should limit drivers and features
to be supported (`support' in this context means tracking security/critical
bug fixes, backporting requested patches, and so on).

One idea is to express the supported features in the form of kernel
config. It is machine-readable, and it's not so difficult to covert
it to human-readable. But I'm not sure if it actually work well because
there are non-configurable features.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

Hi,

(2016/09/22 21:56), Agustin Benito Bethencourt wrote:

Hi,

the below considerations were discussed during the Members meeting.
There is no conclusion yet about how to proceed.

++ Security fixes

+++ Out-of-tree drivers

The embedded systems that CIP will be used in will also often require
out-of-tree drivers and will sometimes include other changes of unknown
quality to their kernel. It needs to be made clear to members that
these modifications are unsupported, and that when they want the CIP
core team to address a bug found in such a modified kernel they must
first demonstrate that it exists in the CIP source release.

I agree on that. CIP don't need to care about out-of-tree drivers.

+++ Security updates

Commercial Linux based distributions like RHEL/SLE typically has
security updates available for the most serious issues (such as
privilege escalation) within a few days of them being publicly known, if
not on the first day.

CIP may be able to achieve this with its source releases but its members
may take much longer to release and deploy binary updates, maybe due to
valid concerns about the risk of regression or limited opportunities to
deploy updates. In the worst case, they may use CIP as an
advertising/compliance point but rarely bother to update. For this
reason Ben H. thinks it's important to draw on the work of the Kernel
Self-Protection Project to add mitigations against common types of
vulnerability. That work is slowly filtering into mainline and at least
some of it could be backported.

I think it is important to prepare for zero-day attacks, and the Kernel
Self-Protection Project would help it. If it could be backported,
doing it is preferred. But does it make the long-term maintenance difficult?

+++ Request to members from maintainer

During the meeting Ben requested CIP members to provide him some
guidelines or policies currently followed to choose security patches.
This information will hopefully provide some light that help maintainers
to define some basic policies for choosing security fixes. This policies
need to be tested over time.

Due to the length of the maintenance period, it is unlikely that the
same person/team maintain the kernel for the entire life cycle so the
main policies at least need to be left written.

Basically, I expect CIP supports all CVEs which are related to supported
features by CIP. But human resources are limited, so CIP members might
need to prioritize them depending on their impact.

Best regards,

Hidehiro Kawai
Hitachi, Ltd. Research & Development Group