Hi,

On 15/09/16 17:44, Paul Sherwood wrote:

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather
than blog posts, since that way people can question, comment and discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members.

If the use cases are so complex that requires a blog or authors want to use them internally for marketing/promo, then a blog would be probably the best option.

Starting a blog is a big commitment over time though. I would start a little smaller using:
* This mailing list, as Paul suggested.
* CIP presentations at events.
** Slides and videos of our talks where the cases are described.

Once we get more robust as project, we can create a blog. Meanwhile, we can post the outcome of our presentations and use this mailing list.

I suggest to bring here (mailing list) a couple of cases and getting one of them into the ELCE or LinuxCon presentation, for instance, or something in this line.

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

On Thu, 2016-09-15 at 17:17 +0100, Paul Sherwood wrote:
[...]
[Greg K-H wrote:]

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

Linux does have a very good record of maintaining application binary
compatibility, which should make it safe to upgrade. But every new
release from Linus brings some or all of the following problems:

- feature regressions
- performance regressions
- security regressions
- increased resource consumption
- removal of deprecated features
- driver API churn

Most of the regressions get fixed quickly, but some linger long after
support for the previous stable branch has ended.

So I don't think it's generally sensible to use the latest stable update
in production, and that's why I care about longterm branches (and I
don't know why Greg does).

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

I agree it's possible to do rolling upgrades, but it's very dependent on
the capability and the will (1) to do regular regression testing on the
target systems and (2) to address the regressions that are found without
waiting for them to be fixed upstream. Where a performance regression
or increased resource consumption stretches an existing system so that
it no longer meets its requirements, this might not be practically
possible. This also applies where non-upstream code is broken by
upstream API changes, and that issue is not going away soon.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.

Hi Agustín and all,

Hi all,

my name is Jan Kiszka. I'm working for Siemens Corporate Technology on
enabling and enhancing open source components for the use in our
embedded products, and that primarily at lower levels like the kernel.

On 2016-09-16 00:31, Ben Hutchings wrote:

On Thu, 2016-09-15 at 17:17 +0100, Paul Sherwood wrote:
[...]
[Greg K-H wrote:]

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

Linux does have a very good record of maintaining application binary
compatibility, which should make it safe to upgrade. But every new
release from Linus brings some or all of the following problems:

- feature regressions
- performance regressions
- security regressions
- increased resource consumption
- removal of deprecated features
- driver API churn

Most of the regressions get fixed quickly, but some linger long after
support for the previous stable branch has ended.

So I don't think it's generally sensible to use the latest stable update
in production, and that's why I care about longterm branches (and I
don't know why Greg does).

Exactly.

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

I agree it's possible to do rolling upgrades, but it's very dependent on
the capability and the will (1) to do regular regression testing on the
target systems and (2) to address the regressions that are found without
waiting for them to be fixed upstream. Where a performance regression
or increased resource consumption stretches an existing system so that
it no longer meets its requirements, this might not be practically
possible. This also applies where non-upstream code is broken by
upstream API changes, and that issue is not going away soon.

And while we are investing in extending long-term support with our SLTS
versions, we have to improve regression testing as well to ensure the
quality of those releases. But nothing prevents to use the results also
for latest upstream versions. That can also contribute to making rolling
updates more realistic in the future, I we should point this out
whenever folks complain that SLTS would be against the upstream trend.

Jan

Hi Paul,

To introduce myself, I'm CEO at Codethink, but to the best of my
ability I'll be commenting here from a personal perspective, independent
of Codethink's official involvement in CIP.

To start the ball rolling, I'm personally worried about how we are
actually going to achieve super-long-term-support and am happy that we
can discuss the issues in public.

Greg K-H, Ben Hutchings and others have contributed a huge amount to
Long Term Stable and followon initiatives in the community over the
years. But when I first started exploring how things like LTS and LTSI
can work for embedded and automotive in 2012/2013, I hit some
fundamental questions, not least - how in practice can a complex
embedded project consume a 'stable' kernel that's being released **
every couple of weeks ** with the words 'users of this series must
upgrade'? I presented some work at an automotive GENIVI event in Oct
2013 [1] but the audience at that time literally refused to accept that
the idea of whole-of-life updates.

As for the embedded systems I deal with, a 2-weeks release is definitely not
required. A 6 six months cycle, complemented with aperiodic patch releases
for really *important* issues, would be good enough.
Of course, different use cases may have different requirements so we
will probably need to reach a consensus on that.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically,
and the work involved in keeping stuff working and testing for problems
increases greatly.”
Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Thanks, interesting article.

" All of which makes perfect sense for traditional embedded projects."

I just wanted to clarify that these 'traditional embedded projects' are actually
in the scope of the CIP project. I believe embedded systems where
continuous updates are hard to implement, should still benefit from other
CIP activities (e.g. testing, RAS, real-time partitioning support or kernel
self-protection).

Best regards,
Daniel

Would be very interested in others' thoughts on this.

br
Paul

[1]
http://www.devcurmudgeon.com/pdfs/mainline-lts-ltsi-genivi-20131025.pdf
[2] https://lwn.net/SubscriberLink/700557/091f804480fab479/
[3] http://www.devcurmudgeon.com/technical-debt-for-whole-systems.html
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

On 2016-09-16 06:15, Daniel Sangorrin wrote:

Greg K-H, Ben Hutchings and others have contributed a huge amount to
Long Term Stable and followon initiatives in the community over the
years. But when I first started exploring how things like LTS and LTSI
can work for embedded and automotive in 2012/2013, I hit some
fundamental questions, not least - how in practice can a complex
embedded project consume a 'stable' kernel that's being released **
every couple of weeks ** with the words 'users of this series must
upgrade'? I presented some work at an automotive GENIVI event in Oct
2013 [1] but the audience at that time literally refused to accept that
the idea of whole-of-life updates.

As for the embedded systems I deal with, a 2-weeks release is definitely not
required. A 6 six months cycle, complemented with aperiodic patch releases
for really *important* issues, would be good enough.
Of course, different use cases may have different requirements so we
will probably need to reach a consensus on that.

I expect there are multiple usecases/scenarios and a one-size-fits-all approach may not be possible. But note that even with six month cycle and periodic patch releases, it seems to me you imply requirements that

a) updates are relatively easy, low effort, low risk
b) updates may be required for the whole production lifetime of the target

I've seen plenty of examples where the real-world LTSI BSP implementation has made the process of updating the kernel 'a nightmare'.

And I've had lots of pushback from people insisting that no updates will be required 'after the first couple of years, when the bugs have been ironed out'.

I'm not yet sure whether CIP usecases mostly involve devices which are connected to the internet or other third-party services. And I'm not sure whether security and integrity of the software over the longterm is expected to be a key concern or not.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically,
and the work involved in keeping stuff working and testing for problems
increases greatly.”
Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Thanks, interesting article.

" All of which makes perfect sense for traditional embedded projects."

I just wanted to clarify that these 'traditional embedded projects'
are actually
in the scope of the CIP project.

Yes, they are.

I'm just suggesting that once we are working with a connected device containing more than tens of millions of lines of code, the principles we learned on self-contained device projects with tens or hundreds of thousands of lines, even if they have worked successfully for decades, may no longer apply.

I believe embedded systems where
continuous updates are hard to implement, should still benefit from other
CIP activities (e.g. testing, RAS, real-time partitioning support or kernel
self-protection).

Absolutely.

Hi all,

My name is Yoshitake Kobayashi. I am working for Toshiba.
I've been working on operating system related topics for more than 20 years.
Currently, I've been leading an embedded Linux team in Toshiba to provide Linux
environments for our embedded products.
For open source related projects, I represent Toshiba for three projects which
include CIP, Debian-LTS [1] and Linux Foundation CE Linux project [2].
I've also made several presentations at Embedded Linux Conferences [3].
In retrospect, the most of my presentations relate to start CIP.

For CIP, I am one of the founding member. So, CIP is really exciting and
challenging project for us. I am looking forward to working with you.

----------------------------------------------------------------
[1] https://www.freexian.com/en/services/debian-lts.html
[2] https://www.linuxfoundation.org/projects/core-embedded-linux
[3] http://www.embeddedlinuxconference.com/
----------------------------------------------------------------

Best regards,
Yoshi

Hi all,

On 2016/09/16 2:36, Agustin Benito Bethencourt wrote:

Hi,

On 15/09/16 17:44, Paul Sherwood wrote:

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather
than blog posts, since that way people can question, comment and discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members.

If the use cases are so complex that requires a blog or authors want to use them internally for marketing/promo, then a blog would be probably the best option.

Starting a blog is a big commitment over time though. I would start a little smaller using:
* This mailing list, as Paul suggested.
* CIP presentations at events.
** Slides and videos of our talks where the cases are described.

Once we get more robust as project, we can create a blog. Meanwhile, we can post the outcome of our presentations and use this mailing list.

I suggest to bring here (mailing list) a couple of cases and getting one of them into the ELCE or LinuxCon presentation, for instance, or something in this line.

I am planning to show a demo at ELC-E which explains a candidate use case of CIP.
The demo will be a full power plant simulator and controller. In this demo,
Linux runs on all controllers to control plant systems.
The above demo is just an example.

It might be better to provide some typical development and maintenance schedule/process
to explain why CIP is important for civil infrastructure systems.

Best regards,
Yoshi

Hi,

On 16/09/16 10:04, KOBAYASHI Yoshitake wrote:

Hi all,

On 2016/09/16 2:36, Agustin Benito Bethencourt wrote:

Hi,

On 15/09/16 17:44, Paul Sherwood wrote:

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather
than blog posts, since that way people can question, comment and
discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members.

If the use cases are so complex that requires a blog or authors want
to use them internally for marketing/promo, then a blog would be
probably the best option.

Starting a blog is a big commitment over time though. I would start a
little smaller using:
* This mailing list, as Paul suggested.
* CIP presentations at events.
** Slides and videos of our talks where the cases are described.

Once we get more robust as project, we can create a blog. Meanwhile,
we can post the outcome of our presentations and use this mailing list.

I suggest to bring here (mailing list) a couple of cases and getting
one of them into the ELCE or LinuxCon presentation, for instance, or
something in this line.

I am planning to show a demo at ELC-E which explains a candidate use
case of CIP.
The demo will be a full power plant simulator and controller. In this demo,
Linux runs on all controllers to control plant systems.
The above demo is just an example.

You just open my appetite for knowing more. If possible, share here some info and I will be more than happy to help shaping the presentation to make an even bigger hit.

It might be better to provide some typical development and maintenance
schedule/process
to explain why CIP is important for civil infrastructure systems.

Agree.

There is another aspect that many do not consider when approaching the maintenance problem which is risk.

We are very use to nowadays to evaluate maintenance costs. But maintenance might involve very high risks in certain environments and tackle them might require a heavy investment. This might heavily affect the maintenance policies.

I am curious about how Toshiba face this topic for the power plan use case you are proposing.

Best regards,
Yoshi

_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

CIP has a new IRC channel. #cip in irc.freenode.net. Please check the logs at https://irclogs.baserock.org/cip/

I would welcome if you can report that you successfully accessed to it through webchat. Some corporations do not provide access to IRC through the default ports. Maybe accessing through web would be the alternative.

Best regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Dear All;

I am Masato Minda of Plat'Home. Please coll me "minmin".

I will attend the ELCE on behalf of my colleagues. And, I will do an exhibition and demonstration in the ELCE.

I have to book a flight and hotel for ELCE.

I would choose the flight to arrive at the venue in the afternoon of October 10. And I will leave Berlin on October 14.

How do you think about this schedule?
(My schedule is not fixed.)

Regards,

Masato Minda

Hi,

On 21/09/16 11:51, Masato Minda wrote:

Dear All;

I am Masato Minda of Plat'Home. Please coll me "minmin".

I will attend the ELCE on behalf of my colleagues. And, I will do an exhibition and demonstration in the ELCE.

I have to book a flight and hotel for ELCE.

I would choose the flight to arrive at the venue in the afternoon of October 10. And I will leave Berlin on October 14.

How do you think about this schedule?

I will be there on the 10th and leave the same day that you. So I guess is fine.

(My schedule is not fixed.)

Regards,

Masato Minda
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

On 21/09/16 14:25, Don Brown wrote:

Hi Everyone,

I am Don Brown and I'm an individual contributor to CIP, although I am
looking for an employer that would allow me to work on CIP as part of my
employment. I am based in Indianapolis, IN. I have a rather unique
skillset that I'll offer to you as we create the use cases.

Welcome Don.

I received my Bachelor of Science in Computer Integrated Manufacturing
Technology (CIMT) from Purdue University. Out of college, I worked for
Allen-Bradley as an Applications Engineer and then worked as a Software
Controls Engineer for several small machine builders until I landed at
the Eli Lilly Corporate Center managing the Utility and Building
Automation Systems. In 2009 I moved to Schneider Electric as a Program
Manager for Energy Saving Projects for the GSA. In 2015, I returned to
the Lilly Corporate Center as an Engineering Specialist responsible for
Research automation equipment.

For the past 9.5 years, I've been teaching at ITT Technical Institute in
both the Information Technology and Electronics Technology programs. I
taught several courses, but the relevant ones for CIP are:

* Linux Networking
* Linux Security
* C Programming
* C++ Programming
* Microprocessors & Microcontrollers
* Programmable Logic Controllers (PLC's)
* Java I & II Programming

I am currently working on my Masters degree in Computer Science with a
specialization in Computer Systems Security at Colorado Technical
University (Distance Learning). I expect to graduate in June 2017.

I am looking forward to working with you all. Please let me know if
there is anything I can do to help develop the use cases. Do we have a
format or template we can use?

Not that I am aware of. Since use cases will be part of the LinuxCon Eu and ELCE CIP talks, maybe that could be a starting point.

Thank you!

Sincerely,

Don Brown. PMP
don.f.brown@... <mailto:don.f.brown@...>
Mobile: (317) 560-0513
Here's to Life, Linux and the Pursuit of Happiness


_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

during the Technical Committee Meeting, last Monday, Ben Hutchings brought to the attention of the participants several topics to consider. I would like to bring them here. This is the first one

++ When do CIP should pick up a kernel?

+++ Maintainability effort

New major versions of commercial Linux Distributions are released at 3-4 year intervals, so that typically only 4 versions need to be supported at one time. Given that CIP's support period is meant to be even longer, it won’t be sustainable to extend every 'long term' branch, but only takes on a new branch every 2-4 years.

+++ Backport effort

The longer the intervals between new CIP branches, the greater need there will be for CIP or individual members to backport new hardware support (which carries its own risks).

+++ Trade-off

This trade-off is perhaps the most difficult issue to decide.


Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

a second consideration discussed during the meeting. Feel free to provide your opinions or considerations about them.

++ How should the kernel be released?

+++ Binaries or sources

While commercial distributions like RHEL/SLE include a small number of supported kernel configurations in binary form, the CIP kernel should be primarily released as source, with the configuration controlled by its users.

+++ Kernel features

The project needs to decide which architectures, drivers and other kernel features are to be supported by the core team and its releases. This could be documented purely as human-readable text or in a machine-readable form so that the kernel build process can warn when building an unsupported configuration. It may also be sensible to specify the supported toolchain versions.


Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

the below considerations were discussed during the Members meeting. There is no conclusion yet about how to proceed.

++ Security fixes

+++ Out-of-tree drivers

The embedded systems that CIP will be used in will also often require out-of-tree drivers and will sometimes include other changes of unknown quality to their kernel. It needs to be made clear to members that these modifications are unsupported, and that when they want the CIP core team to address a bug found in such a modified kernel they must first demonstrate that it exists in the CIP source release.

+++ Security updates

Commercial Linux based distributions like RHEL/SLE typically has security updates available for the most serious issues (such as privilege escalation) within a few days of them being publicly known, if not on the first day.

CIP may be able to achieve this with its source releases but its members may take much longer to release and deploy binary updates, maybe due to valid concerns about the risk of regression or limited opportunities to deploy updates. In the worst case, they may use CIP as an advertising/compliance point but rarely bother to update. For this reason Ben H. thinks it's important to draw on the work of the Kernel Self-Protection Project to add mitigations against common types of vulnerability. That work is slowly filtering into mainline and at least some of it could be backported.

+++ Request to members from maintainer

During the meeting Ben requested CIP members to provide him some guidelines or policies currently followed to choose security patches. This information will hopefully provide some light that help maintainers to define some basic policies for choosing security fixes. This policies need to be tested over time.

Due to the length of the maintenance period, it is unlikely that the same person/team maintain the kernel for the entire life cycle so the main policies at least need to be left written.

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

at CIP we need to have a clear view of what "Support" means in the context of the Super Long Term Support kernel.

++ What kind of support will CIP provide? To whom?

CIP will support its members and their developers, not system administrators or end users. With the current number of members, there should not be a need for a 'first line' of support between them and the CIP core developers, though that may change if membership grows significantly.

Commercial Linux based distributions like RHEL promise that a subset of the kernel module API and ABI remains stable within a major release, so that many out-of-tree modules can be used without needing to update the module source or binaries along with the kernel. Some IHVs rely on this to distribute driver modules in binary form.

CIP should avoid making any such promise because:

* Upstream fixes frequently change the kernel module API and/or ABI and backporting them in a way that does not is difficult and risky - CIP users set their own kernel configurations, so there will not be a single kernel ABI for IHVs to target anyway

* CIP users are responsible for binary releases of both the kernel and out-of-tree modules, so can ensure that they are properly synchronised.

* The criteria for backporting bug fixes will presumably be based on 'stable-kernel-rules.txt'. However, In CIP context, it is recommended to be more precise than that.

Best Regards

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi,

just a remark: we have some proxy restrictions in the company which result in the fact that IRC is not really practically usable. So our main communication channel would be the mailing list I am afraid.

best regards,
Urs