Hi,

On 16/09/16 10:04, KOBAYASHI Yoshitake wrote:

Hi all,

On 2016/09/16 2:36, Agustin Benito Bethencourt wrote:

Hi,

On 15/09/16 17:44, Paul Sherwood wrote:

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather
than blog posts, since that way people can question, comment and
discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members.

If the use cases are so complex that requires a blog or authors want
to use them internally for marketing/promo, then a blog would be
probably the best option.

Starting a blog is a big commitment over time though. I would start a
little smaller using:
* This mailing list, as Paul suggested.
* CIP presentations at events.
** Slides and videos of our talks where the cases are described.

Once we get more robust as project, we can create a blog. Meanwhile,
we can post the outcome of our presentations and use this mailing list.

I suggest to bring here (mailing list) a couple of cases and getting
one of them into the ELCE or LinuxCon presentation, for instance, or
something in this line.

I am planning to show a demo at ELC-E which explains a candidate use
case of CIP.
The demo will be a full power plant simulator and controller. In this demo,
Linux runs on all controllers to control plant systems.
The above demo is just an example.

You just open my appetite for knowing more. If possible, share here some info and I will be more than happy to help shaping the presentation to make an even bigger hit.

It might be better to provide some typical development and maintenance
schedule/process
to explain why CIP is important for civil infrastructure systems.

Agree.

There is another aspect that many do not consider when approaching the maintenance problem which is risk.

We are very use to nowadays to evaluate maintenance costs. But maintenance might involve very high risks in certain environments and tackle them might require a heavy investment. This might heavily affect the maintenance policies.

I am curious about how Toshiba face this topic for the power plan use case you are proposing.

Best regards,
Yoshi

_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi all,

On 2016/09/16 2:36, Agustin Benito Bethencourt wrote:

Hi,

On 15/09/16 17:44, Paul Sherwood wrote:

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather
than blog posts, since that way people can question, comment and discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members.

If the use cases are so complex that requires a blog or authors want to use them internally for marketing/promo, then a blog would be probably the best option.

Starting a blog is a big commitment over time though. I would start a little smaller using:
* This mailing list, as Paul suggested.
* CIP presentations at events.
** Slides and videos of our talks where the cases are described.

Once we get more robust as project, we can create a blog. Meanwhile, we can post the outcome of our presentations and use this mailing list.

I suggest to bring here (mailing list) a couple of cases and getting one of them into the ELCE or LinuxCon presentation, for instance, or something in this line.

I am planning to show a demo at ELC-E which explains a candidate use case of CIP.
The demo will be a full power plant simulator and controller. In this demo,
Linux runs on all controllers to control plant systems.
The above demo is just an example.

It might be better to provide some typical development and maintenance schedule/process
to explain why CIP is important for civil infrastructure systems.

Best regards,
Yoshi

Hi all,

My name is Yoshitake Kobayashi. I am working for Toshiba.
I've been working on operating system related topics for more than 20 years.
Currently, I've been leading an embedded Linux team in Toshiba to provide Linux
environments for our embedded products.
For open source related projects, I represent Toshiba for three projects which
include CIP, Debian-LTS [1] and Linux Foundation CE Linux project [2].
I've also made several presentations at Embedded Linux Conferences [3].
In retrospect, the most of my presentations relate to start CIP.

For CIP, I am one of the founding member. So, CIP is really exciting and
challenging project for us. I am looking forward to working with you.

----------------------------------------------------------------
[1] https://www.freexian.com/en/services/debian-lts.html
[2] https://www.linuxfoundation.org/projects/core-embedded-linux
[3] http://www.embeddedlinuxconference.com/
----------------------------------------------------------------

Best regards,
Yoshi

On 2016-09-16 06:15, Daniel Sangorrin wrote:

Greg K-H, Ben Hutchings and others have contributed a huge amount to
Long Term Stable and followon initiatives in the community over the
years. But when I first started exploring how things like LTS and LTSI
can work for embedded and automotive in 2012/2013, I hit some
fundamental questions, not least - how in practice can a complex
embedded project consume a 'stable' kernel that's being released **
every couple of weeks ** with the words 'users of this series must
upgrade'? I presented some work at an automotive GENIVI event in Oct
2013 [1] but the audience at that time literally refused to accept that
the idea of whole-of-life updates.

As for the embedded systems I deal with, a 2-weeks release is definitely not
required. A 6 six months cycle, complemented with aperiodic patch releases
for really *important* issues, would be good enough.
Of course, different use cases may have different requirements so we
will probably need to reach a consensus on that.

I expect there are multiple usecases/scenarios and a one-size-fits-all approach may not be possible. But note that even with six month cycle and periodic patch releases, it seems to me you imply requirements that

a) updates are relatively easy, low effort, low risk
b) updates may be required for the whole production lifetime of the target

I've seen plenty of examples where the real-world LTSI BSP implementation has made the process of updating the kernel 'a nightmare'.

And I've had lots of pushback from people insisting that no updates will be required 'after the first couple of years, when the bugs have been ironed out'.

I'm not yet sure whether CIP usecases mostly involve devices which are connected to the internet or other third-party services. And I'm not sure whether security and integrity of the software over the longterm is expected to be a key concern or not.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically,
and the work involved in keeping stuff working and testing for problems
increases greatly.”
Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Thanks, interesting article.

" All of which makes perfect sense for traditional embedded projects."

I just wanted to clarify that these 'traditional embedded projects'
are actually
in the scope of the CIP project.

Yes, they are.

I'm just suggesting that once we are working with a connected device containing more than tens of millions of lines of code, the principles we learned on self-contained device projects with tens or hundreds of thousands of lines, even if they have worked successfully for decades, may no longer apply.

I believe embedded systems where
continuous updates are hard to implement, should still benefit from other
CIP activities (e.g. testing, RAS, real-time partitioning support or kernel
self-protection).

Absolutely.

Hi Paul,

To introduce myself, I'm CEO at Codethink, but to the best of my
ability I'll be commenting here from a personal perspective, independent
of Codethink's official involvement in CIP.

To start the ball rolling, I'm personally worried about how we are
actually going to achieve super-long-term-support and am happy that we
can discuss the issues in public.

Greg K-H, Ben Hutchings and others have contributed a huge amount to
Long Term Stable and followon initiatives in the community over the
years. But when I first started exploring how things like LTS and LTSI
can work for embedded and automotive in 2012/2013, I hit some
fundamental questions, not least - how in practice can a complex
embedded project consume a 'stable' kernel that's being released **
every couple of weeks ** with the words 'users of this series must
upgrade'? I presented some work at an automotive GENIVI event in Oct
2013 [1] but the audience at that time literally refused to accept that
the idea of whole-of-life updates.

As for the embedded systems I deal with, a 2-weeks release is definitely not
required. A 6 six months cycle, complemented with aperiodic patch releases
for really *important* issues, would be good enough.
Of course, different use cases may have different requirements so we
will probably need to reach a consensus on that.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically,
and the work involved in keeping stuff working and testing for problems
increases greatly.”
Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Thanks, interesting article.

" All of which makes perfect sense for traditional embedded projects."

I just wanted to clarify that these 'traditional embedded projects' are actually
in the scope of the CIP project. I believe embedded systems where
continuous updates are hard to implement, should still benefit from other
CIP activities (e.g. testing, RAS, real-time partitioning support or kernel
self-protection).

Best regards,
Daniel

Would be very interested in others' thoughts on this.

br
Paul

[1]
http://www.devcurmudgeon.com/pdfs/mainline-lts-ltsi-genivi-20131025.pdf
[2] https://lwn.net/SubscriberLink/700557/091f804480fab479/
[3] http://www.devcurmudgeon.com/technical-debt-for-whole-systems.html
_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev

Hi all,

my name is Jan Kiszka. I'm working for Siemens Corporate Technology on
enabling and enhancing open source components for the use in our
embedded products, and that primarily at lower levels like the kernel.

On 2016-09-16 00:31, Ben Hutchings wrote:

On Thu, 2016-09-15 at 17:17 +0100, Paul Sherwood wrote:
[...]
[Greg K-H wrote:]

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

Linux does have a very good record of maintaining application binary
compatibility, which should make it safe to upgrade. But every new
release from Linus brings some or all of the following problems:

- feature regressions
- performance regressions
- security regressions
- increased resource consumption
- removal of deprecated features
- driver API churn

Most of the regressions get fixed quickly, but some linger long after
support for the previous stable branch has ended.

So I don't think it's generally sensible to use the latest stable update
in production, and that's why I care about longterm branches (and I
don't know why Greg does).

Exactly.

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

I agree it's possible to do rolling upgrades, but it's very dependent on
the capability and the will (1) to do regular regression testing on the
target systems and (2) to address the regressions that are found without
waiting for them to be fixed upstream. Where a performance regression
or increased resource consumption stretches an existing system so that
it no longer meets its requirements, this might not be practically
possible. This also applies where non-upstream code is broken by
upstream API changes, and that issue is not going away soon.

And while we are investing in extending long-term support with our SLTS
versions, we have to improve regression testing as well to ensure the
quality of those releases. But nothing prevents to use the results also
for latest upstream versions. That can also contribute to making rolling
updates more realistic in the future, I we should point this out
whenever folks complain that SLTS would be against the upstream trend.

Jan

Hi Agustín and all,

On Thu, 2016-09-15 at 17:17 +0100, Paul Sherwood wrote:
[...]
[Greg K-H wrote:]

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

Linux does have a very good record of maintaining application binary
compatibility, which should make it safe to upgrade. But every new
release from Linus brings some or all of the following problems:

- feature regressions
- performance regressions
- security regressions
- increased resource consumption
- removal of deprecated features
- driver API churn

Most of the regressions get fixed quickly, but some linger long after
support for the previous stable branch has ended.

So I don't think it's generally sensible to use the latest stable update
in production, and that's why I care about longterm branches (and I
don't know why Greg does).

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

I agree it's possible to do rolling upgrades, but it's very dependent on
the capability and the will (1) to do regular regression testing on the
target systems and (2) to address the regressions that are found without
waiting for them to be fixed upstream. Where a performance regression
or increased resource consumption stretches an existing system so that
it no longer meets its requirements, this might not be practically
possible. This also applies where non-upstream code is broken by
upstream API changes, and that issue is not going away soon.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.

Hi,

On 15/09/16 17:44, Paul Sherwood wrote:

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather
than blog posts, since that way people can question, comment and discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members.

If the use cases are so complex that requires a blog or authors want to use them internally for marketing/promo, then a blog would be probably the best option.

Starting a blog is a big commitment over time though. I would start a little smaller using:
* This mailing list, as Paul suggested.
* CIP presentations at events.
** Slides and videos of our talks where the cases are described.

Once we get more robust as project, we can create a blog. Meanwhile, we can post the outcome of our presentations and use this mailing list.

I suggest to bring here (mailing list) a couple of cases and getting one of them into the ELCE or LinuxCon presentation, for instance, or something in this line.

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

On 2016-09-15 17:17, Paul Sherwood wrote:
<snip>

Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

Oops - I gave the wrong link:

https://lwn.net/SubscriberLink/700530/7f1c21b0fd40489e/

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever
been used to in the past, yet it's actually the best thing they could
do."

On 2016-09-15 09:30, Agustin Benito Bethencourt wrote:

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Daniel, can you tell us a little about yourself?

My name is Agustín Benito Bethencourt. I represent Codethink at CIP.
I've managed teams-programs-projects for some years ago, the last in
the open.

I am looking forward to find out how we are going to deal with this
outstanding challenge, keeping a Linux system alive and healthy for
many years in mission critical environments. I think it is an exciting
one.

To introduce myself, I'm CEO at Codethink, but to the best of my ability I'll be commenting here from a personal perspective, independent of Codethink's official involvement in CIP.

To start the ball rolling, I'm personally worried about how we are actually going to achieve super-long-term-support and am happy that we can discuss the issues in public.

Greg K-H, Ben Hutchings and others have contributed a huge amount to Long Term Stable and followon initiatives in the community over the years. But when I first started exploring how things like LTS and LTSI can work for embedded and automotive in 2012/2013, I hit some fundamental questions, not least - how in practice can a complex embedded project consume a 'stable' kernel that's being released ** every couple of weeks ** with the words 'users of this series must upgrade'? I presented some work at an automotive GENIVI event in Oct 2013 [1] but the audience at that time literally refused to accept that the idea of whole-of-life updates.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically, and the work involved in keeping stuff working and testing for problems increases greatly.”

Just yesterday there was a very interesting post about backports and long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update their kernels to newer releases to keep up with the security and bugfixes? That goes against everything those managers/PMs have ever been used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers over the last few years, with some success, but many ecosystem members are extremely uncomfortable with the whole idea of aligning with mainline. I think this is broadly because as embedded engineers we've learned over many years that it's best to change the platform as little as possible. I wrote an article trying to challenge this traditional embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

br
Paul

[1] http://www.devcurmudgeon.com/pdfs/mainline-lts-ltsi-genivi-20131025.pdf
[2] https://lwn.net/SubscriberLink/700557/091f804480fab479/
[3] http://www.devcurmudgeon.com/technical-debt-for-whole-systems.html

On 2016-09-15 16:23, Noriaki Fukuyasu wrote:

I think this is an extremely helpful post.

Why dont we start bring the stories of "Linux in the civil
infrastructures in the world" together, and start a series of blog
posts at the CIP website?

I'd personally prefer to see the discussion start on this list, rather than blog posts, since that way people can question, comment and discuss.

If people/companies are willing to contribute the stories, I would be
more than happy to set up a blog section to the CIP web.
I would be very interested in hearing & promoting the use cases
regardless of members and non-members. 

I'd recommend that blog posts can distill from and summarise the list discussion.

Then, we can talk about those stories at the events we are
participating in.

Agreed.

--

Hi,

CIP is doing a significant effort to be present in many events to explain what are the motivations and goals of the group. The technical activity is about to start so there is very little to show and it will be the case for some time.

What can we focus on the coming months from the communication point of view?

I think there is a huge value in advertising the use cases that justify the commitment from companies like Toshiba, Hitachi and Siemens in CIP.

Some of those use cases are completely unknown for a wide range of developers, even many of those who are "Open Source veterans".

By exposing some of the challenges involved in those use cases CIP would not just provide a meaningful inside about our environment but also the opportunity to have a fruitful conversation with many of those engineers who might think that CIP idea of a Super Long Term Support kernel (system) is "anti-cultural".

So I would like to encourage CIP main drivers to provide and advertise some of those unique use cases. I find them extremely interesting. I hope many others will too.

Best Regards
--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

--

Hi all,

On Thu, Sep 15, 2016 at 10:30:59AM +0200, Agustin Benito Bethencourt wrote:

On 15/09/16 08:50, Daniel Sangorrin wrote:

Hi Ben,

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Daniel, can you tell us a little about yourself?

My name is Agustín Benito Bethencourt. I represent Codethink at CIP. I've
managed teams-programs-projects for some years ago, the last in the open.

My name is Domenico Andreoli, I've been using Linux since 1995 and
actively contributed to Debian for about a decade. I've always worked
with Linux, on Linux, for Linux, thanks to Linux, mainly in the embedded
context but I learned that without the necessary shared values, Linux
is not fun at all.

Although I'm not really active nowdays, my FOSS imprinting imposes me
to at least observe and keep me informed. So I'm here, just representing
myself, with bags full of 2 cents.

I am looking forward to find out how we are going to deal with this
outstanding challenge, keeping a Linux system alive and healthy for many
years in mission critical environments. I think it is an exciting one.

Yes, very exciting.

Do we recognize that big irons and small IoT leaves are in the same
need also here?

Kind regards,
Domenico

--
3B10 0CA1 8674 ACBA B4FE FCD2 CE5B CF17 9960 DE13

Hi,

On 15/09/16 08:50, Daniel Sangorrin wrote:

Hi Ben,

I've been asked to introduce myself to the list, before getting into
discussions. So here's a very brief summary.

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Daniel, can you tell us a little about yourself?

My name is Agustín Benito Bethencourt. I represent Codethink at CIP. I've managed teams-programs-projects for some years ago, the last in the open.

I am looking forward to find out how we are going to deal with this outstanding challenge, keeping a Linux system alive and healthy for many years in mission critical environments. I think it is an exciting one.

Best Regards

Agustin

Best regards,
Daniel

--
IoT Technology center
Toshiba Corp. Industrial ICT solutions,
Daniel SANGORRIN

--
Ben Hutchings
Software Developer, Codethink Ltd.


_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev

_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk

Hi Ben,

I've been asked to introduce myself to the list, before getting into
discussions. So here's a very brief summary.

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Best regards,
Daniel

--
IoT Technology center
Toshiba Corp. Industrial ICT solutions,
Daniel SANGORRIN

--
Ben Hutchings
Software Developer, Codethink Ltd.


_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev

I've been asked to introduce myself to the list, before getting into
discussions. So here's a very brief summary.

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.