I've been asked to introduce myself to the list, before getting into
discussions. So here's a very brief summary.

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.

Hi Ben,

I've been asked to introduce myself to the list, before getting into
discussions. So here's a very brief summary.

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Best regards,
Daniel

--
IoT Technology center
Toshiba Corp. Industrial ICT solutions,
Daniel SANGORRIN

--
Ben Hutchings
Software Developer, Codethink Ltd.


_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

Hi,

On 15/09/16 08:50, Daniel Sangorrin wrote:

Hi Ben,

I've been asked to introduce myself to the list, before getting into
discussions. So here's a very brief summary.

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Daniel, can you tell us a little about yourself?

My name is Agustín Benito Bethencourt. I represent Codethink at CIP. I've managed teams-programs-projects for some years ago, the last in the open.

I am looking forward to find out how we are going to deal with this outstanding challenge, keeping a Linux system alive and healthy for many years in mission critical environments. I think it is an exciting one.

Best Regards

Agustin

Best regards,
Daniel

--
IoT Technology center
Toshiba Corp. Industrial ICT solutions,
Daniel SANGORRIN

--
Ben Hutchings
Software Developer, Codethink Ltd.


_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@...

Hi all,

On Thu, Sep 15, 2016 at 10:30:59AM +0200, Agustin Benito Bethencourt wrote:

On 15/09/16 08:50, Daniel Sangorrin wrote:

Hi Ben,

I've been working on the Linux kernel for about 10 years, including
co-maintenance of Debian's kernel package for most of that time. Debian
provides security support for each stable release for 5 years, and in
support of that I backported many fixes to 2.6.32-longterm and took on
maintenance of the 3.2-longterm and 3.16-longterm branches.

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Daniel, can you tell us a little about yourself?

My name is Agustín Benito Bethencourt. I represent Codethink at CIP. I've
managed teams-programs-projects for some years ago, the last in the open.

My name is Domenico Andreoli, I've been using Linux since 1995 and
actively contributed to Debian for about a decade. I've always worked
with Linux, on Linux, for Linux, thanks to Linux, mainly in the embedded
context but I learned that without the necessary shared values, Linux
is not fun at all.

Although I'm not really active nowdays, my FOSS imprinting imposes me
to at least observe and keep me informed. So I'm here, just representing
myself, with bags full of 2 cents.

I am looking forward to find out how we are going to deal with this
outstanding challenge, keeping a Linux system alive and healthy for many
years in mission critical environments. I think it is an exciting one.

Yes, very exciting.

Do we recognize that big irons and small IoT leaves are in the same
need also here?

Kind regards,
Domenico

--
3B10 0CA1 8674 ACBA B4FE FCD2 CE5B CF17 9960 DE13

--

On 2016-09-15 09:30, Agustin Benito Bethencourt wrote:

I'm now adding another part-time role as lead kernel maintainer for the
CIP. I look forward to working with you all to establish a project and
process that can extend the support lifetime even further.

Ben.

Welcome to the CIP project. It's great to have such an experienced maintainer
in the project. Looking forward to working with you too!

Daniel, can you tell us a little about yourself?

My name is Agustín Benito Bethencourt. I represent Codethink at CIP.
I've managed teams-programs-projects for some years ago, the last in
the open.

I am looking forward to find out how we are going to deal with this
outstanding challenge, keeping a Linux system alive and healthy for
many years in mission critical environments. I think it is an exciting
one.

To introduce myself, I'm CEO at Codethink, but to the best of my ability I'll be commenting here from a personal perspective, independent of Codethink's official involvement in CIP.

To start the ball rolling, I'm personally worried about how we are actually going to achieve super-long-term-support and am happy that we can discuss the issues in public.

Greg K-H, Ben Hutchings and others have contributed a huge amount to Long Term Stable and followon initiatives in the community over the years. But when I first started exploring how things like LTS and LTSI can work for embedded and automotive in 2012/2013, I hit some fundamental questions, not least - how in practice can a complex embedded project consume a 'stable' kernel that's being released ** every couple of weeks ** with the words 'users of this series must upgrade'? I presented some work at an automotive GENIVI event in Oct 2013 [1] but the audience at that time literally refused to accept that the idea of whole-of-life updates.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically, and the work involved in keeping stuff working and testing for problems increases greatly.”

Just yesterday there was a very interesting post about backports and long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update their kernels to newer releases to keep up with the security and bugfixes? That goes against everything those managers/PMs have ever been used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers over the last few years, with some success, but many ecosystem members are extremely uncomfortable with the whole idea of aligning with mainline. I think this is broadly because as embedded engineers we've learned over many years that it's best to change the platform as little as possible. I wrote an article trying to challenge this traditional embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

br
Paul

[1] http://www.devcurmudgeon.com/pdfs/mainline-lts-ltsi-genivi-20131025.pdf
[2] https://lwn.net/SubscriberLink/700557/091f804480fab479/
[3] http://www.devcurmudgeon.com/technical-debt-for-whole-systems.html

On 2016-09-15 17:17, Paul Sherwood wrote:
<snip>

Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

Oops - I gave the wrong link:

https://lwn.net/SubscriberLink/700530/7f1c21b0fd40489e/

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever
been used to in the past, yet it's actually the best thing they could
do."

On Thu, 2016-09-15 at 17:17 +0100, Paul Sherwood wrote:
[...]
[Greg K-H wrote:]

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

Linux does have a very good record of maintaining application binary
compatibility, which should make it safe to upgrade. But every new
release from Linus brings some or all of the following problems:

- feature regressions
- performance regressions
- security regressions
- increased resource consumption
- removal of deprecated features
- driver API churn

Most of the regressions get fixed quickly, but some linger long after
support for the previous stable branch has ended.

So I don't think it's generally sensible to use the latest stable update
in production, and that's why I care about longterm branches (and I
don't know why Greg does).

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

I agree it's possible to do rolling upgrades, but it's very dependent on
the capability and the will (1) to do regular regression testing on the
target systems and (2) to address the regressions that are found without
waiting for them to be fixed upstream. Where a performance regression
or increased resource consumption stretches an existing system so that
it no longer meets its requirements, this might not be practically
possible. This also applies where non-upstream code is broken by
upstream API changes, and that issue is not going away soon.

Ben.

--
Ben Hutchings
Software Developer, Codethink Ltd.

Hi Agustín and all,

Hi all,

my name is Jan Kiszka. I'm working for Siemens Corporate Technology on
enabling and enhancing open source components for the use in our
embedded products, and that primarily at lower levels like the kernel.

On 2016-09-16 00:31, Ben Hutchings wrote:

On Thu, 2016-09-15 at 17:17 +0100, Paul Sherwood wrote:
[...]
[Greg K-H wrote:]

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

Linux does have a very good record of maintaining application binary
compatibility, which should make it safe to upgrade. But every new
release from Linus brings some or all of the following problems:

- feature regressions
- performance regressions
- security regressions
- increased resource consumption
- removal of deprecated features
- driver API churn

Most of the regressions get fixed quickly, but some linger long after
support for the previous stable branch has ended.

So I don't think it's generally sensible to use the latest stable update
in production, and that's why I care about longterm branches (and I
don't know why Greg does).

Exactly.

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Would be very interested in others' thoughts on this.

I agree it's possible to do rolling upgrades, but it's very dependent on
the capability and the will (1) to do regular regression testing on the
target systems and (2) to address the regressions that are found without
waiting for them to be fixed upstream. Where a performance regression
or increased resource consumption stretches an existing system so that
it no longer meets its requirements, this might not be practically
possible. This also applies where non-upstream code is broken by
upstream API changes, and that issue is not going away soon.

And while we are investing in extending long-term support with our SLTS
versions, we have to improve regression testing as well to ensure the
quality of those releases. But nothing prevents to use the results also
for latest upstream versions. That can also contribute to making rolling
updates more realistic in the future, I we should point this out
whenever folks complain that SLTS would be against the upstream trend.

Jan

Hi Paul,

To introduce myself, I'm CEO at Codethink, but to the best of my
ability I'll be commenting here from a personal perspective, independent
of Codethink's official involvement in CIP.

To start the ball rolling, I'm personally worried about how we are
actually going to achieve super-long-term-support and am happy that we
can discuss the issues in public.

Greg K-H, Ben Hutchings and others have contributed a huge amount to
Long Term Stable and followon initiatives in the community over the
years. But when I first started exploring how things like LTS and LTSI
can work for embedded and automotive in 2012/2013, I hit some
fundamental questions, not least - how in practice can a complex
embedded project consume a 'stable' kernel that's being released **
every couple of weeks ** with the words 'users of this series must
upgrade'? I presented some work at an automotive GENIVI event in Oct
2013 [1] but the audience at that time literally refused to accept that
the idea of whole-of-life updates.

As for the embedded systems I deal with, a 2-weeks release is definitely not
required. A 6 six months cycle, complemented with aperiodic patch releases
for really *important* issues, would be good enough.
Of course, different use cases may have different requirements so we
will probably need to reach a consensus on that.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically,
and the work involved in keeping stuff working and testing for problems
increases greatly.”
Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Thanks, interesting article.

" All of which makes perfect sense for traditional embedded projects."

I just wanted to clarify that these 'traditional embedded projects' are actually
in the scope of the CIP project. I believe embedded systems where
continuous updates are hard to implement, should still benefit from other
CIP activities (e.g. testing, RAS, real-time partitioning support or kernel
self-protection).

Best regards,
Daniel

Would be very interested in others' thoughts on this.

br
Paul

[1]
http://www.devcurmudgeon.com/pdfs/mainline-lts-ltsi-genivi-20131025.pdf
[2] https://lwn.net/SubscriberLink/700557/091f804480fab479/
[3] http://www.devcurmudgeon.com/technical-debt-for-whole-systems.html
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

On 2016-09-16 06:15, Daniel Sangorrin wrote:

Greg K-H, Ben Hutchings and others have contributed a huge amount to
Long Term Stable and followon initiatives in the community over the
years. But when I first started exploring how things like LTS and LTSI
can work for embedded and automotive in 2012/2013, I hit some
fundamental questions, not least - how in practice can a complex
embedded project consume a 'stable' kernel that's being released **
every couple of weeks ** with the words 'users of this series must
upgrade'? I presented some work at an automotive GENIVI event in Oct
2013 [1] but the audience at that time literally refused to accept that
the idea of whole-of-life updates.

As for the embedded systems I deal with, a 2-weeks release is definitely not
required. A 6 six months cycle, complemented with aperiodic patch releases
for really *important* issues, would be good enough.
Of course, different use cases may have different requirements so we
will probably need to reach a consensus on that.

I expect there are multiple usecases/scenarios and a one-size-fits-all approach may not be possible. But note that even with six month cycle and periodic patch releases, it seems to me you imply requirements that

a) updates are relatively easy, low effort, low risk
b) updates may be required for the whole production lifetime of the target

I've seen plenty of examples where the real-world LTSI BSP implementation has made the process of updating the kernel 'a nightmare'.

And I've had lots of pushback from people insisting that no updates will be required 'after the first couple of years, when the bugs have been ironed out'.

I'm not yet sure whether CIP usecases mostly involve devices which are connected to the internet or other third-party services. And I'm not sure whether security and integrity of the software over the longterm is expected to be a key concern or not.

And as Greg said at the time:

"The patches that apply for stuff after 2 years drops off dramatically,
and the work involved in keeping stuff working and testing for problems
increases greatly.”
Just yesterday there was a very interesting post about backports and
long term stable kernels on LWN [2]. Greg is quoted there considering:

"But if we didn't provide an LTS, would companies constantly update
their kernels to newer releases to keep up with the security and
bugfixes? That goes against everything those managers/PMs have ever been
used to in the past, yet it's actually the best thing they could do."

I've been recommending the constant update route route to customers
over the last few years, with some success, but many ecosystem members
are extremely uncomfortable with the whole idea of aligning with
mainline. I think this is broadly because as embedded engineers we've
learned over many years that it's best to change the platform as little
as possible. I wrote an article trying to challenge this traditional
embedded thinking earlier this year [3]

Thanks, interesting article.

" All of which makes perfect sense for traditional embedded projects."

I just wanted to clarify that these 'traditional embedded projects'
are actually
in the scope of the CIP project.

Yes, they are.

I'm just suggesting that once we are working with a connected device containing more than tens of millions of lines of code, the principles we learned on self-contained device projects with tens or hundreds of thousands of lines, even if they have worked successfully for decades, may no longer apply.

I believe embedded systems where
continuous updates are hard to implement, should still benefit from other
CIP activities (e.g. testing, RAS, real-time partitioning support or kernel
self-protection).

Absolutely.

Hi all,

My name is Yoshitake Kobayashi. I am working for Toshiba.
I've been working on operating system related topics for more than 20 years.
Currently, I've been leading an embedded Linux team in Toshiba to provide Linux
environments for our embedded products.
For open source related projects, I represent Toshiba for three projects which
include CIP, Debian-LTS [1] and Linux Foundation CE Linux project [2].
I've also made several presentations at Embedded Linux Conferences [3].
In retrospect, the most of my presentations relate to start CIP.

For CIP, I am one of the founding member. So, CIP is really exciting and
challenging project for us. I am looking forward to working with you.

----------------------------------------------------------------
[1] https://www.freexian.com/en/services/debian-lts.html
[2] https://www.linuxfoundation.org/projects/core-embedded-linux
[3] http://www.embeddedlinuxconference.com/
----------------------------------------------------------------

Best regards,
Yoshi

Hi,

my name is Urs Gleim. I am with Siemens Corporate Technology. I have a history in embedded systems development with all the big proprietary embedded operating systems out there. I worked in projects in the domains automotive (mainly infotainment), industry automation, healthcare and others. Today I am more in the managing role pushing the shift to open source software and (embedded) Linux in particular in the company (lessons learnt from the work with other OS's :-) We run the internal "Corporate Competence Center Embedded Linux" and support businesses units migrating to and maintaining Linux for their products.

The experiences of the last years showed that the way each and every company is building their own customized Linux versions, tool chains, and test infrastructure could be done much more efficiently and more sustainable. Even inside each company there are several solutions for different products.

We have different product cycles and requirements compared to consumer products, data centers, for example. So what every company in the area of systems used in rail, traffic control, healthcare, power generation/transmission, etc. does is: i. adapting Linux for their needs (in terms of build environment, test, features, ...), ii. maintaining these solutions for years.

Furthermore, having standard configurations would lead to a common platform as a basis for others building upon those platform (libraries, network protocols, tools). At the end this is an ecosystem involving and bringing together different communities/projects, suppliers, and the companies creating products. In other domains there are efforts going in the same direction (e.g., AGL, Genivi). So we believe we can do much more by joining forces in the operating system area and Linux will be established even more in the industrial world. One of the strongest pain points today is the effort we spend for (super) long-term maintenance, for each product, in each company. So we start addressing this by creating a  harmonized build and test infrastructure and agreeing on common super long-term maintained kernel versions and patch cycles.

I am looking forward working with you.

Best regards,
Urs