Date   

Re: [C-safe-secure-studygroup] [SystemSafety] Critical systems Linux

Clive Pygott <clivepygott@...>
 

Hi Paul

I'll have a go at your question - FYI my background is system safety management (as in 61508 & DO178) and coding standards (MISRA & JSF++)

You are right that ultimately system safety is a system property. You cannot talk about software doing harm without knowing what its controlling and how it fits into its physical environment. However, a standard like 61508 takes a layered approach to safety. The topmost levels are system specific: how could the system behave (intentionally or under fault conditions) to cause harm? and what features of the architecture (including software requirements) mitigate these risks? This establishes traceability from software requirements to safety.

From the software perspective, under this is the requirement to show that those software requirements related to safety have been implemented correctly, and as usual this has two components:
  • showing that the code implements the requirements (verification - we've built the right thing)
  • showing the code is well behaved under all circumstances (validation - we've built the thing right)
If you are doing full formal semantic verification, the second step is unnecessary, as the semantic proof will consider all possible combinations of input and state. However, in practice formal proof is so onerous that its almost never done. This means that verification is based on testing, which no matter how thorough, is still based on sampling. There is an implied belief that the digital system will behave continuously, even though its known that this isn't true (a good few years ago an early home computer had an implementation of BASIC that had an integer ABS functions that worked perfectly except for ABS(-32768) which gave -32768  and it wasn't because it was limited to 16-bits, ABS(-32769) gave 32769 etc).

The validation part aims to improve the (albeit flawed) belief in contiguous behaviour by:
  • checking that any constraints imposed by the language are respected
  • any deviations from arithmetic logic are identified (i.e. flagging where underflow, overflow, truncation, wraparound or loss of precision may occur)
This is the domain of MISRA and JSF++  checking that the code will behave sensibly, without knowledge of what it should be doing.

To get back to the original discussion, it is staggeringly naive to claim that 'I have a safe system, because I've used a certified OS kernel'.  I'm sure you weren't suggesting that, but I have seen companies try it. What the certified kernel (or any other architectural component) buys you is that someone has done the verification and validation activities on that component, so you can be reasonably confident that that component will behave as advertised - its a level of detail your project doesn't have to look into (though you may want to audit the quality of the certification evidence).

As I read your original message you are asking 'why can't a wide user base be accepted as evidence of correctness?'  The short answer is, do you have any evidence of what features of the component the users are using and in what combination? Is my project about to use some combination of features in an inventive manner that no-one has previously tried, so the wide user base provides no evidence that it will work  (again a good few years ago, colleagues of mine were writing a compiler for a VAX and traced a bug to a particular instruction in the VAX instruction set that had an error in its implementation. No DEC product or other customer had ever used this instruction.  BTW, DEC's solution was to remove it from the instruction set)

Hope this helps

       Clive
       LDRA Inc.

On Thu, Nov 22, 2018 at 9:24 AM Paul Sherwood <paul.sherwood@...> wrote:
Hi again...
>>> The question is:-
>>>
>>> As Linux is monolithic, already written  (with minimal
>>> requirements/design
>>> docs) and not to any coding standard
>>> How would the world go about making a Certifiable Linux?
>
>>> Is it possible?

Sadly most of the followon discussion seems to have stayed only on
systemsafetylist.org [1] which rather reduces its impact IMO.

I cross-posted in the hope that knowledge from the safety community
could be usefully shared with other communities who are (for better or
worse) considering and in some cases already using Linux in
safety-critical systems. For example Linux Foundation is actively
soliciting contributors expressly for an initiative to establish how
best to support safety scenarios, as discussed at ELCE [2] with
contributors from OSADL (e.g. [3]) and others.

Perhaps I'm being stupid but it's still unclear to me, after the
discussion about existing certificates, whether the 'pre-certification'
approach is justifiable at all, for **any** software, not just Linux.

As I understand it, for any particular project/system/service we need to
define safety requirements, and safety architecture. From that we need
to establish constraints and required properties and behaviours of
chosen architecture components (including OS components). On that basis
it seems to me that we must always prepare a specific argument for an
actual system, and cannot safely claim that any generic
pre-certification fits our use-case?

Please could someone from systemsafetylist.org reply-all and spell it
out, preferably without referring to standards and without triggering a
lot of controversy?

br
Paul

[1] http://systemsafetylist.org/4310.htm
[2]
https://www.osadl.org/Linux-in-Safety-Critical-Systems-Summit.lfsummit-elce-safety.0.html
[3]
https://events.linuxfoundation.org/wp-content/uploads/2017/12/Collaborate-on-Linux-for-Use-in-Safety-Critical-Systems-Lukas-Bulwahn-BMW-Car-IT-GmbH-1.pdf


_______________________________________________
C-safe-secure-studygroup mailing list
C-safe-secure-studygroup@...
https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup


Re: [SystemSafety] Critical systems Linux

Paul Sherwood
 

Hi again...
The question is:-
As Linux is monolithic, already written (with minimal requirements/design
docs) and not to any coding standard
How would the world go about making a Certifiable Linux?
Is it possible?
Sadly most of the followon discussion seems to have stayed only on systemsafetylist.org [1] which rather reduces its impact IMO.

I cross-posted in the hope that knowledge from the safety community could be usefully shared with other communities who are (for better or worse) considering and in some cases already using Linux in safety-critical systems. For example Linux Foundation is actively soliciting contributors expressly for an initiative to establish how best to support safety scenarios, as discussed at ELCE [2] with contributors from OSADL (e.g. [3]) and others.

Perhaps I'm being stupid but it's still unclear to me, after the discussion about existing certificates, whether the 'pre-certification' approach is justifiable at all, for **any** software, not just Linux.

As I understand it, for any particular project/system/service we need to define safety requirements, and safety architecture. From that we need to establish constraints and required properties and behaviours of chosen architecture components (including OS components). On that basis it seems to me that we must always prepare a specific argument for an actual system, and cannot safely claim that any generic pre-certification fits our use-case?

Please could someone from systemsafetylist.org reply-all and spell it out, preferably without referring to standards and without triggering a lot of controversy?

br
Paul

[1] http://systemsafetylist.org/4310.htm
[2] https://www.osadl.org/Linux-in-Safety-Critical-Systems-Summit.lfsummit-elce-safety.0.html
[3] https://events.linuxfoundation.org/wp-content/uploads/2017/12/Collaborate-on-Linux-for-Use-in-Safety-Critical-Systems-Lukas-Bulwahn-BMW-Car-IT-GmbH-1.pdf


Re: CIP IRC weekly meeting today

Daniel Sangorrin <daniel.sangorrin@...>
 

Hi SZ,

I won't be able to attend the meeting today.
I just sent a few e-mails for the CIP Software update workgroup and the CIP Core workgroup on the members list.
I want to continue that conversation on the members list for now, before we start development.

Thanks,
Daniel

-----Original Message-----
From: cip-dev-bounces@lists.cip-project.org
<cip-dev-bounces@lists.cip-project.org> On Behalf Of SZ Lin (林上智)
Sent: Thursday, November 22, 2018 1:42 PM
To: cip-dev@lists.cip-project.org
Subject: [cip-dev] CIP IRC weekly meeting today

Hi All,

Kindly be reminded to attend weekly meeting through IRC to discuss technical topics
with CIP kernel today.

*Please note that IRC meeting was rescheduled to 18:00 (JST) starting from first
week of Nov. according F2F meeting discussion in ELCE.*

US-West US-East UK DE TW JP
02:00 05:00 09:00 10:00 17:00 18:00

Channel:
* irc:chat.freenode.net:6667/cip

Agenda:
* AI review
- #action Make sure armhf is available in Debian 10 (or not)
- #action Daniel will send cip-core questions to cip-members ML

* Kernel maintenance updates
* Kernel testing
* Software update
* CIP Core
* AOB

The meeting will take 30 min although it can be extended to an hour if it makes sense
and those involved in the topics can stay. Otherwise, the topic will be taken offline or
in the next meeting..

Best regards,

SZ Lin

_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev


CIP IRC weekly meeting today

SZ Lin (林上智) <SZ.Lin@...>
 

Hi All,

Kindly be reminded to attend weekly meeting through IRC to discuss technical topics with CIP kernel today.

*Please note that IRC meeting was rescheduled to 18:00 (JST) starting from first week of Nov. according F2F meeting discussion in ELCE.*

US-West US-East UK DE TW JP
02:00 05:00 09:00 10:00 17:00 18:00

Channel:
* irc:chat.freenode.net:6667/cip

Agenda:
* AI review
- #action Make sure armhf is available in Debian 10 (or not)
- #action Daniel will send cip-core questions to cip-members ML

* Kernel maintenance updates
* Kernel testing
* Software update
* CIP Core
* AOB

The meeting will take 30 min although it can be extended to an hour if it makes sense and those involved in the topics can stay. Otherwise, the topic will be taken offline or in the next meeting..

Best regards,

SZ Lin


Re: FW: [Lava-users] Git authentication

Daniel Sangorrin <daniel.sangorrin@...>
 

Good to know, thank you Chris.

-----Original Message-----
From: cip-dev-bounces@lists.cip-project.org
<cip-dev-bounces@lists.cip-project.org> On Behalf Of Chris Paterson
Sent: Tuesday, November 20, 2018 8:54 PM
To: cip-dev@lists.cip-project.org
Subject: [cip-dev] FW: [Lava-users] Git authentication

Hello all,

It was asked last week about if it is possible to keep jobs secret in the CIP LAVA test
setup.

Test results/definitions can indeed be kept private, either on an individual or group
bases.
https://lava.ciplatform.org/static/docs/v2/glossary.html#term-visibility

If a test job is kept private it is also possible to share 'secret' information with the
board. For example, a private git repository.
https://lava.ciplatform.org/static/docs/v2/publishing-artifacts.html?highlight=sec
rets

Please see the mail below for more information.

Kind regards, Chris

-----Original Message-----
From: Lava-users <lava-users-bounces@lists.lavasoftware.org> On Behalf Of
Milosz Wasilewski
Sent: 20 November 2018 10:46
To: axel.lebourhis@linaro.org
Cc: lava-users@lists.lavasoftware.org
Subject: Re: [Lava-users] Git authentication

On Tue, 20 Nov 2018 at 10:42, Axel Lebourhis <axel.lebourhis@linaro.org> wrote:

Hi everyone,

Is it possible to handle git authentication in a test job ?
I need LAVA to clone a repo that can't be set to public,
and obviously it won't work because of the authentication step.
So is it possible to specify a password or a token ?
You can use 'secrets' section in the job definition. Docs here:
https://master.lavasoftware.org/static/docs/v2/publishing-artifacts.html?highligh
t=secrets

It might be also a good idea to limit access to the job to user or
group so the password doesn't leak. It has to be plain text
unfortunately.

milosz

_______________________________________________
Lava-users mailing list
Lava-users@lists.lavasoftware.org
https://lists.lavasoftware.org/mailman/listinfo/lava-users
_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev


Re: [SystemSafety] Critical systems Linux

Paul Sherwood
 

Now to attempt to answer the question...

On 2018-11-20 18:45, Paul Sherwood wrote:
The question is:-
As Linux is monolithic, already written (with minimal requirements/design
docs) and not to any coding standard
How would the world go about making a Certifiable Linux?
Is it possible?
Some initiatives have already started down this road, for example SIL2LINUXMP (in cc)

But my personal perspective is

1) it may be the the certifications themselves are inappropriate. It's far from clear to me that the current standards are fit for purpose.

2) there are many cases of folks retrofitting documentation to support compliance with standards, so perhaps that would be a feasible thing to attempt (although there is far too much code in the Linux kernel and associated FOSS tooling and userland components to make this something which could be achieved in a short time)

3) if we could establish justifiable concrete improvements to make in Linux (and the tools, and the userland), we could hope to persuade the upstreams to make them, or accept our patches.

4) we could construct new software to meet the ABI commitments of Linux (and other components) while adhering to some specific standards and/or processes, but I'm unconvinced this could be achieved in a time/cost-effective way.

And the question I asked: why do it at all when there are plenty of other
POSIX Compliant RTOS and OS out there that have full Safety Certification to
61508 SIL3 and Do178 etc.?
My understanding is that existing certified RTOS/OS tend to be microkernels with limited functionality, limited hardware support, and performance limitations for some usecases. I'd be happy to be wrong, and no-doubt advocates of some of those technologies can explain the reality by return.

br
Paul


Re: [SystemSafety] Critical systems Linux

Paul Sherwood
 

On 2018-11-20 17:40, Chris Hills wrote:
A subversion of the thread to answer one of the points raised by Paul and
almost every Linux aficionado

-----Original Message-----
bielefeld.de] On Behalf Of Paul Sherwood
Sent: Sunday, November 4, 2018 8:54 PM
One anti-pattern I've grown a bit tired of is people choosing a
micro-kernel instead of Linux, because of the notional 'safety cert',
and then having to implement tons of custom software in attempting to
match off-the-shelf Linux functionality or performance. When application
of the standards leads to "develop new, from scratch" instead of using
existing code which is widely used and known to be reliable, something
is clearly weird imo.
The question is:-
As Linux is monolithic, already written (with minimal requirements/design
docs) and not to any coding standard
How would the world go about making a Certifiable Linux?
Is it possible?
And the question I asked: why do it at all when there are plenty of other
POSIX Compliant RTOS and OS out there that have full Safety Certification to
61508 SIL3 and Do178 etc.?
While systemsafety may be the leading community for public discussion around systems (and software) safety, it is not the only ML that has an interest in this topic so I'm cross-posting to some other (including Linux) lists in the hope that we may see wider discussion and contribution.


Re: CIP-dev IRC weekly meeting log: week 46

Jan Kiszka
 

Hi Nobuhiro,

On 19.11.18 23:17, Nobuhiro Iwamatsu wrote:
Hi,
2018年11月19日(月) 17:53 Jan Kiszka <jan.kiszka@siemens.com>:

On 19.11.18 09:14, SZ Lin (林上智) wrote:
Hi,

Please find the logs of CIP meeting last week in below link:

https://irclogs.baserock.org/cip/%23cip.2018-11-15.log.html#t2018-11-15T09:00:17

#action Confirm the next official CIP kernel announcement in TSC meeting.
#action Make sure armhf is available in Debian 10 (or not)
https://release.debian.org/buster/arch_qualify.html
Huh?! Can we support them with hardware to avoid that?
We can see the current hardware problem of armel / armhf from the following.
https://lists.debian.org/debian-arm/2018/09/msg00049.html
Hmm, that message does not yet read to me like a blocker. It's a list of complications, yes.

I also wonder what makes this buster-specific. The issues with the build environment should be generic, shared with supported arm* for stretch or even older.

Again, if there is any concrete thing we could help with here, please let us discuss proposals at TSC level and with Debian folks. armhf is critical for most of us, and that for quite a few years onward.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux


FW: [Lava-users] Git authentication

Chris Paterson
 

Hello all,

It was asked last week about if it is possible to keep jobs secret in the CIP LAVA test setup.

Test results/definitions can indeed be kept private, either on an individual or group bases.
https://lava.ciplatform.org/static/docs/v2/glossary.html#term-visibility

If a test job is kept private it is also possible to share 'secret' information with the board. For example, a private git repository.
https://lava.ciplatform.org/static/docs/v2/publishing-artifacts.html?highlight=secrets

Please see the mail below for more information.

Kind regards, Chris

-----Original Message-----
From: Lava-users <lava-users-bounces@lists.lavasoftware.org> On Behalf Of Milosz Wasilewski
Sent: 20 November 2018 10:46
To: axel.lebourhis@linaro.org
Cc: lava-users@lists.lavasoftware.org
Subject: Re: [Lava-users] Git authentication

On Tue, 20 Nov 2018 at 10:42, Axel Lebourhis <axel.lebourhis@linaro.org> wrote:

Hi everyone,

Is it possible to handle git authentication in a test job ?
I need LAVA to clone a repo that can't be set to public,
and obviously it won't work because of the authentication step.
So is it possible to specify a password or a token ?
You can use 'secrets' section in the job definition. Docs here:
https://master.lavasoftware.org/static/docs/v2/publishing-artifacts.html?highlight=secrets

It might be also a good idea to limit access to the job to user or
group so the password doesn't leak. It has to be plain text
unfortunately.

milosz

_______________________________________________
Lava-users mailing list
Lava-users@lists.lavasoftware.org
https://lists.lavasoftware.org/mailman/listinfo/lava-users


Re: CIP-dev IRC weekly meeting log: week 46

Nobuhiro Iwamatsu <nobuhiro.iwamatsu@...>
 

Hi,

2018年11月19日(月) 17:53 Jan Kiszka <jan.kiszka@siemens.com>:

On 19.11.18 09:14, SZ Lin (林上智) wrote:
Hi,

Please find the logs of CIP meeting last week in below link:

https://irclogs.baserock.org/cip/%23cip.2018-11-15.log.html#t2018-11-15T09:00:17

#action Confirm the next official CIP kernel announcement in TSC meeting.
#action Make sure armhf is available in Debian 10 (or not)
https://release.debian.org/buster/arch_qualify.html
Huh?! Can we support them with hardware to avoid that?
We can see the current hardware problem of armel / armhf from the following.

https://lists.debian.org/debian-arm/2018/09/msg00049.html

Best regards,
Nobuhiro



Jan

#action Daniel will send cip-core questions to cip-members ML

SZ Lin, Moxa
_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev


Re: List of patches that failed to apply to 4.4-stable

Agustín Benito Bethencourt <agustin.benito@...>
 

Hi,

On Monday, 19 November 2018 16:53:05 WET Daniel Wagner wrote:
Yesterday, I checked the patches in my "failed to apply to 4.4-stable" list against the actual git repository using a short python script.
It turns out that some of the patches in the "to apply" list were already there applied.

I have updated our wiki page accordingly
https://wiki.linuxfoundation.org/civilinfrastructureplatform/linux-4.4-failed-patches

For the curious minds, you can find my (dirty) script attached to this e-mail.
If you can put the patches list as a repo on gitlab, and since the target repo is already being mirrored, I wonder if we can create a job that executes the script when the origin list gets updated. We can create a notification with the outcome to our testing mailing list. This way we just need to focus on having the origin in shape. If we add to this a way to parser the outcome in .md format and published as result in gitlab when the job "succeeds", we significantly reduce the effort associated to the wiki page.

@Daniel W. is the above possible, based on your experience with Gitlab?
I don't think you can update a .md file via gitlab-ci. Though there is
GitLab pages which seems to be the thing for creating documentation via
a CI/CD job:

https://docs.gitlab.com/ee/user/project/pages/
There are SSGs that digest .md

https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/

this way the report can also be visualized not just as a static page but also as a repo file, like a normal wiki page. In any case, the format of the report is not a key point. HTML alone would work fine too, I guess.

I assume you agree with the approach in general. Thanks for the response.

Best Regards
--
Agustín Benito Bethencourt
Principal Consultant
Codethink Ltd
We respect your privacy. See https://www.codethink.co.uk/privacy.html


Re: List of patches that failed to apply to 4.4-stable

Daniel Wagner <daniel.wagner@...>
 

Yesterday, I checked the patches in my "failed to apply to 4.4-stable" list against the actual git repository using a short python script.
It turns out that some of the patches in the "to apply" list were already there applied.

I have updated our wiki page accordingly
https://wiki.linuxfoundation.org/civilinfrastructureplatform/linux-4.4-failed-patches

For the curious minds, you can find my (dirty) script attached to this e-mail.
If you can put the patches list as a repo on gitlab, and since the target repo is already being mirrored, I wonder if we can create a job that executes the script when the origin list gets updated. We can create a notification with the outcome to our testing mailing list. This way we just need to focus on having the origin in shape. If we add to this a way to parser the outcome in .md format and published as result in gitlab when the job "succeeds", we significantly reduce the effort associated to the wiki page.

@Daniel W. is the above possible, based on your experience with Gitlab?
I don't think you can update a .md file via gitlab-ci. Though there is
GitLab pages which seems to be the thing for creating documentation via
a CI/CD job:

https://docs.gitlab.com/ee/user/project/pages/


Re: List of patches that failed to apply to 4.4-stable

Agustín Benito Bethencourt <agustin.benito@...>
 

Hi,

On Friday, 16 November 2018 06:16:31 WET Daniel Sangorrin wrote:
To kernel team,

@SZLin: thanks for creating a page on the wiki just for that.

Yesterday, I checked the patches in my "failed to apply to 4.4-stable" list against the actual git repository using a short python script.
It turns out that some of the patches in the "to apply" list were already there applied.

I have updated our wiki page accordingly
https://wiki.linuxfoundation.org/civilinfrastructureplatform/linux-4.4-failed-patches

For the curious minds, you can find my (dirty) script attached to this e-mail.
If you can put the patches list as a repo on gitlab, and since the target repo is already being mirrored, I wonder if we can create a job that executes the script when the origin list gets updated. We can create a notification with the outcome to our testing mailing list. This way we just need to focus on having the origin in shape. If we add to this a way to parser the outcome in .md format and published as result in gitlab when the job "succeeds", we significantly reduce the effort associated to the wiki page.

@Daniel W. is the above possible, based on your experience with Gitlab?

Best Regards


--
Agustín Benito Bethencourt
Principal Consultant
Codethink Ltd
We respect your privacy. See https://www.codethink.co.uk/privacy.html


Re: CIP-dev IRC weekly meeting log: week 46

Jan Kiszka
 

On 19.11.18 09:14, SZ Lin (林上智) wrote:
Hi,
Please find the logs of CIP meeting last week in below link:
https://irclogs.baserock.org/cip/%23cip.2018-11-15.log.html#t2018-11-15T09:00:17
#action Confirm the next official CIP kernel announcement in TSC meeting.
#action Make sure armhf is available in Debian 10 (or not)
https://release.debian.org/buster/arch_qualify.html
Huh?! Can we support them with hardware to avoid that?

Jan

#action Daniel will send cip-core questions to cip-members ML
SZ Lin, Moxa
_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux


CIP-dev IRC weekly meeting log: week 46

SZ Lin (林上智) <SZ.Lin@...>
 

Hi,

Please find the logs of CIP meeting last week in below link:

https://irclogs.baserock.org/cip/%23cip.2018-11-15.log.html#t2018-11-15T09:00:17

#action Confirm the next official CIP kernel announcement in TSC meeting.
#action Make sure armhf is available in Debian 10 (or not)
https://release.debian.org/buster/arch_qualify.html
#action Daniel will send cip-core questions to cip-members ML

SZ Lin, Moxa


[Git][cip-project/cip-kernel/cip-kernel-sec][master] 3 commits: webview: Display {intrdoduced, fixed}-by value of 'never' properly

Agustin Benito Bethencourt
 

Ben Hutchings pushed to branch master at cip-project / cip-kernel / cip-kernel-sec

Commits:

  • ce82cf92
    by Ben Hutchings at 2018-11-18T21:33:27Z
    webview: Display {intrdoduced,fixed}-by value of 'never' properly
    
  • 770a43fa
    by Ben Hutchings at 2018-11-18T21:48:44Z
    webview: Move fixed/introduced commit lists into status on issue page
    
  • 606cb07e
    by Ben Hutchings at 2018-11-18T21:50:54Z
    webview: Use green/red colouring for status background rather than text
    

2 changed files:

Changes:

  • scripts/templates/issue.html
    ... ... @@ -71,36 +71,35 @@
    71 71
         </td>
    
    72 72
       </tr>
    
    73 73
       {% endif %}
    
    74
    -  {% if issue['introduced-by'] %}
    
    75 74
       <tr>
    
    76
    -    <th rowspan={{ issue['introduced-by']|length }}>Introduced by</th><
     /span>
    
    77
    -    {% for branch in issue['introduced-by'] %}
    
    75
    +    <th rowspan={{ branches|length }}>Status</th>
    
    76
    +    {% for name, affected in branches %}
    
    78 77
         <th>
    
    79
    -      {{ branch }}
    
    78
    +      <a href="/branch/{{ name }}/">{{ name }}</a>
    
    80 79
         </th>
    
    81 80
         <td>
    
    82
    -      {% for commit in issue['introduced-by'][branch] %}
    
    81
    +      {% if not affected %}
    
    82
    +      {% if issue['fixed-by'] and issue['fixed-by'][name] and issue['fixed-by'][name] != 'never' %}
    
    83
    +      <span class="good">fixed</span> by
    
    84
    +      {% for commit in issue['fixed-by'][name] %}
    
    83 85
           <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
    
    84 86
           {% endfor %}
    
    85
    -    </td>
    
    86
    -    {% if not loop.last %}
    
    87
    -  </tr>
    
    88
    -  <tr>
    
    89
    -    {% endif %}
    
    90
    -    {% endfor %}
    
    91
    -  </tr>
    
    92
    -  {% endif %}
    
    93
    -  {% if issue['fixed-by'] %}
    
    94
    -  <tr>
    
    95
    -    <th rowspan={{ issue['fixed-by']|length }}>Fixed by</th>
    
    96
    -    {% for branch in issue['fixed-by'] %}
    
    97
    -    <th>
    
    98
    -      {{ branch }}
    
    99
    -    </th>
    
    100
    -    <td>
    
    101
    -      {% for commit in issue['fixed-by'][branch] %}
    
    87
    +      {% else %}
    
    88
    +      <span class="good">never affected</span>
    
    89
    +      {% endif %}
    
    90
    +      {% else %}
    
    91
    +      {% if issue.ignore and (issue.ignore.all or issue.ignore[name]) %}
    
    92
    +      <span class="ignored">ignored</span>
    
    93
    +      {% else %}
    
    94
    +      <span class="bad">vulnerable</span>
    
    95
    +      {% endif %}
    
    96
    +      {% if issue['introduced-by'] and issue['introduced-by'][name] and issue['introduced-by'][name] != 'never' %}
    
    97
    +      - introduced by
    
    98
    +      {% for commit in issue['introduced-by'][name] %}
    
    102 99
           <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
    
    103 100
           {% endfor %}
    
    101
    +      {% endif %}
    
    102
    +      {% endif %}
    
    104 103
         </td>
    
    105 104
         {% if not loop.last %}
    
    106 105
       </tr>
    
    ... ... @@ -108,7 +107,6 @@
    108 107
         {% endif %}
    
    109 108
         {% endfor %}
    
    110 109
       </tr>
    
    111
    -  {% endif %}
    
    112 110
       {% if issue['fix-depends-on'] %}
    
    113 111
       <tr>
    
    114 112
         <th>Fix depends on</th>
    
    ... ... @@ -122,26 +120,4 @@
    122 120
         </td>
    
    123 121
       </tr>
    
    124 122
       {% endif %}
    
    125
    -  <tr>
    
    126
    -    <th rowspan={{ branches|length }}>Status</th>
    
    127
    -    {% for name, affected in branches %}
    
    128
    -    <th>
    
    129
    -      <a href="/branch/{{ name }}/">{{ name }}</a>
    
    130
    -    </th>
    
    131
    -    <td>
    
    132
    -      {% if not affected %}
    
    133
    -      {# Currently we don't distinguish between never-affected or fixed #}
    
    134
    -      <span class="good">not affected</span>
    
    135
    -      {% elif issue.ignore and (issue.ignore.all or issue.ignore[name]) %}
    
    136
    -      <span class="ignored">ignored</span>
    
    137
    -      {% else %}
    
    138
    -      <span class="bad">vulnerable</span>
    
    139
    -      {% endif %}
    
    140
    -    </td>
    
    141
    -    {% if not loop.last %}
    
    142
    -  </tr>
    
    143
    -  <tr>
    
    144
    -    {% endif %}
    
    145
    -    {% endfor %}
    
    146
    -  </tr>
    
    147 123
     </table>

  • scripts/templates/style.css
    ... ... @@ -20,10 +20,10 @@ body {
    20 20
     }
    
    21 21
     
    
    22 22
     .good {
    
    23
    -    color: #33cc33;
    
    23
    +    background-color: #40ff40;
    
    24 24
     }
    
    25 25
     .bad {
    
    26
    -    color: #ff0000;
    
    26
    +    background-color: #ff4040;
    
    27 27
     }
    
    28 28
     
    
    29 29
     table {
    


  • [Git][cip-project/cip-kernel/cip-kernel-sec][master] 17 commits: Fix PEP-8 style issues identified by pycodestyle

    Agustin Benito Bethencourt
     

    Ben Hutchings pushed to branch master at cip-project / cip-kernel / cip-kernel-sec

    Commits:

    • 8a32d40b
      by Ben Hutchings at 2018-11-16T15:03:18Z
      Fix PEP-8 style issues identified by pycodestyle
      
      * Add recommended white-space
      * Break lines so they are < 80 characters long
      
    • 0559db08
      by Ben Hutchings at 2018-11-16T15:05:32Z
      Remove unused imports identified by pyflakes
      
    • a216d60d
      by Ben Hutchings at 2018-11-16T15:08:25Z
      kernel_sec.isssue: Use Loader.construct_scalar when loading timestamps
      
      _IssueLoader.construct_yaml_timestamp already called
      Loader.construct_scalar but did not use the result.
      
    • 6e8ecdd9
      by Ben Hutchings at 2018-11-16T16:29:08Z
      webview: Fix variable/attribute names in Branch, Branches
      
    • c000a018
      by Ben Hutchings at 2018-11-16T16:29:19Z
      Move commit/branch mapping from report_affected to kernel_sec.branch module
      
      * Add a function for branch sort keys
      * Add a class for the commit/branch mapping
      
    • 5838d699
      by Ben Hutchings at 2018-11-16T16:29:19Z
      Move commit list comparison from report_affected to kernel_sec.issue module
      
    • 9785ec44
      by Ben Hutchings at 2018-11-16T16:29:19Z
      report_affected: Move branch-independent issue checks out of the branch loop
      
    • e777bee7
      by Ben Hutchings at 2018-11-16T16:33:11Z
      webview: Add list of issues to branch pages
      
    • 3a370f88
      by Ben Hutchings at 2018-11-16T16:45:40Z
      webview: Add issue descriptions to issue lists
      
    • 12e3c3d8
      by Ben Hutchings at 2018-11-16T16:54:24Z
      webview: Add a stylesheet (initially empty)
      
    • 273d4d9c
      by Ben Hutchings at 2018-11-16T18:26:33Z
      webview: Specify text and background colours in stylesheet
      
      We need to specify colours for everything to ensure that more specific
      rules don't conflict with user-defined colours.
      
    • 867887e4
      by Ben Hutchings at 2018-11-16T18:28:07Z
      webview: Grey-out ignored issues in lists
      
      * Set an "ignored" class on list items for ignored issues
      * Mix text/link/visited colours with 40% white for the "ignored" class
      
    • 4d06a7f2
      by Ben Hutchings at 2018-11-16T18:45:25Z
      webview: Move branch names and commit-branch mapping to Root class
      
    • 7cda6dde
      by Ben Hutchings at 2018-11-16T18:57:54Z
      webview: Add per-branch status to issue pages
      
    • 882fe908
      by Ben Hutchings at 2018-11-18T20:59:27Z
      webview: Sort branch list
      
    • 9695e007
      by Ben Hutchings at 2018-11-18T21:17:45Z
      webview: Set table styles
      
      * Enable narrow borders between rows
      * Increase spacing between columns
      * Align headings to top right of cell
      
    • c4d3bc0b
      by Ben Hutchings at 2018-11-18T21:18:24Z
      webview: Use three columns for tables on issue pages
      
      * Insert a column between the field name and details
      * Put comment authors and branch names in this column
      

    16 changed files:

    Changes:

  • scripts/cleanup.py
    ... ... @@ -8,11 +8,13 @@
    8 8
     
    
    9 9
     import kernel_sec.issue
    
    10 10
     
    
    11
    +
    
    11 12
     def main():
    
    12 13
         issues = set(kernel_sec.issue.get_list())
    
    13 14
         for cve_id in issues:
    
    14 15
             issue = kernel_sec.issue.load(cve_id)
    
    15 16
             kernel_sec.issue.save(cve_id, issue)
    
    16 17
     
    
    18
    +
    
    17 19
     if __name__ == '__main__':
    
    18 20
         main()

  • scripts/import_debian.py
    ... ... @@ -20,6 +20,7 @@ import sys
    20 20
     
    
    21 21
     import kernel_sec.issue
    
    22 22
     
    
    23
    +
    
    23 24
     IMPORT_DIR = 'import/debian'
    
    24 25
     
    
    25 26
     LINE_BREAK_RE = re.compile(r'\n\s*')
    
    ... ... @@ -30,6 +31,7 @@ STATUS_RE = re.compile(r'^\s*(?P<state>\S*)'
    30 31
                            r'\s*(\[(?P<changerefs>(\S*,\s*)*\S*)\s*\])'
    
    31 32
                            r'|"(?P<reason>.+)")?')
    
    32 33
     
    
    34
    +
    
    33 35
     def load_debian_issue(f):
    
    34 36
         deb_issue = deb822.Deb822(f)
    
    35 37
         issue = {}
    
    ... ... @@ -86,6 +88,7 @@ def load_debian_issue(f):
    86 88
     
    
    87 89
         return issue
    
    88 90
     
    
    91
    +
    
    89 92
     def main():
    
    90 93
         # Remove obsolete Subversion working directory
    
    91 94
         if os.path.isdir(IMPORT_DIR + '/.svn'):
    
    ... ... @@ -96,10 +99,10 @@ def main():
    96 99
         if os.path.isdir(IMPORT_DIR + '/.git'):
    
    97 100
             subprocess.check_call(['git', 'pull'], cwd=IMPORT_DIR)
    
    98 101
         else:
    
    99
    -        subprocess.check_call(['git', 'clone',
    
    100
    -                               'https://salsa.debian.org/kernel-team/kernel-sec.git',
    
    101
    -                               '.'],
    
    102
    -                              cwd=IMPORT_DIR)
    
    102
    +        subprocess.check_call(
    
    103
    +            ['git', 'clone',
    
    104
    +             'https://salsa.debian.org/kernel-team/kernel-sec.git', '.'],
    
    105
    +            cwd=IMPORT_DIR)
    
    103 106
     
    
    104 107
         our_issues = set(kernel_sec.issue.get_list())
    
    105 108
         their_issues = dict((os.path.basename(name), name) for name in
    
    ... ... @@ -126,9 +129,9 @@ def main():
    126 129
                 # Copy theirs
    
    127 130
                 ours = theirs
    
    128 131
             else:
    
    129
    -            # Merge into ours
    
    132
    +            # Check that it's good to start with, then merge into ours
    
    130 133
                 ours = kernel_sec.issue.load(cve_id)
    
    131
    -            kernel_sec.issue.validate(ours) # check that it's good to start with
    
    134
    +            kernel_sec.issue.validate(ours)
    
    132 135
                 if not kernel_sec.issue.merge_into(ours, theirs):
    
    133 136
                     continue
    
    134 137
     
    
    ... ... @@ -140,5 +143,6 @@ def main():
    140 143
     
    
    141 144
             kernel_sec.issue.save(cve_id, ours)
    
    142 145
     
    
    146
    +
    
    143 147
     if __name__ == '__main__':
    
    144 148
         main()

  • scripts/import_stable.py
    ... ... @@ -13,9 +13,10 @@ import argparse
    13 13
     import io
    
    14 14
     import re
    
    15 15
     import subprocess
    
    16
    -import sys
    
    17 16
     
    
    18
    -import kernel_sec.branch, kernel_sec.issue
    
    17
    +import kernel_sec.branch
    
    18
    +import kernel_sec.issue
    
    19
    +
    
    19 20
     
    
    20 21
     BACKPORT_COMMIT_RE = re.compile(
    
    21 22
         r'^(?:' r'commit (%s) upstream\.'
    
    ... ... @@ -24,10 +25,12 @@ BACKPORT_COMMIT_RE = re.compile(
    24 25
         r')$'
    
    25 26
         % ((r'[0-9a-f]{40}',) * 3))
    
    26 27
     
    
    28
    +
    
    27 29
     def update(git_repo, remote_name):
    
    28 30
         subprocess.check_call(['git', 'remote', 'update', remote_name],
    
    29 31
                               cwd=git_repo)
    
    30 32
     
    
    33
    +
    
    31 34
     def get_backports(git_repo, remote_name):
    
    32 35
         branches = kernel_sec.branch.get_stable_branches(git_repo, remote_name)
    
    33 36
         backports = {}
    
    ... ... @@ -35,7 +38,8 @@ def get_backports(git_repo, remote_name):
    35 38
         for branch_name in branches:
    
    36 39
             base_ver = kernel_sec.branch.get_stable_branch_base_ver(branch_name)
    
    37 40
             log_proc = subprocess.Popen(
    
    38
    -            # Format with hash on one line, body on following lines indented by 1
    
    41
    +            # Format with hash on one line, body on following lines indented
    
    42
    +            # by 1
    
    39 43
                 ['git', 'log', '--no-notes', '--pretty=%H%n%w(0,1,1)%b',
    
    40 44
                  'v%s..%s/%s' % (base_ver, remote_name, branch_name)],
    
    41 45
                 cwd=git_repo, stdout=subprocess.PIPE)
    
    ... ... @@ -49,11 +53,12 @@ def get_backports(git_repo, remote_name):
    49 53
                     if match:
    
    50 54
                         mainline_commit = match.group(1) or match.group(2) \
    
    51 55
                                           or match.group(3)
    
    52
    -                    backports.setdefault(mainline_commit, {}) \
    
    53
    -                        [branch_name] = stable_commit
    
    56
    +                    backports.setdefault(mainline_commit, {})[branch_name] \
    
    57
    +                        = stable_commit
    
    54 58
     
    
    55 59
         return backports
    
    56 60
     
    
    61
    +
    
    57 62
     def add_backports(issue_commits, all_backports):
    
    58 63
         try:
    
    59 64
             mainline_commits = issue_commits['mainline']
    
    ... ... @@ -84,6 +89,7 @@ def add_backports(issue_commits, all_backports):
    84 89
     
    
    85 90
         return changed
    
    86 91
     
    
    92
    +
    
    87 93
     def main(git_repo, remote_name):
    
    88 94
         update(git_repo, remote_name)
    
    89 95
         backports = get_backports(git_repo, remote_name)
    
    ... ... @@ -102,17 +108,20 @@ def main(git_repo, remote_name):
    102 108
             if changed:
    
    103 109
                 kernel_sec.issue.save(cve_id, issue)
    
    104 110
     
    
    111
    +
    
    105 112
     if __name__ == '__main__':
    
    106 113
         parser = argparse.ArgumentParser(
    
    107
    -        description=('Import information about backported fixes and regressions '
    
    108
    -                     'from commit messages on stable branches.'))
    
    114
    +        description=('Import information about backported fixes and '
    
    115
    +                     'regressions from commit messages on stable branches.'))
    
    109 116
         parser.add_argument('--git-repo',
    
    110 117
                             dest='git_repo', default='../kernel',
    
    111
    -                        help='git repository from which to read commit logs (default: ../kernel)',
    
    118
    +                        help=('git repository from which to read commit logs '
    
    119
    +                              '(default: ../kernel)'),
    
    112 120
                             metavar='DIRECTORY')
    
    113 121
         parser.add_argument('--stable-remote',
    
    114 122
                             dest='stable_remote_name', default='stable',
    
    115
    -                        help='git remote for stable branches (default: stable)',
    
    123
    +                        help=('git remote for stable branches '
    
    124
    +                              '(default: stable)'),
    
    116 125
                             metavar='NAME')
    
    117 126
         args = parser.parse_args()
    
    118 127
         main(args.git_repo, args.stable_remote_name)

  • scripts/import_ubuntu.py
    ... ... @@ -21,6 +21,7 @@ import sys
    21 21
     
    
    22 22
     import kernel_sec.issue
    
    23 23
     
    
    24
    +
    
    24 25
     IMPORT_DIR = 'import/ubuntu'
    
    25 26
     
    
    26 27
     BREAK_FIX_RE = re.compile(r'^break-fix: (?:([0-9a-f]{40})|[-\w]+)'
    
    ... ... @@ -29,6 +30,7 @@ DISCOVERED_BY_SEP_RE = re.compile(r'(?:,\s*(?:and\s+)?|\s+and\s+)')
    29 30
     COMMENT_RE = re.compile(r'^(\w+)>\s+(.*)$')
    
    30 31
     DESCRIPTION_ANDROID_RE = re.compile(r'\bAndroid\b')
    
    31 32
     
    
    33
    +
    
    32 34
     # Based on load_cve() in scripts/cve_lib.py
    
    33 35
     def load_cve(cve, strict=False):
    
    34 36
         '''Loads a given CVE into:
    
    ... ... @@ -44,7 +46,7 @@ def load_cve(cve, strict=False):
    44 46
         code = EXIT_OKAY
    
    45 47
     
    
    46 48
         data = dict()
    
    47
    -    data.setdefault('tags',dict())
    
    49
    +    data.setdefault('tags', dict())
    
    48 50
         affected = dict()
    
    49 51
         lastfield = None
    
    50 52
         fields_seen = []
    
    ... ... @@ -64,7 +66,7 @@ def load_cve(cve, strict=False):
    64 66
                 continue
    
    65 67
     
    
    66 68
             try:
    
    67
    -            field, value = line.split(':',1)
    
    69
    +            field, value = line.split(':', 1)
    
    68 70
             except ValueError as e:
    
    69 71
                 msg += "%s: bad line '%s' (%s)\n" % (cve, line, e)
    
    70 72
                 code = EXIT_FAIL
    
    ... ... @@ -78,9 +80,11 @@ def load_cve(cve, strict=False):
    78 80
                 fields_seen.append(field)
    
    79 81
             value = value.strip()
    
    80 82
             if field == 'Candidate':
    
    81
    -            data.setdefault(field,value)
    
    82
    -            if value != "" and not value.startswith('CVE-') and not value.startswith('UEM-') and not value.startswith('EMB-'):
    
    83
    -                msg += "%s: unknown Candidate '%s' (must be /(CVE|UEM|EMB)-/)\n" % (cve, value)
    
    83
    +            data.setdefault(field, value)
    
    84
    +            if value != "" and not value.startswith('CVE-') and \
    
    85
    +               not value.startswith('UEM-') and not value.startswith('EMB-'):
    
    86
    +                msg += ("%s: unknown Candidate '%s' (must be "
    
    87
    +                        "/(CVE|UEM|EMB)-/)\n" % (cve, value))
    
    84 88
                     code = EXIT_FAIL
    
    85 89
             elif 'Priority' in field:
    
    86 90
                 # For now, throw away comments on Priority fields
    
    ... ... @@ -88,25 +92,26 @@ def load_cve(cve, strict=False):
    88 92
                     value = value.split()[0]
    
    89 93
                 if 'Priority_' in field:
    
    90 94
                     try:
    
    91
    -                    foo, pkg = field.split('_',1)
    
    95
    +                    foo, pkg = field.split('_', 1)
    
    92 96
                     except ValueError:
    
    93
    -                    msg += "%s: bad field with 'Priority_': '%s'\n" % (cve, field)
    
    97
    +                    msg += ("%s: bad field with 'Priority_': '%s'\n" %
    
    98
    +                            (cve, field))
    
    94 99
                         code = EXIT_FAIL
    
    95 100
                         continue
    
    96
    -            data.setdefault(field,value)
    
    101
    +            data.setdefault(field, value)
    
    97 102
             elif 'Patches_' in field:
    
    98 103
                 '''These are raw fields'''
    
    99 104
                 try:
    
    100
    -                foo, pkg = field.split('_',1)
    
    105
    +                foo, pkg = field.split('_', 1)
    
    101 106
                 except ValueError:
    
    102 107
                     msg += "%s: bad field with 'Patches_': '%s'\n" % (cve, field)
    
    103 108
                     code = EXIT_FAIL
    
    104 109
                     continue
    
    105
    -            data.setdefault(field,value)
    
    110
    +            data.setdefault(field, value)
    
    106 111
             elif 'Tags_' in field:
    
    107 112
                 '''These are processed into the "tags" hash'''
    
    108 113
                 try:
    
    109
    -                foo, pkg = field.split('_',1)
    
    114
    +                foo, pkg = field.split('_', 1)
    
    110 115
                 except ValueError:
    
    111 116
                     msg += "%s: bad field with 'Tags_': '%s'\n" % (cve, field)
    
    112 117
                     code = EXIT_FAIL
    
    ... ... @@ -116,15 +121,16 @@ def load_cve(cve, strict=False):
    116 121
                     data['tags'][pkg].add(word)
    
    117 122
             elif '_' in field:
    
    118 123
                 try:
    
    119
    -                release, pkg = field.split('_',1)
    
    124
    +                release, pkg = field.split('_', 1)
    
    120 125
                 except ValueError:
    
    121 126
                     msg += "%s: bad field with '_': '%s'\n" % (cve, field)
    
    122 127
                     code = EXIT_FAIL
    
    123 128
                     continue
    
    124 129
                 try:
    
    125
    -                info = value.split(' ',1)
    
    130
    +                info = value.split(' ', 1)
    
    126 131
                 except ValueError:
    
    127
    -                msg += "%s: missing state for '%s': '%s'\n" % (cve, field, value)
    
    132
    +                msg += ("%s: missing state for '%s': '%s'\n" %
    
    133
    +                        (cve, field, value))
    
    128 134
                     code = EXIT_FAIL
    
    129 135
                     continue
    
    130 136
                 state = info[0]
    
    ... ... @@ -145,30 +151,32 @@ def load_cve(cve, strict=False):
    145 151
                     notes = state
    
    146 152
                     state = 'released'
    
    147 153
     
    
    148
    -            if state not in ['needs-triage','needed','active','pending','released','deferred','DNE','ignored','not-affected']:
    
    149
    -                msg += "%s: %s_%s has unknown state: '%s'\n" % (cve, release, pkg, state)
    
    154
    +            if state not in ['needs-triage', 'needed', 'active', 'pending',
    
    155
    +                             'released', 'deferred', 'DNE', 'ignored',
    
    156
    +                             'not-affected']:
    
    157
    +                msg += ("%s: %s_%s has unknown state: '%s'\n" %
    
    158
    +                        (cve, release, pkg, state))
    
    150 159
                     code = EXIT_FAIL
    
    151 160
     
    
    152
    -            # Verify "released" kernels have version notes
    
    153
    -            #if state == 'released' and pkg in kernel_srcs and notes == '':
    
    154
    -            #    msg += "%s: %s_%s has state '%s' but lacks version note\n" % (cve, release, pkg, state)
    
    155
    -            #    code = EXIT_FAIL
    
    156
    -
    
    157 161
                 # Verify "active" states have an Assignee
    
    158 162
                 if state == 'active' and data['Assigned-to'].strip() == "":
    
    159
    -                msg += "%s: %s_%s has state '%s' but lacks 'Assigned-to'\n" % (cve, release, pkg, state)
    
    163
    +                msg += ("%s: %s_%s has state '%s' but lacks 'Assigned-to'\n" %
    
    164
    +                        (cve, release, pkg, state))
    
    160 165
                     code = EXIT_FAIL
    
    161 166
     
    
    162
    -            affected.setdefault(pkg,dict())
    
    163
    -            affected[pkg].setdefault(release,[state,notes])
    
    164
    -        elif field not in ['References', 'Description', 'Ubuntu-Description', 'Notes', 'Bugs', 'Assigned-to', 'Approved-by', 'PublicDate', 'PublicDateAtUSN', 'CRD', 'Discovered-by']:
    
    167
    +            affected.setdefault(pkg, dict())
    
    168
    +            affected[pkg].setdefault(release, [state, notes])
    
    169
    +        elif field not in ['References', 'Description', 'Ubuntu-Description',
    
    170
    +                           'Notes', 'Bugs', 'Assigned-to', 'Approved-by',
    
    171
    +                           'PublicDate', 'PublicDateAtUSN', 'CRD',
    
    172
    +                           'Discovered-by']:
    
    165 173
                 msg += "%s: unknown field '%s'\n" % (cve, field)
    
    166 174
                 code = EXIT_FAIL
    
    167 175
             else:
    
    168
    -            data.setdefault(field,value)
    
    176
    +            data.setdefault(field, value)
    
    169 177
     
    
    170 178
         # Check for required fields
    
    171
    -    for field in ['Candidate','PublicDate','Description']:
    
    179
    +    for field in ['Candidate', 'PublicDate', 'Description']:
    
    172 180
             if field not in data:
    
    173 181
                 msg += "%s: missing field '%s'\n" % (cve, field)
    
    174 182
                 code = EXIT_FAIL
    
    ... ... @@ -181,7 +189,7 @@ def load_cve(cve, strict=False):
    181 189
     
    
    182 190
         # Fill in defaults for missing fields
    
    183 191
         if 'Priority' not in data:
    
    184
    -        data.setdefault('Priority','untriaged')
    
    192
    +        data.setdefault('Priority', 'untriaged')
    
    185 193
         # Perform override fields
    
    186 194
         if 'PublicDateAtUSN' in data:
    
    187 195
             data['PublicDate'] = data['PublicDateAtUSN']
    
    ... ... @@ -194,9 +202,11 @@ def load_cve(cve, strict=False):
    194 202
             raise ValueError(msg.strip())
    
    195 203
         return data
    
    196 204
     
    
    205
    +
    
    197 206
     class NonKernelIssue(Exception):
    
    198 207
         pass
    
    199 208
     
    
    209
    +
    
    200 210
     def load_ubuntu_issue(f):
    
    201 211
         ubu_issue = load_cve(f)
    
    202 212
         issue = {}
    
    ... ... @@ -249,6 +259,7 @@ def load_ubuntu_issue(f):
    249 259
     
    
    250 260
         return issue
    
    251 261
     
    
    262
    +
    
    252 263
     # Ubuntu doesn't seem to retire issues any more, so only include issues
    
    253 264
     # that are active and discovered either this year or last year
    
    254 265
     def get_recent_issues():
    
    ... ... @@ -259,13 +270,15 @@ def get_recent_issues():
    259 270
             if year >= this_year - 1:
    
    260 271
                 yield (cve_id, filename)
    
    261 272
     
    
    273
    +
    
    262 274
     def main():
    
    263 275
         os.makedirs(IMPORT_DIR, 0o777, exist_ok=True)
    
    264 276
         if os.path.isdir(IMPORT_DIR + '/.bzr'):
    
    265 277
             subprocess.check_call(['bzr', 'update'], cwd=IMPORT_DIR)
    
    266 278
         else:
    
    267
    -        subprocess.check_call(['bzr', 'checkout', 'lp:ubuntu-cve-tracker', '.'],
    
    268
    -                              cwd=IMPORT_DIR)
    
    279
    +        subprocess.check_call(
    
    280
    +            ['bzr', 'checkout', 'lp:ubuntu-cve-tracker', '.'],
    
    281
    +            cwd=IMPORT_DIR)
    
    269 282
     
    
    270 283
         our_issues = set(kernel_sec.issue.get_list())
    
    271 284
         their_issues = dict(get_recent_issues())
    
    ... ... @@ -293,9 +306,9 @@ def main():
    293 306
                 # Copy theirs
    
    294 307
                 ours = theirs
    
    295 308
             else:
    
    296
    -            # Merge into ours
    
    309
    +            # Check that it's good to start with, then merge into ours
    
    297 310
                 ours = kernel_sec.issue.load(cve_id)
    
    298
    -            kernel_sec.issue.validate(ours) # check that it's good to start with
    
    311
    +            kernel_sec.issue.validate(ours)
    
    299 312
                 if not kernel_sec.issue.merge_into(ours, theirs):
    
    300 313
                     continue
    
    301 314
     
    
    ... ... @@ -307,5 +320,6 @@ def main():
    307 320
     
    
    308 321
             kernel_sec.issue.save(cve_id, ours)
    
    309 322
     
    
    323
    +
    
    310 324
     if __name__ == '__main__':
    
    311 325
         main()

  • scripts/kernel_sec/branch.py
    ... ... @@ -4,18 +4,25 @@
    4 4
     # Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
    
    5 5
     # for details.
    
    6 6
     
    
    7
    +import io
    
    7 8
     import re
    
    8 9
     import subprocess
    
    9 10
     
    
    11
    +from . import version
    
    12
    +
    
    13
    +
    
    10 14
     _STABLE_BRANCH_RE = re.compile(r'^linux-([\d.]+)\.y$')
    
    11 15
     
    
    16
    +
    
    12 17
     def get_base_ver_stable_branch(base_ver):
    
    13 18
         return 'linux-%s.y' % base_ver
    
    14 19
     
    
    20
    +
    
    15 21
     def get_stable_branch_base_ver(branch_name):
    
    16 22
         match = _STABLE_BRANCH_RE.match(branch_name)
    
    17 23
         return match and match.group(1)
    
    18 24
     
    
    25
    +
    
    19 26
     def get_stable_branches(git_repo, remote_name='stable'):
    
    20 27
         branches = []
    
    21 28
     
    
    ... ... @@ -36,6 +43,7 @@ def get_stable_branches(git_repo, remote_name='stable'):
    36 43
     
    
    37 44
         return branches
    
    38 45
     
    
    46
    +
    
    39 47
     def get_live_stable_branches(*args, **kwargs):
    
    40 48
         # TODO: Pull list of longterm branches from
    
    41 49
         # https://www.kernel.org/category/releases.html ?
    
    ... ... @@ -61,3 +69,49 @@ def get_live_stable_branches(*args, **kwargs):
    61 69
     
    
    62 70
         return [branch_name for branch_name in get_stable_branches(*args, **kwargs)
    
    63 71
                 if branch_name not in dead_branches]
    
    72
    +
    
    73
    +
    
    74
    +def get_sort_key(branch):
    
    75
    +    if branch == 'mainline':
    
    76
    +        return [1000000]
    
    77
    +    base_ver = get_stable_branch_base_ver(branch)
    
    78
    +    assert base_ver is not None
    
    79
    +    return version.get_sort_key(base_ver)
    
    80
    +
    
    81
    +
    
    82
    +def _get_commits(git_repo, end, start=None):
    
    83
    +    if start:
    
    84
    +        list_expr = '%s..%s' % (start, end)
    
    85
    +    else:
    
    86
    +        list_expr = end
    
    87
    +
    
    88
    +    list_proc = subprocess.Popen(['git', 'rev-list', list_expr],
    
    89
    +                                 cwd=git_repo, stdout=subprocess.PIPE)
    
    90
    +    for line in io.TextIOWrapper(list_proc.stdout):
    
    91
    +        yield line.rstrip('\n')
    
    92
    +
    
    93
    +
    
    94
    +class CommitBranchMap:
    
    95
    +    def __init__(self, git_repo, mainline_remote_name, branch_names):
    
    96
    +        # Generate sort key for each branch
    
    97
    +        self._branch_sort_key = {
    
    98
    +            branch: get_sort_key(branch) for branch in branch_names
    
    99
    +        }
    
    100
    +
    
    101
    +        # Generate sort key for each commit
    
    102
    +        self._commit_sort_key = {}
    
    103
    +        start = None
    
    104
    +        for branch in sorted(branch_names, key=get_sort_key):
    
    105
    +            if branch == 'mainline':
    
    106
    +                end = '%s/master' % mainline_remote_name
    
    107
    +            else:
    
    108
    +                base_ver = get_stable_branch_base_ver(branch)
    
    109
    +                assert base_ver is not None
    
    110
    +                end = 'v' + base_ver
    
    111
    +            for commit in _get_commits(git_repo, end, start):
    
    112
    +                self._commit_sort_key[commit] = self._branch_sort_key[branch]
    
    113
    +            start = end
    
    114
    +
    
    115
    +    def is_commit_in_branch(self, commit, branch):
    
    116
    +        return commit in self._commit_sort_key and \
    
    117
    +            self._commit_sort_key[commit] <= self._branch_sort_key[branch]

  • scripts/kernel_sec/issue.py
    ... ... @@ -13,21 +13,27 @@ import yaml
    13 13
     import yaml.dumper
    
    14 14
     import yaml.loader
    
    15 15
     
    
    16
    +
    
    16 17
     # Only SHA-1 for now
    
    17 18
     _GIT_HASH_RE = re.compile(r'^[0-9a-f]{40}$')
    
    19
    +
    
    20
    +
    
    18 21
     def is_git_hash(s):
    
    19 22
         return _GIT_HASH_RE.match(s) is not None
    
    20 23
     
    
    24
    +
    
    21 25
     def _validate_string(name, value):
    
    22 26
         if type(value) is str:
    
    23 27
             return
    
    24 28
         raise ValueError('%s must be a string' % name)
    
    25 29
     
    
    30
    +
    
    26 31
     def _validate_datetime(name, value):
    
    27 32
         if type(value) is datetime.datetime:
    
    28 33
             return
    
    29 34
         raise ValueError('%s must be an ISO8601 date and time' % name)
    
    30 35
     
    
    36
    +
    
    31 37
     def _validate_sequence_string(name, value):
    
    32 38
         if type(value) is list:
    
    33 39
             for entry in value:
    
    ... ... @@ -37,6 +43,7 @@ def _validate_sequence_string(name, value):
    37 43
                 return
    
    38 44
         raise ValueError('%s must be a sequence of strings' % name)
    
    39 45
     
    
    46
    +
    
    40 47
     def _validate_mapping_string(name, value):
    
    41 48
         if type(value) is dict:
    
    42 49
             for v in value.values():
    
    ... ... @@ -46,6 +53,7 @@ def _validate_mapping_string(name, value):
    46 53
                 return
    
    47 54
         raise ValueError('%s must be a mapping to strings' % name)
    
    48 55
     
    
    56
    +
    
    49 57
     def _validate_mapping_sequence_hash(name, value):
    
    50 58
         if type(value) is dict:
    
    51 59
             for seq in value.values():
    
    ... ... @@ -55,15 +63,16 @@ def _validate_mapping_sequence_hash(name, value):
    55 63
                     break
    
    56 64
                 for entry in seq:
    
    57 65
                     if type(entry) is not str or not is_git_hash(entry):
    
    58
    -                    break # to outer break
    
    66
    +                    break  # to outer break
    
    59 67
                 else:
    
    60 68
                     continue
    
    61
    -            break # to top level
    
    69
    +            break  # to top level
    
    62 70
             else:
    
    63 71
                 return
    
    64 72
         raise ValueError('%s must be a mapping to sequences of git hashes' %
    
    65 73
                          name)
    
    66 74
     
    
    75
    +
    
    67 76
     def _validate_hashmapping_string(name, value):
    
    68 77
         if type(value) is dict:
    
    69 78
             for h, v in value.items():
    
    ... ... @@ -76,6 +85,7 @@ def _validate_hashmapping_string(name, value):
    76 85
         raise ValueError('%s must be a mapping from git hashes to strings' %
    
    77 86
                          name)
    
    78 87
     
    
    88
    +
    
    79 89
     _ALL_FIELDS = [
    
    80 90
         ('description',     _validate_string),
    
    81 91
         ('advisory',        _validate_string),
    
    ... ... @@ -94,6 +104,7 @@ _FIELD_VALIDATOR = dict(_ALL_FIELDS)
    94 104
     _FIELD_ORDER = dict((name, i) for i, (name, _) in enumerate(_ALL_FIELDS))
    
    95 105
     _REQUIRED_FIELDS = ['description']
    
    96 106
     
    
    107
    +
    
    97 108
     def validate(issue):
    
    98 109
         for name in _REQUIRED_FIELDS:
    
    99 110
             if name not in issue:
    
    ... ... @@ -107,6 +118,7 @@ def validate(issue):
    107 118
             else:
    
    108 119
                 validator(name, value)
    
    109 120
     
    
    121
    +
    
    110 122
     def merge_into(ours, theirs):
    
    111 123
         changed = False
    
    112 124
     
    
    ... ... @@ -161,6 +173,7 @@ def merge_into(ours, theirs):
    161 173
     
    
    162 174
         return changed
    
    163 175
     
    
    176
    +
    
    164 177
     class _IssueDumper(yaml.dumper.SafeDumper):
    
    165 178
         # Write strings as UTF-8, not ASCII with escapes
    
    166 179
         def __init__(self, *args, **kwargs):
    
    ... ... @@ -174,8 +187,8 @@ class _IssueDumper(yaml.dumper.SafeDumper):
    174 187
             finally:
    
    175 188
                 del self.__root
    
    176 189
     
    
    177
    -    # ISO 8601 specifies 'T' to separate date & time, but for some reason PyYAML
    
    178
    -    # uses ' ' by default
    
    190
    +    # ISO 8601 specifies 'T' to separate date & time, but for some reason
    
    191
    +    # PyYAML uses ' ' by default
    
    179 192
         def represent_datetime(self, data):
    
    180 193
             return self.represent_scalar('tag:yaml.org,2002:timestamp',
    
    181 194
                                          data.isoformat())
    
    ... ... @@ -205,15 +218,18 @@ class _IssueDumper(yaml.dumper.SafeDumper):
    205 218
     
    
    206 219
             return super().represent_sequence(tag, sequence, flow_style)
    
    207 220
     
    
    208
    -_IssueDumper.add_representer(datetime.datetime, _IssueDumper.represent_datetime)
    
    221
    +
    
    222
    +_IssueDumper.add_representer(datetime.datetime,
    
    223
    +                             _IssueDumper.represent_datetime)
    
    209 224
     _IssueDumper.add_representer(str, _IssueDumper.represent_str)
    
    210 225
     
    
    226
    +
    
    211 227
     class _IssueLoader(yaml.loader.SafeLoader):
    
    212 228
         # Keep timezone information instead of adjusting the timestamp and then
    
    213 229
         # discarding it.
    
    214 230
         def construct_yaml_timestamp(self, node):
    
    215 231
             value = self.construct_scalar(node)
    
    216
    -        match = self.timestamp_regexp.match(node.value)
    
    232
    +        match = self.timestamp_regexp.match(value)
    
    217 233
             values = match.groupdict()
    
    218 234
             year = int(values['year'])
    
    219 235
             month = int(values['month'])
    
    ... ... @@ -243,32 +259,73 @@ class _IssueLoader(yaml.loader.SafeLoader):
    243 259
             return datetime.datetime(year, month, day, hour, minute, second,
    
    244 260
                                      fraction, tzinfo)
    
    245 261
     
    
    262
    +
    
    246 263
     _IssueLoader.add_constructor('tag:yaml.org,2002:timestamp',
    
    247 264
                                  _IssueLoader.construct_yaml_timestamp)
    
    248 265
     
    
    266
    +
    
    249 267
     def load_filename(name):
    
    250 268
         with open(name) as f:
    
    251 269
             return yaml.load(f, Loader=_IssueLoader)
    
    252 270
     
    
    271
    +
    
    253 272
     def save_filename(name, issue):
    
    254 273
         with open(name, 'w') as f:
    
    255 274
             yaml.dump(issue, f, Dumper=_IssueDumper)
    
    256 275
     
    
    276
    +
    
    257 277
     def get_list():
    
    258
    -    return [os.path.basename(name)[:-4] for name in glob.glob('issues/CVE-*.yml')]
    
    278
    +    return [os.path.basename(name)[:-4]
    
    279
    +            for name in glob.glob('issues/CVE-*.yml')]
    
    280
    +
    
    259 281
     
    
    260 282
     def get_filename(cve_id):
    
    261 283
         return 'issues/%s.yml' % cve_id
    
    262 284
     
    
    285
    +
    
    263 286
     def load(cve_id):
    
    264 287
         return load_filename(get_filename(cve_id))
    
    265 288
     
    
    289
    +
    
    266 290
     def save(cve_id, issue):
    
    267 291
         save_filename(get_filename(cve_id), issue)
    
    268 292
     
    
    293
    +
    
    269 294
     # Match the "arbitrary digits" after the year
    
    270 295
     _cve_id_arbdig_re = re.compile(r'-(\d+)$')
    
    271 296
     
    
    297
    +
    
    272 298
     # Pad "arbitrary digits" to 6 digits so string comparison works
    
    273 299
     def get_id_sort_key(cve_id):
    
    274 300
         return _cve_id_arbdig_re.sub(lambda m: '-%06d' % int(m.group(1)), cve_id)
    
    301
    +
    
    302
    +
    
    303
    +def affects_branch(issue, branch, is_commit_in_branch):
    
    304
    +    # If it was not introduced on this branch, and was introduced on
    
    305
    +    # mainline after the branch point, branch is not affected
    
    306
    +    introduced = issue.get('introduced-by')
    
    307
    +    if introduced:
    
    308
    +        if introduced.get('mainline') == 'never' and \
    
    309
    +           (branch == 'mainline' or branch not in introduced):
    
    310
    +            return False
    
    311
    +        if branch not in introduced:
    
    312
    +            for commit in introduced['mainline']:
    
    313
    +                if is_commit_in_branch(commit, branch):
    
    314
    +                    break
    
    315
    +            else:
    
    316
    +                return False
    
    317
    +
    
    318
    +    # If it was fixed on this branch, or fixed on mainline before
    
    319
    +    # the branch point, branch is not affected
    
    320
    +    fixed = issue.get('fixed-by', {})
    
    321
    +    if fixed:
    
    322
    +        if fixed.get(branch, 'never') != 'never':
    
    323
    +            return False
    
    324
    +        if fixed.get('mainline', 'never') != 'never':
    
    325
    +            for commit in fixed['mainline']:
    
    326
    +                if not is_commit_in_branch(commit, branch):
    
    327
    +                    break
    
    328
    +            else:
    
    329
    +                return False
    
    330
    +
    
    331
    +    return True

  • scripts/kernel_sec/version.py
    ... ... @@ -6,8 +6,10 @@
    6 6
     
    
    7 7
     import re
    
    8 8
     
    
    9
    +
    
    9 10
     _RC_RE = re.compile('-rc(\d+)$')
    
    10 11
     
    
    12
    +
    
    11 13
     def _split(ver_str):
    
    12 14
         # Split off any '-rc' part; split rest at dots; map to integers
    
    13 15
         match = _RC_RE.search(ver_str)
    
    ... ... @@ -18,6 +20,7 @@ def _split(ver_str):
    18 20
             rc_n = None
    
    19 21
         return [int(comp) for comp in ver_str.split('.')], rc_n
    
    20 22
     
    
    23
    +
    
    21 24
     def get_sort_key(ver_str):
    
    22 25
         # Treat x -rc y as (x-1), (large), y so it sorts between x-1 stable updates
    
    23 26
         # and x
    

  • scripts/report_affected.py
    ... ... @@ -9,26 +9,15 @@
    9 9
     # Report issues affecting each stable branch.
    
    10 10
     
    
    11 11
     import argparse
    
    12
    -import io
    
    13
    -import re
    
    14 12
     import subprocess
    
    15
    -import sys
    
    16 13
     
    
    17
    -import kernel_sec.branch, kernel_sec.issue, kernel_sec.version
    
    14
    +import kernel_sec.branch
    
    15
    +import kernel_sec.issue
    
    16
    +import kernel_sec.version
    
    18 17
     
    
    19
    -def get_commits(git_repo, end, start=None):
    
    20
    -    if start:
    
    21
    -        list_expr = '%s..%s' % (start, end)
    
    22
    -    else:
    
    23
    -        list_expr = end
    
    24
    -
    
    25
    -    list_proc = subprocess.Popen(['git', 'rev-list', list_expr],
    
    26
    -                                 cwd=git_repo, stdout=subprocess.PIPE)
    
    27
    -    for line in io.TextIOWrapper(list_proc.stdout):
    
    28
    -        yield line.rstrip('\n')
    
    29 18
     
    
    30
    -def main(git_repo, mainline_remote_name, stable_remote_name, only_fixed_upstream,
    
    31
    -         include_ignored, *branch_names):
    
    19
    +def main(git_repo, mainline_remote_name, stable_remote_name,
    
    20
    +         only_fixed_upstream, include_ignored, *branch_names):
    
    32 21
         if branch_names:
    
    33 22
             # Support stable release strings as shorthand for stable branches
    
    34 23
             branch_names = [kernel_sec.branch.get_base_ver_stable_branch(name)
    
    ... ... @@ -40,90 +29,45 @@ def main(git_repo, mainline_remote_name, stable_remote_name, only_fixed_upstream
    40 29
             if not only_fixed_upstream:
    
    41 30
                 branch_names.append('mainline')
    
    42 31
     
    
    43
    -    # Generate sort key for each branch
    
    44
    -    branch_sort_key = {}
    
    45
    -    for branch in branch_names:
    
    46
    -        if branch == 'mainline':
    
    47
    -            branch_sort_key[branch] = [1000000]
    
    48
    -        else:
    
    49
    -            base_ver = kernel_sec.branch.get_stable_branch_base_ver(branch)
    
    50
    -            assert base_ver is not None
    
    51
    -            branch_sort_key[branch] = kernel_sec.version.get_sort_key(base_ver)
    
    52
    -
    
    53
    -    branch_names.sort(key=(lambda branch: branch_sort_key[branch]))
    
    54
    -
    
    55
    -    # Generate sort key for each commit
    
    56
    -    commit_sort_key = {}
    
    57
    -    start = None
    
    58
    -    for branch in branch_names:
    
    59
    -        if branch == 'mainline':
    
    60
    -            end = '%s/master' % mainline_remote_name
    
    61
    -        else:
    
    62
    -            base_ver = kernel_sec.branch.get_stable_branch_base_ver(branch)
    
    63
    -            assert base_ver is not None
    
    64
    -            end = 'v' + base_ver
    
    65
    -        for commit in get_commits(git_repo, end, start):
    
    66
    -            commit_sort_key[commit] = branch_sort_key[branch]
    
    67
    -        start = end
    
    32
    +    branch_names.sort(key=kernel_sec.branch.get_sort_key)
    
    33
    +
    
    34
    +    c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, mainline_remote_name,
    
    35
    +                                                branch_names)
    
    68 36
     
    
    69 37
         branch_issues = {}
    
    70 38
         issues = set(kernel_sec.issue.get_list())
    
    71 39
     
    
    72 40
         for cve_id in issues:
    
    73 41
             issue = kernel_sec.issue.load(cve_id)
    
    42
    +        ignore = issue.get('ignore', {})
    
    43
    +        fixed = issue.get('fixed-by', {})
    
    74 44
     
    
    75
    -        for branch in branch_names:
    
    76
    -            # If it was not introduced on this branch, and was introduced on
    
    77
    -            # mainline after the branch point, branch is not affected
    
    78
    -            introduced = issue.get('introduced-by')
    
    79
    -            if introduced:
    
    80
    -                if introduced.get('mainline') == 'never' and \
    
    81
    -                   (branch == 'mainline' or branch not in introduced):
    
    82
    -                    continue
    
    83
    -                if branch not in introduced:
    
    84
    -                    for commit in introduced['mainline']:
    
    85
    -                        if commit in commit_sort_key \
    
    86
    -                           and commit_sort_key[commit] <= branch_sort_key[branch]:
    
    87
    -                            break
    
    88
    -                    else:
    
    89
    -                        continue
    
    90
    -
    
    91
    -            # Check whether it is ignored on this branch, unless we're
    
    92
    -            # overriding that
    
    93
    -            ignore = issue.get('ignore', {})
    
    94
    -            if not include_ignored and (ignore.get('all') or ignore.get(branch)):
    
    95
    -                continue
    
    96
    -
    
    97
    -            fixed = issue.get('fixed-by', {})
    
    45
    +        # Should this issue be ignored?
    
    46
    +        if (not include_ignored and ignore.get('all')) or \
    
    47
    +           (only_fixed_upstream and fixed.get('mainline', 'never') == 'never'):
    
    48
    +            continue
    
    98 49
     
    
    99
    -            if only_fixed_upstream and fixed.get('mainline', 'never') == 'never':
    
    50
    +        for branch in branch_names:
    
    51
    +            # Should this issue be ignored for this branch?
    
    52
    +            if not include_ignored and ignore.get(branch):
    
    100 53
                     continue
    
    101 54
     
    
    102
    -            # If it was fixed on this branch, or fixed on mainline before
    
    103
    -            # the branch point, branch is not affected
    
    104
    -            if fixed:
    
    105
    -                if fixed.get(branch, 'never') != 'never':
    
    106
    -                    continue
    
    107
    -                if fixed.get('mainline', 'never') != 'never':
    
    108
    -                    for commit in fixed['mainline']:
    
    109
    -                        if commit not in commit_sort_key \
    
    110
    -                           or commit_sort_key[commit] > branch_sort_key[branch]:
    
    111
    -                            break
    
    112
    -                    else:
    
    113
    -                        continue
    
    114
    -
    
    115
    -            branch_issues.setdefault(branch, []).append(cve_id)
    
    55
    +            if kernel_sec.issue.affects_branch(
    
    56
    +                    issue, branch, c_b_map.is_commit_in_branch):
    
    57
    +                branch_issues.setdefault(branch, []).append(cve_id)
    
    116 58
     
    
    117 59
         for branch in branch_names:
    
    118 60
             print('%s:' % branch, *sorted(branch_issues.get(branch, []),
    
    119 61
                                           key=kernel_sec.issue.get_id_sort_key))
    
    120 62
     
    
    63
    +
    
    121 64
     if __name__ == '__main__':
    
    122 65
         parser = argparse.ArgumentParser(
    
    123 66
             description='Report unfixed CVEs in Linux kernel branches.')
    
    124 67
         parser.add_argument('--git-repo',
    
    125 68
                             dest='git_repo', default='../kernel',
    
    126
    -                        help='git repository from which to read commit logs (default: ../kernel)',
    
    69
    +                        help=('git repository from which to read commit logs '
    
    70
    +                              '(default: ../kernel)'),
    
    127 71
                             metavar='DIRECTORY')
    
    128 72
         parser.add_argument('--mainline-remote',
    
    129 73
                             dest='mainline_remote_name', default='torvalds',
    
    ... ... @@ -131,7 +75,8 @@ if __name__ == '__main__':
    131 75
                             metavar='NAME')
    
    132 76
         parser.add_argument('--stable-remote',
    
    133 77
                             dest='stable_remote_name', default='stable',
    
    134
    -                        help='git remote for stable branches (default: stable)',
    
    78
    +                        help=('git remote for stable branches '
    
    79
    +                              '(default: stable)'),
    
    135 80
                             metavar='NAME')
    
    136 81
         parser.add_argument('--only-fixed-upstream',
    
    137 82
                             action='store_true',
    
    ... ... @@ -141,7 +86,8 @@ if __name__ == '__main__':
    141 86
                             help='include issues that have been marked as ignored')
    
    142 87
         parser.add_argument('branches',
    
    143 88
                             nargs='*',
    
    144
    -                        help='specific branch to report on (default: all active branches)',
    
    89
    +                        help=('specific branch to report on '
    
    90
    +                              '(default: all active branches)'),
    
    145 91
                             metavar='BRANCH')
    
    146 92
         args = parser.parse_args()
    
    147 93
         main(args.git_repo, args.mainline_remote_name, args.stable_remote_name,
    

  • scripts/templates/branch.html
    1
    +<link rel="stylesheet" href="/static/style.css">
    
    1 2
     <title>Branch {{ name }}</title>
    
    2 3
     <h1>Branch {{ name }}</h1>
    
    3
    -<p>
    
    4
    -  There should be a list of issues here...
    
    5
    -</p>
    4
    +<ul>
    
    5
    +  {% for cve_id, issue in issues %}
    
    6
    +  <li class="{% if issue.ignore and (issue.ignore.all or issue.ignore[name]) %}ignored{% endif %}">
    
    7
    +    <a href="/issues/{{ cve_id }}/">{{ cve_id }}</a> -
    
    8
    +    {{ issue.description|truncate(100) }}
    
    9
    +  </li>
    
    10
    +  {% endfor %}
    
    11
    +</ul>

  • scripts/templates/branches.html
    1
    +<link rel="stylesheet" href="/static/style.css">
    
    1 2
     <title>Branches</title>
    
    2 3
     <h1>Branches</h1>
    
    3 4
     <p>
    

  • scripts/templates/issue.html
    1
    +<link rel="stylesheet" href="/static/style.css">
    
    1 2
     <title>{{ cve_id }} - {{ issue.description|truncate(50) }}</title>
    
    2 3
     <h1>{{ cve_id }} - {{ issue.description|truncate(50) }}</h1>
    
    3 4
     <h2>Summary</h2>
    
    ... ... @@ -10,6 +11,7 @@
    10 11
       {% if issue.references %}
    
    11 12
       <tr>
    
    12 13
         <th>References</th>
    
    14
    +    <td/>
    
    13 15
         <td>
    
    14 16
           {% for url in issue.references %}
    
    15 17
           <a href="{{ url }}">{{ url }}</a>
    
    ... ... @@ -21,6 +23,7 @@
    21 23
       {% if issue.aliases %}
    
    22 24
       <tr>
    
    23 25
         <th>Aliases</th>
    
    26
    +    <td/>
    
    24 27
         <td>
    
    25 28
           {% for alias in issue.aliases %}
    
    26 29
           {{ alias }}{% if not loop.last %},{% endif %}  
    
    ... ... @@ -30,18 +33,25 @@
    30 33
       {% endif %}
    
    31 34
       {% if issue.comments %}
    
    32 35
       <tr>
    
    33
    -    <th>Comments</th>
    
    36
    +    <th rowspan={{ issue.comments|length }}>Comments</th>
    
    37
    +    {% for handle in issue.comments %}
    
    38
    +    <th>
    
    39
    +      {{ handle }}
    
    40
    +    </th>
    
    34 41
         <td>
    
    35
    -      {% for handle in issue.comments %}
    
    36
    -      {{ handle }}: <q>{{ issue.comments[handle] }}</q>
    
    37
    -      {% if not loop.last %}<br/>{% endif %}
    
    38
    -      {% endfor %}
    
    42
    +      <q>{{ issue.comments[handle] }}</q>
    
    39 43
         </td>
    
    44
    +    {% if not loop.last %}
    
    45
    +  </tr>
    
    46
    +  <tr>
    
    47
    +    {% endif %}
    
    48
    +    {% endfor %}
    
    40 49
       </tr>
    
    41 50
       {% endif %}
    
    42 51
       {% if issue.reporters %}
    
    43 52
       <tr>
    
    44 53
         <th>Reporters</th>
    
    54
    +    <td/>
    
    45 55
         <td>
    
    46 56
           {% for rep in issue.reporters %}
    
    47 57
           {# This may include an email address which could be linked #}
    
    ... ... @@ -53,6 +63,7 @@
    53 63
       {% if issue['embargo-end'] %}
    
    54 64
       <tr>
    
    55 65
         <th>Embargo</th>
    
    66
    +    <td/>
    
    56 67
         <td>
    
    57 68
           {# TODO: If embargo-end is in the future we should prominently
    
    58 69
           flag this issue as embargoed #}
    
    ... ... @@ -62,35 +73,46 @@
    62 73
       {% endif %}
    
    63 74
       {% if issue['introduced-by'] %}
    
    64 75
       <tr>
    
    65
    -    <th>Introduced by</th>
    
    76
    +    <th rowspan={{ issue['introduced-by']|length }}>Introduced by</th>
    
    77
    +    {% for branch in issue['introduced-by'] %}
    
    78
    +    <th>
    
    79
    +      {{ branch }}
    
    80
    +    </th>
    
    66 81
         <td>
    
    67
    -      {% for branch in issue['introduced-by'] %}
    
    68
    -      {{ branch }}:
    
    69 82
           {% for commit in issue['introduced-by'][branch] %}
    
    70 83
           <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
    
    71 84
           {% endfor %}
    
    72
    -      {% if not loop.last %}<br/>{% endif %}
    
    73
    -      {% endfor %}
    
    74 85
         </td>
    
    86
    +    {% if not loop.last %}
    
    87
    +  </tr>
    
    88
    +  <tr>
    
    89
    +    {% endif %}
    
    90
    +    {% endfor %}
    
    75 91
       </tr>
    
    76 92
       {% endif %}
    
    77 93
       {% if issue['fixed-by'] %}
    
    78 94
       <tr>
    
    79
    -    <th>Fixed by</th>
    
    95
    +    <th rowspan={{ issue['fixed-by']|length }}>Fixed by</th>
    
    96
    +    {% for branch in issue['fixed-by'] %}
    
    97
    +    <th>
    
    98
    +      {{ branch }}
    
    99
    +    </th>
    
    80 100
         <td>
    
    81
    -      {% for branch in issue['fixed-by'] %}
    
    82
    -      {{ branch }}:
    
    83 101
           {% for commit in issue['fixed-by'][branch] %}
    
    84 102
           <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
    
    85 103
           {% endfor %}
    
    86
    -      {% if not loop.last %}<br/>{% endif %}
    
    87
    -      {% endfor %}
    
    88 104
         </td>
    
    105
    +    {% if not loop.last %}
    
    106
    +  </tr>
    
    107
    +  <tr>
    
    108
    +    {% endif %}
    
    109
    +    {% endfor %}
    
    89 110
       </tr>
    
    90 111
       {% endif %}
    
    91 112
       {% if issue['fix-depends-on'] %}
    
    92 113
       <tr>
    
    93 114
         <th>Fix depends on</th>
    
    115
    +    <td/>
    
    94 116
         <td>
    
    95 117
           {% for commit in issue['fix-depends-on'] %}
    
    96 118
           <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>
    
    ... ... @@ -100,5 +122,26 @@
    100 122
         </td>
    
    101 123
       </tr>
    
    102 124
       {% endif %}
    
    125
    +  <tr>
    
    126
    +    <th rowspan={{ branches|length }}>Status</th>
    
    127
    +    {% for name, affected in branches %}
    
    128
    +    <th>
    
    129
    +      <a href="/branch/{{ name }}/">{{ name }}</a>
    
    130
    +    </th>
    
    131
    +    <td>
    
    132
    +      {% if not affected %}
    
    133
    +      {# Currently we don't distinguish between never-affected or fixed #}
    
    134
    +      <span class="good">not affected</span>
    
    135
    +      {% elif issue.ignore and (issue.ignore.all or issue.ignore[name]) %}
    
    136
    +      <span class="ignored">ignored</span>
    
    137
    +      {% else %}
    
    138
    +      <span class="bad">vulnerable</span>
    
    139
    +      {% endif %}
    
    140
    +    </td>
    
    141
    +    {% if not loop.last %}
    
    142
    +  </tr>
    
    143
    +  <tr>
    
    144
    +    {% endif %}
    
    145
    +    {% endfor %}
    
    146
    +  </tr>
    
    103 147
     </table>
    104
    -{# TODO: show issue status for each branch #}

  • scripts/templates/issues.html
    1
    +<link rel="stylesheet" href="/static/style.css">
    
    1 2
     <title>Issues</title>
    
    2 3
     <h1>Issues</h1>
    
    3 4
     <ul>
    
    4
    -  {% for cve_id in cve_ids %}
    
    5
    -  <li><a href="{{ cve_id }}/">{{ cve_id }}</a></li>
    
    5
    +  {% for cve_id, issue in cve_ids %}
    
    6
    +  <li class="{% if issue.ignore and issue.ignore.all %}ignored{% endif %}">
    
    7
    +    <a href="{{ cve_id }}/">{{ cve_id }}</a> -
    
    8
    +    {{ issue.description|truncate(100) }}
    
    9
    +  </li>
    
    6 10
       {% endfor %}
    
    7 11
     </ul>

  • scripts/templates/root.html
    1
    +<link rel="stylesheet" href="/static/style.css">
    
    1 2
     <title>Kernel security tracker</title>
    
    2 3
     <h1>Kernel security tracker</h1>
    
    3 4
     <p>
    

  • scripts/templates/style.css
    1
    +body {
    
    2
    +    color: #000000;
    
    3
    +    background-color: #ffffff;
    
    4
    +}
    
    5
    +:link {
    
    6
    +    color: #0000ff;
    
    7
    +}
    
    8
    +:visited {
    
    9
    +    color: #8000c0;
    
    10
    +}
    
    11
    +
    
    12
    +.ignored {
    
    13
    +    color: #666666;
    
    14
    +}
    
    15
    +.ignored > :link {
    
    16
    +    color: #6666ff;
    
    17
    +}
    
    18
    +.ignored > :visited {
    
    19
    +    color: #b366d9;
    
    20
    +}
    
    21
    +
    
    22
    +.good {
    
    23
    +    color: #33cc33;
    
    24
    +}
    
    25
    +.bad {
    
    26
    +    color: #ff0000;
    
    27
    +}
    
    28
    +
    
    29
    +table {
    
    30
    +    border-collapse: collapse;
    
    31
    +}
    
    32
    +tr {
    
    33
    +    border-top: solid 1px;
    
    34
    +    border-bottom: solid 1px;
    
    35
    +}
    
    36
    +th {
    
    37
    +    text-align: right;
    
    38
    +    vertical-align: top;
    
    39
    +    white-space: nowrap;
    
    40
    +}
    
    41
    +th, td {
    
    42
    +    padding-left: 0.5em;
    
    43
    +    padding-right: 0.5em;
    
    44
    +}

  • scripts/validate.py
    ... ... @@ -9,11 +9,11 @@
    9 9
     import argparse
    
    10 10
     import sys
    
    11 11
     
    
    12
    -from kernel_sec.issue import get_filename, get_list, load, load_filename, validate
    
    12
    +from kernel_sec.issue \
    
    13
    +    import get_filename, get_list, load_filename, validate
    
    13 14
     
    
    14
    -def main(*names):
    
    15
    -    import glob
    
    16 15
     
    
    16
    +def main(*names):
    
    17 17
         rc = 0
    
    18 18
         if len(names) == 0:
    
    19 19
             names = [get_filename(cve_id) for cve_id in get_list()]
    
    ... ... @@ -27,6 +27,7 @@ def main(*names):
    27 27
     
    
    28 28
         sys.exit(rc)
    
    29 29
     
    
    30
    +
    
    30 31
     if __name__ == '__main__':
    
    31 32
         parser = argparse.ArgumentParser(
    
    32 33
             description='Validate issue files against the schema.')
    

  • scripts/webview.py
    ... ... @@ -12,7 +12,8 @@ import os
    12 12
     import cherrypy
    
    13 13
     import jinja2
    
    14 14
     
    
    15
    -import kernel_sec.branch, kernel_sec.issue
    
    15
    +import kernel_sec.branch
    
    16
    +import kernel_sec.issue
    
    16 17
     
    
    17 18
     
    
    18 19
     _template_env = jinja2.Environment(
    
    ... ... @@ -33,7 +34,8 @@ class IssueCache:
    33 34
             return cache_data
    
    34 35
     
    
    35 36
         def _refresh_keys(self):
    
    36
    -        return self._refresh('issues', lambda: set(kernel_sec.issue.get_list()))
    
    37
    +        return self._refresh('issues',
    
    38
    +                             lambda: set(kernel_sec.issue.get_list()))
    
    37 39
     
    
    38 40
         def _refresh_issue(self, cve_id):
    
    39 41
             filename = kernel_sec.issue.get_filename(cve_id)
    
    ... ... @@ -58,66 +60,97 @@ _issue_cache = IssueCache()
    58 60
     class Branch:
    
    59 61
         _template = _template_env.get_template('branch.html')
    
    60 62
     
    
    61
    -    def __init__(self, branch):
    
    63
    +    def __init__(self, name, root):
    
    62 64
             self._name = name
    
    65
    +        self._root = root
    
    63 66
     
    
    64 67
         @cherrypy.expose
    
    65 68
         def index(self):
    
    66
    -        return self._template.render(name=self._name)
    
    69
    +        return self._template.render(
    
    70
    +            name=self._name,
    
    71
    +            issues=[
    
    72
    +                (cve_id, _issue_cache[cve_id])
    
    73
    +                for cve_id in sorted(_issue_cache.keys(),
    
    74
    +                                     key=kernel_sec.issue.get_id_sort_key)
    
    75
    +                if kernel_sec.issue.affects_branch(
    
    76
    +                        _issue_cache[cve_id], self._name,
    
    77
    +                        self._root.is_commit_in_branch)
    
    78
    +            ])
    
    67 79
     
    
    68 80
     
    
    69 81
     class Branches:
    
    70 82
         _template = _template_env.get_template('branches.html')
    
    71 83
     
    
    72
    -    def __init__(self, git_repo, mainline_remote_name, stable_remote_name):
    
    73
    -        self._names = kernel_sec.branch.get_live_stable_branches(
    
    74
    -            git_repo, stable_remote_name)
    
    75
    -        self._names.append('mainline')
    
    84
    +    def __init__(self, root):
    
    85
    +        self._root = root
    
    76 86
     
    
    77 87
         def _cp_dispatch(self, vpath):
    
    78
    -        if len(vpath) == 1 and vpath[0] in self._branches:
    
    79
    -            return Branch(vpath.pop())
    
    88
    +        if len(vpath) == 1 and vpath[0] in self._root.branch_names:
    
    89
    +            return Branch(vpath.pop(), self._root)
    
    80 90
             return vpath
    
    81 91
     
    
    82 92
         @cherrypy.expose
    
    83 93
         def index(self):
    
    84
    -        return self._template.render(names=self._names)
    
    94
    +        return self._template.render(names=self._root.branch_names)
    
    85 95
     
    
    86 96
     
    
    87 97
     class Issue:
    
    88 98
         _template = _template_env.get_template('issue.html')
    
    89 99
     
    
    90
    -    def __init__(self, cve_id):
    
    100
    +    def __init__(self, cve_id, root):
    
    91 101
             self._cve_id = cve_id
    
    102
    +        self._root = root
    
    92 103
     
    
    93 104
         @cherrypy.expose
    
    94 105
         def index(self):
    
    95
    -        return self._template.render(cve_id=self._cve_id,
    
    96
    -                                     issue=_issue_cache[self._cve_id])
    
    106
    +        issue = _issue_cache[self._cve_id]
    
    107
    +        return self._template.render(
    
    108
    +            cve_id=self._cve_id,
    
    109
    +            issue=issue,
    
    110
    +            branches=[
    
    111
    +                (name,
    
    112
    +                 kernel_sec.issue.affects_branch(
    
    113
    +                     issue, name, self._root.is_commit_in_branch))
    
    114
    +                for name in self._root.branch_names
    
    115
    +            ])
    
    97 116
     
    
    98 117
     
    
    99 118
     class Issues:
    
    100 119
         _template = _template_env.get_template('issues.html')
    
    101 120
     
    
    121
    +    def __init__(self, root):
    
    122
    +        self._root = root
    
    123
    +
    
    102 124
         def _cp_dispatch(self, vpath):
    
    103 125
             if len(vpath) == 1 and vpath[0] in _issue_cache:
    
    104
    -            return Issue(vpath.pop())
    
    126
    +            return Issue(vpath.pop(), self._root)
    
    105 127
             return vpath
    
    106 128
     
    
    107 129
         @cherrypy.expose
    
    108 130
         def index(self):
    
    109 131
             return self._template.render(
    
    110
    -            cve_ids=sorted(_issue_cache.keys(),
    
    111
    -                           key=kernel_sec.issue.get_id_sort_key))
    
    132
    +            cve_ids=[
    
    133
    +                (cve_id, _issue_cache[cve_id])
    
    134
    +                for cve_id in sorted(_issue_cache.keys(),
    
    135
    +                                     key=kernel_sec.issue.get_id_sort_key)
    
    136
    +            ])
    
    112 137
     
    
    113 138
     
    
    114 139
     class Root:
    
    115 140
         _template = _template_env.get_template('root.html')
    
    116 141
     
    
    117 142
         def __init__(self, git_repo, mainline_remote_name, stable_remote_name):
    
    118
    -        self.branches = Branches(git_repo, mainline_remote_name,
    
    119
    -                                 stable_remote_name)
    
    120
    -        self.issues = Issues()
    
    143
    +        self.branch_names = kernel_sec.branch.get_live_stable_branches(
    
    144
    +            git_repo, stable_remote_name)
    
    145
    +        self.branch_names.append('mainline')
    
    146
    +        self.branch_names.sort(key=kernel_sec.branch.get_sort_key)
    
    147
    +
    
    148
    +        c_b_map = kernel_sec.branch.CommitBranchMap(
    
    149
    +            git_repo, mainline_remote_name, self.branch_names)
    
    150
    +        self.is_commit_in_branch = c_b_map.is_commit_in_branch
    
    151
    +
    
    152
    +        self.branches = Branches(self)
    
    153
    +        self.issues = Issues(self)
    
    121 154
     
    
    122 155
         def _cp_dispatch(self, vpath):
    
    123 156
             if vpath[0] == 'branch':
    
    ... ... @@ -138,7 +171,8 @@ if __name__ == '__main__':
    138 171
             description='Report unfixed CVEs in Linux kernel branches.')
    
    139 172
         parser.add_argument('--git-repo',
    
    140 173
                             dest='git_repo', default='../kernel',
    
    141
    -                        help='git repository from which to read commit logs (default: ../kernel)',
    
    174
    +                        help=('git repository from which to read commit logs '
    
    175
    +                              '(default: ../kernel)'),
    
    142 176
                             metavar='DIRECTORY')
    
    143 177
         parser.add_argument('--mainline-remote',
    
    144 178
                             dest='mainline_remote_name', default='torvalds',
    
    ... ... @@ -146,8 +180,20 @@ if __name__ == '__main__':
    146 180
                             metavar='NAME')
    
    147 181
         parser.add_argument('--stable-remote',
    
    148 182
                             dest='stable_remote_name', default='stable',
    
    149
    -                        help='git remote for stable branches (default: stable)',
    
    183
    +                        help=('git remote for stable branches '
    
    184
    +                              '(default: stable)'),
    
    150 185
                             metavar='NAME')
    
    151 186
         args = parser.parse_args()
    
    187
    +
    
    188
    +    conf = {
    
    189
    +        '/static/style.css': {
    
    190
    +            'tools.staticfile.on': True,
    
    191
    +            'tools.staticfile.filename':
    
    192
    +            os.path.abspath('scripts/templates/style.css')
    
    193
    +        }
    
    194
    +    }
    
    195
    +
    
    152 196
         cherrypy.quickstart(Root(args.git_repo, args.mainline_remote_name,
    
    153
    -                             args.stable_remote_name))
    197
    +                             args.stable_remote_name),
    
    198
    +                        '/',
    
    199
    +                        conf)


  • Re: List of patches that failed to apply to 4.4-stable

    Daniel Sangorrin <daniel.sangorrin@...>
     

    To kernel team,

    @SZLin: thanks for creating a page on the wiki just for that.

    Yesterday, I checked the patches in my "failed to apply to 4.4-stable" list against the actual git repository using a short python script.
    It turns out that some of the patches in the "to apply" list were already there applied.

    I have updated our wiki page accordingly
    https://wiki.linuxfoundation.org/civilinfrastructureplatform/linux-4.4-failed-patches

    For the curious minds, you can find my (dirty) script attached to this e-mail.

    Thanks,
    Daniel

    -----Original Message-----
    From: SZ Lin (林上智) <sz.lin@moxa.com>
    Sent: Monday, November 12, 2018 8:04 PM
    To: daniel.sangorrin@toshiba.co.jp
    Cc: cip dev <cip-dev@lists.cip-project.org>
    Subject: Re: [cip-dev] List of patches that failed to apply to 4.4-stable

    Hi Daniel,

    Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp> 於 2018年11月12日 週一 下
    午5:15寫道:

    To: CIP Kernel team

    I have prepared a list of patches that failed to apply to 4.4-stable in the past. The
    list is based on the corresponding e-mails sent to the stable mailing list (since I
    subscribed, there may be more failed patches before my subscription). We can
    discuss today at the TSC meeting, and on Thursday during the IRC meeting whether
    we want to backport any of them or not.

    I've created a draft wiki page for it, feel free to modify the format.

    https://wiki.linuxfoundation.org/civilinfrastructureplatform/linux-4.4-failed-patche
    s

    SZ

    [Note] I still have to check if some of them were applied by any other means.

    Best regards,
    Daniel
    _______________________________________________
    cip-dev mailing list
    cip-dev@lists.cip-project.org
    https://lists.cip-project.org/mailman/listinfo/cip-dev


    CIP IRC weekly meeting today

    SZ Lin (林上智) <SZ.Lin@...>
     

    Hi All,

    Kindly be reminded to attend weekly meeting through IRC to discuss technical topics with CIP kernel today.

    *Please note that IRC meeting was rescheduled to 18:00 (JST) starting from first week of Nov. according F2F meeting discussion in ELCE.*

    US-West US-East UK DE TW JP
    02:00 05:00 09:00 10:00 17:00 18:00

    Channel:
    * irc:chat.freenode.net:6667/cip

    Agenda:
    * Kernel maintenance updates
    * Kernel testing
    * Software update
    * CIP Core
    * AOB

    The meeting will take 30 min although it can be extended to an hour if it makes sense and those involved in the topics can stay. Otherwise, the topic will be taken offline or in the next meeting..

    Best regards,

    SZ Lin


    [Git][cip-project/cip-kernel/cip-kernel-sec][master] 3 commits: Move CVE ID sort key function into kernel_sec.issue module

    Agustin Benito Bethencourt
     

    Ben Hutchings pushed to branch master at cip-project / cip-kernel / cip-kernel-sec

    Commits:

    • 0f9551d9
      by Ben Hutchings at 2018-11-14T18:26:22Z
      Move CVE ID sort key function into kernel_sec.issue module
      
    • 73dd8ff0
      by Ben Hutchings at 2018-11-14T22:18:14Z
      Add simple web application
      
    • 58186e1a
      by Ben Hutchings at 2018-11-14T22:20:04Z
      Document the PyYAML dependency
      

    9 changed files:

    Changes:

  • README.md
    ... ... @@ -14,7 +14,8 @@ The schema is roughly documented in
    14 14
     
    
    15 15
     Various scripts, written in Python 3, are in the `scripts`
    
    16 16
     subdirectory.  Supporting modules are in the `kernel_sec` subdirectory
    
    17
    -beneath that.
    
    17
    +beneath that.  They require PyYAML (packaged in Debian as
    
    18
    +python3-yaml).
    
    18 19
     
    
    19 20
     * `scripts/import_debian.py` - import information from Debian's
    
    20 21
     `kernel_sec` project.  It includes all issues that Debian considers
    
    ... ... @@ -39,6 +40,10 @@ schema.
    39 40
     files.  This should be run after hand-editing files to reduce
    
    40 41
     "noise" in later automated updates.
    
    41 42
     
    
    43
    +* `scripts/webview.py` - run a local web server that allows browsing
    
    44
    +branches and issues.  This requires CherryPy and Jinja2 (packaged
    
    45
    +in Debian as python3-cherrypy3 and python3-jinja2).
    
    46
    +
    
    42 47
     ## Contributions
    
    43 48
     
    
    44 49
     If you have better information about any issue, or additional
    

  • scripts/kernel_sec/issue.py
    ... ... @@ -265,3 +265,10 @@ def load(cve_id):
    265 265
     
    
    266 266
     def save(cve_id, issue):
    
    267 267
         save_filename(get_filename(cve_id), issue)
    
    268
    +
    
    269
    +# Match the "arbitrary digits" after the year
    
    270
    +_cve_id_arbdig_re = re.compile(r'-(\d+)$')
    
    271
    +
    
    272
    +# Pad "arbitrary digits" to 6 digits so string comparison works
    
    273
    +def get_id_sort_key(cve_id):
    
    274
    +    return _cve_id_arbdig_re.sub(lambda m: '-%06d' % int(m.group(1)), cve_id)

  • scripts/report_affected.py
    ... ... @@ -27,10 +27,6 @@ def get_commits(git_repo, end, start=None):
    27 27
         for line in io.TextIOWrapper(list_proc.stdout):
    
    28 28
             yield line.rstrip('\n')
    
    29 29
     
    
    30
    -# Pad last part of CVE ID to 6 digits so string comparison keeps working
    
    31
    -def pad_cve_id(cve_id):
    
    32
    -    return re.sub(r'-(\d+)$', lambda m: '-%06d' % int(m.group(1)), cve_id)
    
    33
    -
    
    34 30
     def main(git_repo, mainline_remote_name, stable_remote_name, only_fixed_upstream,
    
    35 31
              include_ignored, *branch_names):
    
    36 32
         if branch_names:
    
    ... ... @@ -119,7 +115,8 @@ def main(git_repo, mainline_remote_name, stable_remote_name, only_fixed_upstream
    119 115
                 branch_issues.setdefault(branch, []).append(cve_id)
    
    120 116
     
    
    121 117
         for branch in branch_names:
    
    122
    -        print('%s:' % branch, *sorted(branch_issues.get(branch, []), key=pad_cve_id))
    
    118
    +        print('%s:' % branch, *sorted(branch_issues.get(branch, []),
    
    119
    +                                      key=kernel_sec.issue.get_id_sort_key))
    
    123 120
     
    
    124 121
     if __name__ == '__main__':
    
    125 122
         parser = argparse.ArgumentParser(
    

  • scripts/templates/branch.html
    1
    +<title>Branch {{ name }}</title>
    
    2
    +<h1>Branch {{ name }}</h1>
    
    3
    +<p>
    
    4
    +  There should be a list of issues here...
    
    5
    +</p>

  • scripts/templates/branches.html
    1
    +<title>Branches</title>
    
    2
    +<h1>Branches</h1>
    
    3
    +<p>
    
    4
    +  Only the currently maintained branches are shown.
    
    5
    +</p>
    
    6
    +<ul>
    
    7
    +  {% for name in names %}
    
    8
    +  <li><a href="{{ name }}/">{{ name }}</a></li>
    
    9
    +  {% endfor %}
    
    10
    +</ul>

  • scripts/templates/issue.html
    1
    +<title>{{ cve_id }} - {{ issue.description|truncate(50) }}</title>
    
    2
    +<h1>{{ cve_id }} - {{ issue.description|truncate(50) }}</h1>
    
    3
    +<h2>Summary</h2>
    
    4
    +<p>{{ issue.description }}</p>
    
    5
    +{% if issue.advisory %}
    
    6
    +<h2>Advisory</h2>
    
    7
    +<p>{{ issue.advisory }}</p>
    
    8
    +{% endif %}
    
    9
    +<table>
    
    10
    +  {% if issue.references %}
    
    11
    +  <tr>
    
    12
    +    <th>References</th>
    
    13
    +    <td>
    
    14
    +      {% for url in issue.references %}
    
    15
    +      <a href="{{ url }}">{{ url }}</a>
    
    16
    +      {% if not loop.last %}|{% endif %}
    
    17
    +      {% endfor %}
    
    18
    +    </td>
    
    19
    +  </tr>
    
    20
    +  {% endif %}
    
    21
    +  {% if issue.aliases %}
    
    22
    +  <tr>
    
    23
    +    <th>Aliases</th>
    
    24
    +    <td>
    
    25
    +      {% for alias in issue.aliases %}
    
    26
    +      {{ alias }}{% if not loop.last %},{% endif %}  
    
    27
    +      {% endfor %}
    
    28
    +    </td>
    
    29
    +  </tr>
    
    30
    +  {% endif %}
    
    31
    +  {% if issue.comments %}
    
    32
    +  <tr>
    
    33
    +    <th>Comments</th>
    
    34
    +    <td>
    
    35
    +      {% for handle in issue.comments %}
    
    36
    +      {{ handle }}: <q>{{ issue.comments[handle] }}</q>
    
    37
    +      {% if not loop.last %}<br/>{% endif %}
    
    38
    +      {% endfor %}
    
    39
    +    </td>
    
    40
    +  </tr>
    
    41
    +  {% endif %}
    
    42
    +  {% if issue.reporters %}
    
    43
    +  <tr>
    
    44
    +    <th>Reporters</th>
    
    45
    +    <td>
    
    46
    +      {% for rep in issue.reporters %}
    
    47
    +      {# This may include an email address which could be linked #}
    
    48
    +      {{ rep }}{% if not loop.last %},{% endif %}
    
    49
    +      {% endfor %}
    
    50
    +    </td>
    
    51
    +  </tr>
    
    52
    +  {% endif %}
    
    53
    +  {% if issue['embargo-end'] %}
    
    54
    +  <tr>
    
    55
    +    <th>Embargo</th>
    
    56
    +    <td>
    
    57
    +      {# TODO: If embargo-end is in the future we should prominently
    
    58
    +      flag this issue as embargoed #}
    
    59
    +      {{ issue['embargo-end'] }}
    
    60
    +    </td>
    
    61
    +  </tr>
    
    62
    +  {% endif %}
    
    63
    +  {% if issue['introduced-by'] %}
    
    64
    +  <tr>
    
    65
    +    <th>Introduced by</th>
    
    66
    +    <td>
    
    67
    +      {% for branch in issue['introduced-by'] %}
    
    68
    +      {{ branch }}:
    
    69
    +      {% for commit in issue['introduced-by'][branch] %}
    
    70
    +      <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
    
    71
    +      {% endfor %}
    
    72
    +      {% if not loop.last %}<br/>{% endif %}
    
    73
    +      {% endfor %}
    
    74
    +    </td>
    
    75
    +  </tr>
    
    76
    +  {% endif %}
    
    77
    +  {% if issue['fixed-by'] %}
    
    78
    +  <tr>
    
    79
    +    <th>Fixed by</th>
    
    80
    +    <td>
    
    81
    +      {% for branch in issue['fixed-by'] %}
    
    82
    +      {{ branch }}:
    
    83
    +      {% for commit in issue['fixed-by'][branch] %}
    
    84
    +      <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
    
    85
    +      {% endfor %}
    
    86
    +      {% if not loop.last %}<br/>{% endif %}
    
    87
    +      {% endfor %}
    
    88
    +    </td>
    
    89
    +  </tr>
    
    90
    +  {% endif %}
    
    91
    +  {% if issue['fix-depends-on'] %}
    
    92
    +  <tr>
    
    93
    +    <th>Fix depends on</th>
    
    94
    +    <td>
    
    95
    +      {% for commit in issue['fix-depends-on'] %}
    
    96
    +      <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id={{ commit }}">{{ commit[:12] }}</a>
    
    97
    +      ({{ issue['fix-depends-on'][commit] }})
    
    98
    +      {% if not loop.last %}<br/>{% endif %}
    
    99
    +      {% endfor %}
    
    100
    +    </td>
    
    101
    +  </tr>
    
    102
    +  {% endif %}
    
    103
    +</table>
    
    104
    +{# TODO: show issue status for each branch #}

  • scripts/templates/issues.html
    1
    +<title>Issues</title>
    
    2
    +<h1>Issues</h1>
    
    3
    +<ul>
    
    4
    +  {% for cve_id in cve_ids %}
    
    5
    +  <li><a href="{{ cve_id }}/">{{ cve_id }}</a></li>
    
    6
    +  {% endfor %}
    
    7
    +</ul>

  • scripts/templates/root.html
    1
    +<title>Kernel security tracker</title>
    
    2
    +<h1>Kernel security tracker</h1>
    
    3
    +<p>
    
    4
    +  <a href="branch/">View branches</a> | <a href="issue/">View issues</a>
    
    5
    +</p>

  • scripts/webview.py
    1
    +#!/usr/bin/python3
    
    2
    +
    
    3
    +# Copyright 2018 Codethink Ltd.
    
    4
    +#
    
    5
    +# This script is distributed under the terms and conditions of the GNU General
    
    6
    +# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
    
    7
    +# for details.
    
    8
    +
    
    9
    +import argparse
    
    10
    +import os
    
    11
    +
    
    12
    +import cherrypy
    
    13
    +import jinja2
    
    14
    +
    
    15
    +import kernel_sec.branch, kernel_sec.issue
    
    16
    +
    
    17
    +
    
    18
    +_template_env = jinja2.Environment(
    
    19
    +    loader=jinja2.FileSystemLoader('scripts/templates'),
    
    20
    +    autoescape=True)
    
    21
    +
    
    22
    +
    
    23
    +class IssueCache:
    
    24
    +    def __init__(self):
    
    25
    +        self._data = {}
    
    26
    +
    
    27
    +    def _refresh(self, name, loader):
    
    28
    +        file_time = os.stat(name).st_mtime
    
    29
    +        cache_data, cache_time = self._data.get(name, (None, None))
    
    30
    +        if file_time != cache_time:
    
    31
    +            cache_data, cache_time = loader(), file_time
    
    32
    +            self._data[name] = (cache_data, cache_time)
    
    33
    +        return cache_data
    
    34
    +
    
    35
    +    def _refresh_keys(self):
    
    36
    +        return self._refresh('issues', lambda: set(kernel_sec.issue.get_list()))
    
    37
    +
    
    38
    +    def _refresh_issue(self, cve_id):
    
    39
    +        filename = kernel_sec.issue.get_filename(cve_id)
    
    40
    +        return self._refresh(filename,
    
    41
    +                             lambda: kernel_sec.issue.load_filename(filename))
    
    42
    +
    
    43
    +    def keys(self):
    
    44
    +        return iter(self._refresh_keys())
    
    45
    +
    
    46
    +    def __contains__(self, cve_id):
    
    47
    +        return cve_id in self._refresh_keys()
    
    48
    +
    
    49
    +    def __getitem__(self, cve_id):
    
    50
    +        if cve_id not in self._refresh_keys():
    
    51
    +            raise KeyError
    
    52
    +        return self._refresh_issue(cve_id)
    
    53
    +
    
    54
    +
    
    55
    +_issue_cache = IssueCache()
    
    56
    +
    
    57
    +
    
    58
    +class Branch:
    
    59
    +    _template = _template_env.get_template('branch.html')
    
    60
    +
    
    61
    +    def __init__(self, branch):
    
    62
    +        self._name = name
    
    63
    +
    
    64
    +    @cherrypy.expose
    
    65
    +    def index(self):
    
    66
    +        return self._template.render(name=self._name)
    
    67
    +
    
    68
    +
    
    69
    +class Branches:
    
    70
    +    _template = _template_env.get_template('branches.html')
    
    71
    +
    
    72
    +    def __init__(self, git_repo, mainline_remote_name, stable_remote_name):
    
    73
    +        self._names = kernel_sec.branch.get_live_stable_branches(
    
    74
    +            git_repo, stable_remote_name)
    
    75
    +        self._names.append('mainline')
    
    76
    +
    
    77
    +    def _cp_dispatch(self, vpath):
    
    78
    +        if len(vpath) == 1 and vpath[0] in self._branches:
    
    79
    +            return Branch(vpath.pop())
    
    80
    +        return vpath
    
    81
    +
    
    82
    +    @cherrypy.expose
    
    83
    +    def index(self):
    
    84
    +        return self._template.render(names=self._names)
    
    85
    +
    
    86
    +
    
    87
    +class Issue:
    
    88
    +    _template = _template_env.get_template('issue.html')
    
    89
    +
    
    90
    +    def __init__(self, cve_id):
    
    91
    +        self._cve_id = cve_id
    
    92
    +
    
    93
    +    @cherrypy.expose
    
    94
    +    def index(self):
    
    95
    +        return self._template.render(cve_id=self._cve_id,
    
    96
    +                                     issue=_issue_cache[self._cve_id])
    
    97
    +
    
    98
    +
    
    99
    +class Issues:
    
    100
    +    _template = _template_env.get_template('issues.html')
    
    101
    +
    
    102
    +    def _cp_dispatch(self, vpath):
    
    103
    +        if len(vpath) == 1 and vpath[0] in _issue_cache:
    
    104
    +            return Issue(vpath.pop())
    
    105
    +        return vpath
    
    106
    +
    
    107
    +    @cherrypy.expose
    
    108
    +    def index(self):
    
    109
    +        return self._template.render(
    
    110
    +            cve_ids=sorted(_issue_cache.keys(),
    
    111
    +                           key=kernel_sec.issue.get_id_sort_key))
    
    112
    +
    
    113
    +
    
    114
    +class Root:
    
    115
    +    _template = _template_env.get_template('root.html')
    
    116
    +
    
    117
    +    def __init__(self, git_repo, mainline_remote_name, stable_remote_name):
    
    118
    +        self.branches = Branches(git_repo, mainline_remote_name,
    
    119
    +                                 stable_remote_name)
    
    120
    +        self.issues = Issues()
    
    121
    +
    
    122
    +    def _cp_dispatch(self, vpath):
    
    123
    +        if vpath[0] == 'branch':
    
    124
    +            vpath.pop(0)
    
    125
    +            return self.branches
    
    126
    +        if vpath[0] == 'issue':
    
    127
    +            vpath.pop(0)
    
    128
    +            return self.issues
    
    129
    +        return vpath
    
    130
    +
    
    131
    +    @cherrypy.expose
    
    132
    +    def index(self):
    
    133
    +        return self._template.render()
    
    134
    +
    
    135
    +
    
    136
    +if __name__ == '__main__':
    
    137
    +    parser = argparse.ArgumentParser(
    
    138
    +        description='Report unfixed CVEs in Linux kernel branches.')
    
    139
    +    parser.add_argument('--git-repo',
    
    140
    +                        dest='git_repo', default='../kernel',
    
    141
    +                        help='git repository from which to read commit logs (default: ../kernel)',
    
    142
    +                        metavar='DIRECTORY')
    
    143
    +    parser.add_argument('--mainline-remote',
    
    144
    +                        dest='mainline_remote_name', default='torvalds',
    
    145
    +                        help='git remote for mainline (default: torvalds)',
    
    146
    +                        metavar='NAME')
    
    147
    +    parser.add_argument('--stable-remote',
    
    148
    +                        dest='stable_remote_name', default='stable',
    
    149
    +                        help='git remote for stable branches (default: stable)',
    
    150
    +                        metavar='NAME')
    
    151
    +    args = parser.parse_args()
    
    152
    +    cherrypy.quickstart(Root(args.git_repo, args.mainline_remote_name,
    
    153
    +                             args.stable_remote_name))

  • 6701 - 6720 of 8370