Hi,

You can find the following sentence on the page.

an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.

Oh, I missed the sentence. Thank you very much for telling it.

In addition to these, we may need to describe the supported hardware (CPU, boards) and
test environment as well.

Is the backporting limited to the supported hardware?

Best regards,
Hiraku Toyooka

2019年7月11日(木) 8:47 <nobuhiro1.iwamatsu@...>:

Hi,

Thanks for pointed out.
You can find the following sentence on the page.

an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.

I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

Agree.

We think that we need to add the following sentences.
---
We do not apply patches that have not been applied to upstream
(Linus tree:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git).
If you want to add new features or hardware, they need to be applied into upstream.
Also, you may not be able to easily apply the patches on Upstream. In such a case,
you need to describe in the commit message of patch what modifications have been made
from the original patch.
---

In addition to these, we may need to describe the supported hardware (CPU, boards) and
test environment as well.

Best regards,
Nobuhiro

From: Yoshitake Kobayashi [mailto:yoshitake.kobayashi@...]
Sent: Wednesday, July 10, 2019 9:56 PM
To: hiraku.toyooka@...
Cc: cip-dev@...; pavel@...; iwamatsu nobuhiro(岩松 信洋 ○ＳＷＣ□ＯＳＴ) <nobuhiro1.iwamatsu@...>
Subject: Re: [cip-dev] About CIP kernel maintenance policy for new hardware support backporting

Hi,

Thanks for pointed out.
You can find the following sentence on the page.

an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.

I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

Best regards,
Yoshi



2019年7月10日(水) 15:42 <hiraku.toyooka@...>:
Hello,

I have a question about CIP kernel maintenance policy.
Can CIP kernel accept backport patches for new hardware support (not
only fixes)?

I read the following document and it seems to describe only acceptable fixes.
https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipkernelmaintenance

On the other hand, Current linux-4.19.y-cip branch accepts some hardware
support patches such as RZ/G2E(r8a774c0) from upstream.
So I wonder if some policy about new hardware support exists.

--
Hiraku Toyooka
Cybertrust Japan Co., Ltd.
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

--
Hiraku Toyooka
Cybertrust Japan Co., Ltd.

Although the README already contains all the information
that users may need, there are some bits of know-how that
are better expressed through a step-by-step quickstart or
tutorial. This files tries to fill that gap.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
QUICKSTART.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
create mode 100644 QUICKSTART.md

diff --git a/QUICKSTART.md b/QUICKSTART.md
new file mode 100644
index 0000000..c79af41
--- /dev/null
+++ b/QUICKSTART.md
@@ -0,0 +1,132 @@
+# Quickstart
+
+## Overview
+
+This project tracks the status of CVEs in mainline and stable kernels. Each CVE is described in YAML format that includes data such as:
+
+```
+$ cat issues/CVE-2019-1999.yml
+description: 'binder: fix race between munmap() and direct reclaim'
+references:
+- https://source.android.com/security/bulletin/2019-02-01
+comments:
+ Debian-bwh: |-
+ Introduced in 4.14 by f2517eb76f1f "android: binder: Add global lru
+ shrinker to binder". Backports of the fix to stable have incorrect
+ metadata.
+ bwh: Backports to stable have incorrect metadata
+introduced-by:
+ mainline: [f2517eb76f1f2f7f89761f9db2b202e89931738c]
+fixed-by:
+ linux-4.14.y: [33c6b9ca70a8b066a613e2a3d0331ae8f82aa31a]
+ linux-4.19.y: [6bf7d3c5c0c5dad650bfc4345ed553c18b69d59e]
+ linux-5.0.y: [bbb19ca082ce27ce60ca65be016a951806ea947c]
+ mainline: [5cec2d2e5839f9c0fec319c523a911e0a7fd299f]
+```
+
+## Quickstart
+
+Clone `cip-kernel-sec` and install its dependencies:
+
+```
+$ git clone https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec
+$ cd cip-kernel-sec/
+$ sudo apt install python3-yaml python3-html5lib python3-cherrypy3 python3-jinja2
+```
+
+Prepare kernel remote repositories according to `conf/remotes.yml`:
+
+```
+$ ./scripts/prepare_remotes.py
+```
+
+Alternatively, you can do that manually:
+
+```
+$ mkdir ../kernel
+$ cd ../kernel
+$ git remote add torvalds https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+$ git remote add cip https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
+$ cd ../cip-kernel-sec
+```
+
+Download CVE information from [Debian] (https://salsa.debian.org/kernel-team/kernel-sec.git), [Ubuntu] (https://git.launchpad.net/ubuntu-cve-tracker) and Stable:
+
+```
+
+$ ./scripts/import_debian.py
+ -> import/debian
+$ ./scripts/import_ubuntu.py
+ -> import/ubuntu
+$ ./scripts/import_stable.py
+ -> import/stable_branches.yml
+```
+
+Check issues that affect a linux-cip branch:
+
+```
+$ ./scripts/report_affected.py linux-4.4.y
+```
+
+You can show a short description on your report:
+
+```
+$ ./scripts/report_affected.py --show-description linux-4.4.y
+```
+
+Check issues that affect a tag:
+
+```
+$ ./scripts/report_affected.py v4.4.181-cip33
+```
+
+Browse kernel branches and issues interactively:
+
+```
+$ ./scripts/webview.py
+$ firefox http://localhost:8080
+```
+
+[Note] Use Ctr-c to stop the `webview.py` script.
+
+## Kernel maintainer workflow
+
+Import or update the latest CVE information:
+
+```
+$ ./scripts/import_debian.py
+$ ./scripts/import_ubuntu.py
+$ ./scripts/import_stable.py
+```
+
+Edit by hand the newly created issues if you see that some imported information is incorrect or there is missing information:
+
+```
+$ vi issues/CVE-xx.yml
+```
+
+Validate the issue files against the YAML schema.
+
+```
+$ ./scripts/validate.py
+```
+
+YAML allows the same thing to be written in different ways, e.g. bracketed vs bulleted lists. Use `cleanup.py` to make the syntax and ordering of items consistent with the importers, to reduce "noise" in diffs:
+
+```
+$ ./scripts/cleanup.py
+```
+
+Check if the current issues:
+
+```
+$ ./scripts/report_affected.py
+```
+
+## Changelog
+
+- 20190614: First version <daniel.sangorrin@...>
+- 20190618: Add workflow information provided by Ben
+- 20190711: Add tag reporting <daniel.sangorrin@...>
+
--
2.17.1

Hello Ben,

Sorry, I realized that there were a few issues in the Quickstart
so I am resending the patch. Please ignore the previous one.

[cip-kernel-sec][Quickstart v2] docs: add a quickstart with practical

Thanks,
Daniel

Probably this should be squashed into the corresponding
patches.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
README.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 576cc75..8164826 100644
--- a/README.md
+++ b/README.md
@@ -41,7 +41,8 @@ current or previous year or that are already tracked here.
stable and other configured branches, by reading the git commit logs.

* `scripts/report_affected.py` - report which issues affect the
-specified branches, or all active branches.
+specified branches, or all active branches. You can use --show-description
+to obtain a short description for each CVE ID.

* `scripts/validate.py` - validate all issue files against the
schema.
@@ -72,6 +73,7 @@ keys:
* `base_ver`: Stable version that the branch is based on, e.g.
"4.4". This needs to be quoted so that it's a string not a
number.
+* `tag_regexp`: A regular expression that matches tags on a branch.

### Remotes

--
2.17.1

Although the README already contains all the information
that users may need, there are some bits of know-how that
are better expressed through a step-by-step quickstart or
tutorial. This files tries to fill that gap.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
QUICKSTART.md | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 126 insertions(+)
create mode 100644 QUICKSTART.md

diff --git a/QUICKSTART.md b/QUICKSTART.md
new file mode 100644
index 0000000..4bceb97
--- /dev/null
+++ b/QUICKSTART.md
@@ -0,0 +1,126 @@
+# Quickstart
+
+## Overview
+
+This project tracks the status of CVEs in mainline and stable kernels. Each CVE is described in YAML format that includes data such as:
+
+```
+$ cat CVE-2019-1999.yml
+description: 'binder: fix race between munmap() and direct reclaim'
+references:
+- https://source.android.com/security/bulletin/2019-02-01
+comments:
+ Debian-bwh: |-
+ Introduced in 4.14 by f2517eb76f1f "android: binder: Add global lru
+ shrinker to binder". Backports of the fix to stable have incorrect
+ metadata.
+ bwh: Backports to stable have incorrect metadata
+introduced-by:
+ mainline: [f2517eb76f1f2f7f89761f9db2b202e89931738c]
+fixed-by:
+ linux-4.14.y: [33c6b9ca70a8b066a613e2a3d0331ae8f82aa31a]
+ linux-4.19.y: [6bf7d3c5c0c5dad650bfc4345ed553c18b69d59e]
+ linux-5.0.y: [bbb19ca082ce27ce60ca65be016a951806ea947c]
+ mainline: [5cec2d2e5839f9c0fec319c523a911e0a7fd299f]
+```
+
+## Quickstart
+
+Clone `cip-kernel-sec` and install its dependencies:
+
+```
+$ git clone https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec
+$ cd cip-kernel-sec/
+$ sudo apt install python3-yaml and python3-html5lib python3-cherrypy3 python3-jinja2
+```
+
+Prepare kernel remote repositories according to `conf/remotes.yml`:
+
+```
+$ ./scripts/prepare_remotes.py
+```
+
+Alternatively, you can do that manually:
+
+```
+$ mkdir ../kernel
+$ cd ../kernel
+$ git remote add torvalds https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+$ git remote add cip https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
+$ cd ../cip-kernel-sec
+```
+
+Download CVE information from Debian (https://salsa.debian.org/kernel-team/kernel-sec.git), Ubuntu (https://git.launchpad.net/ubuntu-cve-tracker) and Stable:
+
+```
+
+$ ./scripts/import_debian.py
+ -> import/debian
+$ ./scripts/import_ubuntu.py
+ -> import/ubuntu
+$ ./scripts/import_stable.py
+ -> import/stable_branches.yml
+```
+
+Check issues that affect a linux-cip branch:
+
+```
+$ ./scripts/report_affected.py linux-4.4.y
+```
+
+Check issues that affect a tag:
+
+```
+$ ./scripts/report_affected.py v4.4.181-cip33
+```
+
+Browse kernel branches and issues interactively:
+
+```
+$ ./scripts/webview.py
+$ firefox http://localhost:8080
+```
+
+[Note] Use Ctr-c to stop the `webview.py` script.
+
+## Kernel maintainer workflow
+
+Import or update the latest CVE information:
+
+```
+$ ./scripts/import_debian.py
+$ ./scripts/import_ubuntu.py
+$ ./scripts/import_stable.py
+```
+
+Edit by hand the newly created issues if you see that some imported information is incorrect or there is missing information:
+
+```
+$ vi issues/CVE-xx.yml
+```
+
+Validate the issue files against the YAML schema.
+
+```
+$ ./scripts/validate.py
+```
+
+YAML allows the same thing to be written in different ways, e.g. bracketed vs bulleted lists. Use `cleanup.py` to make the syntax and ordering of items consistent with the importers, to reduce "noise" in diffs:
+
+```
+$ ./scripts/cleanup.py
+```
+
+Check if the current issues:
+
+```
+$ ./scripts/report_affected.py
+```
+
+## Changelog
+
+- 20190614: First version <daniel.sangorrin@...>
+- 20190618: Add workflow information provided by Ben
+- 20190711: Add tag reporting <daniel.sangorrin@...>
+
--
2.17.1

Hello Ben,

I was planning to put this into the CIP wiki, but I though it might
be better to store it together with the source code so there you go.

[cip-kernel-sec][quickstart] docs: add a quickstart with practical

Thanks,
Daniel

Reporting on tags is useful for product engineers that
have shipped a kernel with a specific tag and need to know
which issues affect their product after some time.

Examples:
$ ./scripts/report_affected.py v4.4 v4.4.107 v4.4.181-cip33
$ cd ../kernel
$ git tag myproduct-v1 0f13d9b4d0efa9e87381717c113df57718bc92d6
$ cd ../cip-kernel-sec
$ ./scripts/report_affected.py linux-4.19.y-cip:myproduct-v1 v4.19.50-cip3

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
conf/branches.yml | 2 ++
scripts/kernel_sec/branch.py | 8 +++--
scripts/report_affected.py | 68 +++++++++++++++++++++++++++++++-----
3 files changed, 66 insertions(+), 12 deletions(-)

diff --git a/conf/branches.yml b/conf/branches.yml
index 2ed9db6..8197596 100644
--- a/conf/branches.yml
+++ b/conf/branches.yml
@@ -2,7 +2,9 @@
base_ver: "4.4"
git_remote: cip
git_name: linux-4.4.y-cip
+ tag_regexp: '^v4\.4\.\d+-cip\d+$'
- short_name: linux-4.19.y-cip
base_ver: "4.19"
git_remote: cip
git_name: linux-4.19.y-cip
+ tag_regexp: '^v4\.19\.\d+-cip\d+$'
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 9a7bc3a..cd4f423 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -23,11 +23,13 @@ from . import version

def get_base_ver_stable_branch(base_ver):
branch_name = 'linux-%s.y' % base_ver
+ esc_base_ver = re.escape(base_ver)
return {
'short_name': branch_name,
'git_remote': 'stable',
'git_name': branch_name,
- 'base_ver': base_ver
+ 'base_ver': base_ver,
+ 'tag_regexp' : r'(^v%s$|^v%s\.\d+$)' % (esc_base_ver, esc_base_ver)
}


@@ -141,7 +143,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)


-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +172,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0966fe1..27c39ef 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.

import argparse
+import copy
import subprocess
+import re

import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,38 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' not in branch:
+ # no tag_regexp defined, or mainline
+ continue
+
+ # predefined in branches.yml or a stable branch
+ if re.match(branch['tag_regexp'], branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name, tag = branch_name.split(':', 1)
else:
name = branch_name

for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +70,18 @@ def main(git_repo, remotes,

c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())

@@ -65,15 +102,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue

+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break

for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -104,9 +152,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1

Hello Ben,

I think this should be fine.
You need to remove the cached file import/stable_branches.yml

[cip-kernel-sec][RESEND v3] report_affected: add support for

To be honest, I am not 100% sure that the escape code is
perfectly correct. The file stable_branches.yml will have
parameters like this:

tag_regexp: (^v3\.16$|^v3\.16\.\d+$)

Thanks,
Daniel

+ for branch in branches:
+ esc_base_ver = branch['base_ver'].replace('.', re.escape('.'))
+ # example tags: v4.4, v4.19.12
+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (esc_base_ver, esc_base_ver)
+ branch['tag_regexp'] = tag_regexp
+

Sorry, I went too fast. I should have moved the regexp to get_base_ver_stable_branch. I will resend.

Thanks,
Daniel

branches.extend(_get_configured_branches('conf/branches.yml'))
branches.extend(
_get_configured_branches(
@@ -141,7 +148,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)


-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +177,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0966fe1..27c39ef 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.

import argparse
+import copy
import subprocess
+import re

import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,38 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' not in branch:
+ # no tag_regexp defined, or mainline
+ continue
+
+ # predefined in branches.yml or a stable branch
+ if re.match(branch['tag_regexp'], branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name, tag = branch_name.split(':', 1)
else:
name = branch_name

for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +70,18 @@ def main(git_repo, remotes,

c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())

@@ -65,15 +102,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue

+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break

for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -104,9 +152,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1

_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

From: Ben Hutchings <ben.hutchings@...>

[...]

+ if 'tag_regexp' in branch:
+ # predefined in conf/branches.yml
+ tag_regexp = branch['tag_regexp']
+ elif branch['git_remote'] == 'stable':
+ # stable format, e.g. v4.19.12
+ esc_base_ver = branch['base_ver'].replace('.', '\.')

This happens to work now, but '\.' is an unrecognised escape sequence
which is deprecated. You presumably meant r'\.', but it might be
clearer to use re.escape().

Thanks, you are right. I have used re.escape('.'). I didn't know this function, it's really useful not having to remember how to escape characters (I figured out I could have used \\.).

I think that the YAML strings in conf/branches.yml are being read as if they were raw strings, but if there is a problem with them let me know.

+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
+ esc_base_ver, esc_base_ver)

I also expected that you would set tag_regexp for stable branches in
the branch module along with all their other fields. Then there's no
need to handle them specially here.

Ah sorry about that. I have moved that code to branch.py now.

+ else:
+ # no tag_regexp defined, or mainline
+ continue
+
+ if re.match(tag_regexp, branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name_tuple = tuple(branch_name.split(':'))
+ name = name_tuple[0]
+ tag = name_tuple[1]

[...]

You really can do simply:

name, tag = branch_name.split(':', 1)

(Tuple assignment only requires an iterable, not specifically a tuple,
on the right hand side.) So please use that.

Nice trick. I fixed the code.

Thanks,
Daniel

Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom

Rather than looking up each issue file, I would like
to have an overview of what each CVE ID means.

Example:
$ ./scripts/report_affected.py --show-description linux-4.4.y-cip

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
scripts/report_affected.py | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 27c39ef..22a923b 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -18,8 +18,8 @@ import kernel_sec.issue
import kernel_sec.version


-def main(git_repo, remotes,
- only_fixed_upstream, include_ignored, *branch_names):
+def main(git_repo, remotes, only_fixed_upstream,
+ include_ignored, show_description, *branch_names):
live_branches = kernel_sec.branch.get_live_branches()
if branch_names:
branches = []
@@ -121,7 +121,13 @@ def main(git_repo, remotes,
sorted_cve_ids = sorted(
branch_issues.get(branch['full_name'], []),
key=kernel_sec.issue.get_id_sort_key)
- print('%s:' % branch['full_name'], *sorted_cve_ids)
+ if show_description:
+ print('%s:' % branch['full_name'])
+ for cve_id in sorted_cve_ids:
+ print(cve_id, '=>',
+ kernel_sec.issue.load(cve_id).get('description', 'None'))
+ else:
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -150,6 +156,9 @@ if __name__ == '__main__':
parser.add_argument('--include-ignored',
action='store_true',
help='include issues that have been marked as ignored')
+ parser.add_argument('--show-description',
+ action='store_true',
+ help='show the issue description')
parser.add_argument('branches',
nargs='*',
help=('specific branch[:tag] or stable tag to '
@@ -162,5 +171,5 @@ if __name__ == '__main__':
mainline=args.mainline_remote_name,
stable=args.stable_remote_name)
kernel_sec.branch.check_git_repo(args.git_repo, remotes)
- main(args.git_repo, remotes,
- args.only_fixed_upstream, args.include_ignored, *args.branches)
+ main(args.git_repo, remotes, args.only_fixed_upstream,
+ args.include_ignored, args.show_description, *args.branches)
--
2.17.1

Reporting on tags is useful for product engineers that
have shipped a kernel with a specific tag and need to know
which issues affect their product after some time.

Examples:
$ ./scripts/report_affected.py v4.4 v4.4.107 v4.4.181-cip33
$ cd ../kernel
$ git tag myproduct-v1 0f13d9b4d0efa9e87381717c113df57718bc92d6
$ cd ../cip-kernel-sec
$ ./scripts/report_affected.py linux-4.19.y-cip:myproduct-v1 v4.19.50-cip3

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
conf/branches.yml | 2 ++
scripts/kernel_sec/branch.py | 11 ++++--
scripts/report_affected.py | 68 +++++++++++++++++++++++++++++++-----
3 files changed, 70 insertions(+), 11 deletions(-)

diff --git a/conf/branches.yml b/conf/branches.yml
index 2ed9db6..8197596 100644
--- a/conf/branches.yml
+++ b/conf/branches.yml
@@ -2,7 +2,9 @@
base_ver: "4.4"
git_remote: cip
git_name: linux-4.4.y-cip
+ tag_regexp: '^v4\.4\.\d+-cip\d+$'
- short_name: linux-4.19.y-cip
base_ver: "4.19"
git_remote: cip
git_name: linux-4.19.y-cip
+ tag_regexp: '^v4\.19\.\d+-cip\d+$'
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 9a7bc3a..1922419 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -121,6 +121,13 @@ def _get_configured_branches(filename):

def get_live_branches():
branches = _get_live_stable_branches()
+ # add regular expressions to infer a stable branch from a stable tag
+ for branch in branches:
+ esc_base_ver = branch['base_ver'].replace('.', re.escape('.'))
+ # example tags: v4.4, v4.19.12
+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (esc_base_ver, esc_base_ver)
+ branch['tag_regexp'] = tag_regexp
+
branches.extend(_get_configured_branches('conf/branches.yml'))
branches.extend(
_get_configured_branches(
@@ -141,7 +148,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)


-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +177,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0966fe1..27c39ef 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.

import argparse
+import copy
import subprocess
+import re

import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,38 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' not in branch:
+ # no tag_regexp defined, or mainline
+ continue
+
+ # predefined in branches.yml or a stable branch
+ if re.match(branch['tag_regexp'], branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name, tag = branch_name.split(':', 1)
else:
name = branch_name

for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +70,18 @@ def main(git_repo, remotes,

c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())

@@ -65,15 +102,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue

+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break

for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -104,9 +152,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1

Hello Ben,

Thanks again for your detailed reviews. I have fixed the tags
patch with your suggestions. I will reply to them separately.

[cip-kernel-sec][RESEND v2 1/2] report_affected: add support for
[cip-kernel-sec][RESEND v2 2/2] report_affected: add show-description

Thanks,
Daniel

Hi,

Thanks for pointed out.
You can find the following sentence on the page.

an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.

I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

Agree.

We think that we need to add the following sentences.
---
We do not apply patches that have not been applied to upstream
(Linus tree:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git).
If you want to add new features or hardware, they need to be applied into upstream.
Also, you may not be able to easily apply the patches on Upstream. In such a case,
you need to describe in the commit message of patch what modifications have been made
from the original patch.
---

In addition to these, we may need to describe the supported hardware (CPU, boards) and
test environment as well.

Best regards,
Nobuhiro

From: Yoshitake Kobayashi [mailto:yoshitake.kobayashi@...]
Sent: Wednesday, July 10, 2019 9:56 PM
To: hiraku.toyooka@...
Cc: cip-dev@...; pavel@...; iwamatsu nobuhiro(岩松 信洋 ○ＳＷＣ□ＯＳＴ) <nobuhiro1.iwamatsu@...>
Subject: Re: [cip-dev] About CIP kernel maintenance policy for new hardware support backporting

Hi,

Thanks for pointed out.
You can find the following sentence on the page. 

an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.

I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

Best regards,
Yoshi



2019年7月10日(水) 15:42 <hiraku.toyooka@...>:
Hello,

I have a question about CIP kernel maintenance policy.
Can CIP kernel accept backport patches for new hardware support (not
only fixes)?

I read the following document and it seems to describe only acceptable fixes.
https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipkernelmaintenance

On the other hand, Current linux-4.19.y-cip branch accepts some hardware
support patches such as RZ/G2E(r8a774c0) from upstream.
So I wonder if some policy about new hardware support exists.

--
Hiraku Toyooka
Cybertrust Japan Co., Ltd.
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev

On Sat, 2019-06-22 at 22:07 +0200, Pavel Machek wrote:

Hi!

There are a couple of open questions, on which I would like to hear
other's opinions:

* Should the source lists be added to the repository or not? If they
are added, then they should not be changed by the standard "all" and
"clean" targets.
I tend to think that they should be added, because they take a long
time to generate and require cross-compilers etc. to be installed.

I believe they should go to the repository, because we may want to
manually adjust the lists.

I've implemented that option.

I don't believe "example" configurations we have necessarily have
enabled all the options "final products" may need. If some common
option (vfat?) is not enabled in our configurations, because it is not
hardware specific and not needed for testing, we may still want to
review it, because it is common enough that someone will need it...

I hadn't thought of that, but it does seem possible. However, I think
that rather than editing a generated list it would be better to have a
separate manually maintained list that is merged into the generated
list.

Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom

On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:

Hello Ben,

Thank you for the detailed review, and sorry for the delay.
I have modified my patch series taking your comments into account.
Note: I will reply to your comments separately

I re-send to you the whole series to make sure that patches
apply properly.

I've applied patches 1, 2, 3, and 5 (as it didn't depend on 4). Please
re-send the remaining two with my requested changes.

Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom

On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
[...]

--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py

[...]

@@ -22,15 +24,47 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' in branch:
+ # predefined in conf/branches.yml
+ tag_regexp = branch['tag_regexp']
+ elif branch['git_remote'] == 'stable':
+ # stable format, e.g. v4.19.12
+ esc_base_ver = branch['base_ver'].replace('.', '\.')

This happens to work now, but '\.' is an unrecognised escape sequence
which is deprecated. You presumably meant r'\.', but it might be
clearer to use re.escape().

+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
+ esc_base_ver, esc_base_ver)

I also expected that you would set tag_regexp for stable branches in
the branch module along with all their other fields. Then there's no
need to handle them specially here.

+ else:
+ # no tag_regexp defined, or mainline
+ continue
+
+ if re.match(tag_regexp, branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name_tuple = tuple(branch_name.split(':'))
+ name = name_tuple[0]
+ tag = name_tuple[1]

[...]

You really can do simply:

name, tag = branch_name.split(':', 1)

(Tuple assignment only requires an iterable, not specifically a tuple,
on the right hand side.) So please use that.

Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom