Date   

Re: About CIP kernel maintenance policy for new hardware support backporting

Hiraku Toyooka
 

Hi,

You can find the following sentence on the page.
an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.
Oh, I missed the sentence. Thank you very much for telling it.

In addition to these, we may need to describe the supported hardware (CPU, boards) and
test environment as well.
Is the backporting limited to the supported hardware?

Best regards,
Hiraku Toyooka

2019年7月11日(木) 8:47 <nobuhiro1.iwamatsu@...>:


Hi,


Thanks for pointed out.
You can find the following sentence on the page.
an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.
I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.
Agree.

We think that we need to add the following sentences.
---
We do not apply patches that have not been applied to upstream
(Linus tree:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git).
If you want to add new features or hardware, they need to be applied into upstream.
Also, you may not be able to easily apply the patches on Upstream. In such a case,
you need to describe in the commit message of patch what modifications have been made
from the original patch.
---

In addition to these, we may need to describe the supported hardware (CPU, boards) and
test environment as well.

Best regards,
Nobuhiro

From: Yoshitake Kobayashi [mailto:yoshitake.kobayashi@...]
Sent: Wednesday, July 10, 2019 9:56 PM
To: hiraku.toyooka@...
Cc: cip-dev@...; pavel@...; iwamatsu nobuhiro(岩松 信洋 ○SWC□OST) <nobuhiro1.iwamatsu@...>
Subject: Re: [cip-dev] About CIP kernel maintenance policy for new hardware support backporting

Hi,

Thanks for pointed out.
You can find the following sentence on the page.
an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.
I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

Best regards,
Yoshi



2019年7月10日(水) 15:42 <hiraku.toyooka@...>:
Hello,

I have a question about CIP kernel maintenance policy.
Can CIP kernel accept backport patches for new hardware support (not
only fixes)?

I read the following document and it seems to describe only acceptable fixes.
https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipkernelmaintenance

On the other hand, Current linux-4.19.y-cip branch accepts some hardware
support patches such as RZ/G2E(r8a774c0) from upstream.
So I wonder if some policy about new hardware support exists.

--
Hiraku Toyooka
Cybertrust Japan Co., Ltd.
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev


--
Hiraku Toyooka
Cybertrust Japan Co., Ltd.


[cip-kernel-sec][Quickstart v2] docs: add a quickstart with practical information

Daniel Sangorrin <daniel.sangorrin@...>
 

Although the README already contains all the information
that users may need, there are some bits of know-how that
are better expressed through a step-by-step quickstart or
tutorial. This files tries to fill that gap.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
QUICKSTART.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
create mode 100644 QUICKSTART.md

diff --git a/QUICKSTART.md b/QUICKSTART.md
new file mode 100644
index 0000000..c79af41
--- /dev/null
+++ b/QUICKSTART.md
@@ -0,0 +1,132 @@
+# Quickstart
+
+## Overview
+
+This project tracks the status of CVEs in mainline and stable kernels. Each CVE is described in YAML format that includes data such as:
+
+```
+$ cat issues/CVE-2019-1999.yml
+description: 'binder: fix race between munmap() and direct reclaim'
+references:
+- https://source.android.com/security/bulletin/2019-02-01
+comments:
+ Debian-bwh: |-
+ Introduced in 4.14 by f2517eb76f1f "android: binder: Add global lru
+ shrinker to binder". Backports of the fix to stable have incorrect
+ metadata.
+ bwh: Backports to stable have incorrect metadata
+introduced-by:
+ mainline: [f2517eb76f1f2f7f89761f9db2b202e89931738c]
+fixed-by:
+ linux-4.14.y: [33c6b9ca70a8b066a613e2a3d0331ae8f82aa31a]
+ linux-4.19.y: [6bf7d3c5c0c5dad650bfc4345ed553c18b69d59e]
+ linux-5.0.y: [bbb19ca082ce27ce60ca65be016a951806ea947c]
+ mainline: [5cec2d2e5839f9c0fec319c523a911e0a7fd299f]
+```
+
+## Quickstart
+
+Clone `cip-kernel-sec` and install its dependencies:
+
+```
+$ git clone https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec
+$ cd cip-kernel-sec/
+$ sudo apt install python3-yaml python3-html5lib python3-cherrypy3 python3-jinja2
+```
+
+Prepare kernel remote repositories according to `conf/remotes.yml`:
+
+```
+$ ./scripts/prepare_remotes.py
+```
+
+Alternatively, you can do that manually:
+
+```
+$ mkdir ../kernel
+$ cd ../kernel
+$ git remote add torvalds https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+$ git remote add cip https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
+$ cd ../cip-kernel-sec
+```
+
+Download CVE information from [Debian] (https://salsa.debian.org/kernel-team/kernel-sec.git), [Ubuntu] (https://git.launchpad.net/ubuntu-cve-tracker) and Stable:
+
+```
+
+$ ./scripts/import_debian.py
+ -> import/debian
+$ ./scripts/import_ubuntu.py
+ -> import/ubuntu
+$ ./scripts/import_stable.py
+ -> import/stable_branches.yml
+```
+
+Check issues that affect a linux-cip branch:
+
+```
+$ ./scripts/report_affected.py linux-4.4.y
+```
+
+You can show a short description on your report:
+
+```
+$ ./scripts/report_affected.py --show-description linux-4.4.y
+```
+
+Check issues that affect a tag:
+
+```
+$ ./scripts/report_affected.py v4.4.181-cip33
+```
+
+Browse kernel branches and issues interactively:
+
+```
+$ ./scripts/webview.py
+$ firefox http://localhost:8080
+```
+
+[Note] Use Ctr-c to stop the `webview.py` script.
+
+## Kernel maintainer workflow
+
+Import or update the latest CVE information:
+
+```
+$ ./scripts/import_debian.py
+$ ./scripts/import_ubuntu.py
+$ ./scripts/import_stable.py
+```
+
+Edit by hand the newly created issues if you see that some imported information is incorrect or there is missing information:
+
+```
+$ vi issues/CVE-xx.yml
+```
+
+Validate the issue files against the YAML schema.
+
+```
+$ ./scripts/validate.py
+```
+
+YAML allows the same thing to be written in different ways, e.g. bracketed vs bulleted lists. Use `cleanup.py` to make the syntax and ordering of items consistent with the importers, to reduce "noise" in diffs:
+
+```
+$ ./scripts/cleanup.py
+```
+
+Check if the current issues:
+
+```
+$ ./scripts/report_affected.py
+```
+
+## Changelog
+
+- 20190614: First version <daniel.sangorrin@...>
+- 20190618: Add workflow information provided by Ben
+- 20190711: Add tag reporting <daniel.sangorrin@...>
+
--
2.17.1


(No subject)

Daniel Sangorrin <daniel.sangorrin@...>
 

Hello Ben,

Sorry, I realized that there were a few issues in the Quickstart
so I am resending the patch. Please ignore the previous one.

[cip-kernel-sec][Quickstart v2] docs: add a quickstart with practical

Thanks,
Daniel


[cip-kernel-sec] readme: add info about tag_regexp and show-description

Daniel Sangorrin <daniel.sangorrin@...>
 

Probably this should be squashed into the corresponding
patches.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
README.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 576cc75..8164826 100644
--- a/README.md
+++ b/README.md
@@ -41,7 +41,8 @@ current or previous year or that are already tracked here.
stable and other configured branches, by reading the git commit logs.

* `scripts/report_affected.py` - report which issues affect the
-specified branches, or all active branches.
+specified branches, or all active branches. You can use --show-description
+to obtain a short description for each CVE ID.

* `scripts/validate.py` - validate all issue files against the
schema.
@@ -72,6 +73,7 @@ keys:
* `base_ver`: Stable version that the branch is based on, e.g.
"4.4". This needs to be quoted so that it's a string not a
number.
+* `tag_regexp`: A regular expression that matches tags on a branch.

### Remotes

--
2.17.1


[cip-kernel-sec][quickstart] docs: add a quickstart with practical information

Daniel Sangorrin <daniel.sangorrin@...>
 

Although the README already contains all the information
that users may need, there are some bits of know-how that
are better expressed through a step-by-step quickstart or
tutorial. This files tries to fill that gap.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
QUICKSTART.md | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 126 insertions(+)
create mode 100644 QUICKSTART.md

diff --git a/QUICKSTART.md b/QUICKSTART.md
new file mode 100644
index 0000000..4bceb97
--- /dev/null
+++ b/QUICKSTART.md
@@ -0,0 +1,126 @@
+# Quickstart
+
+## Overview
+
+This project tracks the status of CVEs in mainline and stable kernels. Each CVE is described in YAML format that includes data such as:
+
+```
+$ cat CVE-2019-1999.yml
+description: 'binder: fix race between munmap() and direct reclaim'
+references:
+- https://source.android.com/security/bulletin/2019-02-01
+comments:
+ Debian-bwh: |-
+ Introduced in 4.14 by f2517eb76f1f "android: binder: Add global lru
+ shrinker to binder". Backports of the fix to stable have incorrect
+ metadata.
+ bwh: Backports to stable have incorrect metadata
+introduced-by:
+ mainline: [f2517eb76f1f2f7f89761f9db2b202e89931738c]
+fixed-by:
+ linux-4.14.y: [33c6b9ca70a8b066a613e2a3d0331ae8f82aa31a]
+ linux-4.19.y: [6bf7d3c5c0c5dad650bfc4345ed553c18b69d59e]
+ linux-5.0.y: [bbb19ca082ce27ce60ca65be016a951806ea947c]
+ mainline: [5cec2d2e5839f9c0fec319c523a911e0a7fd299f]
+```
+
+## Quickstart
+
+Clone `cip-kernel-sec` and install its dependencies:
+
+```
+$ git clone https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec
+$ cd cip-kernel-sec/
+$ sudo apt install python3-yaml and python3-html5lib python3-cherrypy3 python3-jinja2
+```
+
+Prepare kernel remote repositories according to `conf/remotes.yml`:
+
+```
+$ ./scripts/prepare_remotes.py
+```
+
+Alternatively, you can do that manually:
+
+```
+$ mkdir ../kernel
+$ cd ../kernel
+$ git remote add torvalds https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+$ git remote add cip https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
+$ cd ../cip-kernel-sec
+```
+
+Download CVE information from Debian (https://salsa.debian.org/kernel-team/kernel-sec.git), Ubuntu (https://git.launchpad.net/ubuntu-cve-tracker) and Stable:
+
+```
+
+$ ./scripts/import_debian.py
+ -> import/debian
+$ ./scripts/import_ubuntu.py
+ -> import/ubuntu
+$ ./scripts/import_stable.py
+ -> import/stable_branches.yml
+```
+
+Check issues that affect a linux-cip branch:
+
+```
+$ ./scripts/report_affected.py linux-4.4.y
+```
+
+Check issues that affect a tag:
+
+```
+$ ./scripts/report_affected.py v4.4.181-cip33
+```
+
+Browse kernel branches and issues interactively:
+
+```
+$ ./scripts/webview.py
+$ firefox http://localhost:8080
+```
+
+[Note] Use Ctr-c to stop the `webview.py` script.
+
+## Kernel maintainer workflow
+
+Import or update the latest CVE information:
+
+```
+$ ./scripts/import_debian.py
+$ ./scripts/import_ubuntu.py
+$ ./scripts/import_stable.py
+```
+
+Edit by hand the newly created issues if you see that some imported information is incorrect or there is missing information:
+
+```
+$ vi issues/CVE-xx.yml
+```
+
+Validate the issue files against the YAML schema.
+
+```
+$ ./scripts/validate.py
+```
+
+YAML allows the same thing to be written in different ways, e.g. bracketed vs bulleted lists. Use `cleanup.py` to make the syntax and ordering of items consistent with the importers, to reduce "noise" in diffs:
+
+```
+$ ./scripts/cleanup.py
+```
+
+Check if the current issues:
+
+```
+$ ./scripts/report_affected.py
+```
+
+## Changelog
+
+- 20190614: First version <daniel.sangorrin@...>
+- 20190618: Add workflow information provided by Ben
+- 20190711: Add tag reporting <daniel.sangorrin@...>
+
--
2.17.1


Add quickstart

Daniel Sangorrin <daniel.sangorrin@...>
 

Hello Ben,

I was planning to put this into the CIP wiki, but I though it might
be better to store it together with the source code so there you go.

[cip-kernel-sec][quickstart] docs: add a quickstart with practical

Thanks,
Daniel


[cip-kernel-sec][RESEND v3] report_affected: add support for reporting on tags

Daniel Sangorrin <daniel.sangorrin@...>
 

Reporting on tags is useful for product engineers that
have shipped a kernel with a specific tag and need to know
which issues affect their product after some time.

Examples:
$ ./scripts/report_affected.py v4.4 v4.4.107 v4.4.181-cip33
$ cd ../kernel
$ git tag myproduct-v1 0f13d9b4d0efa9e87381717c113df57718bc92d6
$ cd ../cip-kernel-sec
$ ./scripts/report_affected.py linux-4.19.y-cip:myproduct-v1 v4.19.50-cip3

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
conf/branches.yml | 2 ++
scripts/kernel_sec/branch.py | 8 +++--
scripts/report_affected.py | 68 +++++++++++++++++++++++++++++++-----
3 files changed, 66 insertions(+), 12 deletions(-)

diff --git a/conf/branches.yml b/conf/branches.yml
index 2ed9db6..8197596 100644
--- a/conf/branches.yml
+++ b/conf/branches.yml
@@ -2,7 +2,9 @@
base_ver: "4.4"
git_remote: cip
git_name: linux-4.4.y-cip
+ tag_regexp: '^v4\.4\.\d+-cip\d+$'
- short_name: linux-4.19.y-cip
base_ver: "4.19"
git_remote: cip
git_name: linux-4.19.y-cip
+ tag_regexp: '^v4\.19\.\d+-cip\d+$'
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 9a7bc3a..cd4f423 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -23,11 +23,13 @@ from . import version

def get_base_ver_stable_branch(base_ver):
branch_name = 'linux-%s.y' % base_ver
+ esc_base_ver = re.escape(base_ver)
return {
'short_name': branch_name,
'git_remote': 'stable',
'git_name': branch_name,
- 'base_ver': base_ver
+ 'base_ver': base_ver,
+ 'tag_regexp' : r'(^v%s$|^v%s\.\d+$)' % (esc_base_ver, esc_base_ver)
}


@@ -141,7 +143,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)


-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +172,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0966fe1..27c39ef 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.

import argparse
+import copy
import subprocess
+import re

import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,38 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' not in branch:
+ # no tag_regexp defined, or mainline
+ continue
+
+ # predefined in branches.yml or a stable branch
+ if re.match(branch['tag_regexp'], branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name, tag = branch_name.split(':', 1)
else:
name = branch_name

for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +70,18 @@ def main(git_repo, remotes,

c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())

@@ -65,15 +102,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue

+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break

for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -104,9 +152,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1


(Resend v3) move the tag with the other fields

Daniel Sangorrin <daniel.sangorrin@...>
 

Hello Ben,

I think this should be fine.
You need to remove the cached file import/stable_branches.yml

[cip-kernel-sec][RESEND v3] report_affected: add support for

To be honest, I am not 100% sure that the escape code is
perfectly correct. The file stable_branches.yml will have
parameters like this:

tag_regexp: (^v3\.16$|^v3\.16\.\d+$)

Thanks,
Daniel


Re: [cip-kernel-sec][RESEND v2 1/2] report_affected: add support for reporting on tags

daniel.sangorrin@...
 

+ for branch in branches:
+ esc_base_ver = branch['base_ver'].replace('.', re.escape('.'))
+ # example tags: v4.4, v4.19.12
+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (esc_base_ver, esc_base_ver)
+ branch['tag_regexp'] = tag_regexp
+
Sorry, I went too fast. I should have moved the regexp to get_base_ver_stable_branch. I will resend.

Thanks,
Daniel


branches.extend(_get_configured_branches('conf/branches.yml'))
branches.extend(
_get_configured_branches(
@@ -141,7 +148,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)


-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +177,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0966fe1..27c39ef 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.

import argparse
+import copy
import subprocess
+import re

import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,38 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' not in branch:
+ # no tag_regexp defined, or mainline
+ continue
+
+ # predefined in branches.yml or a stable branch
+ if re.match(branch['tag_regexp'], branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name, tag = branch_name.split(':', 1)
else:
name = branch_name

for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +70,18 @@ def main(git_repo, remotes,

c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())

@@ -65,15 +102,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue

+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break

for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -104,9 +152,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1

_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev


Re: [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags

daniel.sangorrin@...
 

From: Ben Hutchings <ben.hutchings@...>
[...]
+ if 'tag_regexp' in branch:
+ # predefined in conf/branches.yml
+ tag_regexp = branch['tag_regexp']
+ elif branch['git_remote'] == 'stable':
+ # stable format, e.g. v4.19.12
+ esc_base_ver = branch['base_ver'].replace('.', '\.')
This happens to work now, but '\.' is an unrecognised escape sequence
which is deprecated. You presumably meant r'\.', but it might be
clearer to use re.escape().
Thanks, you are right. I have used re.escape('.'). I didn't know this function, it's really useful not having to remember how to escape characters (I figured out I could have used \\.).

I think that the YAML strings in conf/branches.yml are being read as if they were raw strings, but if there is a problem with them let me know.


+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
+ esc_base_ver, esc_base_ver)
I also expected that you would set tag_regexp for stable branches in
the branch module along with all their other fields. Then there's no
need to handle them specially here.
Ah sorry about that. I have moved that code to branch.py now.


+ else:
+ # no tag_regexp defined, or mainline
+ continue
+
+ if re.match(tag_regexp, branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name_tuple = tuple(branch_name.split(':'))
+ name = name_tuple[0]
+ tag = name_tuple[1]
[...]

You really can do simply:

name, tag = branch_name.split(':', 1)

(Tuple assignment only requires an iterable, not specifically a tuple,
on the right hand side.) So please use that.
Nice trick. I fixed the code.

Thanks,
Daniel



Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom


[cip-kernel-sec][RESEND v2 2/2] report_affected: add show-description option

Daniel Sangorrin <daniel.sangorrin@...>
 

Rather than looking up each issue file, I would like
to have an overview of what each CVE ID means.

Example:
$ ./scripts/report_affected.py --show-description linux-4.4.y-cip

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
scripts/report_affected.py | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 27c39ef..22a923b 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -18,8 +18,8 @@ import kernel_sec.issue
import kernel_sec.version


-def main(git_repo, remotes,
- only_fixed_upstream, include_ignored, *branch_names):
+def main(git_repo, remotes, only_fixed_upstream,
+ include_ignored, show_description, *branch_names):
live_branches = kernel_sec.branch.get_live_branches()
if branch_names:
branches = []
@@ -121,7 +121,13 @@ def main(git_repo, remotes,
sorted_cve_ids = sorted(
branch_issues.get(branch['full_name'], []),
key=kernel_sec.issue.get_id_sort_key)
- print('%s:' % branch['full_name'], *sorted_cve_ids)
+ if show_description:
+ print('%s:' % branch['full_name'])
+ for cve_id in sorted_cve_ids:
+ print(cve_id, '=>',
+ kernel_sec.issue.load(cve_id).get('description', 'None'))
+ else:
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -150,6 +156,9 @@ if __name__ == '__main__':
parser.add_argument('--include-ignored',
action='store_true',
help='include issues that have been marked as ignored')
+ parser.add_argument('--show-description',
+ action='store_true',
+ help='show the issue description')
parser.add_argument('branches',
nargs='*',
help=('specific branch[:tag] or stable tag to '
@@ -162,5 +171,5 @@ if __name__ == '__main__':
mainline=args.mainline_remote_name,
stable=args.stable_remote_name)
kernel_sec.branch.check_git_repo(args.git_repo, remotes)
- main(args.git_repo, remotes,
- args.only_fixed_upstream, args.include_ignored, *args.branches)
+ main(args.git_repo, remotes, args.only_fixed_upstream,
+ args.include_ignored, args.show_description, *args.branches)
--
2.17.1


[cip-kernel-sec][RESEND v2 1/2] report_affected: add support for reporting on tags

Daniel Sangorrin <daniel.sangorrin@...>
 

Reporting on tags is useful for product engineers that
have shipped a kernel with a specific tag and need to know
which issues affect their product after some time.

Examples:
$ ./scripts/report_affected.py v4.4 v4.4.107 v4.4.181-cip33
$ cd ../kernel
$ git tag myproduct-v1 0f13d9b4d0efa9e87381717c113df57718bc92d6
$ cd ../cip-kernel-sec
$ ./scripts/report_affected.py linux-4.19.y-cip:myproduct-v1 v4.19.50-cip3

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
---
conf/branches.yml | 2 ++
scripts/kernel_sec/branch.py | 11 ++++--
scripts/report_affected.py | 68 +++++++++++++++++++++++++++++++-----
3 files changed, 70 insertions(+), 11 deletions(-)

diff --git a/conf/branches.yml b/conf/branches.yml
index 2ed9db6..8197596 100644
--- a/conf/branches.yml
+++ b/conf/branches.yml
@@ -2,7 +2,9 @@
base_ver: "4.4"
git_remote: cip
git_name: linux-4.4.y-cip
+ tag_regexp: '^v4\.4\.\d+-cip\d+$'
- short_name: linux-4.19.y-cip
base_ver: "4.19"
git_remote: cip
git_name: linux-4.19.y-cip
+ tag_regexp: '^v4\.19\.\d+-cip\d+$'
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 9a7bc3a..1922419 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -121,6 +121,13 @@ def _get_configured_branches(filename):

def get_live_branches():
branches = _get_live_stable_branches()
+ # add regular expressions to infer a stable branch from a stable tag
+ for branch in branches:
+ esc_base_ver = branch['base_ver'].replace('.', re.escape('.'))
+ # example tags: v4.4, v4.19.12
+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (esc_base_ver, esc_base_ver)
+ branch['tag_regexp'] = tag_regexp
+
branches.extend(_get_configured_branches('conf/branches.yml'))
branches.extend(
_get_configured_branches(
@@ -141,7 +148,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)


-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +177,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0966fe1..27c39ef 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.

import argparse
+import copy
import subprocess
+import re

import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,38 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' not in branch:
+ # no tag_regexp defined, or mainline
+ continue
+
+ # predefined in branches.yml or a stable branch
+ if re.match(branch['tag_regexp'], branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name, tag = branch_name.split(':', 1)
else:
name = branch_name

for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +70,18 @@ def main(git_repo, remotes,

c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())

@@ -65,15 +102,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue

+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break

for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)


if __name__ == '__main__':
@@ -104,9 +152,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1


(Resend v2) report issues for tags

Daniel Sangorrin <daniel.sangorrin@...>
 

Hello Ben,

Thanks again for your detailed reviews. I have fixed the tags
patch with your suggestions. I will reply to them separately.

[cip-kernel-sec][RESEND v2 1/2] report_affected: add support for
[cip-kernel-sec][RESEND v2 2/2] report_affected: add show-description

Thanks,
Daniel


CIP IRC weekly meeting today

SZ Lin (林上智) <SZ.Lin@...>
 

Hi all,

 

Kindly be reminded to attend the weekly meeting through IRC to discuss technical topics with CIP kernel today.

 

*Please note that IRC meeting was rescheduled to UTC (GMT) 09:00 starting from the first week of Apr. according to TSC meeting*

https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019&month=7&day11&hour=9&min=0&sec=0&p1=241&p2=137&p3=179&p4=136&p5=37&p6=248

 

US-West US-East   UK     DE     TW     JP

02:00    05:00   10:00   11:00   17:00   18:00

 

Channel:

* irc:chat.freenode.net:6667/cip

 

Agenda:

 

* Action item

1. Provide the script for CIP kernel config collection - bwh

#link https://lists.cip-project.org/pipermail/cip-dev/2019-June/002506.html

2. List real time kernel questions to ask Daniel Wagner - szlin

3. Try updating CIP RT kernel to 4.19.50 - Pavel

#link https://lists.cip-project.org/pipermail/cip-dev/2019-June/002548.html

4. Work out a solution for LAVA master backups - patersonc

* Kernel maintenance updates

* Kernel testing

* CIP Core

* Software update

* AOB

 

The meeting will take 30 min, although it can be extended to an hour if it makes sense and those involved in the topics can stay. Otherwise, the topic will be taken offline or in the next meeting.

 

Best regards,

 

SZ Lin, Moxa.


Re: About CIP kernel maintenance policy for new hardware support backporting

Nobuhiro Iwamatsu
 

Hi,


Thanks for pointed out.
You can find the following sentence on the page.
an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.
I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.
Agree.

We think that we need to add the following sentences.
---
We do not apply patches that have not been applied to upstream
(Linus tree:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git).
If you want to add new features or hardware, they need to be applied into upstream.
Also, you may not be able to easily apply the patches on Upstream. In such a case,
you need to describe in the commit message of patch what modifications have been made
from the original patch.
---

In addition to these, we may need to describe the supported hardware (CPU, boards) and
test environment as well.

Best regards,
Nobuhiro

From: Yoshitake Kobayashi [mailto:yoshitake.kobayashi@...]
Sent: Wednesday, July 10, 2019 9:56 PM
To: hiraku.toyooka@...
Cc: cip-dev@...; pavel@...; iwamatsu nobuhiro(岩松 信洋 ○SWC□OST) <nobuhiro1.iwamatsu@...>
Subject: Re: [cip-dev] About CIP kernel maintenance policy for new hardware support backporting

Hi,

Thanks for pointed out.
You can find the following sentence on the page. 
an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.
I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

Best regards,
Yoshi



2019年7月10日(水) 15:42 <hiraku.toyooka@...>:
Hello,

I have a question about CIP kernel maintenance policy.
Can CIP kernel accept backport patches for new hardware support (not
only fixes)?

I read the following document and it seems to describe only acceptable fixes.
https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipkernelmaintenance

On the other hand, Current linux-4.19.y-cip branch accepts some hardware
support patches such as RZ/G2E(r8a774c0) from upstream.
So I wonder if some policy about new hardware support exists.

--
Hiraku Toyooka
Cybertrust Japan Co., Ltd.
_______________________________________________
cip-dev mailing list
cip-dev@...
https://lists.cip-project.org/mailman/listinfo/cip-dev


Re: [RFC] Script to find used sources in the kernel

Ben Hutchings <ben.hutchings@...>
 

On Sat, 2019-06-22 at 22:07 +0200, Pavel Machek wrote:
Hi!

There are a couple of open questions, on which I would like to hear
other's opinions:

* Should the source lists be added to the repository or not? If they
are added, then they should not be changed by the standard "all" and
"clean" targets.
I tend to think that they should be added, because they take a long
time to generate and require cross-compilers etc. to be installed.
I believe they should go to the repository, because we may want to
manually adjust the lists.
I've implemented that option.

I don't believe "example" configurations we have necessarily have
enabled all the options "final products" may need. If some common
option (vfat?) is not enabled in our configurations, because it is not
hardware specific and not needed for testing, we may still want to
review it, because it is common enough that someone will need it...
I hadn't thought of that, but it does seem possible. However, I think
that rather than editing a generated list it would be better to have a
separate manually maintained list that is merged into the generated
list.

Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom


Re: Add support for cip branches and tags

Ben Hutchings <ben.hutchings@...>
 

On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
Hello Ben,

Thank you for the detailed review, and sorry for the delay.
I have modified my patch series taking your comments into account.
Note: I will reply to your comments separately

I re-send to you the whole series to make sure that patches
apply properly.
I've applied patches 1, 2, 3, and 5 (as it didn't depend on 4). Please
re-send the remaining two with my requested changes.

Ben.

--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom


[Git][cip-project/cip-kernel/cip-kernel-sec][master] 4 commits: check_git_repo: add checks to the local repository

Agustin Benito Bethencourt
 

Ben Hutchings pushed to branch master at cip-project / cip-kernel / cip-kernel-sec

Commits:

  • 6b530a7c
    by Daniel Sangorrin at 2019-07-10T14:43:03Z
    check_git_repo: add checks to the local repository
    
    Add checks to make sure that the local repository exists
    and has the configured remotes in place.
    
    Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
    Signed-off-by: Ben Hutchings <ben.hutchings@...>
    
  • c3495333
    by Daniel Sangorrin at 2019-07-10T14:43:03Z
    prepare_remotes: helper script to prepare local repo
    
    Helper script that prepares the local git repository
    with the configured remote branches. Expert developers
    and kernel maintainers will probably have their own
    worktrees but for new users or a quickstart, this
    script should be helpful.
    
    Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
    Signed-off-by: Ben Hutchings <ben.hutchings@...>
    
  • 3eaa6089
    by Daniel Sangorrin at 2019-07-10T14:43:03Z
    report_affected: fix code when branches are specified
    
    The previous code could not handle branches with names
    other than stable branch names. For example, passing
    "linux-4.4.y-cip" as a branch would return an error.
    
    Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
    Signed-off-by: Ben Hutchings <ben.hutchings@...>
    
  • ddf0f91c
    by Daniel Sangorrin at 2019-07-10T14:43:03Z
    pep8: fix pep8-related errors such as too long lines
    
    These were distracting when checking new code.
    
    Signed-off-by: Daniel Sangorrin <daniel.sangorrin@...>
    Signed-off-by: Ben Hutchings <ben.hutchings@...>
    

7 changed files:

Changes:

  • README.md
    ... ... @@ -23,6 +23,10 @@ this is assumed to be in `../kernel`, with remotes configured in
    23 23
     stable and cip repositories. These can be overridden by command-line options
    
    24 24
     or configuration (`~/.config/kernel-sec/remotes.yml`).
    
    25 25
     
    
    26
    +* `scripts/prepare_remotes.py` - creates the local git repository
    
    27
    +and adds all configured remotes. You may prefer to skip this script
    
    28
    +and configure the repository by hand.
    
    29
    +
    
    26 30
     * `scripts/import_debian.py` - import information from Debian's
    
    27 31
     `kernel_sec` project.  It includes all issues that Debian considers
    
    28 32
     active or that are already tracked here.
    
    ... ... @@ -81,6 +85,7 @@ with the keys:
    81 85
       branch from this remote.
    
    82 86
     * `git_name`: (optional) The name actually used for this git
    
    83 87
       remote, if it's different from the default.
    
    88
    +* `git_repo_url`: URL of the remote git repository.
    
    84 89
     
    
    85 90
     ## Contributions
    
    86 91
     
    

  • conf/remotes.yml
    1 1
     torvalds:
    
    2 2
       commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
    
    3
    +  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    
    3 4
     stable:
    
    4 5
       commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id=
    
    6
    +  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
    
    5 7
     cip:
    
    6 8
       commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/commit?id=
    
    9
    +  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git

  • scripts/import_stable.py
    ... ... @@ -31,11 +31,6 @@ BACKPORT_COMMIT_BOTTOM_RE = re.compile(
    31 31
         .format(**RE_USE))
    
    32 32
     
    
    33 33
     
    
    34
    -def update(git_repo, remote_name):
    
    35
    -    subprocess.check_call(['git', 'remote', 'update', remote_name],
    
    36
    -                          cwd=git_repo)
    
    37
    -
    
    38
    -
    
    39 34
     def get_backports(git_repo, remotes, branches, debug=False):
    
    40 35
         backports = {}
    
    41 36
     
    
    ... ... @@ -69,7 +64,8 @@ def get_backports(git_repo, remotes, branches, debug=False):
    69 64
                         backports.setdefault(mainline_commit, {})[branch_name] \
    
    70 65
                             = stable_commit
    
    71 66
                     if line.strip() != '':
    
    72
    -                    commit_re = BACKPORT_COMMIT_BOTTOM_RE  # next line is not top
    
    67
    +                    # next line is not top
    
    68
    +                    commit_re = BACKPORT_COMMIT_BOTTOM_RE
    
    73 69
     
    
    74 70
         return backports
    
    75 71
     
    
    ... ... @@ -125,7 +121,8 @@ def add_backports(branches, c_b_map, issue_commits, all_backports,
    125 121
                     if debug_context:
    
    126 122
                         print('%s/%s: recording commits' %
    
    127 123
                               (debug_context, branch_name))
    
    128
    -                issue_commits.setdefault(branch_name, []).extend(branch_commits)
    
    124
    +                issue_commits.setdefault(
    
    125
    +                    branch_name, []).extend(branch_commits)
    
    129 126
                     changed = True
    
    130 127
                 else:
    
    131 128
                     if debug_context:
    
    ... ... @@ -140,7 +137,8 @@ def main(git_repo, remotes, debug=False):
    140 137
         remote_names = set(branch['git_remote'] for branch in branches)
    
    141 138
     
    
    142 139
         for remote_name in remote_names:
    
    143
    -        update(git_repo, remotes[remote_name]['git_name'])
    
    140
    +        kernel_sec.branch.remote_update(
    
    141
    +            git_repo, remotes[remote_name]['git_name'])
    
    144 142
         backports = get_backports(git_repo, remotes, branches, debug)
    
    145 143
         c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
    
    146 144
     
    
    ... ... @@ -173,7 +171,7 @@ if __name__ == '__main__':
    173 171
                             metavar='DIRECTORY')
    
    174 172
         parser.add_argument('--remote-name',
    
    175 173
                             dest='remote_name', action='append', default=[],
    
    176
    -                        help='git remote name mappings, e.g. stable:korg-stable',
    
    174
    +                        help='git remote name mappings, e.g. stable:mystable',
    
    177 175
                             metavar='NAME:OTHER-NAME')
    
    178 176
         parser.add_argument('--mainline-remote',
    
    179 177
                             dest='mainline_remote_name',
    
    ... ... @@ -190,4 +188,5 @@ if __name__ == '__main__':
    190 188
         remotes = kernel_sec.branch.get_remotes(args.remote_name,
    
    191 189
                                                 mainline=args.mainline_remote_name,
    
    192 190
                                                 stable=args.stable_remote_name)
    
    191
    +    kernel_sec.branch.check_git_repo(args.git_repo, remotes)
    
    193 192
         main(args.git_repo, remotes, args.debug)

  • scripts/kernel_sec/branch.py
    ... ... @@ -4,10 +4,12 @@
    4 4
     # Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
    
    5 5
     # for details.
    
    6 6
     
    
    7
    +import argparse
    
    7 8
     import io
    
    8 9
     import os
    
    9 10
     import re
    
    10 11
     import subprocess
    
    12
    +import sys
    
    11 13
     import time
    
    12 14
     import urllib.error
    
    13 15
     import urllib.request
    
    ... ... @@ -19,9 +21,6 @@ import yaml
    19 21
     from . import version
    
    20 22
     
    
    21 23
     
    
    22
    -_STABLE_BRANCH_RE = re.compile(r'^linux-([\d.]+)\.y$')
    
    23
    -
    
    24
    -
    
    25 24
     def get_base_ver_stable_branch(base_ver):
    
    26 25
         branch_name = 'linux-%s.y' % base_ver
    
    27 26
         return {
    
    ... ... @@ -32,11 +31,6 @@ def get_base_ver_stable_branch(base_ver):
    32 31
             }
    
    33 32
     
    
    34 33
     
    
    35
    -def get_stable_branch(branch_name):
    
    36
    -    match = _STABLE_BRANCH_RE.match(branch_name)
    
    37
    -    return match and get_base_ver_stable_branch(match.group(1))
    
    38
    -
    
    39
    -
    
    40 34
     def _extract_live_stable_branches(doc):
    
    41 35
         xhtml_ns = 'http://www.w3.org/1999/xhtml'
    
    42 36
         ns = {'html': xhtml_ns}
    
    ... ... @@ -68,8 +62,8 @@ def _extract_live_stable_branches(doc):
    68 62
                 if match:
    
    69 63
                     version = match.group(1)
    
    70 64
                     eol = match.group(2) is not None
    
    71
    -        if branch_type not in ['mainline', 'stable', 'longterm', 'linux-next'] \
    
    72
    -           or version is None:
    
    65
    +        if branch_type not in ['mainline', 'stable', 'longterm',
    
    66
    +                               'linux-next'] or version is None:
    
    73 67
                 raise ValueError('failed to parse releases row text %r' % row_text)
    
    74 68
     
    
    75 69
             # Filter out irrelevant branches
    
    ... ... @@ -219,3 +213,32 @@ def get_remotes(mappings, mainline=None, stable=None):
    219 213
         if stable:
    
    220 214
             remotes['stable']['git_name'] = stable
    
    221 215
         return remotes
    
    216
    +
    
    217
    +
    
    218
    +def remote_update(git_repo, remote_name):
    
    219
    +    subprocess.check_call(['git', 'remote', 'update', remote_name],
    
    220
    +                          cwd=git_repo)
    
    221
    +
    
    222
    +
    
    223
    +def remote_add(git_repo, remote_name, remote_url):
    
    224
    +    subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
    
    225
    +                          cwd=git_repo)
    
    226
    +
    
    227
    +
    
    228
    +def check_git_repo(git_repo, remotes):
    
    229
    +    if not os.path.isdir(git_repo):
    
    230
    +        msg = "directory %r not present" % git_repo
    
    231
    +        raise argparse.ArgumentError(None, msg)
    
    232
    +    # .git could be a regular file (worktrees) or a directory
    
    233
    +    if not os.path.exists(os.path.join(git_repo, '.git')):
    
    234
    +        msg = "directory %r is not a git repository" % git_repo
    
    235
    +        raise argparse.ArgumentError(None, msg)
    
    236
    +
    
    237
    +    current_remotes = subprocess.check_output(
    
    238
    +        ['git', 'remote', 'show'], cwd=git_repo).decode(
    
    239
    +            sys.stdout.encoding).strip().split('\n')
    
    240
    +    for key in remotes.keys():
    
    241
    +        remote = remotes[key]  # __getitem__ will add git_name
    
    242
    +        if remote['git_name'] not in current_remotes:
    
    243
    +            msg = "remote %r not in git repository" % remote['git_name']
    
    244
    +            raise argparse.ArgumentError(None, msg)

  • scripts/prepare_remotes.py
    1
    +#!/usr/bin/python3
    
    2
    +
    
    3
    +# Copyright 2019 Toshiba corp.
    
    4
    +# Based on import_stable.py by Codethink Ltd.
    
    5
    +#
    
    6
    +# This script is distributed under the terms and conditions of the GNU General
    
    7
    +# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
    
    8
    +# for details.
    
    9
    +
    
    10
    +# Helper script that prepares the local git repository with the configured
    
    11
    +# remote branches
    
    12
    +
    
    13
    +import argparse
    
    14
    +import os
    
    15
    +import subprocess
    
    16
    +
    
    17
    +import kernel_sec.branch
    
    18
    +
    
    19
    +
    
    20
    +def main(git_repo, remotes):
    
    21
    +    if os.path.isdir(git_repo):
    
    22
    +        msg = "directory %r already exists" % git_repo
    
    23
    +        raise argparse.ArgumentError(None, msg)
    
    24
    +    else:
    
    25
    +        os.mkdir(git_repo)
    
    26
    +        subprocess.check_call(['git', 'init', '.'], cwd=git_repo)
    
    27
    +
    
    28
    +    for key in remotes.keys():
    
    29
    +        remote = remotes[key]  # __getitem__ will add git_name
    
    30
    +        kernel_sec.branch.remote_add(
    
    31
    +            git_repo, remote['git_name'], remote['git_repo_url'])
    
    32
    +        kernel_sec.branch.remote_update(git_repo, remote['git_name'])
    
    33
    +
    
    34
    +    # self-check
    
    35
    +    kernel_sec.branch.check_git_repo(git_repo, remotes)
    
    36
    +
    
    37
    +
    
    38
    +if __name__ == '__main__':
    
    39
    +    parser = argparse.ArgumentParser(
    
    40
    +        description=('Prepare local git repository with configured remotes.'))
    
    41
    +    parser.add_argument('--git-repo',
    
    42
    +                        dest='git_repo', default='../kernel',
    
    43
    +                        help=('local git repository location '
    
    44
    +                              '(default: ../kernel)'),
    
    45
    +                        metavar='DIRECTORY')
    
    46
    +    parser.add_argument('--remote-name',
    
    47
    +                        dest='remote_name', action='append', default=[],
    
    48
    +                        help='git remote name mappings, e.g. stable:mystable',
    
    49
    +                        metavar='NAME:OTHER-NAME')
    
    50
    +    parser.add_argument('--mainline-remote',
    
    51
    +                        dest='mainline_remote_name',
    
    52
    +                        help="git remote name to use instead of 'torvalds'",
    
    53
    +                        metavar='OTHER-NAME')
    
    54
    +    parser.add_argument('--stable-remote',
    
    55
    +                        dest='stable_remote_name',
    
    56
    +                        help="git remote name to use instead of 'stable'",
    
    57
    +                        metavar='OTHER-NAME')
    
    58
    +    args = parser.parse_args()
    
    59
    +    remotes = kernel_sec.branch.get_remotes(args.remote_name,
    
    60
    +                                            mainline=args.mainline_remote_name,
    
    61
    +                                            stable=args.stable_remote_name)
    
    62
    +    main(args.git_repo, remotes)

  • scripts/report_affected.py
    ... ... @@ -18,14 +18,25 @@ import kernel_sec.version
    18 18
     
    
    19 19
     def main(git_repo, remotes,
    
    20 20
              only_fixed_upstream, include_ignored, *branch_names):
    
    21
    +    live_branches = kernel_sec.branch.get_live_branches()
    
    21 22
         if branch_names:
    
    22
    -        # Support stable release strings as shorthand for stable branches
    
    23
    -        branches = [kernel_sec.branch.get_base_ver_stable_branch(name)
    
    24
    -                    if name[0].isdigit()
    
    25
    -                    else kernel_sec.branch.get_stable_branch(name)
    
    26
    -                    for name in branch_names]
    
    23
    +        branches = []
    
    24
    +        for branch_name in branch_names:
    
    25
    +            if branch_name[0].isdigit():
    
    26
    +                # 4.4 is mapped to linux-4.4.y
    
    27
    +                name = 'linux-%s.y' % branch_name
    
    28
    +            else:
    
    29
    +                name = branch_name
    
    30
    +
    
    31
    +            for branch in live_branches:
    
    32
    +                if branch['short_name'] == name:
    
    33
    +                    branches.append(branch)
    
    34
    +                    break
    
    35
    +            else:
    
    36
    +                msg = "Branch %s could not be found" % branch_name
    
    37
    +                raise argparse.ArgumentError(None, msg)
    
    27 38
         else:
    
    28
    -        branches = kernel_sec.branch.get_live_branches()
    
    39
    +        branches = live_branches
    
    29 40
             if only_fixed_upstream:
    
    30 41
                 branches = [branch for branch in branches
    
    31 42
                             if branch['short_name'] != 'mainline']
    
    ... ... @@ -75,7 +86,7 @@ if __name__ == '__main__':
    75 86
                             metavar='DIRECTORY')
    
    76 87
         parser.add_argument('--remote-name',
    
    77 88
                             dest='remote_name', action='append', default=[],
    
    78
    -                        help='git remote name mappings, e.g. stable:korg-stable',
    
    89
    +                        help='git remote name mappings, e.g. stable:mystable',
    
    79 90
                             metavar='NAME:OTHER-NAME')
    
    80 91
         parser.add_argument('--mainline-remote',
    
    81 92
                             dest='mainline_remote_name',
    
    ... ... @@ -100,5 +111,6 @@ if __name__ == '__main__':
    100 111
         remotes = kernel_sec.branch.get_remotes(args.remote_name,
    
    101 112
                                                 mainline=args.mainline_remote_name,
    
    102 113
                                                 stable=args.stable_remote_name)
    
    114
    +    kernel_sec.branch.check_git_repo(args.git_repo, remotes)
    
    103 115
         main(args.git_repo, remotes,
    
    104 116
              args.only_fixed_upstream, args.include_ignored, *args.branches)

  • scripts/webview.py
    ... ... @@ -205,7 +205,7 @@ if __name__ == '__main__':
    205 205
                             metavar='DIRECTORY')
    
    206 206
         parser.add_argument('--remote-name',
    
    207 207
                             dest='remote_name', action='append', default=[],
    
    208
    -                        help='git remote name mappings, e.g. stable:korg-stable',
    
    208
    +                        help='git remote name mappings, e.g. stable:mystable',
    
    209 209
                             metavar='NAME:OTHER-NAME')
    
    210 210
         parser.add_argument('--mainline-remote',
    
    211 211
                             dest='mainline_remote_name',
    
    ... ... @@ -219,6 +219,7 @@ if __name__ == '__main__':
    219 219
         remotes = kernel_sec.branch.get_remotes(args.remote_name,
    
    220 220
                                                 mainline=args.mainline_remote_name,
    
    221 221
                                                 stable=args.stable_remote_name)
    
    222
    +    kernel_sec.branch.check_git_repo(args.git_repo, remotes)
    
    222 223
     
    
    223 224
         conf = {
    
    224 225
             '/static/style.css': {
    


  • Re: [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags

    Ben Hutchings <ben.hutchings@...>
     

    On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
    [...]
    --- a/scripts/report_affected.py
    +++ b/scripts/report_affected.py
    [...]
    @@ -22,15 +24,47 @@ def main(git_repo, remotes,
    if branch_names:
    branches = []
    for branch_name in branch_names:
    + tag = None
    if branch_name[0].isdigit():
    # 4.4 is mapped to linux-4.4.y
    name = 'linux-%s.y' % branch_name
    + elif branch_name[0] == 'v':
    + # an official tag, e.g. v4.4.92-cip11
    + # infer branch from tag (regexp's must be specific)
    + for branch in live_branches:
    + if 'tag_regexp' in branch:
    + # predefined in conf/branches.yml
    + tag_regexp = branch['tag_regexp']
    + elif branch['git_remote'] == 'stable':
    + # stable format, e.g. v4.19.12
    + esc_base_ver = branch['base_ver'].replace('.', '\.')
    This happens to work now, but '\.' is an unrecognised escape sequence
    which is deprecated. You presumably meant r'\.', but it might be
    clearer to use re.escape().

    + tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
    + esc_base_ver, esc_base_ver)
    I also expected that you would set tag_regexp for stable branches in
    the branch module along with all their other fields. Then there's no
    need to handle them specially here.

    + else:
    + # no tag_regexp defined, or mainline
    + continue
    +
    + if re.match(tag_regexp, branch_name):
    + tag = branch_name
    + name = branch['short_name']
    + break
    + else:
    + raise ValueError('Failed to match tag %r' % branch_name)
    + elif ':' in branch_name:
    + # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
    + name_tuple = tuple(branch_name.split(':'))
    + name = name_tuple[0]
    + tag = name_tuple[1]
    [...]

    You really can do simply:

    name, tag = branch_name.split(':', 1)

    (Tuple assignment only requires an iterable, not specifically a tuple,
    on the right hand side.) So please use that.

    Ben.

    --
    Ben Hutchings, Software Developer Codethink Ltd
    https://www.codethink.co.uk/ Dale House, 35 Dale Street
    Manchester, M1 2HF, United Kingdom


    Re: About CIP kernel maintenance policy for new hardware support backporting

    Yoshitake Kobayashi
     

    Hi,

    Thanks for pointed out.
    You can find the following sentence on the page. 
    > an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches.

    I think we need to write much clear to mention what kind of patches we can accept to CIP SLTS tree and how to approve(?) it by CIP.

    Best regards,
    Yoshi




    2019年7月10日(水) 15:42 <hiraku.toyooka@...>:

    Hello,

    I have a question about CIP kernel maintenance policy.
    Can CIP kernel accept backport patches for new hardware support (not
    only fixes)?

    I read the following document and it seems to describe only acceptable fixes.
    https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipkernelmaintenance

    On the other hand, Current linux-4.19.y-cip branch accepts some hardware
    support patches such as RZ/G2E(r8a774c0) from upstream.
    So I wonder if some policy about new hardware support exists.

    --
    Hiraku Toyooka
    Cybertrust Japan Co., Ltd.
    _______________________________________________
    cip-dev mailing list
    cip-dev@...
    https://lists.cip-project.org/mailman/listinfo/cip-dev

    7561 - 7580 of 10158