Date
1 - 7 of 7
5.10.110-rc2 mirroring/testing
Chris Paterson
Hi Pavel,
Originally we had a linux-stable-rc mirror that sent a webhook/trigger to linux-stable-rc-ci when there was an update.
This was the only purpose for CIP to have a copy of the linux-stable-rc repository.
The new method instead just polls the kernel.org linux-stable-rc repository directly.
If there is an update then the usual build jobs are kicked off.
Downsides of the new approach:
- Currently polling is set to once every 2 hours, so potentially it's slower then the old method (I think GitLab mirrors poll every 30 minutes).
- Because I just use git remote-ls to see if there is an update to upstream linux-stable-rc, we don't have as detailed information in the linux-stable-rc-ci commits. We only have the SHA.
Inside the jobs however we can see the actual log oneliner to help identify the RC version.
Old: Linux 4.4.302 (a09b2d8f61ea)
New: Upstream SHA: a09b2d8f61ea
Upsides of the new approach:
+ We no longer need linux-stable-rc - we could delete this repo and stop all the warnings/errors from it being too large.
+ The new trigger method has much less data transfer. We don't need to clone all of the branches from linux-stable-rc just to see if there is an update.
The trigger has already built all 4 LTS trees we monitor and build/test jobs have gone through.
You can see the latest here: https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/pipelines
In the background I'm trying to follow the steps in [0] to reduce our original linux-stable-rc repo size, but perhaps we're better off killing it.
I've no idea if this will have an impact on anyone else though?
[0] https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html
There are probably many other ways of doing things, but today this is what I've come up with :)
Kind regards, Chris
From: Pavel Machek <pavel@...>Today I've implemented a new triggering method for our stable-rc testing.
Sent: 08 April 2022 16:59
Hi!I see there are some changes:5.10.110-rc2 is out, and it would be good to test it, but somehow weAlready linux-stable-rc uses 11G by now.
are still at -rc1...?
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-
5.10.y
but it seems to say "authored 12 minutes ago". Hour ago it also said
"authored 12 minutes ago", so I believe it will not be able to finish
the tests.
Originally we had a linux-stable-rc mirror that sent a webhook/trigger to linux-stable-rc-ci when there was an update.
This was the only purpose for CIP to have a copy of the linux-stable-rc repository.
The new method instead just polls the kernel.org linux-stable-rc repository directly.
If there is an update then the usual build jobs are kicked off.
Downsides of the new approach:
- Currently polling is set to once every 2 hours, so potentially it's slower then the old method (I think GitLab mirrors poll every 30 minutes).
- Because I just use git remote-ls to see if there is an update to upstream linux-stable-rc, we don't have as detailed information in the linux-stable-rc-ci commits. We only have the SHA.
Inside the jobs however we can see the actual log oneliner to help identify the RC version.
Old: Linux 4.4.302 (a09b2d8f61ea)
New: Upstream SHA: a09b2d8f61ea
Upsides of the new approach:
+ We no longer need linux-stable-rc - we could delete this repo and stop all the warnings/errors from it being too large.
+ The new trigger method has much less data transfer. We don't need to clone all of the branches from linux-stable-rc just to see if there is an update.
The trigger has already built all 4 LTS trees we monitor and build/test jobs have gone through.
You can see the latest here: https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/pipelines
In the background I'm trying to follow the steps in [0] to reduce our original linux-stable-rc repo size, but perhaps we're better off killing it.
I've no idea if this will have an impact on anyone else though?
[0] https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html
There are probably many other ways of doing things, but today this is what I've come up with :)
Kind regards, Chris
Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Pavel Machek
Hi!
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-5.10.y
but it seems to say "authored 12 minutes ago". Hour ago it also said
"authored 12 minutes ago", so I believe it will not be able to finish
the tests.
Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
I see there are some changes:5.10.110-rc2 is out, and it would be good to test it, but somehow weAlready linux-stable-rc uses 11G by now.
are still at -rc1...?
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-5.10.y
but it seems to say "authored 12 minutes ago". Hour ago it also said
"authored 12 minutes ago", so I believe it will not be able to finish
the tests.
Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Jan Kiszka
On 07.04.22 12:36, Chris Paterson wrote:
this, we just didn't apply for it yet:
https://about.gitlab.com/solutions/open-source/join/
I'm not yet spotting if that would raise storage quotas, though.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Hello,Or we could ask - as an OSS project - for extra benefits. Gitlab offersFrom: Chris PatersonI've gone through and deleted all the tags/branches we don't care about but GitLab hasn’t updated their storage calculations yet.
Sent: 07 April 2022 09:51
Hello Pavel,From: Jan Kiszka <jan.kiszka@...>https://gitlab
Sent: 07 April 2022 07:44
On 07.04.22 00:06, Pavel Machek wrote:Hi!
5.10.110-rc2 is out, and it would be good to test it, but somehow we
are still at -rc1...?.com%2Fcip-project%2Fcip-testing%2Flinux-stable-rc-ci%2F-5.10.y&data=04%7C01%7CChris.Paterson2%40renesas.com%7Ccda7f53
%2Ftree%2Flinux-
12d69483e751208da1861f34f%7C53d82571da1947e49cb4625a166a4a2a%7C0
%7C0%7C637849106225416510%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
amp;sdata=p0tXQDyIDY1SqRyIOHrtFzygGDqTbRphnrVzn34XVy0%3D&reserved=0As Jan hinted, the mirroring has stopped because the linux-stable-rc
Does mirroring need to be kicked again?
repository has now gone over the 10GB size limit.
I guess we could remove all the branches/tags we don't care about, then
protect the branches we do care about (LTS) and configure GitLab to only
mirror protected branches?
Hopefully that'll reduce the size a lot and mirroring will work again.
For now we'll have to wait until they recalculate. Hopefully the repo size will drop enough.
The alternative is that CIP buys extra storage, but I'm not sure if that's done at the top-level for all of cip-project and the individual project limit of 10GB still applies.
this, we just didn't apply for it yet:
https://about.gitlab.com/solutions/open-source/join/
I'm not yet spotting if that would raise storage quotas, though.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Chris Paterson
Hello,
For now we'll have to wait until they recalculate. Hopefully the repo size will drop enough.
The alternative is that CIP buys extra storage, but I'm not sure if that's done at the top-level for all of cip-project and the individual project limit of 10GB still applies.
Kind regards, Chris
From: Chris PatersonI've gone through and deleted all the tags/branches we don't care about but GitLab hasn’t updated their storage calculations yet.
Sent: 07 April 2022 09:51
Hello Pavel,From: Jan Kiszka <jan.kiszka@...>https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
Sent: 07 April 2022 07:44
On 07.04.22 00:06, Pavel Machek wrote:Hi!
5.10.110-rc2 is out, and it would be good to test it, but somehow we
are still at -rc1...?.com%2Fcip-project%2Fcip-testing%2Flinux-stable-rc-ci%2F-5.10.y&data=04%7C01%7CChris.Paterson2%40renesas.com%7Ccda7f53
%2Ftree%2Flinux-
12d69483e751208da1861f34f%7C53d82571da1947e49cb4625a166a4a2a%7C0
%7C0%7C637849106225416510%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
amp;sdata=p0tXQDyIDY1SqRyIOHrtFzygGDqTbRphnrVzn34XVy0%3D&reserved=0As Jan hinted, the mirroring has stopped because the linux-stable-rc
Does mirroring need to be kicked again?
repository has now gone over the 10GB size limit.
I guess we could remove all the branches/tags we don't care about, then
protect the branches we do care about (LTS) and configure GitLab to only
mirror protected branches?
Hopefully that'll reduce the size a lot and mirroring will work again.
For now we'll have to wait until they recalculate. Hopefully the repo size will drop enough.
The alternative is that CIP buys extra storage, but I'm not sure if that's done at the top-level for all of cip-project and the individual project limit of 10GB still applies.
Kind regards, Chris
ChrisMaybe the problem is rather this:https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fgroups%2Fcip-project%2F-%2Fusage_quotas%23storage-quota-tab&data=04%7C01%7CChris.Paterson2%40renesas.com%7Ccda7f5312d
69483e751208da1861f34f%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C
0%7C637849106225416510%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
sdata=pzhx2y7WhVSd97cJak6p%2BTfM41%2BoYbRLe1gQwoX%2BxuI%3D&reserved=0
Already linux-stable-rc uses 11G by now.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Chris Paterson
Hello Pavel,
I guess we could remove all the branches/tags we don't care about, then protect the branches we do care about (LTS) and configure GitLab to only mirror protected branches?
Hopefully that'll reduce the size a lot and mirroring will work again.
Chris
From: Jan Kiszka <jan.kiszka@...>As Jan hinted, the mirroring has stopped because the linux-stable-rc repository has now gone over the 10GB size limit.
Sent: 07 April 2022 07:44
On 07.04.22 00:06, Pavel Machek wrote:Hi!https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
5.10.110-rc2 is out, and it would be good to test it, but somehow we
are still at -rc1...?
.com%2Fcip-project%2Fcip-testing%2Flinux-stable-rc-ci%2F-
%2Ftree%2Flinux-
5.10.y&data=04%7C01%7CChris.Paterson2%40renesas.com%7Ccda7f53
12d69483e751208da1861f34f%7C53d82571da1947e49cb4625a166a4a2a%7C0
%7C0%7C637849106225416510%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
amp;sdata=p0tXQDyIDY1SqRyIOHrtFzygGDqTbRphnrVzn34XVy0%3D&r
eserved=0
Does mirroring need to be kicked again?
I guess we could remove all the branches/tags we don't care about, then protect the branches we do care about (LTS) and configure GitLab to only mirror protected branches?
Hopefully that'll reduce the size a lot and mirroring will work again.
Chris
Maybe the problem is rather this:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fgroups%2Fcip-project%2F-%2Fusage_quotas%23storage-quota-
tab&data=04%7C01%7CChris.Paterson2%40renesas.com%7Ccda7f5312d
69483e751208da1861f34f%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C
0%7C637849106225416510%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
sdata=pzhx2y7WhVSd97cJak6p%2BTfM41%2BoYbRLe1gQwoX%2BxuI%3D&a
mp;reserved=0
Already linux-stable-rc uses 11G by now.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Jan Kiszka
On 07.04.22 00:06, Pavel Machek wrote:
https://gitlab.com/groups/cip-project/-/usage_quotas#storage-quota-tab
Already linux-stable-rc uses 11G by now.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Hi!Maybe the problem is rather this:
5.10.110-rc2 is out, and it would be good to test it, but somehow we
are still at -rc1...?
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-5.10.y
Does mirroring need to be kicked again?
https://gitlab.com/groups/cip-project/-/usage_quotas#storage-quota-tab
Already linux-stable-rc uses 11G by now.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Pavel Machek
Hi!
5.10.110-rc2 is out, and it would be good to test it, but somehow we
are still at -rc1...?
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-5.10.y
Does mirroring need to be kicked again?
Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
5.10.110-rc2 is out, and it would be good to test it, but somehow we
are still at -rc1...?
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-5.10.y
Does mirroring need to be kicked again?
Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany