2021.05.02 00:36 "[Tiff] SIMD optimizations", by Larry Bank

2021.05.04 07:50 "Re: [Tiff] SIMD optimizations", by Roger Leigh

Does GitLab offer any facilities which might allow adding the target-specific code to the existing continuous integration testing? Does GitLab offer access to alternate CPU types as would be available to a Debian (or some other well-supported OS) maintainer?

I'm not really versed into GitLab CI (I saw mentions of gitlab and arm collaborating together, but not sure if that extends to the free CI runners of gitlab.com). I know that Travis-CI has Arm64 configurations (which I actually use for GDAL testing), but I don't think we can couple GitLab with that (but could be used at least as a workaround by pushing into a github clone). A potential fallback would be to use a x86 runner with cross compilation & emulation chain. Maybe not a 5 minute job to setup, but definitely doable I believe, and that could be a setup used for anyone wanting to test locally such code.

Locally on my machine, I've also used recently docker buildx capabilities to create containers for other architectures and run them locally (which uses qemu user mode emulation underneath), but that required to use a docker extension and some configuration tweaking, which might not be possible in a GitLab CI VM. Hum, actually looking at https://www.docker.com/blog/multi-arch-build-what-about-gitlab-ci/, it might actually be possible.

I think having comprehensive CI testing of such SIMD codepaths is essential.

Setting up qemu-system-arm to run cross-compiled standalone binaries isn’t a five minute job, but it’s absolutely possible to do so. We can create a custom docker image with the cross-toolchain and build and run the libtiff tests within the emulator. (I’ve set up such a system very recently. It was a pain to get working, but once properly set up it works very well). While I have been testing with 32-bit Cortex-type processors, qemu now also supports Aarch64 armv8 as well.

Alternatively, you could add a Raspberry Pi or other similar ARM-based Linux or BSD system, and add it to GitLab as a runner. Then you can build and test natively without any cross-compilation or emulation hassle. The downside is that someone would need to host the runner. I do have spare RPi hardware I could set up for doing that, but the docker approach is going to be easier to scale and likely more reliable overall since it’s not dependent upon my home internet (which is pretty good, but not as good as the cloud providers), or my time to maintain it.

We might possibly need to test coverage of thumb vs full ARM variants? Does the SIMD code manage both?

Looking at

https://docs.gitlab.com/ee/user/gitlab_com/index.html#shared-runners <https://docs.gitlab.com/ee/user/gitlab_com/index.html#shared-runners>
https://developer.arm.com/solutions/infrastructure/developer-resources/ci-cd/gitlab <https://developer.arm.com/solutions/infrastructure/developer-resources/ci-cd/gitlab>

it doesn’t look like free ARM runners are provided at present. You are suggested to add your own, either AWS graviton hosts or your own physical hardware. So the choice is either emulation or an RPi-equivalent.

Regards,

Roger