As someone mentioned our build process is quite fast. Especially when compared to building Linux, Singularity or other operating systems. However we often need to perform builds dozens of times in a single hour. This is why I have repeatedly put so much emphasis on the need to make our build process even faster. Every bit of speed increase that we can squeeze out of our build process will return huge dividends to us in other areas.
We know what we want to do. A full optimization of our build process including caching of compilation, completion of our internal x86 assembler, and dynamic loading. However this is quite a big process. Because of this the following phased approach is now being proposed to give us speed increases without the need for such massive changes to the build system. We are all anxious to work on networking, file systems, and so forth.
Phases presented in order of implementation. Each phase is fully contained and will deliver an increase in build speed.
Check for inefficient methods in our build process and see if they can be optimized by using better data structures or refactoring.
Determine the difficulty of implementing Phase 4. If possible, skip Phase 3. If Phase 4 is a bit involved, go for the quickie and implement phase 3 so we will benefit from a speed improvement while Phase 4 is under way.
Scanning methods by far is the longest process right now. Lets make an option that generates the full asm, ie does not scan and include only what we need. It then creates one big .asm PER input assembly and saves the timestamp of the input assembly. If it doesnt change, we don't regenerate the .asm for that assembly. This information can be written into the build config we already use.
Special note about plugs. Matthijs thinks some work might need to be done here, however plugs are in separate assemblies already so hopefully there will be no issues here.
Next we cache the .asm or .obj files. Currently we do not use a linker and we let nasm do the linking. nasm currently runs very quickly, so potentially the best option for this phase likely is to cache the .asm files, and just feed them all at once to nasm and let it recompile and link them rather than caching .obj files and externalizing the link process again. Keep in mind this phase is a temporary step to loading dynamic libraries where caching of .asm or .obj becomes moot.
We need to add support for dynamic loading of binaries at boot time. If it is only at boot time and not while processes are running, that is adequate for this phase and in fact preferred. For dynamic loading at runtime there are many other things we need to complete to match the Cosmos security model, not to mention things like file systems, memory management, etc.
A mod exists for syslinux to allow dynamic loading of ELF modules. It is planned to be part of syslinux 4.
Unfortunately I have not been able to locate any documentation on how to use this mod, even browsing the source tree. There are a few very basic notes in the authors journal. The rest I found by looking at the main and print_help routines in:
syslinux-elf\com32\elflink\test_com32.c
This will allow us to cache libraries created from system assemblies and just redeploy them each time without the need to scan, compile, or link them. This is the preferred option.
We already use ELF format, so its a matter of splitting our output into multiple ELF files. One bit I was able to extract is that for the mod to load, the modules have to load in order of dependency. That is if module 1 depends on module 2, module 2 must be specified and loaded before module 1.
This poses a serious issue for us. For example, plugs for file system calls. That makes mscorlib dependent on the plug, but all code is dependent on mscorlib. So there is a circular reference which is unresolvable.
So we need to verify if this limitation still exists in the syslinux mod. If so we need to modify it ourselves and contribute our changes back into syslinux.
Enable threading to take advantage of multi core in the build process. This is much less important now than it was before we did phase 1, but as the size of our projects grow it will become important again and it is not that difficult to do.