Previously darkserver project had a koji plugin, which used to collect the information from the koji-hubs itself. It used to process each rpm, find ELF files and extract GNU BUILD-ID from them.
But the major disadvantage on this method we found was package build time increase. For bigger packages like kernel it used to take more than one hour extra. Koji does not allow to do direct async-plugins, so I had to look for a complete separate import process.
The following image explains the new process
We have a darkproducer process running, which is checking koji for any new build (using API(s) provided by koji), if it finds a build is on, then it creates an entry in the Buildqueue, if the build is completed then it creates an entery in the Jobqueue.
darkbuildworker process is monitoring Buildqueue, and if it finds any completed job then it moves it to the Jobqueue or else puts it back in the Buildqueue.
Then finally we can have many darkjobworker processes, they can monitor the Jobqueue and imports any jobs landing in the queue. All of the processes work as daemons.
Broke the previous monolithic design into simple producer-consumer one. Using Redis to maintain the queues. I am also running a dev instance to to test the design
The codebase which contains queue creation and task management, is being moved into a separate python module to create/use redis based queues. I hope it will be useful for someone else trying to solve similar kind of problem.
