Calgary is working.
Sometime near 4AM on Sunday night I got half dynamic mappings working. Half dynamic mappings are still mapping from the physical address in the PCI address space to the same physical address in the system memory space (like static mappings), but only map those addresses the driver requests from the DMA API (unlike static mappings which map everything). The name is courtesy of Olof Johansson, who did a large chunk of the PPC IOMMU work (Hi Olof :-)).
Once I had half dynamic mode working, I started experimenting with simple transformations on the addresses. When even those worked, which showed that none of the drivers are bypassing the DMA API, I was in a real quandary – why do they work, and fully dynamic, which is just a different transformation on the address, doesn’t?
At this point in time my machine’s remote console dropped off the network and I went to sleep.
On Monday morning, inspiration hit: fully dynamic really is just another transformation on the address – but it’s the only one that I tried that was *0 based*! I immediately made a trivial change to my brain dead TCE allocator: instead of starting to allocate from address 0, start from 0x7000000 (arbitrary, I have 2GB of memory in the victim machine). When this worked, I knew I hit the jackpot.
Investigation later showed that 1MB was the cut-off point. As long as I allocated TCEs above 1MB, everything was fine. 1MB… there’s something about this address… BIOS? and indeed further investigation showed that the region between 640KB and 1MB is special and requires special handling. Now my allocator simply starts from 1MB and all is well with the world.
Next dual steps: moving the guts of Calgary into Xen so that we can use it for isolation and preparing it for submission to the mainline kernel.
Can you tell I’m happy? 🙂
Muli Ben-Yehuda's journal 