Muli Ben-Yehuda's journal

10/12/06 – the big news is that CalIOC2 DMAs are now going through!
the HW guys hooked up a logic analyzer and figured out that Linux is
misprogramming the bridge to have a base/limit of 0/0, which caused a
nasty interaction with the 0-1MB MMIO region on the bridge, since TCE
addresses are allocated from 0 and up. The
workaround is simple, start allocating TCEs from address 1MB and up,
and voila, DMAs start coming through.

I’m now trying to
understand why Linux is misprogramming the bridge. It only happens
when TCEs are enabled, and a diff of dmesg from the same kernel with
TCEs enabled and disabled has this to say:

+PCI: Ignore bogus resource 7 [0:0] of 0000:0a:00.0 
+PCI: Ignore bogus resource 8 [0:0] of 0000:0a:00.0
 PCI: Bridge: 0000:0a:00.0
-  IO window: disabled.
-  MEM window: disabled.
+  IO window: 0000-0000
+  MEM window: 00000000-00000000
   PREFETCH window: d4100000-d41fffff

After I figure this one out, I need to implement a new cache flush
routine for CalIOC2 and another workaround the designer recommends,
stress test it, and then if all goes well it can go in, probably for
2.6.21.

In other news, my hacker’s diet and workout regime is working very
well. I’m losing weight at a steady rate of 0.6kgs per week and
feeling better every day. Starting the day with a walk on the
sea-shore before the sun rises, masochistic as it may be, is also
highly enjoyable. The more I suffer, the more I enjoy it. I’m going to
add swimming too next week, albeit in a heated pool, not in the cold
Mediterranean.

In other other news, we decided to go ahead with the USENIX
paper. Deadlines, here we come!

05/12/06 – took half a day off and spent it pleasantly hacking the now-defunct Hacker’s Diet Tracker to run in my environment.

04/12/06 – finally got back to Xen-IOMMU hacking. Fixed the xenlinux tree to boot as domU, not just dom0, and cleaned up the IOMMU initialization code to allow multiple IOMMUs to be compiled in and used at run time (e.g., Calgary used by default but use swiotlb if iommu=soft is specified). Next step is assigning a PCI device to the xenlinux kernel that has translation enabled in the IOMMU and settling back to watch the fireworks. In the evening did a quick backport of the 2.6.20 Calgary tree to the SLES 10 SP1 kernel (based on 2.6.16.29). When I last dealt with SLES kernels a few years back they were crammed full of weird patches that made them look nothing like mainline. I was glad to see the situation has improved.

Summary of last week: continued working on CalIOC2 support. It’s pretty obvious that we’re seeing a HW problem and the ball is in the HW folks’s court. Switching back to xen-iommu for the time being.

27/11/06 – woke up at 6 AM again, despite a most enjoyable evening spent at the pub the day before with yrk and 10 Year Old Springbank. Surprised myself and actually went our for a brisk walk and exercise. Spent the rest of the day in various meetings and conference calls, occasionally stealing a few minutes to work on CalIOC2.

If I was going to lca, which I’m not, upon hearing who the first keynote speaker is, I’d request – nay, demand – a dramatic reenactment in person!

26/11/06 – Started the morning with a vigorous 4 KM walk in the neighborhood. I now ache all over, but it was worth it.

Spent a happy few hours with the hardware specs convincing myself the CalIOC2 code is correct. The more I stare at it, the more I’m convinced it’s a hardware bug. The only reason I haven’t thrown it on the hardware guys lap yet is that I was convinced the HW was at fault at least three times during the initial Calgary bringup, and it was a software bug each time, which I eventually figured out by staring at it long enough. So… more staring required.

19/11/06 – dealt with some of the less enjoyable by-products of work such as patent searches for two new disclosures. Also spent some time thinking about the IOMMU performance paper we want to put together for USENIX.

20/11/06 – watched Yael in the morning, then headed into work to listen to a very interesting talk by Yaron Weinsberg on his NIC operating system work. Yaron will also give the talk in Haifa at the lab in a few weeks, ping me if you’re local and would like to attend. Spent the rest of the day in various meetings. Came up with a Pretty Cool Idea in one of them (heresy, I know – a useful meeting?!), now to see if the rest of the team buys into it. Had some hopes that the CalIOC2 machine will come back up after a lab reorg in the evening, but alas, it didn’t happen. Hopefully I’ll be able to get back to the CalIOC2 bringup and the rest of the Xen IOMMU work tomorrow.

13/11 – Avi Kivity visited the lab and gave an ejoyable talk on talk on KVM. From a theoretical POV, I think hypervisors should be small and lean, not based on full-fledged OS kernels. However, from a pragmatic point of view – after dealing with Xen for so long – it’s certainly nice to see the Linux kernel gaining a subset of Xen’s capabilities, while remaining the Linux kernel we know and love.

Spent the rest of the day bringing up the CalIOC2 IOMMU on the Beaverton machine. The bridge issue previously mentioned was pretty easy to solve, we just check if the bus the device is on is a bridge, and if yes, use the IO address space on the bridge’s parent. There was also a nasty issue with the format of an undocumented Calgary/CalIOC2 register initialized by the BIOS which is initialized to the wrong value on CalIOC2. Once that was taken care of, the kernel came up and Calgary worked fine, but DMA’s weren’t going through for the guinea pig PCI-e NIC attached to CalIOC2. Here is my current debug patch calgary-support-calioc2.

14/11 – more CalIOC2 debugging but no discernible progress. Eventually gave up on the flaky machine for the day and moved on to other things. In the evening spent some time talking with various people about plans for next year.

12/1/06 – went to the doctor with the little rascal in the morning and then to breakfast with Orna (we both ate! at the same time! with both hands! if you don’t understand why this is significant, wait until you have a baby…). Dealt with various odds and ends for the rest of the day, but didn’t get any code written. Some days are like this.

Weekend update – taking care of a very fussy yet adorable baby. Who knew babies were this much work? also managed to update my website for the first time in years, including the new index page (does anyone recognize the inspiration?), publications page and new and improved CV. Not that I’m looking, the old one was just shabby.