Shachar Shemesh has a few comments on the hacking contest. He thinks that “the kernel challenge was not planned properly… what Muli did to the kernel pales in comparison to things you see in the real world”. I guess what Shachar doesn’t understand is that it was supposed to be hard, not impossible. We could’ve been a lot more evil, and made their lives much harder, but what would’ve been the point in that? it was supposed to be fun, and solvable within an hour.
We intended for the teams to understand that the kernel has been tampered with, find in what way it was tampered with, find the backdoor in the tampering, and finish this stage. This is exactly what the winning team did. We anticipated the teams trying to boot with a clean kernel (this is exactly what we would’ve done in their stead) and took steps to prevent that from working (only our kernel would agree to mount the minix file system that the file resided on – a one bit change in the magic field in the superblock is all it took).
Shachar’s team “solved” this stage in two ways – the first, by removing the loopback mount and creating a different file at /usr/local/august/stage3.tmp. This is something we considered a low-quality solution, since it did not solve the original problem, only worked around it. Their second solution was “we did something and the file changed, and we don’t know what it was”. We accepted it, since the file did appear to be changed, but it would’ve been nice if they would’ve known how they did it. It’s possible that they exploited a bug in my patch, or tricked us somehow, but since the winning team rm -rf’d their machine at the end, we will never know.
It was fun and we certainly learned a few things for the next time. And Shachar, I could’ve written a complete root kit, but considering how long it took you guys to handle this relatively simple challenge – what would’ve been the point?
Muli Ben-Yehuda's journal 
Congratulations for cool ideas for the hacking contest.
Judging from the times needed by the hackers to overcome the challenges, and from the close ties in most of the stages, I’d say that the contest was very well planned and executed.
The next challenge to designers of hacking contests:
Not only make ‘root’ not root (user ID different from 0) but also make it difficult for the hacker to find out about this by feeding him a fake /etc/passwd file and somehow hiding the true /etc/passwd.
Comment by tddpirate — August 8, 2004 @ 6:46 AM |
Oh, there are plenty of things we can do – I’m thinking chroot jails, obsolete linux versions (think 0.01…) running inside emulators you need to break out of, etc, etc. As long as it’s solvable, I’m game š
Comment by mulix — August 8, 2004 @ 6:50 AM |
Namecalling does not become you
It really doesn’t.
Shachar
Comment by Anonymous — August 8, 2004 @ 6:51 AM |
Re: Namecalling does not become you
It was supposed to be tongue-in-cheek. I’m sorry if you’re offended – I’ll edit it. I have to admit I was a bit offended when I read your blog entry, and then I decided to take it humourously.
Comment by mulix — August 8, 2004 @ 6:53 AM |
Re: Namecalling does not become you
Here’s a “rule of thumb” for you to know whether your joke is going to be appreciated or offend. If you are offended by something, it is highly likely that any joke you are going to make about these matters is going to turn out offending too. Please believe me, this rule is the only reason my original summary did not contain any equivalent name-calling in your direction, despite being written between 2 and 4 am, when I couldn’t sleep.
Regarding the content of your claims: Any organizing party of a hacking contest should be aware (or at least hope) that the participants are as smart and he is. As such, the common rule for hacking contests (and I consider Obfuscated C as the prime example) are instructed to encourage solutions that are “out of the box”. In fact, the very fact a solution causes the organizers to rewrite next year’s rules is usually a sure-sign that it is a winning one. See the “hacker test” for more info.
As for the inode claim – it was Zafrir (a member of the winning party, mind you. I guess you can call him a “sore winner”) who said that their editing, after changing the date, changed the inode number too.
He also said that you told him that they won’t be getting full points, because they only solved it the way you planned them to solve it, and did not come up with a different one. I think you should make up your mind.
Shachar
Comment by Anonymous — August 8, 2004 @ 11:39 PM
Re: Namecalling does not become you
Well, EMACS does change the inode, a carnal sin I could have considered a full justification not to use EMACS, if I did not expect it to have a very handy elisp command to disable this misfeature[1]. vi, on the other hand, correctly refrains from changing the inode number. And Tzafrir is an EMACS user, if I am not mistaken.
Regarding the contest, I think the challenge could have been phrased clearer. Does “change the file” mean “change the inode”, or “get the system to a state such that a file with the same pathname contains modified contents”? I think it’s closer to the first than the second (it’s UNIX, afterall, not NT), but then again, this interpretation also means that if you unlink the file by mistake or by purpose (which I cannot recall whether our/red team did or did not) you lose the contest.
Maybe the contest managers’ interpretation should be coded in code, and the teams know they’ve succeeded when the scoreboard process sends a greeting to the console.
[1] EMACS users, is my expectation correct?
Comment by adistav — August 9, 2004 @ 2:09 AM
Re: Namecalling does not become you
Actually, you weren’t supposed to be able to unlink the file either, so… š
And yes, it looks like emacs does unlink / rename. This is actually useful when you’re editing something that’s hard linked elsewhere (e.g. kernel sources) and you want emacs to change the link (copy on write…).
Comment by mulix — August 9, 2004 @ 3:54 AM
Re: Namecalling does not become you
> In fact, the very fact a solution causes the organizers to rewrite next
> year’s rules is usually a sure-sign that it is a winning one. See the
> “hacker test” for more info.
You have a good point here and I’m willing to concede it might’ve been an error in my judgement to deprecate your first solution.
> As for the inode claim – it was Zafrir (a member of the winning party,
> mind you. I guess you can call him a “sore winner”) who said that their
> editing, after changing the date, changed the inode number too.
That’s an unintended side effect of editing it with an editor that does unlink and rename, rather than in-place editing. Editing it with ‘echo > stage3.tmp’ wouldn’t have changed teh inode.
> He also said that you told him that they won’t be getting full points,
> because they only solved it the way you planned them to solve it, and
> did not come up with a different one. I think you should make up your
> mind.
I answered this in my respone to him in your blog. I would’ve given them 9/10. 10/10 I would’ve given to a team that managed to change the file (*inode*) without using the backdoor, by somehow disabling or exploiting a bug in the patch in a way that we did not anticipate. Your first solution did not change the inode; the second, you don’t know what happened, so it kind of defeats the purpose. If you can reproduce it and tell me how you did it, I’ll give you a 10/10 and buy you dinner. Deal?
Comment by mulix — August 9, 2004 @ 3:49 AM
Re: Namecalling does not become you
I’m game, if it means “any way that…”, instead of “the way that eventually worked during APIII”.
It’s something we actually would have done, except the file was zeroed out by a bug in the security module.
I’ll give you a step by step for the way I know would work, though:
1. Unmount the filesystem
2. Use “hexeditor” (was on the machine, we used it several times) to find the text in the file.
3. Change the “Locked” to “Unlocked”.
4. Remount the filesystem.
Like I said, the only reason we did not do this was that trying to edit the file with “vi” caused it to lose all of its content before we had a chance to think about making a copy of the unmounted minix filesystem. None of us knows minix well enough to hack the bare filesystem using hex editor.
As for the bug in the patch – I’m not sure what caused it. What we did that worked, however, was along the lines of creating a file with the correct name and content elsewhere (using a clean kernel), and either copying or moving it to the right place. Like I said above – I’m not sure what exactly.
Shachar
Comment by Anonymous — August 9, 2004 @ 4:12 AM
Re: Namecalling does not become you
> It’s something we actually would have done, except the file was zeroed
> out by a bug in the security module.
Actually, it’s by design… meaning that we are aware of it and decided to leave it in as an added difficulty for the contestants to deal with.
> 1. Unmount the filesystem
> 2. Use “hexeditor” (was on the machine, we used it several times) to find > the text in the file.
> 3. Change the “Locked” to “Unlocked”.
> 4. Remount the filesystem.
It would’ve worked only if the file wasn’t zero’d first (which it was). Since it was zero’d, it would’ve worked only if you would’ve changed the relevant inode metadata (e.g. file size) with the hex editor as well, after a clean reboot, without accessing the file first. Not an easy thing to do.
> As for the bug in the patch – I’m not sure what caused it. What we did
> that worked, however, was along the lines of creating a file with the
> correct name and content elsewhere (using a clean kernel), and either
> copying or moving it to the right place. Like I said above – I’m not
> sure what exactly.
Neither should have worked – ISTR we tried both of them and they don’t work. I’ll try it again right now – possibly if it’s the *first* access to the file after a reboot it could work, although I doubt it.
>
Comment by mulix — August 9, 2004 @ 4:21 AM
Re: Namecalling does not become you
Why the clean reboot? The kernel should not save any caches of an inode on an unmounted filesystem.
Also, does that mean we get dinner?
Shachar
Comment by Anonymous — August 9, 2004 @ 6:57 AM
Re: Namecalling does not become you
> Why the clean reboot? The kernel should not save any caches of an inode on an unmounted filesystem.
It has to do with how the patch works. On the first access to a relevant inode, it marks it as “tainted”. Then, subsequent accesses to tainted inodes are denied. It turns that if the first access is a copy, I don’t catch it properly. I’m working on a fix.
> Also, does that mean we get dinner?
You (singular) do.
Cheers,
Muli
Comment by mulix — August 11, 2004 @ 1:45 AM
Re: Namecalling does not become you
>It has to do with how the patch works. On the first access to a relevant inode,
>it marks it as “tainted”. Then, subsequent accesses to tainted inodes are
>denied. It turns that if the first access is a copy, I don’t catch it properly.
>I’m working on a fix.
Wow, that sounds terribly flawed to me. In fact, that sounds “kernel infrastructure” flawed to me.
First – how the !$(&!(@# does it know it’s the same inode when I unmount the filesystem? Does that mean that if I have a different file with the same inode number on a different filesystem, it’s going to be affected too?
Second, and much worse, why should the plugin writer think of all possible combinations of access? This means that, had this change been aimed to doing something useful, in less than 1 hour of tweaking with the system we found a security bug. The security infrastructure, as far as I see it, should have made a security-bug free patch easy to write.
To me, this sounds seriously flawed, and not with your patch.
>You (singular) do.
I’ll take what I can get š
Shachar
Comment by Anonymous — August 12, 2004 @ 4:18 AM
Re: Namecalling does not become you
> Wow, that sounds terribly flawed to me. In fact, that sounds “kernel
> infrastructure” flawed to me.
I’m still struggling to figure out how the VFS data structures that I get via the LSM hooks behave. That’s why I haven’t posted a fixed patch yet. I won’t blame the kernel infrastructure until I figure out that there’s no way to do what I want, which doesn’t seem likely.
> First – how the !$(&!(@# does it know it’s the same inode when I
> unmount the filesystem? Does that mean that if I have a different file
> with the same inode number on a different filesystem, it’s going to be
> affected too?
I am referring to the kernel’s ‘struct inode’ structure, not to the actual inode (as identified by its number) on the disk. What I do is marking inodes (struct inodes) that match a given name (stage3.tmp) as ‘tainted’ by setting inode->i_security to non-NULL, and then checking inode->i_security on subsequent accesses. It worked in testing, but obviously it does not work in all cases. It’s also a poor subsitute to figuring out how the VFS works, but I was working under a tight deadline š
> Second, and much worse, why should the plugin writer think of all
> possible combinations of access? This means that, had this change been
> aimed to doing something useful, in less than 1 hour of tweaking with
> the system we found a security bug. The security infrastructure, as far
> as I see it, should have made a security-bug free patch easy to write.
The security infrastructure aims for flexibility (SELinux), not convenience. Don’t blame the infrastructure for my flawed patch š
> To me, this sounds seriously flawed, and not with your patch.
As I said, the jury is still out š
> I’ll take what I can get š
You’re on š
Comment by mulix — August 12, 2004 @ 1:40 PM
dude, is it possible for me to download the tampered kernel or to reproduce the same challenge over here? fingers itching.
Comment by ideawerkz — August 8, 2004 @ 7:56 AM |
Hi,
The hacking challenge looks pretty intresting (altough pretty shallow for a full-fledge challenge, unlike the 1-hour they had there), how about you using your kernel expertise to publish a weekly challenge where you change something on a working linux system (perhaps disribute a mini image, or a kernel image), and give a challenge for 7 days, then publish the results and the answer ?
Comment by Anonymous — August 8, 2004 @ 11:27 AM |
weekly challenge – no way, it takes far too much time to prepare. Every so often – interesting idea, I’ll keep it in mind.
Comment by mulix — August 8, 2004 @ 2:03 PM |