Alan Hargreaves' Blog

The ramblings of an Australian SaND TSC* Principal Field Technologist

The in.telnetd vulnerability/exploit (3rd update)

Before I get into the meat of this posting, let me acknowledge that, yes, this was an almighty cock up and should not have happened. It did happen. Let’s move on.

Also, while I might not agree with the publication of zero day exploits. Again, It happened. There’s really not much I can do about that. There’s really no point in being upset about it.

The upside to the posted exploit was the fact that because the code was available, the poster included an analysis of what was going wrong, pointing at the code that was broken. This almost certainly saved us some time in troubleshooting the issue. For this part of the post, you have my thanks.

I would certainly be interested if the person who posted the exploit could tell us how he found the problem; for no other reason, than I’m simply interested.

Anyway, this blog is supposed to be about getting it fixed.

All the times below are Australia/NSW.

One of our National SSEs (Rodney) was on-site with a large customer yesterday. This customer had asked him about a telnet exploit and described the problem to him. Rodney gave me a call and asked me about it at about 1pm. I hadn’t and on hearing the description (initially only described to him as a root exploit) Ttwo of us (thanks for your help Chris) dove into the code to start looking at how telnet -l-froot could behave as it did. At this point I did not know about the zero day exploit posting. Once we worked out what was going on, I called Rodney back and explained the full implications of the bug to him so he could explain it to the customer.

We told them that they could block the root vulnerability by uncommenting the CONSOLE= line in /etc/default/login. Note that this has been the default since Solaris 10 update 2 almost forever. However, I still see lots of customer configurations where it is commented out. The only other way to protect against the other implications (login as any user without a password) would be to disable the telnet service until we could fix the issue. eg

# svcadm disable svc:/network/telnet:default

We then started looking through the code to determine the best way to fix this. I logged an internal escalation and was in the process of logging a high priority bug when I saw the following in the SCCS history of usr/src/cmd/cmd-inet/usr.sbin/in.telnetd.c

D 1.67  07/02/11 19:46:41 danmcd        90 89   00009/00010/04896
6523815 LARGE vulnerability in telnetd

I immediately had a look at the bug and banged of an email to Dan stating that I could probably get IDR patches built for on10 pretty quickly. After a brief discussion of the bug and the fix, he pointed me at the manager of the group responsible for the backport and I started the backport (actually a very simple fix).

I got the IDRs built and basic testing done by about 5pm and started writing the Sun Alert.

The documentation for how to write a Sun Alert and specifically the actions that need to be taken to get interim fixes available were spot on and I sent off the initial draft at about 6:45 along with sending a request for getting the IDR patches turned into ISR patches (Interim Security Relief) and getting them published.

Just before 9pm, I started getting into discussions with UK based folk in Derrick Scholl’s group about getting the Sun Alert out and what needed to happen for me to get a gate open to get the fix back into the patch gate.

Thanks to Angela, Paul, Brent and Bill for working hard to get to the point that I could log the RTI at 10:10, and start doing the minor nit type stuff that needed to be done before Bill could pass the RTI onto the on10-patch gatekeepers and I could go home (at about 11:15).

As an aside, I missed my train connection by four minutes due to a late running North Shore train and spent an hour sat on a blacked out Hornsby station, getting home at about 2:30am)

While I was traveling (and sleeping) the heads up went out to the gatekeepers and all folk who needed to know about this so that when I came online at 8 this morning, I had very little to do before doing the actual putback into the patch gate (which happened at about 8:30).

The gatekeepers immediately closed the gate and started work on a patch.

The reason that I’ve detailed what I’ve been through with this is to point out one thing.

The speed at which I was able to do this and get to the point that an ISR patch will be shortly available publicly, is nothing short of phenomenal. For Sun to respond to and address a vulnerability like this in around 24 hours would have been completely unheard of even two to three years ago.

But it’s not just the processes here. What really made for speed here was an incredibly focussed and helpful who had an interested in rapidly getting this addressed. Without the help of folks like Dan, Rodney, Chris, Angela, Paul, Brent, Bill and Seth, and not forgetting the gatekeeping team for pulling out the stops to start building a formal patch, none of this would be possible. If I’ve missed anyone, please forgive me, I didn’t get a lot of sleep last night πŸ™‚

I love working for a company that has people like this.


update 1

The ISR patches are available for free download from http://sunsolve.sun.com/tpatches. The details of the patches are:

IDR125457-01 SunOS i386_x86: in.telnetd can call login with an
option given as a username
IDR125456-01 in.telnetd can call login with an option given as a
username

update 2

Sun Alert 102802 is publicly available talking about this issue. Section 4 should shortly be modified to add the following paragraph:

Interim Security Relief (ISRs) are available from
http://sunsolve.sun.com/tpatches for the following releases:

SPARC Platform

Solaris 10 IDR125456-01

x86 Platform

Solaris 10 IDR125457-01

Note: This document refers to one or more Interim Security Relief (ISRs) which are designed to address the concerns identified herein. Sun has limited experience with these (ISRs) due to their interim nature. As such, you should only install the (ISRs) on systems meeting the configurations described above. Sun may release full patches at a later date, however, Sun is under no obligation whatsoever to create, release, or distribute any such patch.

Update 3

I’ve just been informed that the formal patches are release ready and should be released to sunsolve in the next few hours. Keep an eye out for:

120068-02 SunOS 5.10_sparc: in.telnetd Patch
120069-02 SunOS 5.10_x86: in.telnetd Patch

As these are security patches, they will be publicly available.

Technorati Tags:
,
,
,

Advertisements

Written by Alan

February 12, 2007 at 4:25 pm

Posted in OpenSolaris

16 Responses

Subscribe to comments with RSS.

  1. Hey, mate… nice job done! I told blokes at work that Sun would probably have at least a IDR patch issued in short order. Sure enough. πŸ™‚
    Was nice reading details of how it occurred. Well done, all.

    Dan

    February 12, 2007 at 10:31 pm

  2. Good job, nice to see you blokes fix bugs better than you play cricket!

    Ian

    February 12, 2007 at 11:47 pm

  3. Good job? … hmmm… a 13 years old bug fixed now… according to osvdb.org (id 1007): http://osvdb.org/displayvuln.php?osvdb_id=1007

    Csaba

    February 13, 2007 at 2:29 am

  4. One question which I have to ask is when and how this bug was added to Solaris 10, since I believe that Solaris 9 and below had no such error.
    Thanks again for the quick fix.

    Andrew Watkins

    February 13, 2007 at 4:32 am

  5. Dan: Thank you.
    Ian: Need I remind you of what happened in the ashes series πŸ˜‰
    Csaba: No, not 13. See my answer to Andrew below. I think you also missed the point I was making, in that Sun’s reaction time to reported security bugs has phenomenally improved over the last few years. I certainly don’t make any excuses for the bug itself, which I believe was one of the first things I stated. Would you perhaps have been happier if we had taken six months to react to and publish a fix? I didn’t think so. The code slipped through, the exploit was posted. Shit happens. All that anyone can do is react and move on.
    Andrew: The bug was inserted as part of kerberizing in.telnetd very early in the Solaris 10 build cycle (November 2002). This code does not exist in the Solaris 9 source.
    Alan.

    Alan Hargreaves

    February 13, 2007 at 4:42 am

  6. Alan: Congratulation for the quick fix. I didn’t wanted to offend you. Of course I am happier with a quick fix instead of a fix taking a few months πŸ™‚

    Csaba

    February 13, 2007 at 5:47 am

  7. It’s really nice to see this level of transparency into the process, folks.
    Could it not be the case that the problem was discovered via a code audit of services the auditor had reason to think would be exposed?
    (not sure why someone would expect telnet to be exposed, but hey…)

    Chris

    February 13, 2007 at 7:57 am

  8. Why it doesn’t work in Solaris 9? If I had to guess (I dont have access to Solaris 9 code, obviously), the exploit was there in in.telnetd.c, but not exploitable. The string is not parsed for a payload within in.telnetd.c so you end up running as root with user -froot which expands the option to include -froot.
    Then you move to login.c and you see why it suddenly is exploitable in Solaris 10: zones. -f was probably reenabled from the original 1995 feature add, then disabled because of the exploit and later reenabled to help do a quick implementation of zlogin.
    Again, pure speculation, but it does sound plausible πŸ™‚
    I’d tie all security advisories to source control, to be able to review. Maybe lock old versions even if they would reintroduce a major exploit… Of course easier said than done.

    Francois Dion

    February 13, 2007 at 9:10 am

  9. It seems that Solaris 10 has some big issues with “oldies but goldies” bugs.
    Ping of death 2007:
    http://sunsolve.sun.com/search/document.do?assetkey=1-26-102697-1

    denial

    February 13, 2007 at 1:50 pm

  10. I read in the patch description that a reboot is required. Although the only thing that’s being updated is /usr/sbin/in.telnetd. It seems to me like you could just shut down telnet, patch, then restart telnet. What am I missing? In today’s enterprise environment, outages cost big bucks and are very difficult to . For what seems like a trivial workaround- I would have expected to see an alternative to a reboot. This is one of the major gripes all my customers have with solaris’s patching methodology. It’s just too painful.

    Eric

    February 13, 2007 at 2:16 pm

  11. Eric: Arggggh. I pushed all the right buttons to say that this patch could be installed in multi-user without a reboot, and that is how I tested it. It looks like the reboot after comes from the prior revision of the patch. It works fine for me without having to reboot.
    Francois: As I responded to Andrew earlier, 9 is not vulnerable because the code in question in in.telnetd.c was added early in the Solaris 10 builds and never backported.

    Alan Hargreaves

    February 13, 2007 at 2:31 pm

  12. Nice to se the quick fix of this.
    What remains to see is that
    this securily fix really get inot
    the list of security patches that
    is fetched and used by

    smpatch update
    

    At present it is not (along with some others older security patches). It will be
    interesting to see how long this takes. This is also an important
    part of Solaris security that really should work.

    Klas Heggemann

    February 13, 2007 at 8:30 pm

  13. Very nice job!

    Jim Grisanzio

    February 14, 2007 at 6:19 am

  14. Alan, you say that it is great working for a company with people like those who helped you.

    Well, I’d like to say its great working for comapnmy with people like YOU! Great effort and great dedication to get the patch out asap.

    Also, I love the transparency that OpenSolaris brings. Root-cause problems get identified sooner and fixes rollout quicker.

    Come on IBM and HP – time to bring AIX and HP-UX out into the open. πŸ˜‰

    James E.

    February 14, 2007 at 2:41 pm

  15. Hi Alan,
    nice work on quick patch!
    You’ve said that “The bug was inserted as part of kerberizing in.telnetd very early in the Solaris 10 build cycle (November 2002). This code does not exist in the Solaris 9 source.” and “the code in question in in.telnetd.c was added…”.
    But this’s happened because login was updated (-f switch was added), not telnetd itself, right? Just like Chris said – the bug was there but was not exploitable. Is this correct?
    cheers!

    Tomasz Muras

    February 15, 2007 at 8:13 am

  16. I know this vulnerability could have been present in any network service on Solaris, however Why is it that Sun sees fit to include in.telnetd as part of the default install and why are they patching an inherently flawed protocol. Surely it would be better to remove in.telnetd from service all together due to the well known issues with clear text data transmissions.

    Dylan Johnson

    February 15, 2007 at 8:34 am


Comments are closed.

%d bloggers like this: