Discussion:
How does WSUS determine a patch is missing?
(too old to reply)
D Riberdy
2007-06-26 13:39:00 UTC
Permalink
I am trying to figure out how WSUS currently determines how a patch is
missing from a server. Does it check the individual files in each patch and
then report it as missing if the files are at a previous version?

I have seen in the past that software can install previous versions of dlls,
etc that "break" patches already applied. Windows Update from the web does
not recognize them as being needed again but the vulnerability still exists.
Lawrence Garvin (MVP)
2007-06-27 01:33:14 UTC
Permalink
Post by D Riberdy
I am trying to figure out how WSUS currently determines how a patch is
missing from a server.
The Windows Update Agent tells it that it's missing.
Post by D Riberdy
Does it check the individual files in each patch and
then report it as missing if the files are at a previous version?
The Windows Update Agent does a file level version check based on the
metadata stored in the update detection logic, which is obtained from the
WSUS Server during a detection event, provided that the update detection
metadata contains file level version data.
Post by D Riberdy
I have seen in the past that software can install previous versions of dlls,
etc that "break" patches already applied.
Only stupid software from stupid software developers on systems prior to
Windows 2000, where the stupid software developer failed to check the
version of the installed DLL prior to overwriting it with an older version
contained in the stupid software developer's installation package.
Post by D Riberdy
Windows Update from the web does
not recognize them as being needed again but the vulnerability still exists.
That really depends on what the update is; how old the update is; what
detection methodology is coded into the update package's detection logic;
what version of the operating system you're using; what version of the
Windows Update Agent you're using; and whether or not the affected DLL(s)
are actually part of any given update package.

Do you have a *specific* example you'd like to discuss?
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E

Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx

And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
D Riberdy
2007-06-27 14:16:03 UTC
Permalink
The actual incident in question was installing Microsoft Application Center
SP1 that undid the fix for the Blaster virus on all the servers we installed
it on. Even though Microsoft listed it as installed and said we were
protected an audit found us vulnerable. We had to switch to a 3rd party tool
becuase the customer felt that we and Microsoft were unable to properly
manage the patching and healthiness of the environment.

So, If a, as you put it "Only stupid software from stupid software
developers" were to overlay a newer patches dll with an older version, will
WSUS then show it as missing? Or is that entirely dependent on the people
packaging the patches to insert all the proper data for WSUS to do that type
of checking ala "The Windows Update Agent does a file level version check
based on the metadata stored in the update detection logic, which is obtained
from the
WSUS Server during a detection event, provided that the update detection
metadata contains file level version data."

DR
Post by Lawrence Garvin (MVP)
Post by D Riberdy
I am trying to figure out how WSUS currently determines how a patch is
missing from a server.
The Windows Update Agent tells it that it's missing.
Post by D Riberdy
Does it check the individual files in each patch and
then report it as missing if the files are at a previous version?
The Windows Update Agent does a file level version check based on the
metadata stored in the update detection logic, which is obtained from the
WSUS Server during a detection event, provided that the update detection
metadata contains file level version data.
Post by D Riberdy
I have seen in the past that software can install previous versions of dlls,
etc that "break" patches already applied.
Only stupid software from stupid software developers on systems prior to
Windows 2000, where the stupid software developer failed to check the
version of the installed DLL prior to overwriting it with an older version
contained in the stupid software developer's installation package.
Post by D Riberdy
Windows Update from the web does
not recognize them as being needed again but the vulnerability still exists.
That really depends on what the update is; how old the update is; what
detection methodology is coded into the update package's detection logic;
what version of the operating system you're using; what version of the
Windows Update Agent you're using; and whether or not the affected DLL(s)
are actually part of any given update package.
Do you have a *specific* example you'd like to discuss?
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
.....
Lawrence Garvin (MVP)
2007-06-27 23:43:18 UTC
Permalink
Post by D Riberdy
The actual incident in question was installing Microsoft Application Center
SP1 that undid the fix for the Blaster virus on all the servers we installed
it on.
Interesting.... but a rather ancient problem, wouldn't you agree? The
Blaster virus is almost four years old; Application Center 2000 SP1
(released: Oct 2001) actually pre-dates the Blaster virus by almost two
years; and Application Center 2000 SP2 was released four years ago to
support AppCenter2000 on Windows Server 2003.

As for the Blaster virus itself, it actually exploits an issue that was
fixed by MS03-026 (KB823980), which isn't even an active update anymore,
having been superceded by a Service Pack on every operating system except
Windows 2000.

So, lacking a *current* example, let's take this scenario in the context it
deserves.
Post by D Riberdy
Even though Microsoft listed it as installed and said we were
protected an audit found us vulnerable.
So, if I understand you correctly, you directly installed Application Center
2000 SP1 on top of a machine already containing MS03-026, and then were
surprised when a 2 year old service pack replaced some files with older
versions?

Okay... so lessons learned: [a] Always install updates in chronological
order of release. [b] Always run a security scan after applying any old
updates.

Also, keep in mind that the OS reporting an update as "Installed", merely
means that the installer successfully ran, and that the necessary registry
values were created. It indicates nothing about whether the component files
are physically present, corrupted, or overwritten with an older version by
some other installer.
Post by D Riberdy
We had to switch to a 3rd party tool
becuase the customer felt that we and Microsoft were unable to properly
manage the patching and healthiness of the environment.
Hmmm... and, in reality, it was probably nothing more than a deployment
error in the installation of the AppCenter 2000 service pack. :-)

To that extent, the customer was probably partially correct in expressing
concern about your inability to properly manage the patching and healthiness
of the enviornment. :-\
Post by D Riberdy
So, If a, as you put it "Only stupid software from stupid software
developers" were to overlay a newer patches dll with an older version, will
WSUS then show it as missing?
*TODAY*, with *WSUS* (or SMS, or WU/MU, or MBSA -- all of which use exactly
the same scanning technology) (neither of which existed when you were
installing AppCenter service packs on top of Blaster security vulnerability
patches), all patches are *DETECTED* based on the requisite file contents
and version numbers of those files.
Post by D Riberdy
Or is that entirely dependent on the people
packaging the patches to insert all the proper data for WSUS to do that type
of checking
But, yes, WSUS/WUA is wholly dependent on the publisher of the UPDATE (which
is generally the product group responsible for the specific product being
updated), to properly configure the metadata for the detection logic of that
update. Ergo, the Windows Update Agent can only do what the product team
tells it to do via the update detection logic.
Post by D Riberdy
"The Windows Update Agent does a file level version check
based on the metadata stored in the update detection logic, which is obtained
from the
WSUS Server during a detection event, provided that the update detection
metadata contains file level version data."
Exactly.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E

Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx

And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
D Riberdy
2007-06-28 12:24:00 UTC
Permalink
Post by Lawrence Garvin (MVP)
But, yes, WSUS/WUA is wholly dependent on the publisher of the UPDATE (which
is generally the product group responsible for the specific product being
updated), to properly configure the metadata for the detection logic of that
update. Ergo, the Windows Update Agent can only do what the product team
tells it to do via the update detection logic.
So, what you are saying is that I am at the mercy of a patch developer doing
their job correctly to ensure that the patches will scan correctly in WSUS?
Is this a standard routine for them to do or is there currently cases where
it doesn't have the correct information? Microsoft has been known to
re-re-release patches due to improperly packaged files before.
Post by Lawrence Garvin (MVP)
So, if I understand you correctly, you directly installed Application Center
2000 SP1 on top of a machine already containing MS03-026, and then were
surprised when a 2 year old service pack replaced some files with older
versions?
Actually, you understood me incorrectly. The above patch MS03-026 was
installed when the base machine was created with Windows Update. We
installed Microsoft Application Center 2000 SP1 as a complete application -
not just the SP1 patch. When we rescanned the machines at the end of the
build process, there were no updates to be found - ergo the logic built into
either the MS03-026 patch itself OR Windows Update was flawed.

Therein lies their concern as well as ours that if we hang our hats on the
WSUS product, which by the looks of the forum posts here for version 3, has
less than spectaular commentary, that it will ensure that we are patched to
the fullest extent and rescan previously installed patches to make sure
nothing like the above happens. If it falls on the product team or WSUS
directly, it still is a "Microsoft" problem that things are not scanned
correctly.
Asher_N
2007-06-28 13:48:22 UTC
Permalink
Post by D Riberdy
Post by Lawrence Garvin (MVP)
But, yes, WSUS/WUA is wholly dependent on the publisher of the UPDATE
(which is generally the product group responsible for the specific
product being updated), to properly configure the metadata for the
detection logic of that update. Ergo, the Windows Update Agent can
only do what the product team tells it to do via the update detection
logic.
So, what you are saying is that I am at the mercy of a patch developer
doing their job correctly to ensure that the patches will scan
correctly in WSUS? Is this a standard routine for them to do or is
there currently cases where it doesn't have the correct information?
Microsoft has been known to re-re-release patches due to improperly
packaged files before.
Post by Lawrence Garvin (MVP)
So, if I understand you correctly, you directly installed Application
Center 2000 SP1 on top of a machine already containing MS03-026, and
then were surprised when a 2 year old service pack replaced some
files with older versions?
Actually, you understood me incorrectly. The above patch MS03-026 was
installed when the base machine was created with Windows Update. We
installed Microsoft Application Center 2000 SP1 as a complete
application - not just the SP1 patch. When we rescanned the machines
at the end of the build process, there were no updates to be found -
ergo the logic built into either the MS03-026 patch itself OR Windows
Update was flawed.
Well, Windows Update does what the patch tells it to do to detect, so
it's not flawed.

The update will look for either file versions, or more likely registry
entries. So installing MS03-026 created the requisite entries and folders
in the Windows folder. Then you load a 7 year old piece of software that
blindly clobbers a bunch of DLLs. The MS03-026 registry entries are still
there. What do you expect the detection to do? Scan version of every file
it replaces? For large patches and SPs, it would consume too much
resources on the clients.

If you want to escape responsibility and lay the blame on Microsoft, then
blame App Centre.

Personnaly at this point, I'd upgrade App Centre to the cuttent version.
Or at least, install it immediately after the OS, before patches.
Post by D Riberdy
Therein lies their concern as well as ours that if we hang our hats on
the WSUS product, which by the looks of the forum posts here for
version 3, has less than spectaular commentary, that it will ensure
that we are patched to the fullest extent and rescan previously
installed patches to make sure nothing like the above happens. If it
falls on the product team or WSUS directly, it still is a "Microsoft"
problem that things are not scanned correctly.
D Riberdy
2007-06-28 18:28:04 UTC
Permalink
*sigh* as Garvin noted this was an old problem. The point being we have used
other products and are now wanting to investigate WSUS as a possible solution
for patching.

Yes, I would expect that if I were to call for a scan of missing patches on
a server, it would check the file versions to ensure that they match with
what is is expected to be there. Other tools do the exact same thing and
don't seem to be consuming too many resources.

So far, I haven't really heard any good reason to vary from what we are
using to go to WSUS since we cannot expect that it would confidently tell us
what we want to know. I guess if you vary from the norm and don't kiss the
ring of Microsoft you get belittled on these forums.
Post by Asher_N
Post by D Riberdy
Post by Lawrence Garvin (MVP)
But, yes, WSUS/WUA is wholly dependent on the publisher of the UPDATE
(which is generally the product group responsible for the specific
product being updated), to properly configure the metadata for the
detection logic of that update. Ergo, the Windows Update Agent can
only do what the product team tells it to do via the update detection
logic.
So, what you are saying is that I am at the mercy of a patch developer
doing their job correctly to ensure that the patches will scan
correctly in WSUS? Is this a standard routine for them to do or is
there currently cases where it doesn't have the correct information?
Microsoft has been known to re-re-release patches due to improperly
packaged files before.
Post by Lawrence Garvin (MVP)
So, if I understand you correctly, you directly installed Application
Center 2000 SP1 on top of a machine already containing MS03-026, and
then were surprised when a 2 year old service pack replaced some
files with older versions?
Actually, you understood me incorrectly. The above patch MS03-026 was
installed when the base machine was created with Windows Update. We
installed Microsoft Application Center 2000 SP1 as a complete
application - not just the SP1 patch. When we rescanned the machines
at the end of the build process, there were no updates to be found -
ergo the logic built into either the MS03-026 patch itself OR Windows
Update was flawed.
Well, Windows Update does what the patch tells it to do to detect, so
it's not flawed.
The update will look for either file versions, or more likely registry
entries. So installing MS03-026 created the requisite entries and folders
in the Windows folder. Then you load a 7 year old piece of software that
blindly clobbers a bunch of DLLs. The MS03-026 registry entries are still
there. What do you expect the detection to do? Scan version of every file
it replaces? For large patches and SPs, it would consume too much
resources on the clients.
If you want to escape responsibility and lay the blame on Microsoft, then
blame App Centre.
Personnaly at this point, I'd upgrade App Centre to the cuttent version.
Or at least, install it immediately after the OS, before patches.
Post by D Riberdy
Therein lies their concern as well as ours that if we hang our hats on
the WSUS product, which by the looks of the forum posts here for
version 3, has less than spectaular commentary, that it will ensure
that we are patched to the fullest extent and rescan previously
installed patches to make sure nothing like the above happens. If it
falls on the product team or WSUS directly, it still is a "Microsoft"
problem that things are not scanned correctly.
Harry Johnston
2007-06-28 18:38:21 UTC
Permalink
Post by Asher_N
Well, Windows Update does what the patch tells it to do to detect, so
it's not flawed.
The detection logic is part of the Windows Update product, speaking in the
broader sense.
Post by Asher_N
What do you expect the detection to do? Scan version of every file
it replaces? For large patches and SPs, it would consume too much
resources on the clients.
MBSA 1.2.1 seemed to manage to do this without being very resource-hungry.
(Granted service packs are a special case.)

Probably qfecheck is the right tool to check for this class of problem.
However, it does seem that this functionality should be built into WUA, or
perhaps the OS.

... actually the other thing that puzzles me is why Windows File Protection
didn't kick in.
Post by Asher_N
Personnaly at this point, I'd upgrade App Centre to the cuttent version.
Or at least, install it immediately after the OS, before patches.
Are you suggesting that whenever we need to run a new application we should buy
a new server? I don't think that's a feasible solution in general. :-)

Harry.
Lawrence Garvin (MVP)
2007-06-29 03:31:16 UTC
Permalink
Post by Harry Johnston
Post by Asher_N
Well, Windows Update does what the patch tells it to do to detect, so
it's not flawed.
The detection logic is part of the Windows Update product, speaking in the
broader sense.
Actually, this is a misconception.

The =engine= is contained within the Windows Update Agent code.

However, the =data= that tells that engine what to do is contained within
the update package itself, and is unique to each update package.

It's generally this data that is the reason updates have "revisions", and
MSRC documents are revised. Case in point -- just today Microsoft revised
MS07-022 for issues affecting people running Windows 2000 Service Pack 4 on
NEC 98 systems.
Post by Harry Johnston
Post by Asher_N
What do you expect the detection to do? Scan version of every file it
replaces? For large patches and SPs, it would consume too much resources
on the clients.
MBSA 1.2.1 seemed to manage to do this without being very resource-hungry.
(Granted service packs are a special case.)
Thus my comments, elsewhere, indicting the "scanning tools" used to
determine whether the post-install AppCenter2000SP1 system was secure. MBSA
v1.2.1 didn't exist in 2003 when this incident happened. I know that it was
2003, because if it had been 2004 they surely would have installed
AppCenter2000SP2 -- although that wouldn't have been any guarantee either,
since AppCenter2000SP2 (Jun 2003) also predated MS03-026 (Jul 2003).

Hehe... so AppCenterSP2 was released *before* MS03-026, and they chose to
install a slipstreamed *downlevel* version of the application. Go figger...
but it was suicidal, at best (and, of course, we also have the benefit of
hindsight to support that appelation).

Which then makes me want to ask when this really did occur, but either way,
I think it would just complicate the decisions behind the deployment
choices -- either way, the *current* version of the application was not
installed.
Post by Harry Johnston
Probably qfecheck is the right tool to check for this class of problem.
However, it does seem that this functionality should be built into WUA, or
perhaps the OS.
... actually the other thing that puzzles me is why Windows File
Protection didn't kick in.
Windows File Protection on a Windows 2000 Service Pack 3 system? Service
Pack 4 was only released in June, 2003, and I suspect not yet installed on
the subject system, given that they also were not installing AppCenterSP2
(Jun 2003).
Post by Harry Johnston
Post by Asher_N
Personnaly at this point, I'd upgrade App Centre to the cuttent version.
Or at least, install it immediately after the OS, before patches.
Are you suggesting that whenever we need to run a new application we
should buy a new server? I don't think that's a feasible solution in
general. :-)
No, I don't think that's what Asher is suggesting, but then I also think
that he's misunderstood that this incident is a legacy incident, that
occurred four years ago, not in the recent past.

It's also irrelevant, because AppCenter2000 isn't a supported product at
all, any more. Mainstream Support expired in July, 2006. Only Security
Updates for AppCenter2000SP2 are available, now.

However, in either situation, the *correct* installation methodology would
have surely alleviated some of the issues experienced. The *application*
being installed was at a SP level dated from October, 2001, on top of a
patch released in July, 2003 (and possibly, even, an unsupported SP level if
this all happened after June 2004).

That means, de facto, the *application* had no knowledge of any updates
applicable to that system beyond that date -- even the most current SP for
that application could not have known.

The logical conclusion (at least to me) would be that *anything* released
after that date was subject to having been corrupted. At a minimum I would
have (RE)INSTALLED Service Pack 4 (Jun 2003), and all security patches
released after Service Pack 4 (which would have included MS03-026) -- but
then I would have also installed the product's most recent service pack as
well.

The second problem was trusting the patch tools available at that point
(Windows Update) to properly identify the deficiencies in the patch level of
the system. In 2003, all that WU did was check a registry value to determine
if a patch had been "installed" or "not installed". The only certain way to
know was to personally verify the file versions and/or simply reapply the
update(s) potentially affected -- and certainly any Critical Security
Updates -- like a Blaster patch!
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E

Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx

And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Harry Johnston
2007-06-29 03:59:19 UTC
Permalink
Post by Lawrence Garvin (MVP)
Post by Harry Johnston
The detection logic is part of the Windows Update product, speaking in the
broader sense.
Actually, this is a misconception.
The =engine= is contained within the Windows Update Agent code.
However, the =data= that tells that engine what to do is contained within
the update package itself, and is unique to each update package.
Yeah, but I'd define the Windows Update *product* to include both the Windows
Update Agent and the update packages - neither can function without the other,
after all.
Post by Lawrence Garvin (MVP)
Post by Harry Johnston
... actually the other thing that puzzles me is why Windows File
Protection didn't kick in.
Windows File Protection on a Windows 2000 Service Pack 3 system?
Ah. That would explain it. I hadn't quite picked up on quite how far back into
the dark ages we were looking here.
Post by Lawrence Garvin (MVP)
However, in either situation, the *correct* installation methodology would
have surely alleviated some of the issues experienced.
Sure, but that's not my point. I think that if patch corruption isn't reliably
detected - regardless of what may have caused the corruption - this is (or was!)
a legitimate concern. (It could - presumably - just as easily have happened as
the result of a recent but badly written third-party installer.)

Does Windows File Protection address this problem? It seems that it should, but
I don't know a lot about exactly what it actually does. Certainly I've seen
QFECHECK report a problem when WFP was silent - but I'm still not sure whether
or not that was a false negative or a false positive, and slipstreaming was
involved which I suppose bypasses WFP anyway.

Harry.
Lawrence Garvin (MVP)
2007-06-29 04:38:53 UTC
Permalink
Post by Harry Johnston
Post by Lawrence Garvin (MVP)
Post by Harry Johnston
The detection logic is part of the Windows Update product, speaking in
the broader sense.
Actually, this is a misconception.
The =engine= is contained within the Windows Update Agent code.
However, the =data= that tells that engine what to do is contained within
the update package itself, and is unique to each update package.
Yeah, but I'd define the Windows Update *product* to include both the
Windows Update Agent and the update packages - neither can function
without the other, after all.
I understand that propensity for the common perception -- but the
distinction is actually very significant.

There's this natural inclination to want to blame WSUS, WUA, or the WSUS/WUA
dev teams for failings that appear to be in the product, that are actually
defects in the detection -parameters- defined by the update itself.

Perhaps the most classic example I can recall off the top of my head is the
(still existing) Snafu with the JVM update (KB816093). See this article for
additional info: http://wsusinfo.onsitechsolutions.com/articles/004.htm

Those update detection parameters are solely the output of the individual
product team's Sustained Engineering groups -- and totally outside the
purview of the WSUS/WUA teams. So, in the above example, the flaw lies in
the work product of the JVM SE group -- no doubt which was long disbanded by
the time the detection flaws in KB816093 were discovered by WSUS 2 admins in
July, 2005.

It's kind of like a beer distributor. They bring the Budweiser to you, and
they're responsible if the glass is broken, or the cans are dented, but they
got nothing to do with whether the beer tastes good, or whether you prefer
MGD to Bud Select.
Post by Harry Johnston
Post by Lawrence Garvin (MVP)
Post by Harry Johnston
... actually the other thing that puzzles me is why Windows File
Protection didn't kick in.
Windows File Protection on a Windows 2000 Service Pack 3 system?
Ah. That would explain it. I hadn't quite picked up on quite how far
back into the dark ages we were looking here.
<VBG>
Post by Harry Johnston
Post by Lawrence Garvin (MVP)
However, in either situation, the *correct* installation methodology
would have surely alleviated some of the issues experienced.
Sure, but that's not my point. I think that if patch corruption isn't
reliably detected - regardless of what may have caused the corruption -
this is (or was!) a legitimate concern.
I absolutely agree with you.

But the appropriate procedures in 2003-2004 are entirely different than the
appropriate procedures in 2007.

In 2003, the situation require a much greater involvment of *human*
verification and validation.
Post by Harry Johnston
(It could - presumably - just as easily have happened as the result of a
recent but badly written third-party installer.)
Granted, but either way, there's still a certain amount of undenyable
responsibility that sits in the lap of the person at the console.
Post by Harry Johnston
Does Windows File Protection address this problem?
*Today* on a Win2003 or WinXP system, yes, WFP would definitely have
prevented that snafu.

Actually, the whole updated patch mechanism built into XP and 2003, which
makes use of the %windir%\$hf_mig$ update cache folder, would have
automatically replaced that downgraded DLL needed by MS03-026.

In fact, just the other day I hit a wall with WFP trying to downgrade a WUA
to do testing on the WUA upgrade patch for WSUS 3.0. Because the wups.dll
files were installed from the WUA v7 package, and covered under WFP, when
the WUA v5.8 installer tried to overwrite them via use of the /wuforce
switch -- or when I tried to manually delete them and replace them with the
v5.8 files -- WFP simply replaced the v7 files.

In fact, it's this very technology that's obsoleted the legacy practice (on
WinNT and Win2000) of having to 'reinstall' the latest service pack after
every OS "life changing" event.
Post by Harry Johnston
and slipstreaming was involved which I suppose bypasses WFP anyway.
Possibly. I'd have to look at this. WFP is supposed to be protecting core
"RTM" files also, so by that standard, slipstreamed updates should also be
protected. On the other hand, if slipstreaming doesn't properly update the
WFP registration database, WFP would have no way of knowing that a newer
version had been replaced. Also, WFP is quite dependent on the "cache" copy
of those files in the dllcache folder, and if they're not there, they
wouldn't be replaceable.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E

Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx

And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Harry Johnston
2007-06-29 07:08:25 UTC
Permalink
Post by Lawrence Garvin (MVP)
Post by Harry Johnston
Yeah, but I'd define the Windows Update *product* to include both the
Windows Update Agent and the update packages - neither can function
without the other, after all.
I understand that propensity for the common perception -- but the
distinction is actually very significant.
There's this natural inclination to want to blame WSUS, WUA, or the WSUS/WUA
dev teams for failings that appear to be in the product, that are actually
defects in the detection -parameters- defined by the update itself.
I hadn't thought of looking at it that way. Certainly the actual update is not
part of the Windows Update product; for example, if the update doesn't actually
close the vulnerability it is supposed to address, that certainly isn't a
Windows Update problem. By analogy, then, the detection parameters also aren't
part of Windows Update, though the implementation of those parameters is.

Harry.
Lawrence Garvin (MVP)
2007-06-30 00:12:51 UTC
Permalink
Post by Harry Johnston
Post by Lawrence Garvin (MVP)
There's this natural inclination to want to blame WSUS, WUA, or the
WSUS/WUA dev teams for failings that appear to be in the product, that
are actually defects in the detection -parameters- defined by the update
itself.
I hadn't thought of looking at it that way. Certainly the actual update
is not part of the Windows Update product; for example, if the update
doesn't actually close the vulnerability it is supposed to address, that
certainly isn't a Windows Update problem. By analogy, then, the detection
parameters also aren't part of Windows Update, though the implementation
of those parameters is.
Exactly.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E

Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx

And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Lawrence Garvin (MVP)
2007-06-29 02:54:38 UTC
Permalink
Post by D Riberdy
Post by Lawrence Garvin (MVP)
But, yes, WSUS/WUA is wholly dependent on the publisher of the UPDATE (which
is generally the product group responsible for the specific product being
updated), to properly configure the metadata for the detection logic of that
update. Ergo, the Windows Update Agent can only do what the product team
tells it to do via the update detection logic.
So, what you are saying is that I am at the mercy of a patch developer doing
their job correctly to ensure that the patches will scan correctly in WSUS?
You'd be at their mercy even if you used Microsoft Update!

You'd also be at their mercy if you solely relied on the verbage in the MSRC
or KB document.

At some point ya either gotta trust your OS vendor to some point, or ya
gotta find a new OS vendor.

I'm afraid I can't help you much with your paranoia. ;-)
Post by D Riberdy
Post by Lawrence Garvin (MVP)
So, if I understand you correctly, you directly installed Application Center
2000 SP1 on top of a machine already containing MS03-026, and then were
surprised when a 2 year old service pack replaced some files with older
versions?
Actually, you understood me incorrectly. The above patch MS03-026 was
installed when the base machine was created with Windows Update. We
installed Microsoft Application Center 2000 SP1 as a complete
application -
not just the SP1 patch.
Okay.. so either way.. what you're telling me is that you installed a
product whose release date preceeded the release date of security updates on
the system, and you never considered the need to evaluate whether those
security updates needed to be reinstalled?
Post by D Riberdy
When we rescanned the machines at the end of the
build process, there were no updates to be found - ergo the logic built into
either the MS03-026 patch itself OR Windows Update was flawed.
Or, you misunderstood the significance of whatever tool you used to do the
"rescan".

But, again, that was in 2003, four years ago. It's really a pointless
discussion now.
Post by D Riberdy
Therein lies their concern as well as ours that if we hang our hats on the
WSUS product, which by the looks of the forum posts here for version 3, has
less than spectaular commentary,
Eh?
Post by D Riberdy
that it will ensure that we are patched to
the fullest extent and rescan previously installed patches to make sure
nothing like the above happens. If it falls on the product team or WSUS
directly, it still is a "Microsoft" problem that things are not scanned
correctly.
Like I said above.. it matters not what the PRODUCT used to scan or install
the patch is -- ultimately, in *ANY* circumstance -- you're entirely
dependent upon the Microsoft Sustained Engineeering team(s) for the various
products to forsee every possible installation environment, and be able to
detect every possible installation scenario -- including somebody who'd want
to install a two year old product onto current hotfix security patches.

Bottom line is: That's why every "best practice" document in the world says
TEST TEST TEST TEST TEST.

Prove to YOURSELF that the patch installs correctly, doesn't break anything
on your systems, and does what you need it to do, and then, when you've done
that, deploy it to production systems.


I really don't think there's much else I can say.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E

Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx

And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Al Wilson
2010-09-24 14:58:22 UTC
Permalink
I would like to dicuss a specific Example:

I have a scanner from a third party, and it indicates that a server is not patched with (MS08-069) Vulnerabilities In Microsoft XML Core Services Could Allow Remote Code Execution (955218) because


-> VULN: File path (%systemroot%\system32\msxml6.dll) is NOT fixed.


-> Function: DoesExistingFileMatch


-> Required Version: 6.20.1099.0


-> Existing Version: 6.10.1200.0

With WSUS saying that the machine is patched, I wonder how it knows that? I suspect it is looking for the "uninstall" registry key.

Can anyone help me out with a source of information?
Post by D Riberdy
I am trying to figure out how WSUS currently determines how a patch is
missing from a server. Does it check the individual files in each patch and
then report it as missing if the files are at a previous version?
I have seen in the past that software can install previous versions of dlls,
etc that "break" patches already applied. Windows Update from the web does
not recognize them as being needed again but the vulnerability still exists.
Post by Lawrence Garvin (MVP)
The Windows Update Agent tells it that it's missing.
The Windows Update Agent does a file level version check based on the
metadata stored in the update detection logic, which is obtained from the
WSUS Server during a detection event, provided that the update detection
metadata contains file level version data.
Only stupid software from stupid software developers on systems prior to
Windows 2000, where the stupid software developer failed to check the
version of the installed DLL prior to overwriting it with an older version
contained in the stupid software developer's installation package.
That really depends on what the update is; how old the update is; what
detection methodology is coded into the update package's detection logic;
what version of the operating system you're using; what version of the
Windows Update Agent you're using; and whether or not the affected DLL(s)
are actually part of any given update package.
Do you have a *specific* example you'd like to discuss?
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Post by D Riberdy
The actual incident in question was installing Microsoft Application Center
SP1 that undid the fix for the Blaster virus on all the servers we installed
it on. Even though Microsoft listed it as installed and said we were
protected an audit found us vulnerable. We had to switch to a 3rd party tool
becuase the customer felt that we and Microsoft were unable to properly
manage the patching and healthiness of the environment.
So, If a, as you put it "Only stupid software from stupid software
developers" were to overlay a newer patches dll with an older version, will
WSUS then show it as missing? Or is that entirely dependent on the people
packaging the patches to insert all the proper data for WSUS to do that type
of checking ala "The Windows Update Agent does a file level version check
based on the metadata stored in the update detection logic, which is obtained
from the
WSUS Server during a detection event, provided that the update detection
metadata contains file level version data."
DR
Post by Lawrence Garvin (MVP)
Interesting.... but a rather ancient problem, wouldn't you agree? The
Blaster virus is almost four years old; Application Center 2000 SP1
(released: Oct 2001) actually pre-dates the Blaster virus by almost two
years; and Application Center 2000 SP2 was released four years ago to
support AppCenter2000 on Windows Server 2003.
As for the Blaster virus itself, it actually exploits an issue that was
fixed by MS03-026 (KB823980), which isn't even an active update anymore,
having been superceded by a Service Pack on every operating system except
Windows 2000.
So, lacking a *current* example, let's take this scenario in the context it
deserves.
So, if I understand you correctly, you directly installed Application Center
2000 SP1 on top of a machine already containing MS03-026, and then were
surprised when a 2 year old service pack replaced some files with older
versions?
Okay... so lessons learned: [a] Always install updates in chronological
order of release. [b] Always run a security scan after applying any old
updates.
Also, keep in mind that the OS reporting an update as "Installed", merely
means that the installer successfully ran, and that the necessary registry
values were created. It indicates nothing about whether the component files
are physically present, corrupted, or overwritten with an older version by
some other installer.
Hmmm... and, in reality, it was probably nothing more than a deployment
error in the installation of the AppCenter 2000 service pack. :-)
To that extent, the customer was probably partially correct in expressing
concern about your inability to properly manage the patching and healthiness
of the enviornment. :-\
*TODAY*, with *WSUS* (or SMS, or WU/MU, or MBSA -- all of which use exactly
the same scanning technology) (neither of which existed when you were
installing AppCenter service packs on top of Blaster security vulnerability
patches), all patches are *DETECTED* based on the requisite file contents
and version numbers of those files.
But, yes, WSUS/WUA is wholly dependent on the publisher of the UPDATE (which
is generally the product group responsible for the specific product being
updated), to properly configure the metadata for the detection logic of that
update. Ergo, the Windows Update Agent can only do what the product team
tells it to do via the update detection logic.
Exactly.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Post by D Riberdy
So, what you are saying is that I am at the mercy of a patch developer doing
their job correctly to ensure that the patches will scan correctly in WSUS?
Is this a standard routine for them to do or is there currently cases where
it doesn't have the correct information? Microsoft has been known to
re-re-release patches due to improperly packaged files before.
Actually, you understood me incorrectly. The above patch MS03-026 was
installed when the base machine was created with Windows Update. We
installed Microsoft Application Center 2000 SP1 as a complete application -
not just the SP1 patch. When we rescanned the machines at the end of the
build process, there were no updates to be found - ergo the logic built into
either the MS03-026 patch itself OR Windows Update was flawed.
Therein lies their concern as well as ours that if we hang our hats on the
WSUS product, which by the looks of the forum posts here for version 3, has
less than spectaular commentary, that it will ensure that we are patched to
the fullest extent and rescan previously installed patches to make sure
nothing like the above happens. If it falls on the product team or WSUS
directly, it still is a "Microsoft" problem that things are not scanned
correctly.
Post by Asher_N
Well, Windows Update does what the patch tells it to do to detect, so
it's not flawed.
The update will look for either file versions, or more likely registry
entries. So installing MS03-026 created the requisite entries and folders
in the Windows folder. Then you load a 7 year old piece of software that
blindly clobbers a bunch of DLLs. The MS03-026 registry entries are still
there. What do you expect the detection to do? Scan version of every file
it replaces? For large patches and SPs, it would consume too much
resources on the clients.
If you want to escape responsibility and lay the blame on Microsoft, then
blame App Centre.
Personnaly at this point, I'd upgrade App Centre to the cuttent version.
Or at least, install it immediately after the OS, before patches.
Post by D Riberdy
*sigh* as Garvin noted this was an old problem. The point being we have used
other products and are now wanting to investigate WSUS as a possible solution
for patching.
Yes, I would expect that if I were to call for a scan of missing patches on
a server, it would check the file versions to ensure that they match with
what is is expected to be there. Other tools do the exact same thing and
don't seem to be consuming too many resources.
So far, I haven't really heard any good reason to vary from what we are
using to go to WSUS since we cannot expect that it would confidently tell us
what we want to know. I guess if you vary from the norm and don't kiss the
ring of Microsoft you get belittled on these forums.
Post by Harry Johnston
The detection logic is part of the Windows Update product, speaking in the
broader sense.
MBSA 1.2.1 seemed to manage to do this without being very resource-hungry.
(Granted service packs are a special case.)
Probably qfecheck is the right tool to check for this class of problem.
However, it does seem that this functionality should be built into WUA, or
perhaps the OS.
... actually the other thing that puzzles me is why Windows File Protection
didn't kick in.
Are you suggesting that whenever we need to run a new application we should buy
a new server? I don't think that's a feasible solution in general. :-)
Harry.
Post by Lawrence Garvin (MVP)
You'd be at their mercy even if you used Microsoft Update!
You'd also be at their mercy if you solely relied on the verbage in the MSRC
or KB document.
At some point ya either gotta trust your OS vendor to some point, or ya
gotta find a new OS vendor.
I'm afraid I can't help you much with your paranoia. ;-)
Okay.. so either way.. what you're telling me is that you installed a
product whose release date preceeded the release date of security updates on
the system, and you never considered the need to evaluate whether those
security updates needed to be reinstalled?
Or, you misunderstood the significance of whatever tool you used to do the
"rescan".
But, again, that was in 2003, four years ago. It's really a pointless
discussion now.
Eh?
Like I said above.. it matters not what the PRODUCT used to scan or install
the patch is -- ultimately, in *ANY* circumstance -- you're entirely
dependent upon the Microsoft Sustained Engineeering team(s) for the various
products to forsee every possible installation environment, and be able to
detect every possible installation scenario -- including somebody who'd want
to install a two year old product onto current hotfix security patches.
Bottom line is: That's why every "best practice" document in the world says
TEST TEST TEST TEST TEST.
Prove to YOURSELF that the patch installs correctly, doesn't break anything
on your systems, and does what you need it to do, and then, when you've done
that, deploy it to production systems.
I really don't think there's much else I can say.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Post by Lawrence Garvin (MVP)
Actually, this is a misconception.
The =engine= is contained within the Windows Update Agent code.
However, the =data= that tells that engine what to do is contained within
the update package itself, and is unique to each update package.
It's generally this data that is the reason updates have "revisions", and
MSRC documents are revised. Case in point -- just today Microsoft revised
MS07-022 for issues affecting people running Windows 2000 Service Pack 4 on
NEC 98 systems.
Thus my comments, elsewhere, indicting the "scanning tools" used to
determine whether the post-install AppCenter2000SP1 system was secure. MBSA
v1.2.1 didn't exist in 2003 when this incident happened. I know that it was
2003, because if it had been 2004 they surely would have installed
AppCenter2000SP2 -- although that wouldn't have been any guarantee either,
since AppCenter2000SP2 (Jun 2003) also predated MS03-026 (Jul 2003).
Hehe... so AppCenterSP2 was released *before* MS03-026, and they chose to
install a slipstreamed *downlevel* version of the application. Go figger...
but it was suicidal, at best (and, of course, we also have the benefit of
hindsight to support that appelation).
Which then makes me want to ask when this really did occur, but either way,
I think it would just complicate the decisions behind the deployment
choices -- either way, the *current* version of the application was not
installed.
Windows File Protection on a Windows 2000 Service Pack 3 system? Service
Pack 4 was only released in June, 2003, and I suspect not yet installed on
the subject system, given that they also were not installing AppCenterSP2
(Jun 2003).
No, I don't think that's what Asher is suggesting, but then I also think
that he's misunderstood that this incident is a legacy incident, that
occurred four years ago, not in the recent past.
It's also irrelevant, because AppCenter2000 isn't a supported product at
all, any more. Mainstream Support expired in July, 2006. Only Security
Updates for AppCenter2000SP2 are available, now.
However, in either situation, the *correct* installation methodology would
have surely alleviated some of the issues experienced. The *application*
being installed was at a SP level dated from October, 2001, on top of a
patch released in July, 2003 (and possibly, even, an unsupported SP level if
this all happened after June 2004).
That means, de facto, the *application* had no knowledge of any updates
applicable to that system beyond that date -- even the most current SP for
that application could not have known.
The logical conclusion (at least to me) would be that *anything* released
after that date was subject to having been corrupted. At a minimum I would
have (RE)INSTALLED Service Pack 4 (Jun 2003), and all security patches
released after Service Pack 4 (which would have included MS03-026) -- but
then I would have also installed the product's most recent service pack as
well.
The second problem was trusting the patch tools available at that point
(Windows Update) to properly identify the deficiencies in the patch level of
the system. In 2003, all that WU did was check a registry value to determine
if a patch had been "installed" or "not installed". The only certain way to
know was to personally verify the file versions and/or simply reapply the
update(s) potentially affected -- and certainly any Critical Security
Updates -- like a Blaster patch!
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Post by Harry Johnston
Yeah, but I'd define the Windows Update *product* to include both the Windows
Update Agent and the update packages - neither can function without the other,
after all.
Ah. That would explain it. I hadn't quite picked up on quite how far back into
the dark ages we were looking here.
Sure, but that's not my point. I think that if patch corruption isn't reliably
detected - regardless of what may have caused the corruption - this is (or was!)
a legitimate concern. (It could - presumably - just as easily have happened as
the result of a recent but badly written third-party installer.)
Does Windows File Protection address this problem? It seems that it should, but
I don't know a lot about exactly what it actually does. Certainly I've seen
QFECHECK report a problem when WFP was silent - but I'm still not sure whether
or not that was a false negative or a false positive, and slipstreaming was
involved which I suppose bypasses WFP anyway.
Harry.
Post by Lawrence Garvin (MVP)
I understand that propensity for the common perception -- but the
distinction is actually very significant.
There's this natural inclination to want to blame WSUS, WUA, or the WSUS/WUA
dev teams for failings that appear to be in the product, that are actually
defects in the detection -parameters- defined by the update itself.
Perhaps the most classic example I can recall off the top of my head is the
(still existing) Snafu with the JVM update (KB816093). See this article for
additional info: http://wsusinfo.onsitechsolutions.com/articles/004.htm
Those update detection parameters are solely the output of the individual
product team's Sustained Engineering groups -- and totally outside the
purview of the WSUS/WUA teams. So, in the above example, the flaw lies in
the work product of the JVM SE group -- no doubt which was long disbanded by
the time the detection flaws in KB816093 were discovered by WSUS 2 admins in
July, 2005.
It's kind of like a beer distributor. They bring the Budweiser to you, and
they're responsible if the glass is broken, or the cans are dented, but they
got nothing to do with whether the beer tastes good, or whether you prefer
MGD to Bud Select.
<VBG>
I absolutely agree with you.
But the appropriate procedures in 2003-2004 are entirely different than the
appropriate procedures in 2007.
In 2003, the situation require a much greater involvment of *human*
verification and validation.
Granted, but either way, there's still a certain amount of undenyable
responsibility that sits in the lap of the person at the console.
*Today* on a Win2003 or WinXP system, yes, WFP would definitely have
prevented that snafu.
Actually, the whole updated patch mechanism built into XP and 2003, which
makes use of the %windir%\$hf_mig$ update cache folder, would have
automatically replaced that downgraded DLL needed by MS03-026.
In fact, just the other day I hit a wall with WFP trying to downgrade a WUA
to do testing on the WUA upgrade patch for WSUS 3.0. Because the wups.dll
files were installed from the WUA v7 package, and covered under WFP, when
the WUA v5.8 installer tried to overwrite them via use of the /wuforce
switch -- or when I tried to manually delete them and replace them with the
v5.8 files -- WFP simply replaced the v7 files.
In fact, it's this very technology that's obsoleted the legacy practice (on
WinNT and Win2000) of having to 'reinstall' the latest service pack after
every OS "life changing" event.
Possibly. I'd have to look at this. WFP is supposed to be protecting core
"RTM" files also, so by that standard, slipstreamed updates should also be
protected. On the other hand, if slipstreaming doesn't properly update the
WFP registration database, WFP would have no way of knowing that a newer
version had been replaced. Also, WFP is quite dependent on the "cache" copy
of those files in the dllcache folder, and if they're not there, they
wouldn't be replaceable.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Post by Harry Johnston
I hadn't thought of looking at it that way. Certainly the actual update is not
part of the Windows Update product; for example, if the update doesn't actually
close the vulnerability it is supposed to address, that certainly isn't a
Windows Update problem. By analogy, then, the detection parameters also aren't
part of Windows Update, though the implementation of those parameters is.
Harry.
Post by Lawrence Garvin (MVP)
Exactly.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
....
Submitted via EggHeadCafe - Software Developer Portal of Choice
Lucene.Net Indexing Searching Entry Level Tutorial
http://www.eggheadcafe.com/tutorials/aspnet/c69ef65f-e3c6-409e-ab97-168897c74f83/lucenenet-indexing-searching-entry-level-tutorial.aspx
Harry Johnston
2010-09-27 01:22:01 UTC
Permalink
Post by Al Wilson
I have a scanner from a third party, and it indicates that a server is not
patched with (MS08-069) Vulnerabilities In Microsoft XML Core Services Could
Allow Remote Code Execution (955218) because
-> VULN: File path (%systemroot%\system32\msxml6.dll) is NOT fixed.
-> Function: DoesExistingFileMatch
-> Required Version: 6.20.1099.0
-> Existing Version: 6.10.1200.0
With WSUS saying that the machine is patched, I wonder how it knows that? I
suspect it is looking for the "uninstall" registry key.
WSUS doesn't distinguish between updates that are installed and updates that
aren't applicable, so perhaps your system doesn't meet the requirements? It
might also turn out that MSXML6 isn't actually installed, but the file got left
behind.

Which OS is this on? Which service pack? If Windows 2003, does MSXML6 appear
in Add/Remove Programs?

Harry.
--
PS: if it seems quiet in here, that's because Microsoft have closed this newsgroup.

You might want to consider visiting the appropriate web forum instead.

WSUS: http://go.microsoft.com/fwlink/?LinkID=161025

Windows Update: http://social.answers.microsoft.com/Forums/en-US/vistawu/threads
Loading...