MdeModulePkg/PciHostBridgeDxe: Add readback after final Cfg-Write (#599)

## Description

Problem Report:

On AARCH64, there is no ordering guarantee between configuration
space (ECAM) writes and memory space reads (MMIO). ARM AMBA CHI
only guarantees ordering for reads and writes within a single address
region,
however, on some systems MMIO and ECAM may be split into separate
address regions.

A problem may arise when an ECAM write is issued a completion before a
subsequent
MMIO read is issued and receives a completion.

For example, a typical PCI software flow is the following:

1. ECAM write to device command register to enable memory space
2. MMIO read from device memory space for which access was enabled
   in step 1.

There is no guarantee that step 2. will not begin before the completion
of step 1.
on systems where ECAM/MMIO are specified as separate address regions,
even
if both spaces have the memory attributes device-nGnRnE.

- Add a read after the final PCI Configuration space write
in RootBridgeIoPciAccess. Configuration space reads should not have
side-efects.

- When configuration space is strongly ordered, this ensures
that program execution cannot continue until the completion
is received for the previous Cfg-Write, which may have side-effects.

- Risk of reading a "write-only" register and causing a CA which leaves
the device
unresponsive. The expectation based on the PCI Base Spec v6.1 section
7.4 is that
all PCI Spec-defined registers will be readable, however, there may
exist
design-specific registers that fall into this category.

- [X] Impacts functionality?
- **Functionality** - Does the change ultimately impact how firmware
functions?
- Examples: Add a new library, publish a new PPI, update an algorithm,
...
- [ ] Impacts security?
- **Security** - Does the change have a direct security impact on an
application,
    flow, or firmware?
  - Examples: Crypto algorithm change, buffer overflow fix, parameter
    validation improvement, ...
- [ ] Breaking change?
- **Breaking change** - Will anyone consuming this change experience a
break
    in build or boot behavior?
- Examples: Add a new library class, move a module to a different repo,
call
    a function in a new library class in a pre-existing module, ...
- [ ] Includes tests?
  - **Tests** - Does the change include any explicit test code?
  - Examples: Unit tests, integration tests, robot tests, ...
- [ ] Includes documentation?
- **Documentation** - Does the change contain explicit documentation
additions
    outside direct code modifications (and comments)?
- Examples: Update readme file, add feature readme file, link to
documentation
    on an a separate Web page, ...

## How This Was Tested

- AARCH64 platform on real silicon

## Integration Instructions

N/A

---------

Co-authored-by: joelopez <joelopez@microsoft.com>

MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c

@@ -1246,6 +1246,17 @@ RootBridgeIoPciAccess (
     }
   }
 
+  // MU_CHANGE BEGIN
+  //
+  // Perform readback after write to confirm completion was received for the last write
+  // before subsequent memory operations can be issued.
+  //
+  if (!Read) {
+    PciSegmentRead8 (Address - InStride);
+  }
+
+  // MU_CHANGE END
+
   return EFI_SUCCESS;
 }