PVQ: Fix incorrect calculation of rd_stats

When PVQ is on, we reencode at the end of choose_tx_size_type_from_rd to
get the entropy contexts right, previously this was done using
txfm_rd_in_plane but this is different from the encodes done in the loop
which use txfm_yrd, the result is that rd_stats is set incorrectly at
the end of choose_tx_size_type_from_rd when PVQ is on.

Results on objective-1-fast with --limit=5:

   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.5803 | -1.0598 | -1.4565 |  -0.3377 | -0.8153 | -0.5934 |    -0.9943

See https://goo.gl/Hvv0E2

Change-Id: Iccc7b0afaff849f959a0084eb48dbb838bc3cb1a
This commit is contained in:
Guillaume Martres 2016-12-06 20:26:22 -10:00 коммит произвёл Guillaume Martres
Родитель 6dbacf2859
Коммит 930118c5c5
1 изменённых файлов: 1 добавлений и 2 удалений

Просмотреть файл

@ -1818,8 +1818,7 @@ static void choose_tx_size_type_from_rd(const AV1_COMP *const cpi,
#endif
#if CONFIG_PVQ
if (best_rd != INT64_MAX) {
txfm_rd_in_plane(x, cpi, rd_stats, ref_best_rd, 0, bs, best_tx,
cpi->sf.use_fast_coef_costing);
txfm_yrd(cpi, x, rd_stats, ref_best_rd, bs, best_tx_type, best_tx);
}
#endif
}