gcm_avx_init: zero the ghash state after hashing the IV

IVs != 96 bits get hashed with GHASH to bring them to 96 bits. Any call
to GHASH will mix the ghash state in gcm_ghash. This is expected to be
zero at first use in an encrypt or decrypt operation, so it needs to be
zeroed after using GHASH in setup.

gcm_init() does this, but gcm_avx_init() zeroed it before setup, not
after, resulting in incorrect encrypt/decrypt results when using AVX GCM
with an IV != 96 bits.

OpenZFS _always_ uses a 96 bit IV (ZIO_DATA_IV_LEN) so this will never
have been hit in any real-world use, which is extremely fortunate, as we
would have incorrectly-encrypted data on-disk. Still, as long as we have
this code here we should make sure it's correct.

Thanks-to: Joel Low <[email protected]>
Sponsored-by: https://despairlabs.com/sponsor/
Signed-off-by: Rob Norris <[email protected]>
Reviewed-by: Tony Hutter <[email protected]>
Reviewed-by: Tino Reichardt <[email protected]>
Reviewed-by: Attila Fülöp <[email protected]>

module/icp/algs/modes/gcm.c

@@ -1426,7 +1426,6 @@ gcm_init_avx(gcm_ctx_t *ctx, const uint8_t *iv, size_t iv_len,
 	    B_FALSE);
 
 	/* Init H (encrypt zero block) and create the initial counter block. */
-	memset(ctx->gcm_ghash, 0, sizeof (ctx->gcm_ghash));
 	memset(H, 0, sizeof (ctx->gcm_H));
 	kfpu_begin();
 	aes_encrypt_intel(keysched, aes_rounds,
@@ -1455,6 +1454,8 @@ gcm_init_avx(gcm_ctx_t *ctx, const uint8_t *iv, size_t iv_len,
 		kfpu_begin();
 	}
 
+	memset(ctx->gcm_ghash, 0, sizeof (ctx->gcm_ghash));
+
 	/* Openssl post increments the counter, adjust for that. */
 	gcm_incr_counter_block(ctx);