kris/boringssl
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
302ef5ee12
boringssl/crypto/fipsmodule/aes/asm/aesv8-armx.pl

1009 lines
22 KiB
Perl
Raw Normal View History

Sync ARM assembly up to 609b0852e4d50251857dbbac3141ba042e35a9ae. This change was made by copying over the files as of that commit and then discarding the parts of the diff which corresponding to our own changes. Change-Id: I28c5d711f7a8cec30749b8174687434129af5209 Reviewed-on: https://boringssl-review.googlesource.com/17111 Reviewed-by: Adam Langley <agl@google.com>
2017-06-12 23:53:40 +01:00
#! /usr/bin/env perl
# Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the OpenSSL license (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
#
# ====================================================================
# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
# project. The module is, however, dual licensed under OpenSSL and
# CRYPTOGAMS licenses depending on where you obtain it. For further
# details see http://www.openssl.org/~appro/cryptogams/.
# ====================================================================
#
# This module implements support for ARMv8 AES instructions. The
# module is endian-agnostic in sense that it supports both big- and
# little-endian cases. As does it support both 32- and 64-bit modes
# of operation. Latter is achieved by limiting amount of utilized
# registers to 16, which implies additional NEON load and integer
# instructions. This has no effect on mighty Apple A7, where results
# are literally equal to the theoretical estimates based on AES
# instruction latencies and issue rates. On Cortex-A53, an in-order
# execution core, this costs up to 10-15%, which is partially
# compensated by implementing dedicated code path for 128-bit
# CBC encrypt case. On Cortex-A57 parallelizable mode performance
# seems to be limited by sheer amount of NEON instructions...
#
# Performance in cycles per byte processed with 128-bit key:
#
# CBC enc CBC dec CTR
# Apple A7 2.39 1.20 1.20
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
# Cortex-A53 1.32 1.29 1.46
# Cortex-A57(*) 1.95 0.85 0.93
# Denver 1.96 0.86 0.80
Sync ARM assembly up to 609b0852e4d50251857dbbac3141ba042e35a9ae. This change was made by copying over the files as of that commit and then discarding the parts of the diff which corresponding to our own changes. Change-Id: I28c5d711f7a8cec30749b8174687434129af5209 Reviewed-on: https://boringssl-review.googlesource.com/17111 Reviewed-by: Adam Langley <agl@google.com>
2017-06-12 23:53:40 +01:00
# Mongoose 1.33 1.20 1.20
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
#
# (*) original 3.64/1.34/1.32 results were for r0p0 revision
# and are still same even for updated module;
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
$flavour = shift;
Add arm-xlate.pl and initial iOS asm support. This is as partial import of upstream's 9b05cbc33e7895ed033b1119e300782d9e0cf23c. It includes the perlasm changes, but not the CPU feature detection bits as we do those differently. This is largely so we don't diverge from upstream, but it'll help with iOS assembly in the future. sha512-armv8.pl is modified slightly from upstream to switch from conditioning on the output file to conditioning on an extra argument. This makes our previous change from upstream (removing the 'open STDOUT' line) more explicit. BUG=338886 Change-Id: Ic8ca1388ae20e94566f475bad3464ccc73f445df Reviewed-on: https://boringssl-review.googlesource.com/4405 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 18:25:46 +01:00
$output = shift;
$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
( $xlate="${dir}arm-xlate.pl" and -f $xlate ) or
Move AES code into the FIPS module. Change-Id: Id94e71bce4dca25e77f52f38c07e0489ca072d2d Reviewed-on: https://boringssl-review.googlesource.com/15027 Reviewed-by: Adam Langley <agl@google.com> Commit-Queue: Adam Langley <agl@google.com> CQ-Verified: CQ bot account: commit-bot@chromium.org <commit-bot@chromium.org>
2017-04-13 19:00:24 +01:00
( $xlate="${dir}../../../perlasm/arm-xlate.pl" and -f $xlate) or
Add arm-xlate.pl and initial iOS asm support. This is as partial import of upstream's 9b05cbc33e7895ed033b1119e300782d9e0cf23c. It includes the perlasm changes, but not the CPU feature detection bits as we do those differently. This is largely so we don't diverge from upstream, but it'll help with iOS assembly in the future. sha512-armv8.pl is modified slightly from upstream to switch from conditioning on the output file to conditioning on an extra argument. This makes our previous change from upstream (removing the 'open STDOUT' line) more explicit. BUG=338886 Change-Id: Ic8ca1388ae20e94566f475bad3464ccc73f445df Reviewed-on: https://boringssl-review.googlesource.com/4405 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 18:25:46 +01:00
die "can't locate arm-xlate.pl";
open OUT,"| \"$^X\" $xlate $flavour $output";
*STDOUT=*OUT;
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
Add PPC64LE assembly for AES-GCM. This change adds AES and GHASH assembly from upstream, with the aim of speeding up AES-GCM. The PPC64LE assembly matches the interface of the ARMv8 assembly so I've changed the prefix of both sets of asm functions to be the same ("aes_hw_"). Otherwise, the new assmebly files and Perlasm match exactly those from upstream's c536b6be1a (from their master branch). Before: Did 1879000 AES-128-GCM (16 bytes) seal operations in 1000428us (1878196.1 ops/sec): 30.1 MB/s Did 61000 AES-128-GCM (1350 bytes) seal operations in 1006660us (60596.4 ops/sec): 81.8 MB/s Did 11000 AES-128-GCM (8192 bytes) seal operations in 1072649us (10255.0 ops/sec): 84.0 MB/s Did 1665000 AES-256-GCM (16 bytes) seal operations in 1000591us (1664016.6 ops/sec): 26.6 MB/s Did 52000 AES-256-GCM (1350 bytes) seal operations in 1006971us (51640.0 ops/sec): 69.7 MB/s Did 8840 AES-256-GCM (8192 bytes) seal operations in 1013294us (8724.0 ops/sec): 71.5 MB/s After: Did 4994000 AES-128-GCM (16 bytes) seal operations in 1000017us (4993915.1 ops/sec): 79.9 MB/s Did 1389000 AES-128-GCM (1350 bytes) seal operations in 1000073us (1388898.6 ops/sec): 1875.0 MB/s Did 319000 AES-128-GCM (8192 bytes) seal operations in 1000101us (318967.8 ops/sec): 2613.0 MB/s Did 4668000 AES-256-GCM (16 bytes) seal operations in 1000149us (4667304.6 ops/sec): 74.7 MB/s Did 1202000 AES-256-GCM (1350 bytes) seal operations in 1000646us (1201224.0 ops/sec): 1621.7 MB/s Did 269000 AES-256-GCM (8192 bytes) seal operations in 1002804us (268247.8 ops/sec): 2197.5 MB/s Change-Id: Id848562bd4e1aa79a4683012501dfa5e6c08cfcc Reviewed-on: https://boringssl-review.googlesource.com/11262 Reviewed-by: Adam Langley <agl@google.com> Commit-Queue: Adam Langley <agl@google.com> CQ-Verified: CQ bot account: commit-bot@chromium.org <commit-bot@chromium.org>
2016-09-23 20:47:24 +01:00
$prefix="aes_hw";
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
$code=<<___;
Move arm_arch.h and fix up lots of include paths. arm_arch.h is included from ARM asm files, but lives in crypto/, not openssl/include/. Since the asm files are often built from a different location than their position in the source tree, relative include paths are unlikely to work so, rather than having crypto/ be a de-facto, second global include path, this change moves arm_arch.h to include/openssl/. It also removes entries from many include paths because they should be needed as relative includes are always based on the locations of the source file. Change-Id: I638ff43d641ca043a4fc06c0d901b11c6ff73542 Reviewed-on: https://boringssl-review.googlesource.com/5746 Reviewed-by: Adam Langley <agl@google.com>
2015-08-25 02:03:17 +01:00
#include <openssl/arm_arch.h>
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
#if __ARM_MAX_ARCH__>=7
.text
___
Remove clang assembler .arch workaround. This makes it difficult to build against the NDK's toolchain file. The problem is __clang__ just means Clang is the frontend and implies nothing about which assembler. When using as, it is fine. When using clang-as on Linux, one needs a clang-as from this year. The only places where we case about clang's integrated assembler are iOS (where perlasm strips out .arch anyway) and build environments like Chromium which have a regularly-updated clang. Thus we can remove this now. Bug: 39 Update-Note: Holler if this breaks the build. If it doesn't break the build, you can probably remove any BORINGSSL_CLANG_SUPPORTS_DOT_ARCH or explicit -march armv8-a+crypto lines in your BoringSSL build. Change-Id: I21ce54b14c659830520c2f1d51c7bd13e0980c68 Reviewed-on: https://boringssl-review.googlesource.com/24124 Commit-Queue: Adam Langley <agl@google.com> Reviewed-by: Adam Langley <agl@google.com> CQ-Verified: CQ bot account: commit-bot@chromium.org <commit-bot@chromium.org>
2017-12-13 21:28:25 +00:00
$code.=".arch armv8-a+crypto\n" if ($flavour =~ /64/);
ARMv4 assembly pack: harmonize Thumb-ification of iOS build. Three modules were left behind in I59df0b567e8e80befe5c399f817d6410ddafc577. (Imported from upstream's c93f06c12f10c07cea935abd78a07a037e27f155.) This actually meant functions defined in those two files were non-functional. I'm guessing no one noticed upstream because, if you go strictly by iOS compile-time capabilities, all this code is unreachable on ios32, only ios64. Change-Id: I55035edf2aebf96d14bdf66161afa2374643d4ec Reviewed-on: https://boringssl-review.googlesource.com/17113 Reviewed-by: David Benjamin <davidben@google.com>
2017-06-13 00:09:57 +01:00
$code.=<<___ if ($flavour !~ /64/);
.arch armv7-a // don't confuse not-so-latest binutils with argv8 :-)
.fpu neon
.code 32
#undef __thumb2__
___
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
# Assembler mnemonics are an eclectic mix of 32- and 64-bit syntax,
# NEON is mostly 32-bit mnemonics, integer - mostly 64. Goal is to
# maintain both 32- and 64-bit codes within single module and
# transliterate common code to either flavour with regex vodoo.
#
{{{
my ($inp,$bits,$out,$ptr,$rounds)=("x0","w1","x2","x3","w12");
my ($zero,$rcon,$mask,$in0,$in1,$tmp,$key)=
$flavour=~/64/? map("q$_",(0..6)) : map("q$_",(0..3,8..10));
$code.=<<___;
Move .align directives next to their labels for ARM. 2ab24a2d40fd193a25d7c94072596a1180cbb460 added sections to ARM assembly files. However, in cases where .align directives were not next to the labels that they were intended to apply to, the section directives would cause them to be ignored. Change-Id: I32117f6747ff8545b80c70dd3b8effdc6e6f67e0 Reviewed-on: https://boringssl-review.googlesource.com/6050 Reviewed-by: David Benjamin <davidben@chromium.org> Reviewed-by: Adam Langley <agl@google.com>
2015-09-30 18:38:38 +01:00
.align 5
Add arm-xlate.pl and initial iOS asm support. This is as partial import of upstream's 9b05cbc33e7895ed033b1119e300782d9e0cf23c. It includes the perlasm changes, but not the CPU feature detection bits as we do those differently. This is largely so we don't diverge from upstream, but it'll help with iOS assembly in the future. sha512-armv8.pl is modified slightly from upstream to switch from conditioning on the output file to conditioning on an extra argument. This makes our previous change from upstream (removing the 'open STDOUT' line) more explicit. BUG=338886 Change-Id: Ic8ca1388ae20e94566f475bad3464ccc73f445df Reviewed-on: https://boringssl-review.googlesource.com/4405 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 18:25:46 +01:00
.Lrcon:
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
.long 0x01,0x01,0x01,0x01
.long 0x0c0f0e0d,0x0c0f0e0d,0x0c0f0e0d,0x0c0f0e0d // rotate-n-splat
.long 0x1b,0x1b,0x1b,0x1b
.globl ${prefix}_set_encrypt_key
.type ${prefix}_set_encrypt_key,%function
.align 5
${prefix}_set_encrypt_key:
.Lenc_key:
___
$code.=<<___ if ($flavour =~ /64/);
stp x29,x30,[sp,#-16]!
add x29,sp,#0
___
$code.=<<___;
mov $ptr,#-1
cmp $inp,#0
b.eq .Lenc_key_abort
cmp $out,#0
b.eq .Lenc_key_abort
mov $ptr,#-2
cmp $bits,#128
b.lt .Lenc_key_abort
cmp $bits,#256
b.gt .Lenc_key_abort
tst $bits,#0x3f
b.ne .Lenc_key_abort
Add arm-xlate.pl and initial iOS asm support. This is as partial import of upstream's 9b05cbc33e7895ed033b1119e300782d9e0cf23c. It includes the perlasm changes, but not the CPU feature detection bits as we do those differently. This is largely so we don't diverge from upstream, but it'll help with iOS assembly in the future. sha512-armv8.pl is modified slightly from upstream to switch from conditioning on the output file to conditioning on an extra argument. This makes our previous change from upstream (removing the 'open STDOUT' line) more explicit. BUG=338886 Change-Id: Ic8ca1388ae20e94566f475bad3464ccc73f445df Reviewed-on: https://boringssl-review.googlesource.com/4405 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 18:25:46 +01:00
adr $ptr,.Lrcon
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
cmp $bits,#192
veor $zero,$zero,$zero
vld1.8 {$in0},[$inp],#16
mov $bits,#8 // reuse $bits
vld1.32 {$rcon,$mask},[$ptr],#32
b.lt .Loop128
b.eq .L192
b .L256
.align 4
.Loop128:
vtbl.8 $key,{$in0},$mask
vext.8 $tmp,$zero,$in0,#12
vst1.32 {$in0},[$out],#16
aese $key,$zero
subs $bits,$bits,#1
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $key,$key,$rcon
veor $in0,$in0,$tmp
vshl.u8 $rcon,$rcon,#1
veor $in0,$in0,$key
b.ne .Loop128
vld1.32 {$rcon},[$ptr]
vtbl.8 $key,{$in0},$mask
vext.8 $tmp,$zero,$in0,#12
vst1.32 {$in0},[$out],#16
aese $key,$zero
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $key,$key,$rcon
veor $in0,$in0,$tmp
vshl.u8 $rcon,$rcon,#1
veor $in0,$in0,$key
vtbl.8 $key,{$in0},$mask
vext.8 $tmp,$zero,$in0,#12
vst1.32 {$in0},[$out],#16
aese $key,$zero
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $key,$key,$rcon
veor $in0,$in0,$tmp
veor $in0,$in0,$key
vst1.32 {$in0},[$out]
add $out,$out,#0x50
mov $rounds,#10
b .Ldone
.align 4
.L192:
vld1.8 {$in1},[$inp],#8
vmov.i8 $key,#8 // borrow $key
vst1.32 {$in0},[$out],#16
vsub.i8 $mask,$mask,$key // adjust the mask
.Loop192:
vtbl.8 $key,{$in1},$mask
vext.8 $tmp,$zero,$in0,#12
vst1.32 {$in1},[$out],#8
aese $key,$zero
subs $bits,$bits,#1
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in0,$in0,$tmp
vdup.32 $tmp,${in0}[3]
veor $tmp,$tmp,$in1
veor $key,$key,$rcon
vext.8 $in1,$zero,$in1,#12
vshl.u8 $rcon,$rcon,#1
veor $in1,$in1,$tmp
veor $in0,$in0,$key
veor $in1,$in1,$key
vst1.32 {$in0},[$out],#16
b.ne .Loop192
mov $rounds,#12
add $out,$out,#0x20
b .Ldone
.align 4
.L256:
vld1.8 {$in1},[$inp]
mov $bits,#7
mov $rounds,#14
vst1.32 {$in0},[$out],#16
.Loop256:
vtbl.8 $key,{$in1},$mask
vext.8 $tmp,$zero,$in0,#12
vst1.32 {$in1},[$out],#16
aese $key,$zero
subs $bits,$bits,#1
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in0,$in0,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $key,$key,$rcon
veor $in0,$in0,$tmp
vshl.u8 $rcon,$rcon,#1
veor $in0,$in0,$key
vst1.32 {$in0},[$out],#16
b.eq .Ldone
vdup.32 $key,${in0}[3] // just splat
vext.8 $tmp,$zero,$in1,#12
aese $key,$zero
veor $in1,$in1,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in1,$in1,$tmp
vext.8 $tmp,$zero,$tmp,#12
veor $in1,$in1,$tmp
veor $in1,$in1,$key
b .Loop256
.Ldone:
str $rounds,[$out]
mov $ptr,#0
.Lenc_key_abort:
mov x0,$ptr // return value
`"ldr x29,[sp],#16" if ($flavour =~ /64/)`
ret
.size ${prefix}_set_encrypt_key,.-${prefix}_set_encrypt_key
.globl ${prefix}_set_decrypt_key
.type ${prefix}_set_decrypt_key,%function
.align 5
${prefix}_set_decrypt_key:
___
$code.=<<___ if ($flavour =~ /64/);
stp x29,x30,[sp,#-16]!
add x29,sp,#0
___
$code.=<<___ if ($flavour !~ /64/);
stmdb sp!,{r4,lr}
___
$code.=<<___;
bl .Lenc_key
cmp x0,#0
b.ne .Ldec_key_abort
sub $out,$out,#240 // restore original $out
mov x4,#-16
add $inp,$out,x12,lsl#4 // end of key schedule
vld1.32 {v0.16b},[$out]
vld1.32 {v1.16b},[$inp]
vst1.32 {v0.16b},[$inp],x4
vst1.32 {v1.16b},[$out],#16
.Loop_imc:
vld1.32 {v0.16b},[$out]
vld1.32 {v1.16b},[$inp]
aesimc v0.16b,v0.16b
aesimc v1.16b,v1.16b
vst1.32 {v0.16b},[$inp],x4
vst1.32 {v1.16b},[$out],#16
cmp $inp,$out
b.hi .Loop_imc
vld1.32 {v0.16b},[$out]
aesimc v0.16b,v0.16b
vst1.32 {v0.16b},[$inp]
eor x0,x0,x0 // return value
.Ldec_key_abort:
___
$code.=<<___ if ($flavour !~ /64/);
ldmia sp!,{r4,pc}
___
$code.=<<___ if ($flavour =~ /64/);
ldp x29,x30,[sp],#16
ret
___
$code.=<<___;
.size ${prefix}_set_decrypt_key,.-${prefix}_set_decrypt_key
___
}}}
{{{
sub gen_block () {
my $dir = shift;
my ($e,$mc) = $dir eq "en" ? ("e","mc") : ("d","imc");
my ($inp,$out,$key)=map("x$_",(0..2));
my $rounds="w3";
my ($rndkey0,$rndkey1,$inout)=map("q$_",(0..3));
$code.=<<___;
.globl ${prefix}_${dir}crypt
.type ${prefix}_${dir}crypt,%function
.align 5
${prefix}_${dir}crypt:
ldr $rounds,[$key,#240]
vld1.32 {$rndkey0},[$key],#16
vld1.8 {$inout},[$inp]
sub $rounds,$rounds,#2
vld1.32 {$rndkey1},[$key],#16
.Loop_${dir}c:
aes$e $inout,$rndkey0
aes$mc $inout,$inout
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {$rndkey0},[$key],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
subs $rounds,$rounds,#2
aes$e $inout,$rndkey1
aes$mc $inout,$inout
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {$rndkey1},[$key],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
b.gt .Loop_${dir}c
aes$e $inout,$rndkey0
aes$mc $inout,$inout
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {$rndkey0},[$key]
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aes$e $inout,$rndkey1
veor $inout,$inout,$rndkey0
vst1.8 {$inout},[$out]
ret
.size ${prefix}_${dir}crypt,.-${prefix}_${dir}crypt
___
}
&gen_block("en");
&gen_block("de");
}}}
{{{
my ($inp,$out,$len,$key,$ivp)=map("x$_",(0..4)); my $enc="w5";
my ($rounds,$cnt,$key_,$step,$step1)=($enc,"w6","x7","x8","x12");
my ($dat0,$dat1,$in0,$in1,$tmp0,$tmp1,$ivec,$rndlast)=map("q$_",(0..7));
my ($dat,$tmp,$rndzero_n_last)=($dat0,$tmp0,$tmp1);
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
my ($key4,$key5,$key6,$key7)=("x6","x12","x14",$key);
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
### q8-q15 preloaded key schedule
$code.=<<___;
.globl ${prefix}_cbc_encrypt
.type ${prefix}_cbc_encrypt,%function
.align 5
${prefix}_cbc_encrypt:
___
$code.=<<___ if ($flavour =~ /64/);
stp x29,x30,[sp,#-16]!
add x29,sp,#0
___
$code.=<<___ if ($flavour !~ /64/);
mov ip,sp
stmdb sp!,{r4-r8,lr}
vstmdb sp!,{d8-d15} @ ABI specification says so
ldmia ip,{r4-r5} @ load remaining args
___
$code.=<<___;
subs $len,$len,#16
mov $step,#16
b.lo .Lcbc_abort
cclr $step,eq
cmp $enc,#0 // en- or decrypting?
ldr $rounds,[$key,#240]
and $len,$len,#-16
vld1.8 {$ivec},[$ivp]
vld1.8 {$dat},[$inp],$step
vld1.32 {q8-q9},[$key] // load key schedule...
sub $rounds,$rounds,#6
add $key_,$key,x5,lsl#4 // pointer to last 7 round keys
sub $rounds,$rounds,#2
vld1.32 {q10-q11},[$key_],#32
vld1.32 {q12-q13},[$key_],#32
vld1.32 {q14-q15},[$key_],#32
vld1.32 {$rndlast},[$key_]
add $key_,$key,#32
mov $cnt,$rounds
b.eq .Lcbc_dec
cmp $rounds,#2
veor $dat,$dat,$ivec
veor $rndzero_n_last,q8,$rndlast
b.eq .Lcbc_enc128
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {$in0-$in1},[$key_]
add $key_,$key,#16
add $key4,$key,#16*4
add $key5,$key,#16*5
aese $dat,q8
aesmc $dat,$dat
add $key6,$key,#16*6
add $key7,$key,#16*7
b .Lenter_cbc_enc
.align 4
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
.Loop_cbc_enc:
aese $dat,q8
aesmc $dat,$dat
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vst1.8 {$ivec},[$out],#16
.Lenter_cbc_enc:
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $dat,q9
aesmc $dat,$dat
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat,$in0
aesmc $dat,$dat
vld1.32 {q8},[$key4]
cmp $rounds,#4
aese $dat,$in1
aesmc $dat,$dat
vld1.32 {q9},[$key5]
b.eq .Lcbc_enc192
aese $dat,q8
aesmc $dat,$dat
vld1.32 {q8},[$key6]
aese $dat,q9
aesmc $dat,$dat
vld1.32 {q9},[$key7]
nop
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
.Lcbc_enc192:
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $dat,q8
aesmc $dat,$dat
subs $len,$len,#16
aese $dat,q9
aesmc $dat,$dat
cclr $step,eq
aese $dat,q10
aesmc $dat,$dat
aese $dat,q11
aesmc $dat,$dat
vld1.8 {q8},[$inp],$step
aese $dat,q12
aesmc $dat,$dat
veor q8,q8,$rndzero_n_last
aese $dat,q13
aesmc $dat,$dat
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q9},[$key_] // re-pre-load rndkey[1]
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $dat,q14
aesmc $dat,$dat
aese $dat,q15
veor $ivec,$dat,$rndlast
b.hs .Loop_cbc_enc
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vst1.8 {$ivec},[$out],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
b .Lcbc_done
.align 5
.Lcbc_enc128:
vld1.32 {$in0-$in1},[$key_]
aese $dat,q8
aesmc $dat,$dat
b .Lenter_cbc_enc128
.Loop_cbc_enc128:
aese $dat,q8
aesmc $dat,$dat
vst1.8 {$ivec},[$out],#16
.Lenter_cbc_enc128:
aese $dat,q9
aesmc $dat,$dat
subs $len,$len,#16
aese $dat,$in0
aesmc $dat,$dat
cclr $step,eq
aese $dat,$in1
aesmc $dat,$dat
aese $dat,q10
aesmc $dat,$dat
aese $dat,q11
aesmc $dat,$dat
vld1.8 {q8},[$inp],$step
aese $dat,q12
aesmc $dat,$dat
aese $dat,q13
aesmc $dat,$dat
aese $dat,q14
aesmc $dat,$dat
veor q8,q8,$rndzero_n_last
aese $dat,q15
veor $ivec,$dat,$rndlast
b.hs .Loop_cbc_enc128
vst1.8 {$ivec},[$out],#16
b .Lcbc_done
___
{
my ($dat2,$in2,$tmp2)=map("q$_",(10,11,9));
$code.=<<___;
.align 5
.Lcbc_dec:
vld1.8 {$dat2},[$inp],#16
subs $len,$len,#32 // bias
add $cnt,$rounds,#2
vorr $in1,$dat,$dat
vorr $dat1,$dat,$dat
vorr $in2,$dat2,$dat2
b.lo .Lcbc_dec_tail
vorr $dat1,$dat2,$dat2
vld1.8 {$dat2},[$inp],#16
vorr $in0,$dat,$dat
vorr $in1,$dat1,$dat1
vorr $in2,$dat2,$dat2
.Loop3x_cbc_dec:
aesd $dat0,q8
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q8},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
subs $cnt,$cnt,#2
aesd $dat0,q9
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q9},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
b.gt .Loop3x_cbc_dec
aesd $dat0,q8
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
veor $tmp0,$ivec,$rndlast
subs $len,$len,#0x30
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
veor $tmp1,$in0,$rndlast
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
mov.lo x6,$len // x6, $cnt, is zero at this point
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesd $dat0,q9
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
veor $tmp2,$in1,$rndlast
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
add $inp,$inp,x6 // $inp is adjusted in such way that
// at exit from the loop $dat1-$dat2
// are loaded with last "words"
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vorr $ivec,$in2,$in2
mov $key_,$key
aesd $dat0,q12
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q12
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q12
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
vld1.8 {$in0},[$inp],#16
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat0,q13
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q13
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q13
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
vld1.8 {$in1},[$inp],#16
aesd $dat0,q14
aesimc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat1,q14
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q14
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.8 {$in2},[$inp],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesd $dat0,q15
aesd $dat1,q15
aesd $dat2,q15
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q8},[$key_],#16 // re-pre-load rndkey[0]
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
add $cnt,$rounds,#2
veor $tmp0,$tmp0,$dat0
veor $tmp1,$tmp1,$dat1
veor $dat2,$dat2,$tmp2
vld1.32 {q9},[$key_],#16 // re-pre-load rndkey[1]
vst1.8 {$tmp0},[$out],#16
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vorr $dat0,$in0,$in0
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
vst1.8 {$tmp1},[$out],#16
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vorr $dat1,$in1,$in1
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
vst1.8 {$dat2},[$out],#16
vorr $dat2,$in2,$in2
b.hs .Loop3x_cbc_dec
cmn $len,#0x30
b.eq .Lcbc_done
nop
.Lcbc_dec_tail:
aesd $dat1,q8
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q8},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
subs $cnt,$cnt,#2
aesd $dat1,q9
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q9},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
b.gt .Lcbc_dec_tail
aesd $dat1,q8
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aesd $dat1,q9
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
aesd $dat1,q12
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q12
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
cmn $len,#0x20
aesd $dat1,q13
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q13
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
veor $tmp1,$ivec,$rndlast
aesd $dat1,q14
aesimc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesd $dat2,q14
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesimc $dat2,$dat2
veor $tmp2,$in1,$rndlast
aesd $dat1,q15
aesd $dat2,q15
b.eq .Lcbc_dec_one
veor $tmp1,$tmp1,$dat1
veor $tmp2,$tmp2,$dat2
vorr $ivec,$in2,$in2
vst1.8 {$tmp1},[$out],#16
vst1.8 {$tmp2},[$out],#16
b .Lcbc_done
.Lcbc_dec_one:
veor $tmp1,$tmp1,$dat2
vorr $ivec,$in2,$in2
vst1.8 {$tmp1},[$out],#16
.Lcbc_done:
vst1.8 {$ivec},[$ivp]
.Lcbc_abort:
___
}
$code.=<<___ if ($flavour !~ /64/);
vldmia sp!,{d8-d15}
ldmia sp!,{r4-r8,pc}
___
$code.=<<___ if ($flavour =~ /64/);
ldr x29,[sp],#16
ret
___
$code.=<<___;
.size ${prefix}_cbc_encrypt,.-${prefix}_cbc_encrypt
___
}}}
{{{
my ($inp,$out,$len,$key,$ivp)=map("x$_",(0..4));
my ($rounds,$cnt,$key_)=("w5","w6","x7");
my ($ctr,$tctr0,$tctr1,$tctr2)=map("w$_",(8..10,12));
my $step="x12"; # aliases with $tctr2
my ($dat0,$dat1,$in0,$in1,$tmp0,$tmp1,$ivec,$rndlast)=map("q$_",(0..7));
my ($dat2,$in2,$tmp2)=map("q$_",(10,11,9));
my ($dat,$tmp)=($dat0,$tmp0);
### q8-q15 preloaded key schedule
$code.=<<___;
.globl ${prefix}_ctr32_encrypt_blocks
.type ${prefix}_ctr32_encrypt_blocks,%function
.align 5
${prefix}_ctr32_encrypt_blocks:
___
$code.=<<___ if ($flavour =~ /64/);
stp x29,x30,[sp,#-16]!
add x29,sp,#0
___
$code.=<<___ if ($flavour !~ /64/);
mov ip,sp
stmdb sp!,{r4-r10,lr}
vstmdb sp!,{d8-d15} @ ABI specification says so
ldr r4, [ip] @ load remaining arg
___
$code.=<<___;
ldr $rounds,[$key,#240]
ldr $ctr, [$ivp, #12]
vld1.32 {$dat0},[$ivp]
vld1.32 {q8-q9},[$key] // load key schedule...
sub $rounds,$rounds,#4
mov $step,#16
cmp $len,#2
add $key_,$key,x5,lsl#4 // pointer to last 5 round keys
sub $rounds,$rounds,#2
vld1.32 {q12-q13},[$key_],#32
vld1.32 {q14-q15},[$key_],#32
vld1.32 {$rndlast},[$key_]
add $key_,$key,#32
mov $cnt,$rounds
cclr $step,lo
#ifndef __ARMEB__
rev $ctr, $ctr
#endif
vorr $dat1,$dat0,$dat0
add $tctr1, $ctr, #1
vorr $dat2,$dat0,$dat0
add $ctr, $ctr, #2
vorr $ivec,$dat0,$dat0
rev $tctr1, $tctr1
vmov.32 ${dat1}[3],$tctr1
b.ls .Lctr32_tail
rev $tctr2, $ctr
sub $len,$len,#3 // bias
vmov.32 ${dat2}[3],$tctr2
b .Loop3x_ctr32
.align 4
.Loop3x_ctr32:
aese $dat0,q8
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat2,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q8},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
subs $cnt,$cnt,#2
aese $dat0,q9
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat2,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat2,$dat2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q9},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
b.gt .Loop3x_ctr32
aese $dat0,q8
aesmc $tmp0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.8 {$in0},[$inp],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
vorr $dat0,$ivec,$ivec
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat2,q8
aesmc $dat2,$dat2
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
vld1.8 {$in1},[$inp],#16
vorr $dat1,$ivec,$ivec
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $tmp0,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp0,$tmp0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $tmp1,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp1,$tmp1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.8 {$in2},[$inp],#16
mov $key_,$key
aese $dat2,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp2,$dat2
vorr $dat2,$ivec,$ivec
add $tctr0,$ctr,#1
aese $tmp0,q12
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesmc $tmp0,$tmp0
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $tmp1,q12
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesmc $tmp1,$tmp1
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
veor $in0,$in0,$rndlast
add $tctr1,$ctr,#2
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $tmp2,q12
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp2,$tmp2
veor $in1,$in1,$rndlast
add $ctr,$ctr,#3
aese $tmp0,q13
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesmc $tmp0,$tmp0
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $tmp1,q13
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesmc $tmp1,$tmp1
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
veor $in2,$in2,$rndlast
rev $tctr0,$tctr0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $tmp2,q13
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp2,$tmp2
vmov.32 ${dat0}[3], $tctr0
rev $tctr1,$tctr1
aese $tmp0,q14
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesmc $tmp0,$tmp0
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $tmp1,q14
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aesmc $tmp1,$tmp1
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
vmov.32 ${dat1}[3], $tctr1
rev $tctr2,$ctr
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $tmp2,q14
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $tmp2,$tmp2
vmov.32 ${dat2}[3], $tctr2
subs $len,$len,#3
aese $tmp0,q15
aese $tmp1,q15
aese $tmp2,q15
veor $in0,$in0,$tmp0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q8},[$key_],#16 // re-pre-load rndkey[0]
vst1.8 {$in0},[$out],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
veor $in1,$in1,$tmp1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
mov $cnt,$rounds
vst1.8 {$in1},[$out],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
veor $in2,$in2,$tmp2
vld1.32 {q9},[$key_],#16 // re-pre-load rndkey[1]
vst1.8 {$in2},[$out],#16
b.hs .Loop3x_ctr32
adds $len,$len,#3
b.eq .Lctr32_done
cmp $len,#1
mov $step,#16
cclr $step,eq
.Lctr32_tail:
aese $dat0,q8
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q8},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
subs $cnt,$cnt,#2
aese $dat0,q9
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.32 {q9},[$key_],#16
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
b.gt .Lctr32_tail
aese $dat0,q8
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q8
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
aese $dat0,q9
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q9
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
vld1.8 {$in0},[$inp],$step
aese $dat0,q12
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q12
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
vld1.8 {$in1},[$inp]
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aese $dat0,q13
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q13
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
veor $in0,$in0,$rndlast
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat0,q14
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat0,$dat0
aes/asm/aesv8-armx.pl: optimize for Cortex-A5x. ARM has optimized Cortex-A5x pipeline to favour pairs of complementary AES instructions. While modified code improves performance of post-r0p0 Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts original r0p0. We favour later revisions, because one can't prevent future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%, while new code is not slower on r0p0, or Apple A7 for that matter. [Update even SHA results for latest Cortex-A53.] (Imported from upstream's 94376cccb4ed5b376220bffe0739140ea9dad8c8) Change-Id: I581c65b566116b1f4211fb1bd5a1a54479889d70 Reviewed-on: https://boringssl-review.googlesource.com/4481 Reviewed-by: Adam Langley <agl@google.com>
2015-04-20 22:40:54 +01:00
aese $dat1,q14
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
aesmc $dat1,$dat1
veor $in1,$in1,$rndlast
aese $dat0,q15
aese $dat1,q15
cmp $len,#1
veor $in0,$in0,$dat0
veor $in1,$in1,$dat1
vst1.8 {$in0},[$out],#16
b.eq .Lctr32_done
vst1.8 {$in1},[$out]
.Lctr32_done:
___
$code.=<<___ if ($flavour !~ /64/);
vldmia sp!,{d8-d15}
ldmia sp!,{r4-r10,pc}
___
$code.=<<___ if ($flavour =~ /64/);
ldr x29,[sp],#16
ret
___
$code.=<<___;
.size ${prefix}_ctr32_encrypt_blocks,.-${prefix}_ctr32_encrypt_blocks
___
}}}
$code.=<<___;
#endif
___
########################################
if ($flavour =~ /64/) { ######## 64-bit code
my %opcode = (
"aesd" => 0x4e285800, "aese" => 0x4e284800,
"aesimc"=> 0x4e287800, "aesmc" => 0x4e286800 );
local *unaes = sub {
my ($mnemonic,$arg)=@_;
$arg =~ m/[qv]([0-9]+)[^,]*,\s*[qv]([0-9]+)/o &&
sprintf ".inst\t0x%08x\t//%s %s",
$opcode{$mnemonic}|$1|($2<<5),
$mnemonic,$arg;
};
foreach(split("\n",$code)) {
s/\`([^\`]*)\`/eval($1)/geo;
s/\bq([0-9]+)\b/"v".($1<8?$1:$1+8).".16b"/geo; # old->new registers
s/@\s/\/\//o; # old->new style commentary
#s/[v]?(aes\w+)\s+([qv].*)/unaes($1,$2)/geo or
s/cclr\s+([wx])([^,]+),\s*([a-z]+)/csel $1$2,$1zr,$1$2,$3/o or
s/mov\.([a-z]+)\s+([wx][0-9]+),\s*([wx][0-9]+)/csel $2,$3,$2,$1/o or
s/vmov\.i8/movi/o or # fix up legacy mnemonics
s/vext\.8/ext/o or
s/vrev32\.8/rev32/o or
s/vtst\.8/cmtst/o or
s/vshr/ushr/o or
s/^(\s+)v/$1/o or # strip off v prefix
s/\bbx\s+lr\b/ret/o;
Sync up AES assembly. This syncs up with OpenSSL master as of 50ea9d2b3521467a11559be41dcf05ee05feabd6. The non-license non-spelling changes are CFI bits, which were added in upstream in b84460ad3a3e4fcb22efaa0a8365b826f4264ecf. Change-Id: I42280985f834d5b9133eacafc8ff9dbd2f0ea59a Reviewed-on: https://boringssl-review.googlesource.com/25704 Reviewed-by: Adam Langley <agl@google.com>
2018-02-03 00:12:10 +00:00
# fix up remaining legacy suffixes
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
s/\.[ui]?8//o;
m/\],#8/o and s/\.16b/\.8b/go;
s/\.[ui]?32//o and s/\.16b/\.4s/go;
s/\.[ui]?64//o and s/\.16b/\.2d/go;
s/\.[42]([sd])\[([0-3])\]/\.$1\[$2\]/o;
print $_,"\n";
}
} else { ######## 32-bit code
my %opcode = (
"aesd" => 0xf3b00340, "aese" => 0xf3b00300,
"aesimc"=> 0xf3b003c0, "aesmc" => 0xf3b00380 );
local *unaes = sub {
my ($mnemonic,$arg)=@_;
if ($arg =~ m/[qv]([0-9]+)[^,]*,\s*[qv]([0-9]+)/o) {
my $word = $opcode{$mnemonic}|(($1&7)<<13)|(($1&8)<<19)
|(($2&7)<<1) |(($2&8)<<2);
# since ARMv7 instructions are always encoded little-endian.
# correct solution is to use .inst directive, but older
# assemblers don't implement it:-(
sprintf ".byte\t0x%02x,0x%02x,0x%02x,0x%02x\t@ %s %s",
$word&0xff,($word>>8)&0xff,
($word>>16)&0xff,($word>>24)&0xff,
$mnemonic,$arg;
}
};
sub unvtbl {
my $arg=shift;
$arg =~ m/q([0-9]+),\s*\{q([0-9]+)\},\s*q([0-9]+)/o &&
sprintf "vtbl.8 d%d,{q%d},d%d\n\t".
Remove trailing whitespace from Perl files. Upstream did this in 609b0852e4d50251857dbbac3141ba042e35a9ae and it's easier to apply patches if we do also. Change-Id: I5142693ed1e26640987ff16f5ea510e81bba200e Reviewed-on: https://boringssl-review.googlesource.com/13771 Commit-Queue: Adam Langley <agl@google.com> Reviewed-by: David Benjamin <davidben@google.com>
2017-02-09 20:21:08 +00:00
"vtbl.8 d%d,{q%d},d%d", 2*$1,$2,2*$3, 2*$1+1,$2,2*$3+1;
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
}
sub unvdup32 {
my $arg=shift;
$arg =~ m/q([0-9]+),\s*q([0-9]+)\[([0-3])\]/o &&
Remove trailing whitespace from Perl files. Upstream did this in 609b0852e4d50251857dbbac3141ba042e35a9ae and it's easier to apply patches if we do also. Change-Id: I5142693ed1e26640987ff16f5ea510e81bba200e Reviewed-on: https://boringssl-review.googlesource.com/13771 Commit-Queue: Adam Langley <agl@google.com> Reviewed-by: David Benjamin <davidben@google.com>
2017-02-09 20:21:08 +00:00
sprintf "vdup.32 q%d,d%d[%d]",$1,2*$2+($3>>1),$3&1;
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
}
sub unvmov32 {
my $arg=shift;
$arg =~ m/q([0-9]+)\[([0-3])\],(.*)/o &&
Remove trailing whitespace from Perl files. Upstream did this in 609b0852e4d50251857dbbac3141ba042e35a9ae and it's easier to apply patches if we do also. Change-Id: I5142693ed1e26640987ff16f5ea510e81bba200e Reviewed-on: https://boringssl-review.googlesource.com/13771 Commit-Queue: Adam Langley <agl@google.com> Reviewed-by: David Benjamin <davidben@google.com>
2017-02-09 20:21:08 +00:00
sprintf "vmov.32 d%d[%d],%s",2*$1+($2>>1),$2&1,$3;
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
}
foreach(split("\n",$code)) {
s/\`([^\`]*)\`/eval($1)/geo;
s/\b[wx]([0-9]+)\b/r$1/go; # new->old registers
s/\bv([0-9])\.[12468]+[bsd]\b/q$1/go; # new->old registers
s/\/\/\s?/@ /o; # new->old style commentary
Sync up AES assembly. This syncs up with OpenSSL master as of 50ea9d2b3521467a11559be41dcf05ee05feabd6. The non-license non-spelling changes are CFI bits, which were added in upstream in b84460ad3a3e4fcb22efaa0a8365b826f4264ecf. Change-Id: I42280985f834d5b9133eacafc8ff9dbd2f0ea59a Reviewed-on: https://boringssl-review.googlesource.com/25704 Reviewed-by: Adam Langley <agl@google.com>
2018-02-03 00:12:10 +00:00
# fix up remaining new-style suffixes
aarch64 support. This is an initial cut at aarch64 support. I have only qemu to test it however—hopefully hardware will be coming soon. This also affects 32-bit ARM in that aarch64 chips can run 32-bit code and we would like to be able to take advantage of the crypto operations even in 32-bit mode. AES and GHASH should Just Work in this case: the -armx.pl files can be built for either 32- or 64-bit mode based on the flavour argument given to the Perl script. SHA-1 and SHA-256 don't work like this however because they've never support for multiple implementations, thus BoringSSL built for 32-bit won't use the SHA instructions on an aarch64 chip. No dedicated ChaCha20 or Poly1305 support yet. Change-Id: Ib275bc4894a365c8ec7c42f4e91af6dba3bd686c Reviewed-on: https://boringssl-review.googlesource.com/2801 Reviewed-by: Adam Langley <agl@google.com>
2015-01-09 23:44:37 +00:00
s/\{q([0-9]+)\},\s*\[(.+)\],#8/sprintf "{d%d},[$2]!",2*$1/eo or
s/\],#[0-9]+/]!/o;
s/[v]?(aes\w+)\s+([qv].*)/unaes($1,$2)/geo or
s/cclr\s+([^,]+),\s*([a-z]+)/mov$2 $1,#0/o or
s/vtbl\.8\s+(.*)/unvtbl($1)/geo or
s/vdup\.32\s+(.*)/unvdup32($1)/geo or
s/vmov\.32\s+(.*)/unvmov32($1)/geo or
s/^(\s+)b\./$1b/o or
s/^(\s+)mov\./$1mov/o or
s/^(\s+)ret/$1bx\tlr/o;
print $_,"\n";
}
}
close STDOUT;
Reference in New Issue Copy Permalink