Some micro-perf-optimizations#55
Conversation
The (uint)-cast will bring any negative number in the range > int.MaxValue, so it will be outside '9' - '0'. See https://sharplab.io/#v2:EYLgxg9gTgpgtADwGwBYA0AXEBDAzgWwB8ABAJgEYBYAKGIGYACMhgYQYG8aHuGBLAOwwwo/bABsm5JA2AQIE3rgD6AoQHNhACjAALbFAZgAlAwC8APgba9UI9oZwGAcgAMTkwB5TzgJxOHzm4A3Fw8qsKiEsRSMnIKyuEaUOTW+oYmFlYArqp2YAGu7gxevv6OhSHUPDQAvkA==
…check that isn't needed As the copy-length is <= 8 bytes, this check may be dominant, so get rid of it. Cf. https://github.com/dotnet/runtime/blob/8a52f1e948b6f22f418817ec1068f07b8dae2aa5/src/libraries/System.Private.CoreLib/src/System/Buffer.cs#L120-L127
gfoidl
left a comment
gfoidl
left a comment
There was a problem hiding this comment.
Some notes for review.
| return Unsafe.AddByteOffset(ref tableRef, (IntPtr)n); | ||
| } | ||
|
|
||
| private static ReadOnlySpan<byte> powersTable => new byte[19] { |
There was a problem hiding this comment.
Note for new byte[19]: the count isn't necessary, but sometime I made the mistake to enter to less / to much elements in the array and the compiler won't complain. With the count the compiler complains if the element count doesn't match. So it's a little helper.
| { | ||
| Buffer.MemoryCopy(&word, &d, sizeof(double), sizeof(double)); | ||
| } | ||
| Unsafe.CopyBlockUnaligned(&d, &word, sizeof(double)); |
There was a problem hiding this comment.
Buffer.MemoryCopy does an argument-check, that will be dominant with the ptr-size copy. So this method is used, that doesn't perform argument validation.
| { | ||
| internal unsafe struct DigitsBuffer | ||
| { | ||
| private fixed byte digits[(int)Constants.max_digits]; |
There was a problem hiding this comment.
This is C#'s fixed sized buffer that can be accessed withou bound-checks.
But it's stack-allocated with 768 bytes. I'm not sure if this can result in any problems if other methods in the call-stack already rely on heavy stack-allocation. I assume for standard machines this won't be a problem, and as the discussion over in .NET-repo for integrating this algorithm looks like for Wasm / Blazor this may not be used in favor of a size-opt variant.
Alternatively we can heap allocate in the POH (pinned object heap, as special segment from the GC that pre-pins objects), so we could expose the pointer from here.
But that's an heap-allocation which should be avoided if possible, as there's a cost at collecting the "garbage".
There was a problem hiding this comment.
@gfoidl If I understand correctly, you replaced digits = new max_digits[...] with a stack-allocated struct, using the fact that max_digits is a compile-time constant. Of course, that's how the C++ equivalent is done.
This is a bit like the readonly byte span that were introduced elsewhere... that is, it is a way to avoid heap allocations.
I do hope that it should be fine. This makes the whole thing "never allocating" and you can compute exactly how much memory you do use (in the worst case).
I am not sure why that would be a problem with WASM... With WASM, if you know precisely the memory usage (as a fixed quantity), you are golden. Aren't you?
There was a problem hiding this comment.
You understood correct!
With WASM, if you know precisely the memory usage
But I don't know WASM, so that's my problem 😉 (expect form wikipedia).
There was a problem hiding this comment.
could you make the indexers simpler by using something like::
public ref byte this[uint index]
{
get
{
Debug.Assert(index < Constants.max_digits);
return ref digits[index];
}
}There was a problem hiding this comment.
I thought about this, but didn't follow that approach and didn't measure it.
My fear was that we may hit some JIT-limits, if the ref byte will be used in some non-direct ways like casting. That's why the canonical way was gone.
But maybe my fear is without merit -- feel free to validate it.
4a0f92f to
caaccaa
Compare
|
That's very impressive !!!! |
| internal static ulong get_power_of_five_128(int index) | ||
| { | ||
| #if NET5_0 | ||
| Debug.Assert(index < power_of_five_128.Length); |
There was a problem hiding this comment.
This would only kick in debug mode, right?
There was a problem hiding this comment.
Yep, it's removed in Release-builds by the compiler.
Here it's just as safety-net to get a crash in debug-build if something went wrong and would access outside of bounds.
|
I expect that the sizeable performance gains can be attributed to two reasons: |
|
I didn't measure each step, but from gut feeling I'd say
* the branch predictor will do a good job -- at least I assume so -- but tighter loops may be a reason for the boost, maybe we are lucky and the loops will be properly aligned. I don't know the current loop-alignment rules in the JIT, and there were some recent changes, so I'm not up to date here. ** maybe I'm wrong here and the effect is greater, as you assume. It compiles down to mem -> reg -> mem, so ideal. |
|
Ah, almost forgot. Removing the bound-checks has another effect. Would need to investigate how much this effect is actually. |
| } | ||
|
|
||
| return d; | ||
| return BitConverter.Int64BitsToDouble((long)word); |
There was a problem hiding this comment.
This is even better than the previous Unsafe.CopyBlockUnaligned as this boils down to vmovq xmm0, rdx and avoid the stack-usage (a fresh head unleashes more thought on this 😉).
It's only possible for the 64-bit values. On the float-counterpart I used Unsafe.Copy now, as codegen seems a bit better (not measurable).
They are just slow bloat, and aren't emitted by these "meaningless" casts.
gfoidl
changed the title
|
Next perf steps would be looking into it with VTune, PerfView, etc. |
VTune
ParsedNumberString.ParseNumberStringIf I'm not off it's
FastDoubleParser.ComputeFloat
FastDoubleParser.ParseNumber
|
| // able to optimize it well. | ||
| [MethodImpl(MethodImplOptions.AggressiveInlining)] | ||
| internal static bool is_integer(char c) => (char)(c - '0') <= '9' - '0'; | ||
| internal static bool is_integer(char c, out uint cMinus0) |
There was a problem hiding this comment.
The re-use of `cMinus0' produces better code.
Before (a loop):
M03_L25:
cmp rbx,13
jle near ptr M03_L33
mov byte ptr [rsp+12],1
xor r11d,r11d
mov rdx,r10
jmp short M03_L27
M03_L26:
lea r11,[r11+r11*4]
add eax,0FFFFFFD0
lea r11,[rax+r11*2]
add rdx,2
mov rax,0B6B3A7640000
cmp r11,rax
jae short M03_L28
M03_L27:
cmp rdx,r8
je short M03_L28
movzx eax,word ptr [rdx]
mov r9d,eax
add r9d,0FFFFFFD0
cmp r9d,9
jbe short M03_L26With this change:
M03_L22:
cmp rdi,13
jle near ptr M03_L29
mov byte ptr [rsp+12],1
xor r11d,r11d
mov rdx,r10
jmp short M03_L24
M03_L23:
mov rax,0B6B3A7640000
cmp r11,rax
jae short M03_L25
M03_L24:
cmp rdx,r8
je short M03_L25
movzx eax,word ptr [rdx]
add eax,0FFFFFFD0
cmp eax,9
ja short M03_L25
lea r11,[r11+r11*4]
mov eax,eax
lea r11,[rax+r11*2]
add rdx,2
jmp short M03_L23So one add eax,0FFFFFFD0 is eliminated and code layed out a bit different.
Need to re-run the benchmarks later today.
There was a problem hiding this comment.
@gfoidl : just a side question please: May I ask how you've setuped vTune to display source code and JIT side by side. I've tried without any luck. It always complains about my .pdb(s). Maybe it's the version I have 2021.1....
Integration within Visual Studio 2019 is also a problem and Intel team are working on it (I wrote to Intel community support on that).
There was a problem hiding this comment.
I have VTune 2020 Update 3 -- didn't know that there's a newer version.
The key points are
- set the application directory to the build-output dir, as there are the pdb(s) too
- set binaries and sources (marked with the arrow) in the "configure analysis" screen
(Please apologize that I didn't use your name in the path...)
In the collection log it display some warnings about pdbs too, but so far this never was a problem.
Then in Bottom-up screen double-click on a function (or choose "View Source" from the context menu).
Hitting the "Assembly"-button shows it side by side:
If you ran into troubles, don't hesitate to ask.
Integration within Visual Studio 2019 is also a problem
I tried this some time ago, but since then I use VTune as standalone app.
BTW: the profiler in Visual Studio is really good for finding hot-methods -- especially in larger applications.
There was a problem hiding this comment.
i'll give it another try. Maybe I haven't specified enough source directories....
|
@gfoidl You would think that the Well. We are talking about ~5% to ~15% so it is not like these sort of optimizations have fully predictable results from first principles. (Note : ~5% to ~15% is huge for the use case here. I am not dismissing the achievement.) |
|
@CarlVerret I recommend merging. The |
|
I'm just running benchmarks, looks promising. The
😉 On the other side for me it's quite obvious, but I'm working with C# / .NET for quite a long time, so some of the inner workings are familiar to me. Overall the biggest player for the perf-improvements is the removal of bound-checks. |
|
Latest numbers: |
You might be pessimistic in the value of your optimization. I do not not have the numbers for C#, but in C++, this code is at about 280 instructions per float... so saving, say, 20 instructions, would be 5% to 10% of the total instruction count. Of course, instruction counts do not translate into better performance... normally... but if you look at the number of instructions retired per cycle for this code, in C++, we pretty much max out the width of the processor (>=4 ins. per cycle on Intel/AMD, 8 ins. per cycle on ARM M1). Look at the If we were assumed that it was ASCII instead of the pesky UTF-16, we could go faster even ! |
Let us take canada. You went from 6.849 ms to 5.985 ms a saving of 0.86 ms. The bound checking part should not benefit ParseNumberString and it went from 3.319 ms to 3.672 ms, a saving of 0.35 ms. So I would expect that 0.5 ms of the gain (or ~2/3) is due to the unsafe copy and the non-bound-checking exact_power_of_ten. Now, mesh is going to benefit much more from the non-bound-checking exact_power_of_ten because it relies entirely on the fast path. Yet of the two sets, you see the larger benefits with canada (14%) as opposed to mesh (9%). In any case, this is fantastic work. People often say that bound checking is free, and you are definitively showing that it is not case. |
|
@gfoidl Can you sync with the main branch? |
|
For reference: asm generated for `FastDoubleParser.ParseDouble` with canada; csFastFloat.Benchmark.FFBencmark.FastParser_()
push rdi
push rsi
push rbx
sub rsp,30
vzeroupper
vmovaps [rsp+20],xmm6
vmovsd xmm6,qword ptr [7FFA9B380AB0]
mov rsi,[rcx+8]
xor edi,edi
mov ebx,[rsi+8]
test ebx,ebx
jle short M00_L03
M00_L00:
movsxd rcx,edi
mov rcx,[rsi+rcx*8+10]
mov edx,3
mov r8d,2E
call csFastFloat.FastDoubleParser.ParseDouble(System.String, csFastFloat.Enums.chars_format, Char)
vucomisd xmm0,xmm6
ja short M00_L01
jmp short M00_L02
M00_L01:
vmovaps xmm6,xmm0
M00_L02:
inc edi
cmp ebx,edi
jg short M00_L00
M00_L03:
vmovaps xmm0,xmm6
vmovaps xmm6,[rsp+20]
add rsp,30
pop rbx
pop rsi
pop rdi
ret
; Total bytes of code 97
; csFastFloat.FastDoubleParser.ParseDouble(System.String, csFastFloat.Enums.chars_format, Char)
sub rsp,38
vzeroupper
xor eax,eax
mov [rsp+30],rax
mov eax,edx
test rcx,rcx
je short M01_L00
lea r9,[rcx+0C]
mov [rsp+30],r9
mov r9,[rsp+30]
mov ecx,[rcx+8]
mov [rsp+28],r9
lea rdx,[r9+rcx*2]
movzx r9d,r8w
mov rcx,[rsp+28]
mov r8d,eax
call csFastFloat.FastDoubleParser.ParseNumber(Char*, Char*, csFastFloat.Enums.chars_format, Char)
nop
add rsp,38
ret
M01_L00:
call csFastFloat.FastDoubleParser.<ParseDouble>g__ThrowArgumentNull|4_0()
int 3
; Total bytes of code 76
; csFastFloat.FastDoubleParser.ParseNumber(Char*, Char*, csFastFloat.Enums.chars_format, Char)
push r14
push rdi
push rsi
push rbp
push rbx
sub rsp,6B0
vzeroupper
mov rsi,rcx
mov rdi,rdx
mov eax,r8d
mov ebx,r9d
jmp short M02_L01
M02_L00:
add rsi,2
M02_L01:
cmp rsi,rdi
je short M02_L02
movzx ecx,byte ptr [rsi]
mov rdx,1783F046884
cmp byte ptr [rcx+rdx],0
jne short M02_L00
M02_L02:
cmp rsi,rdi
je near ptr M02_L08
mov dword ptr [rsp+20],2E
lea rcx,[rsp+698]
mov rdx,rsi
mov r8,rdi
mov r9d,eax
call csFastFloat.Structures.ParsedNumberString.ParseNumberString(Char*, Char*, csFastFloat.Enums.chars_format, Char)
cmp byte ptr [rsp+6A9],0
jne short M02_L03
mov rcx,rsi
mov rdx,rdi
call csFastFloat.FastDoubleParser.HandleInvalidInput(Char*, Char*)
nop
add rsp,6B0
pop rbx
pop rbp
pop rsi
pop rdi
pop r14
ret
M02_L03:
cmp qword ptr [rsp+698],0FFFFFFFFFFFFFFEA
jl short M02_L04
cmp qword ptr [rsp+698],16
jg short M02_L04
mov rcx,0
cmp [rsp+6A0],rcx
ja short M02_L04
cmp byte ptr [rsp+6AA],0
jne short M02_L04
vmovdqu xmm0,xmmword ptr [rsp+698]
vmovdqu xmmword ptr [rsp+350],xmm0
mov rcx,[rsp+6A8]
mov [rsp+360],rcx
lea rcx,[rsp+350]
call csFastFloat.FastDoubleParser.FastPath(csFastFloat.Structures.ParsedNumberString)
nop
add rsp,6B0
pop rbx
pop rbp
pop rsi
pop rdi
pop r14
ret
M02_L04:
lea rcx,[rsp+688]
mov rdx,[rsp+698]
mov r8,[rsp+6A0]
call csFastFloat.FastDoubleParser.ComputeFloat(Int64, UInt64)
cmp byte ptr [rsp+6AA],0
je short M02_L06
mov rbp,[rsp+688]
mov r14d,[rsp+690]
mov r8,[rsp+6A0]
inc r8
lea rcx,[rsp+678]
mov rdx,[rsp+698]
call csFastFloat.FastDoubleParser.ComputeFloat(Int64, UInt64)
mov r9,[rsp+678]
mov ecx,[rsp+680]
cmp rbp,r9
jne short M02_L05
cmp r14d,ecx
setne r9b
movzx r9d,r9b
test r9d,r9d
je short M02_L06
M02_L05:
mov dword ptr [rsp+690],0FFFFFFFF
M02_L06:
cmp dword ptr [rsp+690],0
jge short M02_L07
movzx r9d,bx
lea rcx,[rsp+368]
mov rdx,rsi
mov r8,rdi
call csFastFloat.Structures.DecimalInfo.parse_decimal(Char*, Char*, Char)
lea rsi,[rsp+688]
lea rcx,[rsp+40]
lea rdx,[rsp+368]
mov r8d,30C
call CORINFO_HELP_MEMCPY
mov rcx,rsi
lea rdx,[rsp+40]
call csFastFloat.FastDoubleParser.ComputeFloat(csFastFloat.Structures.DecimalInfo)
M02_L07:
movzx ecx,byte ptr [rsp+6A8]
vmovupd xmm0,[rsp+688]
vmovupd [rsp+30],xmm0
lea rdx,[rsp+30]
call csFastFloat.FastDoubleParser.ToFloat(Boolean, csFastFloat.Structures.AdjustedMantissa)
nop
add rsp,6B0
pop rbx
pop rbp
pop rsi
pop rdi
pop r14
ret
M02_L08:
call csFastFloat.FastDoubleParser.ThrowArgumentException()
int 3
; Total bytes of code 513
; csFastFloat.Structures.ParsedNumberString.ParseNumberString(Char*, Char*, csFastFloat.Enums.chars_format, Char)
push r15
push r14
push rdi
push rsi
push rbp
push rbx
sub rsp,18
vzeroupper
mov eax,[rsp+70]
xor r10d,r10d
vxorps xmm0,xmm0,xmm0
vmovdqu xmmword ptr [rsp],xmm0
mov [rsp+10],r10
movzx r10d,word ptr [rdx]
cmp r10d,2D
sete r11b
movzx r11d,r11b
mov [rsp+10],r11b
cmp r10d,2D
je short M03_L00
cmp r10d,2B
jne short M03_L02
M03_L00:
add rdx,2
cmp rdx,r8
jne short M03_L01
vmovdqu xmm0,xmmword ptr [rsp]
vmovdqu xmmword ptr [rcx],xmm0
mov rdx,[rsp+10]
mov [rcx+10],rdx
jmp near ptr M03_L30
M03_L01:
movzx r10d,word ptr [rdx]
mov r11d,r10d
add r11d,0FFFFFFD0
cmp r11d,9
jbe short M03_L02
movzx r11d,ax
cmp r10d,r11d
je short M03_L02
vmovdqu xmm0,xmmword ptr [rsp]
vmovdqu xmmword ptr [rcx],xmm0
mov rdx,[rsp+10]
mov [rcx+10],rdx
jmp near ptr M03_L30
M03_L02:
mov r10,rdx
xor r11d,r11d
M03_L03:
cmp rdx,r8
je short M03_L04
movzx esi,word ptr [rdx]
add esi,0FFFFFFD0
cmp esi,9
ja short M03_L04
lea r11,[r11+r11*4]
mov esi,esi
lea r11,[rsi+r11*2]
add rdx,2
jmp short M03_L03
M03_L04:
mov rsi,rdx
mov rdi,rsi
sub rdi,r10
mov rbx,rdi
shr rbx,3F
add rdi,rbx
sar rdi,1
xor ebx,ebx
cmp rsi,r8
je short M03_L07
movzx ebp,word ptr [rsi]
movzx r14d,ax
cmp ebp,r14d
jne short M03_L07
lea rdx,[rsi+2]
M03_L05:
cmp rdx,r8
je short M03_L06
movzx ebx,word ptr [rdx]
add ebx,0FFFFFFD0
cmp ebx,9
ja short M03_L06
movzx ebx,bl
add rdx,2
lea r11,[r11+r11*4]
mov ebx,ebx
lea r11,[rbx+r11*2]
jmp short M03_L05
M03_L06:
lea rbx,[rsi+2]
sub rbx,rdx
mov rbp,rbx
shr rbp,3F
add rbx,rbp
sar rbx,1
sub rdi,rbx
M03_L07:
test rdi,rdi
jne short M03_L08
vmovdqu xmm0,xmmword ptr [rsp]
vmovdqu xmmword ptr [rcx],xmm0
mov r9,[rsp+10]
mov [rcx+10],r9
jmp near ptr M03_L30
M03_L08:
xor ebp,ebp
mov r14d,r9d
and r14d,1
cmp r14d,1
jne near ptr M03_L17
cmp rdx,r8
je near ptr M03_L17
movzx r15d,word ptr [rdx]
cmp r15d,65
je short M03_L09
cmp r15d,45
jne near ptr M03_L17
M03_L09:
add rdx,2
xor r14d,r14d
cmp rdx,r8
je short M03_L10
cmp word ptr [rdx],2D
jne short M03_L10
mov r14d,1
add rdx,2
jmp short M03_L11
M03_L10:
cmp rdx,r8
je short M03_L11
cmp word ptr [rdx],2B
jne short M03_L11
add rdx,2
M03_L11:
cmp rdx,r8
je short M03_L12
movzx r15d,word ptr [rdx]
add r15d,0FFFFFFD0
cmp r15d,9
jbe short M03_L13
M03_L12:
cmp r9d,2
je near ptr M03_L18
vmovdqu xmm0,xmmword ptr [rsp]
vmovdqu xmmword ptr [rcx],xmm0
mov rdx,[rsp+10]
mov [rcx+10],rdx
jmp near ptr M03_L30
M03_L13:
cmp rdx,r8
je short M03_L15
movzx r9d,word ptr [rdx]
add r9d,0FFFFFFD0
cmp r9d,9
ja short M03_L15
movzx r9d,r9b
cmp rbp,10000
jge short M03_L14
lea rbp,[rbp+rbp*4]
mov r9d,r9d
lea rbp,[r9+rbp*2]
M03_L14:
add rdx,2
jmp short M03_L13
M03_L15:
test r14d,r14d
je short M03_L16
neg rbp
M03_L16:
add rbx,rbp
jmp short M03_L18
M03_L17:
cmp r14d,1
jne short M03_L18
test r9b,2
jne short M03_L18
vmovdqu xmm0,xmmword ptr [rsp]
vmovdqu xmmword ptr [rcx],xmm0
mov r10,[rsp+10]
mov [rcx+10],r10
jmp near ptr M03_L30
M03_L18:
mov byte ptr [rsp+11],1
cmp rdi,13
jle near ptr M03_L29
mov rdx,r10
jmp short M03_L21
M03_L19:
cmp r9d,30
jne short M03_L20
dec rdi
M03_L20:
add rdx,2
M03_L21:
cmp rdx,r8
je short M03_L22
movzx r9d,word ptr [rdx]
cmp r9d,30
je short M03_L19
movzx r14d,ax
cmp r9d,r14d
je short M03_L19
M03_L22:
cmp rdi,13
jle near ptr M03_L29
mov byte ptr [rsp+12],1
xor r11d,r11d
mov rdx,r10
jmp short M03_L24
M03_L23:
mov rax,0B6B3A7640000
cmp r11,rax
jae short M03_L25
M03_L24:
cmp rdx,r8
je short M03_L25
movzx eax,word ptr [rdx]
add eax,0FFFFFFD0
cmp eax,9
ja short M03_L25
lea r11,[r11+r11*4]
mov eax,eax
lea r11,[rax+r11*2]
add rdx,2
jmp short M03_L23
M03_L25:
mov rax,0B6B3A7640000
cmp r11,rax
jb short M03_L26
sub rsi,rdx
mov rdx,rsi
shr rdx,3F
add rdx,rsi
sar rdx,1
lea rbx,[rdx+rbp]
jmp short M03_L29
M03_L26:
add rdx,2
mov rbx,rdx
M03_L27:
mov rax,0B6B3A7640000
cmp r11,rax
jae short M03_L28
cmp rdx,r8
je short M03_L28
movzx eax,word ptr [rdx]
add eax,0FFFFFFD0
cmp eax,9
ja short M03_L28
lea r11,[r11+r11*4]
mov eax,eax
lea r11,[rax+r11*2]
add rdx,2
jmp short M03_L27
M03_L28:
sub rbx,rdx
mov rax,rbx
shr rax,3F
add rbx,rax
sar rbx,1
add rbx,rbp
M03_L29:
mov [rsp],rbx
mov [rsp+8],r11
vmovdqu xmm0,xmmword ptr [rsp]
vmovdqu xmmword ptr [rcx],xmm0
mov rax,[rsp+10]
mov [rcx+10],rax
M03_L30:
mov rax,rcx
add rsp,18
pop rbx
pop rbp
pop rsi
pop rdi
pop r14
pop r15
ret
; Total bytes of code 842
; csFastFloat.FastDoubleParser.FastPath(csFastFloat.Structures.ParsedNumberString)
vzeroupper
mov rax,[rcx+8]
vxorps xmm0,xmm0,xmm0
vcvtsi2sd xmm0,xmm0,rax
test rax,rax
jge short M04_L00
vaddsd xmm0,xmm0,qword ptr [7FFA9B3852A0]
M04_L00:
mov rax,[rcx]
test rax,rax
jge short M04_L01
neg rax
mov rdx,1784F1D7998
mov rdx,[rdx]
cmp [rdx],edx
add rdx,10
vdivsd xmm0,xmm0,[rdx+rax*8]
jmp short M04_L02
M04_L01:
mov rdx,1784F1D7998
mov rdx,[rdx]
cmp [rdx],edx
add rdx,10
vmulsd xmm0,xmm0,qword ptr [rdx+rax*8]
M04_L02:
cmp byte ptr [rcx+10],0
je short M04_L03
vmovsd xmm1,qword ptr [7FFA9B3852B0]
vxorps xmm0,xmm0,xmm1
M04_L03:
ret
; Total bytes of code 109
; csFastFloat.FastDoubleParser.ComputeFloat(Int64, UInt64)
push rdi
push rsi
sub rsp,38
vzeroupper
mov rax,rcx
vxorps xmm0,xmm0,xmm0
vmovdqu xmmword ptr [rsp+28],xmm0
test r8,r8
je short M05_L00
cmp rdx,0FFFFFFFFFFFFFEAA
jge short M05_L01
M05_L00:
xor edx,edx
mov [rax],rdx
mov [rax+8],edx
jmp near ptr M05_L13
M05_L01:
cmp rdx,134
jle short M05_L02
mov dword ptr [rsp+30],7FF
xor r8d,r8d
mov [rsp+28],r8
vmovdqu xmm0,xmmword ptr [rsp+28]
vmovdqu xmmword ptr [rax],xmm0
jmp near ptr M05_L13
M05_L02:
xor ecx,ecx
lzcnt rcx,r8
mov r9d,ecx
mov ecx,r9d
shl r8,cl
mov [rsp+58],rdx
lea ecx,[rdx*2+2AC]
mov r10,1784F1D79A8
mov r10,[r10]
cmp [r10],r10d
add r10,10
mov r11d,ecx
mov r10,[r10+r11*8]
lea r11,[rsp+10]
mov rdx,r8
mulx rdx,rsi,r10
mov [r11],rsi
mov r10,[rsp+10]
mov [rsp+18],r10
mov [rsp+20],rdx
mov rdx,[rsp+20]
and edx,1FF
cmp edx,1FF
jne short M05_L03
mov rdx,1784F1D79A8
mov rdx,[rdx]
cmp [rdx],edx
add rdx,10
inc ecx
mov ecx,ecx
mov rcx,[rdx+rcx*8]
lea r10,[rsp+8]
mov rdx,r8
mulx rcx,r11,rcx
mov [r10],r11
lea rdx,[rsp+18]
add [rdx],rcx
cmp rcx,[rsp+18]
jbe short M05_L03
lea rcx,[rsp+20]
inc qword ptr [rcx]
M05_L03:
mov rdx,[rsp+18]
mov r8,[rsp+20]
cmp rdx,0FFFFFFFFFFFFFFFF
jne short M05_L05
mov r10,[rsp+58]
cmp r10,0FFFFFFFFFFFFFFE5
jl short M05_L04
cmp r10,37
setle cl
movzx ecx,cl
test ecx,ecx
mov [rsp+58],r10
jne short M05_L05
M05_L04:
mov dword ptr [rsp+30],0FFFFFFFF
vmovdqu xmm0,xmmword ptr [rsp+28]
vmovdqu xmmword ptr [rax],xmm0
jmp near ptr M05_L13
M05_L05:
mov rcx,r8
shr rcx,3F
mov r11d,ecx
lea esi,[r11+9]
and esi,3F
mov ecx,esi
mov rdi,r8
shr rdi,cl
mov [rsp+28],rdi
mov r10,[rsp+58]
imul ecx,r10d,3526A
sar ecx,10
lea ecx,[rcx+r11+3F]
sub ecx,r9d
add ecx,3FF
mov [rsp+30],ecx
cmp dword ptr [rsp+30],0
jg near ptr M05_L09
mov ecx,[rsp+30]
neg ecx
inc ecx
cmp ecx,40
jl short M05_L06
xor ecx,ecx
mov [rsp+30],ecx
mov [rsp+28],rcx
vmovdqu xmm0,xmmword ptr [rsp+28]
vmovdqu xmmword ptr [rax],xmm0
jmp near ptr M05_L13
M05_L06:
lea rdx,[rsp+28]
shr qword ptr [rdx],cl
lea r10,[rsp+28]
mov rcx,[rsp+28]
and rcx,1
add [r10],rcx
lea rcx,[rsp+28]
shr qword ptr [rcx],1
lea rcx,[rsp+28]
mov r8,0
cmp [rsp+28],r8
jb short M05_L07
mov edx,1
jmp short M05_L08
M05_L07:
xor edx,edx
M05_L08:
mov [rcx+8],edx
vmovdqu xmm0,xmmword ptr [rsp+28]
vmovdqu xmmword ptr [rax],xmm0
jmp near ptr M05_L13
M05_L09:
cmp rdx,1
ja short M05_L10
cmp r10,0FFFFFFFFFFFFFFFC
jl short M05_L10
cmp r10,17
jg short M05_L10
mov rcx,[rsp+28]
and ecx,3
cmp ecx,1
jne short M05_L10
mov rdx,[rsp+28]
mov ecx,esi
shl rdx,cl
cmp rdx,r8
jne short M05_L10
lea rdx,[rsp+28]
and qword ptr [rdx],0FFFFFFFFFFFFFFFE
M05_L10:
lea rdx,[rsp+28]
mov rcx,[rsp+28]
and rcx,1
add [rdx],rcx
lea rdx,[rsp+28]
shr qword ptr [rdx],1
mov rdx,0
cmp [rsp+28],rdx
jb short M05_L11
mov rdx,0
mov [rsp+28],rdx
lea rdx,[rsp+30]
inc dword ptr [rdx]
M05_L11:
lea rdx,[rsp+28]
mov rcx,0FFFFFFFFFFFF
and [rdx],rcx
cmp dword ptr [rsp+30],7FF
jl short M05_L12
mov dword ptr [rsp+30],7FF
xor edx,edx
mov [rsp+28],rdx
M05_L12:
vmovdqu xmm0,xmmword ptr [rsp+28]
vmovdqu xmmword ptr [rax],xmm0
M05_L13:
add rsp,38
pop rsi
pop rdi
ret
; Total bytes of code 703
; csFastFloat.FastDoubleParser.ToFloat(Boolean, csFastFloat.Structures.AdjustedMantissa)
vzeroupper
mov rax,[rdx]
mov edx,[rdx+8]
shl rdx,34
or rax,rdx
test cl,cl
jne short M06_L00
mov rdx,rax
jmp short M06_L01
M06_L00:
mov rdx,0
or rdx,rax
M06_L01:
vmovq xmm0,rdx
ret
; Total bytes of code 44Method was not JITted yet. There's some room for improvement (on the JIT-side). Some stack-work could be reduced (note: there are already work-items addressing this). |
| Buffer.MemoryCopy(&word, &d, sizeof(float), sizeof(float)); | ||
| } | ||
| float d = 0; | ||
| Unsafe.Copy(ref d, &word); |
There was a problem hiding this comment.
could we instead use float d = Unsafe.ReadUnaligned<float>(&word);
It generates less code.
There was a problem hiding this comment.
Note that I have been using Unsafe.ReadUnaligned in the ASCII PR.
There was a problem hiding this comment.
AFAIK it generates the same machine code. C# code is less, and that would be the obvious method to use (the whole copy-thing made me forgot to switch to this, trees and forests...).
4 participants
Skimmed through the code base (really impressive algorithm) and fixed some low hanging fruits which jumped into my eye.
Basically it's exercising some C# and JIT-tricks, as well some new APIs in .NET 5, especially for array-accesses where the bound-checks are avoided. They can sum-up if done for each access to an array or span.
Please see individual commits, they have links in the message that help to explain / demonstrate the difference in generated asm-code.
Benchmarks
This PR + #54
Based on caaccaa with #54 merged into that commit.
Current master
Current master is 788081c