Skip to content

Fill in the neglected bitmap commands: BITFIELD, and an open-ended BITPOS - #3196

Merged
mgravell merged 7 commits into
mainfrom
marc/bitops
Aug 25, 2026
Merged

mgravell merged 7 commits into
mainfrom
marc/bitops

Conversation

@mgravell

@mgravell mgravell commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator

Two gaps in the bitmap family. Of the seven commands in that group, five were already covered (SETBIT, GETBIT, BITCOUNT, BITPOS, BITOP - the latter already current with the 8.2 DIFF/DIFF1/ANDOR/ONE additions from #2900). This adds the missing two, and fixes one thing BITPOS could not say.

BITFIELD / BITFIELD_RO

Previously unimplemented; the only trace in the repo was a commented-out line in the envoy exclusion list.

long?         StringBitField(RedisKey key, BitFieldOperation operation, CommandFlags flags = CommandFlags.None);
Lease<long?>  StringBitField(RedisKey key, ReadOnlyMemory<BitFieldOperation> operations, CommandFlags flags = CommandFlags.None);
// ... plus StringBitFieldAsync equivalents

with four new value types:

BitFieldEncoding.Signed(int) / Unsigned(int)
BitFieldEncoding.Int8/Int16/Int32/Int64, UInt8/UInt16/UInt32/UInt63   // + Width, IsSigned
BitFieldOffset.Bits(long) / Index(long)                              // Index is the server's #N form
BitFieldOverflow.Wrap | Saturate | Fail
BitFieldOperation.Get(enc, off) / Set(enc, off, value, overflow = Wrap) / IncrementBy(...)

Usage:

db.StringBitField(key, BitFieldOperation.IncrementBy(BitFieldEncoding.Int8, 0, 1));

using var results = db.StringBitField(key, new[]
{
    BitFieldOperation.Set(BitFieldEncoding.UInt8, BitFieldOffset.Index(3), 255),
    BitFieldOperation.IncrementBy(BitFieldEncoding.Int16, 64, -1, BitFieldOverflow.Saturate),
    BitFieldOperation.Get(BitFieldEncoding.UInt63, 0),
});

Design notes

  • Signedness belongs with the width, not in a flags enum. u8 is one wire token and one semantic unit; overflow is a separate, sticky, kind-scoped concept that GET ignores entirely. A single flags bag would permit Signed | Fail on a Get, where half is meaningless. A named encoding type also stops (width, offset) transposition, both being bare integers otherwise.
  • #N belongs on the offset, for the same reason: a flag that reinterprets a different argument is the hazard, not the ceremony.
  • Results are plain long. Every legal encoding is i1-i64 or u1-u63, and u63's maximum is exactly long.MaxValue - which is why u64 does not exist (the reply is a signed 64-bit integer). So there is nothing to disambiguate and no need for a struct with signed/unsigned accessors. Elements are nullable only because OVERFLOW FAIL reports a nil for the operation it skipped; the batch itself is non-null - the reply is always an array with one element per operation, so an empty lease covers both an empty batch and a non-observable result such as fire-and-forget.
  • There is a named UInt63 where UInt64 would sit. The double-take is the point: it documents the ceiling exactly where someone reaches for the obvious thing, and Unsigned(64) throws with a message naming UInt63 and Int64 as the alternatives.
  • OVERFLOW is emitted only when it changes, and never for a leading run of WRAP (the server default). The sticky state is neither reset nor consumed by an intervening GET, so the tracker carries across them.
  • All-GET batches go out as BITFIELD_RO when the selected server is 6.0+ and the command is not disabled/renamed, so a replica can serve them; anything that writes stays as BITFIELD, which is primary-only (the server treats BITFIELD as a write even when every sub-operation is a GET - hence BITFIELD_RO existing at all). Write ops are detected client-side, so BITFIELD_RO is never issued with something it would reject.
  • Retry category follows the payload, on a separate axis from routing: ReadOnly when every sub-operation is a GET (whichever of the two commands is issued - the server treating BITFIELD as a write governs which servers accept it, not whether replaying is safe), WriteLastWins when there is no INCRBY (SET is positional, so a replay lands on the same value), and WriteAccumulating otherwise. Follows the existing "demoted where we can see the args" pattern from XADD.
  • An empty batch short-circuits locally to an empty lease - no round-trip.
  • A default(BitFieldEncoding) is rejected at operation construction and in the writer, so a default(BitFieldOperation) sitting in an unfilled array cannot reach the wire as i0.

Not included, deliberately - both non-breaking to add later if wanted: scalar StringBitFieldGet/Set/Increment helpers (the single-op overload subsumes them; the only loss is long? rather than long for a pure Get), and public accessors on BitFieldOperation (Kind/Encoding/Offset/Value/Overflow are internal).

The command is rendered by a Message subclass in WriteImpl (the HotKeysStartMessage shape) rather than via an intermediate RedisValue[], so the batch form aliases the caller's memory and the single-operation form holds the operation inline; sub-command and overflow tokens are framed u8 spans written raw, and the encoding is one framed write into a scratch buffer. Verified against the IL on all six TFMs that neither path allocates - notably, a [operation] collection expression does allocate a one-element array on every framework without the by-ref span constructor (net461, netstandard2.0, net472, net6.0), so that path takes in BitFieldOperation instead.

The two StringBitField overloads need #pragma warning disable RS0026, with the same "disambiguated via parameter types" justification already used in RedisChannel.

StringBitPosition: an open-ended end

StringBitPosition always sent an explicit end, and for a clear-bit search the server distinguishes the two cases:

BITPOS key 0        -> 8     # first clear bit past the end of the string
BITPOS key 0 0      -> 8     # start alone: still open-ended
BITPOS key 0 0 -1   -> -1    # explicit range: no clear bit in the string

(single all-ones byte). The end parameter documented -1 as "unlimited", which was the wrong mental model - -1 is the last byte, and the open-ended result was unreachable.

Passing StringIndex.Unbounded as end now omits it. That requires StringIndexType.Byte, since BYTE/BIT is only legal after an explicit end; silently dropping the token would reinterpret start from bits to bytes, so it throws instead.

A sentinel rather than an overload because a no-end overload would be equally applicable to the existing one at every call site, with both needing defaults substituted - i.e. CS0121 for callers. The sentinel also costs no new interface members and flows through the legacy overload, IBatch/ITransaction and the keyspace-isolation wrappers for free.

BITCOUNT is untouched: its start/end are an inseparable pair on the wire (BITCOUNT key 0 is a syntax error), so a sentinel there could only silently discard start, and 0 -1 is anyway equivalent to no range.

Testing

  • Exact-bytes round-trip tests (no server): OVERFLOW dedup and stickiness across a GET, the #N and i64/u63 tokens, nil parsing, payload classification, BITFIELD_RO vs BITFIELD primary-only-ness, and the BITPOS omitted-end layout.
  • Type unit tests: width limits and their messages, offset forms, default/unknown-value rejection.
  • Live integration: SET/GET/INCRBY semantics, OVERFLOW FAIL nils, index offsets, the empty batch, and a profiler assertion that all-GET calls left as BITFIELD_RO to the replica while the write left as BITFIELD to the primary.

dotnet build Build.csproj -c Release /p:CI=true is clean across all TFMs. Full suite on net10.0: 6022 passed, 149 skipped, with one failure that differs run-to-run (RetryTests.WithRetry_FailsOverBetweenGroupsOnLoading once, InProcPubSubTests.TestBasicPubSub once; each passes in isolation) - neither touches bitmaps, and both look like pre-existing flakiness under parallel load rather than anything here, though I have not bisected them.

…TPOS

Two gaps in the bitmap family (https://redis.io/docs/latest/commands/?group=bitmap).

BITFIELD and BITFIELD_RO were entirely unimplemented - the only trace in the
repo was a commented-out line in the envoy exclusion list. Added as:

    long?         StringBitField(key, BitFieldOperation, flags)
    Lease<long?>? StringBitField(key, ReadOnlyMemory<BitFieldOperation>, flags)

plus BitFieldEncoding (Signed/Unsigned, and named Int8..Int64/UInt8..UInt63),
BitFieldOffset (Bits/Index, the latter being the server's #N form),
BitFieldOverflow (Wrap/Saturate/Fail) and BitFieldOperation (Get/Set/
IncrementBy). Signedness lives with the width rather than in a flags enum, and
#N lives on the offset, so no flag silently reinterprets another argument.

Every legal encoding (i1-i64, u1-u63) fits losslessly in a signed 64-bit
integer, which is exactly why u64 does not exist, so the results are plain
long - nullable only because OVERFLOW FAIL reports a nil for the operation it
skipped. There is deliberately a named UInt63 where UInt64 would sit, to
document the ceiling at the point someone reaches for it.

The OVERFLOW token is sticky server-side and untouched by GET, so it is
emitted only when the required mode changes, and never for a leading run of
WRAP (the server default). An all-GET batch is issued as BITFIELD_RO where the
server supports it (6.0+) so a replica can serve it; anything that writes stays
as BITFIELD, which is primary-only. An empty batch short-circuits locally.
Retry category follows the payload: accumulating by default, demoted to
last-wins when there is no INCRBY.

Separately, StringBitPosition could not express an open-ended range. It always
sent an explicit end, and for a clear-bit search the server distinguishes the
two: with an explicit end an all-set string yields -1, whereas with no end the
result is the first bit past the end of the string. Passing
StringIndex.Unbounded as the end now omits it; that requires StringIndexType
.Byte, since BYTE/BIT is only legal after an explicit end, and silently
dropping the token would reinterpret start from bits to bytes. Implemented as a
sentinel rather than an overload because a no-end overload would be
CS0121-ambiguous with the existing one at every call site.
Bits/Index both said the wrong thing: Index named the mechanism ("an index")
when the question is *of what* - bits or elements - and Bits(100) read as a
quantity ("100 bits") rather than a position ("bit 100").

Singular noun plus ordinal, symmetrically: Bit(600) is bit 600, Element(3) is
element 3, i.e. the #3 form the server multiplies by the width. Element rather
than Field because "field" is already carrying weight in BitFieldEncoding and
BitFieldOperation, and would read as naming a field rather than counting one.
The reply is always an array with one element per operation - empty for no
operations - so a null *array* was unreachable; the only way to see one was a
non-observable result such as fire-and-forget, where ExecuteSync hands back
default(T). That made callers null-check for a state that never carries
information, while the inner long? (an operation skipped by OVERFLOW FAIL) is
the nullability that actually means something.

Use the established defaultValue: idiom instead - as GeoPosition/GeoHash do for
their arrays - so the batch returns a non-null Lease<long?> and yields an empty
lease when there is nothing to observe. The single-operation overload keeps its
long?, where null already means FAIL.
Follows the HotKeysStartMessage shape: a Message subclass that renders itself in
WriteImpl, with ArgCount computed (and the payload validated) in the
constructor, so a bad operation still throws at the caller rather than mid-write
on the connection.

This drops the intermediate RedisValue[] entirely - the batch form keeps the
caller's ReadOnlyMemory, as the existing keyed-values messages do with caller
arrays, and the single-operation form holds the operation inline, so that path
now allocates nothing but the message. It also obviates the literals: the
sub-command and overflow tokens are framed compile-time u8 spans written raw,
which retires the RedisLiterals additions and the 128-entry encoding table
behind ToLiteral.

The encoding is written as one framed bulk string into a scratch buffer hoisted
out of the loop: the width is 1-64, so the length prefix is always a single
digit and it is either $2\r\nXX\r\n or $3\r\nXXX\r\n. The element offset shares
that buffer for its '#' plus digits, but keeps WriteBulkString since its own
length prefix varies.

Round-trip tests now build the messages directly, as SetCardinalityRoundTrip
does, which also retires the internal payload-builder seam that only existed for
testability; new cases cover the encoding and offset framing at both prefix
widths, ArgCount, and that a single operation writes byte-for-byte the same as a
one-element batch.
The kind switch used its default arm for INCRBY, so an operation that was
somehow neither GET nor SET would have been written as the most destructive of
the three. Reject it instead: guessing is worse than failing here, because a
wrong sub-command corrupts data and one of the wrong arity corrupts the
connection - a GET written where the header promised four arguments leaves the
server parsing the next command's bytes as arguments.

The overflow arm keeps WRAP, and that asymmetry is deliberate: its argument
count comes from the transition (op.Overflow != overflow) rather than the mode
itself, so an unknown mode is shape-neutral and the server's own default is the
harmless answer.

CountArgs already rejected this at construction, but only watertight for the
single-operation form, which copies. The batch form aliases the caller's memory,
and unlike the messages holding a RedisValue[] - where ArgCount is the array
length and content cannot change the shape - here the shape depends on the
contents, so mutating the array between issue and write would desync the frame.
Re-check before the header is written, so a violated contract fails with nothing
written rather than corrupting the connection.
`[operation]` for a ReadOnlySpan parameter only avoids the heap where the by-ref
span constructor exists. Checking the IL per framework: net8.0 and net10.0 emit
ReadOnlySpan<T>::.ctor(!0&) over a stack local, but net461, netstandard2.0,
net472 and net6.0 all emit `newarr BitFieldOperation` plus the array-based
constructor - twice per call, once counting in the constructor and once writing -
which is exactly the allocation that holding the operation inline was meant to
avoid, on four of the six frameworks we ship.

So don't take a span of operations at all on that path: the count and write steps
become per-operation helpers taking `in BitFieldOperation`, and the batch form
loops them over its span. The IL now shows no newarr in either message class on
any framework, the only localloc being the intended scratch buffer.

Also note in passing that the batch form's pre-write re-check becomes redundant
once the write path serializes on the calling thread, since that removes the
interval in which the aliased operations could be mutated at all.

@mgravell mgravell left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Routing and replay safety are separate axes, and the fallback path conflated
them. When every sub-operation is a GET but BITFIELD_RO is unavailable - a
server below 6.0, or the command disabled or renamed in the command map - the
message went out as BITFIELD and inherited its WriteAccumulating default,
despite there being nothing to accumulate. The server insisting BITFIELD is a
write governs which servers will accept it, not whether replaying it is safe.

Apply CommandRetryReadOnly for any all-GET payload, whichever of the two
commands we end up issuing; BITFIELD_RO already had it by default, so this only
changes the fallback. An explicitly requested category still wins, as
WithCategory has always done.

The decision is now a pure static taking the availability as a parameter, which
is the only way to observe it - the flags a message ends up with are not visible
from outside - so all four payload shapes plus the explicit-category case are
covered directly.
@mgravell
mgravell merged commit c00000e into main Aug 25, 2026
8 of 10 checks passed
@mgravell
mgravell deleted the marc/bitops branch August 25, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant