# `IDataView` Type System ## Overview The *IDataView system* consists of a set of interfaces and classes that provide efficient, compositional transformation of and cursoring through schematized data, as required by many machine-learning and data analysis applications. It is designed to gracefully and efficiently handle both extremely high dimensional data and very large data sets. It does not directly address distributed data, but is suitable for single node processing of data partitions belonging to larger distributed data sets. While `IDataView` is one interface in this system, colloquially, the term IDataView is frequently used to refer to the entire system. In this document, the specific interface is written using fixed pitch font as `IDataView`. IDataView is the data pipeline machinery for ML.NET. The ML.NET codebase has an extensive library of IDataView related components (loaders, transforms, savers, trainers, predictors, etc.). More are being worked on. The name IDataView was inspired from the database world, where the term table typically indicates a mutable body of data, while a view is the result of a query on one or more tables or views, and is generally immutable. Note that both tables and views are schematized, being organized into typed columns and rows conforming to the column types. Views differ from tables in several ways: * Views are immutable; tables are mutable. * Views are composable -- new views can be formed by applying transformations (queries) to other views. Forming a new table from an existing table involves copying data, making them decoupled—the new table is not linked to the original table in any way. * Views are virtual; tables are fully realized/persisted. Note that immutability and compositionality are critical enablers of technologies that require reasoning over transformation, like query optimization and remoting. Immutability is also key for concurrency and thread safety. This document includes a very brief introduction to some of the basic concepts of IDataView, but then focuses primarily on the IDataView type system. Why does IDataView need a special type system? The .NET type system is not well suited to machine-learning and data analysis needs. For example, while one could argue that `typeof(double[])` indicates a vector of double values, it explicitly does not include the dimensionality of the vector/array. Similarly, there is no good way to indicate a subset of an integer type, for example integers from 1 to 100, as a .NET type. In short, there is no reasonable way to encode complete range and dimensionality information in a `System.Type`. In addition, a well-defined type system, including complete specification of standard data types and conversions, enables separately authored components to seamlessly work together without surprises. ### Basic Concepts `IDataView`, in the narrow sense, is an interface implemented by many components. At a high level, it is analogous to the .Net interface `IEnumerable`, with some very significant differences. While `IEnumerable` is a sequence of objects of type `T`, `IDataView` is a sequence of rows. An `IDataView` object has an associated `ISchema` object that defines the `IDataView`'s columns, including their names, types, indices, and associated metadata. Each row of the `IDataView` has a value for each column defined by the schema. Just as `IEnumerable` has an associated enumerator interface, namely `IEnumerator`, `IDataView` has an associated cursor interface, namely `IRowCursor`. In the enumerable world, an enumerator object implements a Current property that returns the current value of the iteration as an object of type `T`. In the IDataView world, an `IRowCursor` object encapsulates the current row of the iteration. There is no separate object that represents the current row. Instead, the cursor implements methods that provide the values of the current row, when requested. Additionally, the methods that serve up values do not require memory allocation on each invocation, but use sharable buffers. This scheme significantly reduces the memory allocations needed to cursor through data. Both `IDataView` and `IEnumerable` present a read-only view on data, in the sense that a sequence presented by each is not directly mutable. "Modifications" to the sequence are accomplished by additional operators or transforms applied to the sequence, so do not modify any underlying data. For example, to normalize a numeric column in an `IDataView` object, a normalization transform is applied to the sequence to form a new `IDataView` object representing the composition. In the new view, the normalized values are contained in a new column. Often, the new column has the same name as the original source column and "replaces" the source column in the new view. Columns that are not involved in the transformation are simply "passed through" from the source `IDataView` to the new one. Detailed specifications of the `IDataView`, `ISchema`, and `IRowCursor` interfaces are in other documents. ### Column Types Each column in an `IDataView` has an associated column type. The collection of column types is open, in the sense that new code can introduce new column types without requiring modification of all `IDataView` related components. While introducing new types is possible, we expect it will also be relatively rare. All column type implementations derive from the abstract class `ColumnType`. Primitive column types are those whose implementation derives from the abstract class `PrimitiveType`, which derives from `ColumnType`. ### Representation Type A column type has an associated .Net type, known as its representation type or raw type. Note that a column type often contains much more information than the associated .Net representation type. Moreover, many distinct column types can use the same representation type. Consequently, code should not assume that a particular .Net type implies a particular column type. ### Standard Column Types There is a set of predefined standard column types, divided into standard primitive types and vector types. Note that there can be types that are neither primitive nor vector types. These types are not standard types and may require extra care when handling them. For example, a `PictureType` value might require disposing when it is no longer needed. Standard primitive types include the text type, the boolean type, numeric types, and key types. Numeric types are further split into floating-point types, signed integer types, and unsigned integer types. A vector type has an associated item type that must be a primitive type, but need not be a standard primitive type. Note that vector types are not primitive types, so vectors of vectors are not supported. Note also that vectors are homogeneous—all elements are of the same type. In addition to its item type, a vector type contains dimensionality information. At the basic level, this dimensionality information indicates the length of the vector type. A length of zero means that the vector type is variable length, that is, different values may have different lengths. Additional detail of vector types is in a subsequent section. Vector types are instances of the sealed class `VectorType`, which derives from `ColumnType`. This document uses convenient shorthand for standard types: * `TX`: text * `BL`: boolean * `R4`, `R8`: single and double precision floating-point * `I1`, `I2`, `I4`, `I8`: signed integer types with the indicated number of bytes * `U1`, `U2`, `U4`, `U8`: unsigned integer types with the indicated number of bytes * `UG`: unsigned type with 16-bytes, typically used as a unique ID * `TS`: timespan, a period of time * `DT`: datetime, a date and time but no timezone * `DZ`: datetime zone, a date and time with a timezone * `U4[100-199]`: A key type based on `U4` representing legal values from 100 to 199, inclusive * `V`: A vector type with item type `R4` and dimensionality information [3,2] See the sections on the specific types for more detail. The IDataView system includes many standard conversions between standard primitive types. A later section contains a full specification of these conversions. ### Default Value Each column type has an associated default value corresponding to the default value of its representation type, as defined by the .Net (C# and CLR) specifications. The standard conversions map source default values to destination default values. For example, the standard conversion from `TX` to `R8` maps the empty text value to the value zero. Note that the empty text value is distinct from the missing text value, as discussed next. ### Missing Value Most of the standard primitive types support the notion of a missing value. In particular, the text type, floating-point types, signed integer types, and key types all have an internal representation of missing. We follow R's lead and denote such values as `NA`. Unlike R, the standard primitive types do not distinguish between missing and invalid. For example, in floating-point arithmetic, computing zero divided by zero, or infinity minus infinity, produces an invalid value known as a `NaN` (for Not-a-Number). R uses a specific `NaN` value to represent its `NA` value, with all other `NaN` values indicating invalid. The IDataView standard floating-point types do not distinguish between the various `NaN` values, treating them all as missing/invalid. A standard conversion from a source type with `NA` to a destination type with `NA` maps `NA` to `NA`. A standard conversion from a source type with `NA` to a destination type without `NA` maps `NA` to the default value of the destination type. For example, converting a text `NA` value to `R4` produces a `NaN`, but converting a text `NA` to `U4` results in zero. Note that this specification does not address diagnostic user messages, so, in certain environments, the latter situation may generate a warning to the user. Note that a vector type does not support a representation of missing, but may contain `NA` values of its item type. Generally, there is no standard mechanism faster than O(N) for determining whether a vector with N items contains any missing values. For further details on missing value representations, see the sections detailing the particular standard primitive types. ### Vector Representations Values of a vector type may be represented either sparsely or densely. A vector type does not mandate denseness or sparsity, nor does it imply that one is favored over the other. A sparse representation is semantically equivalent to a dense representation having the suppressed entries filled in with the *default* value of the item type. Note that the values of the suppressed entries are emphatically *not* the missing/`NA` value of the item type, unless the missing and default values are identical, as they are for key types. ### Metadata A column in an `ISchema` can have additional column-wide information, known as metadata. For each string value, known as a metadata kind, a column may have a value associated with that metadata kind. The value also has an associated type, which is a compatible column type. For example: * A column may indicate that it is normalized, by providing a `BL` valued piece of metadata named `IsNormalized`. * A column whose type is `V`, meaning a vector of length 17 whose items are single-precision floating-point values, might have `SlotNames` metadata of type `V`, meaning a vector of length 17 whose items are text. * A column produced by a scorer may have several pieces of associated metadata, indicating the "scoring column group id" that it belongs to, what kind of scorer produced the column (for example, binary classification), and the precise semantics of the column (for example, predicted label, raw score, probability). The `ISchema` interface, including the metadata API, is fully specified in another document. ## Text Type The text type, denoted by the shorthand `TX`, represents text values. The `TextType` class derives from `PrimitiveType` and has a single instance, exposed as `TextType.Instance`. The representation type of `TX` is an immutable struct known as `DvText`. A `DvText` value represents a sequence of characters whose length is contained in its `Length` field. The missing/`NA` value has a `Length` of -1, while all other values have a non-negative `Length`. The default value has a `Length` of zero and represents an empty sequence of characters. In text processing transformations, it is very common to split text into pieces. A key advantage of using `DvText` instead of `System.String` for text values is that these splits require no memory allocation—the derived `DvText` references the same underlying `System.String` as the original `DvText` does. Another reason that `System.String` is not ideal for text is that we want the default value to be empty and not `NA`. For `System.String`, the default value is null, which would be a more natural representation for `NA` than for empty text. By using a custom struct wrapper around a portion (or span) of a `System.String`, we address both the memory efficiency and default value problems. ## Boolean Type The standard boolean type, denoted by the shorthand `BL`, represents true/false values. The `BooleanType` class derives from `PrimitiveType` and has a single instance, exposed as `BooleanType.Instance`. The representation type of `BL` is the `DvBool` enumeration type, logically stored as `sbyte`: `DvBool` | `sbyte` Value --------:|:------------- `NA` | -128 `False` | 0 `True` | 1 The default value of `BL` is `DvBool.False` and the `NA` value of `BL` is `DvBool.NA`. Note that the underlying type of the `DvBool` `enum` is signed byte and the default and `NA` values of `BL` align with the default and `NA` values of `I1`. There is a standard conversion from `TX` to `BL`. There are standard conversions from `BL` to all signed integer and floating point numeric types, with `DvBool.False` mapping to zero, `DvBool.True` mapping to one, and `DvBool.NA` mapping to `NA`. ## Number Types The standard number types are all instances of the sealed class NumberType, which is derived from PrimitiveType. There are two standard floating-point types, four standard signed integer types, and four standard unsigned integer types. Each of these is represented by a single instance of NumberType and there are static properties of NumberType to access each instance. For example, to test whether a variable type represents `I4`, use the C# code `type == NumberType.I4`. Floating-point arithmetic has a well-deserved reputation for being troublesome. This is primarily because it is imprecise, in the sense that the result of most operations must be rounded to the nearest representable value. This rounding means, among other side effects, that floating-point addition and multiplication are not associate, nor satisfy the distributive property. However, in many ways, floating-point arithmetic is the best-suited system for arithmetic computation. For example, the IEEE 754 specification mandates precise graceful overflow behavior—as results grow, they lose resolution in the least significant digits, and eventually overflow to a special infinite value. In contrast, when integer arithmetic overflows, the result is a non- sense value. Trapping and handling integer overflow is expensive, both in runtime and development costs. The IDataView system supports integer numeric types mostly for data interchange convenience, but we strongly discourage performing arithmetic on those values without first converting to floating-point. ### Floating-point Types The floating-point types, `R4` and `R8`, have representation types `System.Single` and `System.Double`. Their default values are zero. Any `NaN` is considered an `NA` value, with the specific `Single.NaN` and `Double.NaN` values being the canonical `NA` values. There are standard conversions from each floating-point type to the other floating-point type. There are also standard conversions from text to each floating-point type and from each integer type to each floating-point type. ### Signed Integer Types The signed integer types, `I1`, `I2`, `I4`, and `I8`, have representation types Sytem.SByte, `System.Int16`, `System.Int32`, and `System.Int64`. The default value of each of these is zero. Each of these has a non-zero value that is its own additive inverse, namely `(-2)^^{8n-1}`, where `n` is the number of bytes in the representation type. This is the minimum value of each of these types. We follow R's lead and use these values as the `NA` values. There are standard conversions from each signed integer type to every other signed integer type. There are also standard conversions from text to each signed integer type and from each signed integer type to each floating-point type. Note that we have not defined standard conversions from floating-point types to signed integer types. ### Unsigned Integer Types The unsigned integer types, `U1`, `U2`, `U4`, and `U8`, have representation types Sytem.Byte, `System.UInt16`, `System.UInt32`, and `System.UInt64`, respectively. The default value of each of these is zero. These types do not have an `NA` value. There are standard conversions from each unsigned integer type to every other unsigned integer type. There are also standard conversions from text to each unsigned integer type and from each unsigned integer type to each floating- point type. Note that we have not defined standard conversions from floating-point types to unsigned integer types, or between signed integer types and unsigned integer types. ## Key Types Key types are used for data that is represented numerically, but where the order and/or magnitude of the values is not semantically meaningful. For example, hash values, social security numbers, and the index of a term in a dictionary are all best modeled with a key type. The representation type of a key type, also called its underlying type, must be one of the standard four .Net unsigned integer types. The `NA` and default values of a key type are the same value, namely the representational value zero. Key types are instances of the sealed class `KeyType`, which derives from `PrimitiveType`. In addition to its underlying type, a key type specifies: * A count value, between `0` and `int.MaxValue`, inclusive * A "minimum" value, between `0` and `ulong.MaxValue`, inclusive * A Boolean value indicating whether the values of the key type are contiguous Regardless of the minimum and count values, the representational value zero always means `NA` and the representational value one is always the first valid value of the key type. Notes: * The `Count` property returns the count of the key type. This is of type `int`, but is required to be non-negative. When `Count` is zero, the key type has no known or useful maximum value. Otherwise, the legal representation values are from one up to and including `Count`. The `Count` is required to be representable in the underlying type, so, for example, the `Count` value of a key type based on `System.Byte` must not exceed `255`. As an example of the usefulness of the `Count` property, consider the `KeyToVector` transform implemented as part of ML.NET. It maps from a key type value to an indicator vector. The length of the vector is the `Count` of the key type, which is required to be positive. For a key value of `k`, with `1 ≤ k ≤ Count`, the resulting vector has a value of one in the (`k-1`)th slot, and zero in all other slots. An `NA` value (with representation zero) is mapped to the all- zero vector of length `Count`. * For a key type with positive `Count`, a representation value should be between `0` and `Count`, inclusive, with `0` meaning `NA`. When processing values from an untrusted source, it is best to guard against values bigger than `Count` and treat such values as equivalent to `NA`. * The `Min` property returns the minimum semantic value of the key type. This is used exclusively for transforming from a representation value, where the valid values start at one, to user facing values, which might start at any non-negative value. The most common values for `Min` are zero and one. * The boolean `Contiguous` property indicates whether values of the key type are generally contiguous in the sense that a complete sampling of representation values of the key type would cover most, if not all, values from one up to their max. A `true` value indicates that using an array to implement a map from the key type values is a reasonable choice. When `false`, it is likely more prudent to use a hash table. * A key type can be non-`Contiguous` only if `Count` is zero. The converse however is not true. A key type that is contiguous but has `Count` equal to zero is one where there is a reasonably small maximum, but that maximum is unknown. In this case, an array might be a good choice for a map from the key type. * The shorthand for a key type with representation type `U1`, and semantic values from `1000` to `1099`, inclusive, is `U1[1000-1099]`. Note that the `Min` value of this key type is outside the range of the underlying type, `System.Byte`, but the `Count` value is only `100`, which is representable in a `System.Byte`. Recall that the representation values always start at 1 and extend up to `Count`, in this case `100`. * For a key type with representation type `System.UInt32` and semantic values starting at `1000`, with no known maximum, the shorthand is `U4[1000-*]`. There are standard conversions from text to each key type. This conversion parses the text as a standard non-negative integer value and honors the `Min` and `Count` values of the key type. If a parsed numeric value falls outside the range indicated by `Min` and `Count`, or if the text is not parsable as a non-negative integer, the result is `NA`. There are standard conversions from one key type to another, provided: * The source and destination key types have the same `Min` and `Count` values. * Either the number of bytes in the destination's underlying type is greater than the number of bytes in the source's underlying type, or the `Count` value is positive. In the latter case, the `Count` is necessarily less than 2k, where k is the number of bits in the destination type's underlying type. For example, `U1[1-*]` can be converted to `U2[1-*]`, but `U2[1-*]` cannot be converted to `U1[1-*]`. Also, `U1[1-100]` and `U2[1-100]` can be converted in both directions. ## Vector Types ### Introduction Vector types are one of the key innovations of the IDataView system and are critical for high dimensional machine-learning applications. For example, when processing text, it is common to hash all or parts of the text and encode the resulting hash values, first as a key type, then as indicator or bag vectors using the `KeyToVector` transform. Using a `k`-bit hash produces a key type with `Count` equal to `2^^k`, and vectors of the same length. It is common to use `20` or more hash bits, producing vectors of length a million or more. The vectors are typically very sparse. In systems that do not support vector-valued columns, each of these million or more values is placed in a separate (sparse) column, leading to a massive explosion of the column space. Most tabular systems are not designed to scale to millions of columns, and the user experience also suffers when displaying such data. Moreover, since the vectors are very sparse, placing each value in its own column means that, when a row is being processed, each of those sparse columns must be queried or scanned for its current value. Effectively the sparse matrix of values has been needlessly transposed. This is very inefficient when there are just a few (often one) non-zero entries among the column values. Vector types solve these issues. A vector type is an instance of the sealed `VectorType` class, which derives from `ColumnType`. The vector type contains its `ItemType`, which must be a `PrimitiveType`, and its dimensionality information. The dimensionality information consists of one or more non-negative integer values. The `VectorSize` is the product of the dimensions. A dimension value of zero means that the true value of that dimension can vary from value to value. For example, tokenizing a text by splitting it into multiple terms generates a vector of text of varying/unknown length. The result type shorthand is `V`. Hashing this using `6` bits then produces the vector type `V`. Applying the `KeyToVector` transform then produces the vector type `V`. Each of these vector types has a `VectorSize` of zero, indicating that the total number of slots varies, but the latter still has potentially useful dimensionality information: the vector slots are partitioned into an unknown number of runs of consecutive slots each of length `64`. As another example, consider an image data set. The data starts with a `TX` column containing URLs for images. Applying an `ImageLoader` transform generates a column of a custom (non-standard) type, `Picture<*,*,4>`, where the asterisks indicate that the picture dimensions are unknown. The last dimension of `4` indicates that there are four channels in each pixel: the three color components, plus the alpha channel. Applying an `ImageResizer` transform scales and crops the images to a specified size, for example, `100x100`, producing a type of `Picture<100,100,4>`. Finally, applying a `ImagePixelExtractor` transform (and specifying that the alpha channel should be dropped), produces the vector type `V`. In this example, the `ImagePixelExtractor` re-organized the color information into separate planes, and divided each pixel value by 256 to get pixel values between zero and one. ### Equivalence Note that two vector types are equivalent when they have equivalent item types and have identical dimensionality information. To test for compatibility, instead of equivalence, in the sense that the total `VectorSize` should be the same, use the `SameSizeAndItem` method instead of the Equals method (see the `ColumnType` code below). ### Representation Type The representation type of a vector type is the struct `VBuffer`, where `T` is the representation type of the item type. For example, the representation type of `V` is `VBuffer`. When the vector type's `VectorSize` is positive, each value of the type will have length equal to the `VectorSize`. The struct `VBuffer`, sketched below, provides both dense and sparse representations and encourages cooperative buffer sharing. A complete discussion of `VBuffer` and associated coding idioms is in another document. Notes: * `VBuffer` contains four public readonly fields: `Length`, `Count`, `Values`, and `Indices`. * `Length` is the logical length of the vector, and must be non-negative. * `Count` is the number of items explicitly represented in the vector. `Count` is non-negative and less than or equal to Length. * When `Count` is equal to Length, the vector is dense. Otherwise, the vector is sparse. * The `Values` array contains the explicitly represented item values. The length of the `Values` array is at least `Count`, but not necessarily equal to `Count`. Only the first `Count` items in `Values` are part of the vector; any remaining items are garbage and should be ignored. Note that when `Count` is zero, `Values` may be null. * The `Indices` array is only relevant when the vector is sparse. In the sparse case, `Indices` is parallel to `Values`, only the first `Count` items are meaningful, the indices must be non-negative and less than `Length`, and the indices must be strictly increasing. Note that when `Count` is zero, `Indices` may be null. In the dense case, `Indices` is not meaningful and may or may not be null. * It is very common for the arrays in a `VBuffer` to be larger than needed for their current value. A special case of this is when a dense `VBuffer` has a non-null `Indices` array. The extra items in the arrays are not meaningful and should be ignored. Allowing these buffers to be larger than currently needed reduces the need to reallocate buffers for different values. For example, when cursoring through a vector valued column with `VectorSize` of 100, client code could pre-allocate values and indices arrays and seed a `VBuffer` with those arrays. When fetching values, the client code passes the `VBuffer` by reference. The called code can re-use those arrays, filling them with the current values. * Generally, vectors should use a sparse representation only when the number of non-default items is at most half the value of Length. However, this guideline is not a mandate. See the full `IDataView` technical specification for additional details on `VBuffer`, including complete discussion of programming idioms, and information on helper classes for building and manipulating vectors. ## Standard Conversions The `IDataView` system includes the definition and implementation of many standard conversions. Standard conversions are required to map source default values to destination default values. When both the source type and destination type have an `NA` value, the conversion must map `NA` to `NA`. When the source type has an `NA` value, but the destination type does not, the conversion must map `NA` to the default value of the destination type. Most standard conversions are implemented by the singleton class `Conversions` in the namespace `Microsoft.MachineLearning.Data.Conversion`. The standard conversions are exposed by the `ConvertTransform`. ### From Text There are standard conversions from `TX` to the standard primitive types, `R4`, `R8`, `I1`, `I2`, `I4`, `I8`, `U1`, `U2`, `U4`, `U8`, and `BL`. For non- empty, non-missing `TX` values, these conversions use standard parsing of floating-point and integer values. For `BL`, the mapping is case insensitive, maps text values `{ true, yes, t, y, 1, +1, + }` to `DvBool.True`, and maps the values `{ false, no, f, n, 0, -1, - }` to `DvBool.False`. If parsing fails, the result is the `NA` value for floating-point, signed integer types, and boolean, and zero for unsigned integer types. Note that overflow of an integer type is considered failure of parsing, so produces an `NA` (or zero for unsigned). These conversions map missing/`NA` text to `NA`, for floating-point and signed integer types, and to zero for unsigned integer types. These conversions are required to map empty text (the default value of `TX`) to the default value of the destination, which is zero for all numeric types and DvBool.False for `BL`. This may seem unfortunate at first glance, but leads to some nice invariants. For example, when loading a text file with sparse row specifications, it's desirable for the result to be the same whether the row is first processed entirely as `TX` values, then parsed, or processed directly into numeric values, that is, parsing as the row is processed. In the latter case, it is simple to map implicit items (suppressed due to sparsity) to zero. In the former case, these items are first mapped to the empty text value. To get the same result, we need empty text to map to zero. ### Floating Point There are standard conversions from `R4` to `R8` and from `R8` to `R4`. These are the standard IEEE 754 conversions (using unbiased round-to-nearest in the case of `R8` to `R4`). ### Signed Integer There are standard conversions from each signed integer type to each other signed integer type. These conversions map `NA` to `NA`, map any other numeric value that fits in the destination type to the corresponding value, and maps any numeric value that does not fit in the destination type to `NA`. For example, when mapping from `I1` to `I2`, the source `NA` value, namely 0x80, is mapped to the destination `NA` value, namely 0x8000, and all other numeric values are mapped as expected. When mapping from `I2` to `I1`, any value that is too large in magnitude to fit in `I1`, such as 312, is mapped to `NA`, namely 0x80. ### Signed Integer to Floating Point There are standard conversions from each signed integer type to each floating- point type. These conversions map `NA` to `NA`, and map all other values according to the IEEE 754 specification using unbiased round-to-nearest. ### Unsigned Integer There are standard conversions from each unsigned integer type to each other unsigned integer type. These conversions map any numeric value that fits in the destination type to the corresponding value, and maps any numeric value that does not fit in the destination type to zero. For example, when mapping from `U2` to `U1`, any value that is too large in magnitude to fit in `U1`, such as 312, is mapped to zero. ### Unsigned Integer to Floating Point There are standard conversions from each unsigned integer type to each floating-point type. These conversions map all values according to the IEEE 754 specification using unbiased round-to-nearest. ### Key Types There are standard conversions from one key type to another, provided: * The source and destination key types have the same `Min` and `Count` values. * Either the number of bytes in the destination's underlying type is greater than the number of bytes in the source's underlying type, or the `Count` value is positive. In the latter case, the `Count` is necessarily less than `2^^k`, where `k` is the number of bits in the destination type's underlying type. For example, `U1[1-*]` can be converted to `U2[1-*]`, but `U2[1-*]` cannot be converted to `U1[1-*]`. Also, `U1[1-100]` and `U2[1-100]` can be converted in both directions. The conversion maps source representation values to the corresponding destination representation values. There are no special cases, because of the requirements above. ### Boolean to Numeric There are standard conversions from `BL` to each of the signed integer and floating point numeric. These map `DvBool.True` to one, `DvBool.False` to zero, and `DvBool.NA` to the numeric type's `NA` value. ## Type Classes This chapter contains information on the C# classes used to represent column types. Since the IDataView type system is extensible this list describes only the core data types. ### `ColumnType` Abstract Class The IDataView system includes the abstract class `ColumnType`. This is the base class for all column types. `ColumnType` has several convenience properties that simplify testing for common patterns. For example, the `IsVector` property indicates whether the `ColumnType` is an instance of `VectorType`. In the following notes, the symbol `type` is a variable of type `ColumnType`. * The `type.RawType` property indicates the representation type of the column type. Its use should generally be restricted to constructing generic type and method instantiations. In particular, testing whether `type.RawType == typeof(int)` is not sufficient to test for the standard `U4` type. The proper test is `type == NumberType.I4`, since there is a single universal instance of the `I4` type. * Certain .Net types have a corresponding `DataKind` `enum` value. The value of the `type.RawKind` property is consistent with `type.RawType`. For .Net types that do not have a corresponding `DataKind` value, the `type.RawKind` property returns zero. The `type.RawKind` property is particularly useful when switching over raw type possibilities, but only after testing for the broader kind of the type (key type, numeric type, etc.). * The `type.IsVector` property is equivalent to `type is VectorType`. * The `type.IsNumber` property is equivalent to `type is NumberType`. * The `type.IsText` property is equivalent to `type is TextType`. There is a single instance of the `TextType`, so this is also equivalent to `type == TextType.Instance`. * The `type.IsBool` property is equivalent to `type is BoolType`. There is a single instance of the `BoolType`, so this is also equivalent to `type == BoolType.Instance`. * Type `type.IsKey` property is equivalent to `type is KeyType`. * If `type` is a key type, then `type.KeyCount` is the same as `((KeyType)type).Count`. If `type` is not a key type, then `type.KeyCount` is zero. Note that a key type can have a `Count` value of zero, indicating that the count is unknown, so `type.KeyCount` being zero does not imply that `type` is not a key type. In summary, `type.KeyCount` is equivalent to: `type is KeyType ? ((KeyType)type).Count : 0`. * The `type.ItemType` property is the item type of the vector type, if `type` is a vector type, and is the same as `type` otherwise. For example, to test for a type that is either `TX` or a vector of `TX`, one can use `type.ItemType.IsText`. * The `type.IsKnownSizeVector` property is equivalent to `type.VectorSize > 0`. * The `type.VectorSize` property is zero if either `type` is not a vector type or if `type` is a vector type of unknown/variable length. Otherwise, it is the length of vectors belonging to the type. * The `type.ValueCount` property is one if `type` is not a vector type and the same as `type.VectorSize` if `type` is a vector type. * The `Equals` method returns whether the types are semantically equivalent. Note that for vector types, this requires the dimensionality information to be identical. * The `SameSizeAndItemType` method is the same as `Equals` for non-vector types. For vector types, it returns true iff the two types have the same item type and have the same `VectorSize` values. For example, for the two vector types `V` and `V`, `Equals` returns false but `SameSizeAndItemType` returns true. ### `PrimitiveType` Abstract Class The `PrimitiveType` abstract class derives from `ColumnType` and is the base class of all primitive type implementations. ### `TextType` Sealed Class The `TextType` sealed class derives from `PrimitiveType` and is a singleton- class for the standard text type. The instance is exposed by the static `TextType.Instance` property. ### `BooleanType` Sealed Class The `BooleanType` sealed class derives from `PrimitiveType` and is a singleton-class for the standard boolean type. The instance is exposed by the static `BooleanType.Instance` property. ### `NumberType` Sealed Class The `NumberType` sealed class derives from `PrimitiveType` and exposes single instances of each of the standard numeric types, `R4`, `R8`, `I1`, `I2`, `I4`, `I8`, `U1`, `U2`, `U4`, `U8`, and `UG`. ### `DateTimeType` Sealed Class The `DateTimeType` sealed class derives from `PrimitiveType` and is a singleton-class for the standard datetime type. The instance is exposed by the static `DateTimeType.Instance` property. ### `DateTimeZoneType` Sealed Class The `DateTimeZoneType` sealed class derives from `PrimitiveType` and is a singleton-class for the standard datetime timezone type. The instance is exposed by the static `DateTimeType.Instance` property. ### `TimeSpanType` Sealed Class The `TimeSpanType` sealed class derives from `PrimitiveType` and is a singleton-class for the standard datetime timezone type. The instance is exposed by the static `TimeSpanType.Instance` property. ### `KeyType` Sealed Class The `KeyType` sealed class derives from `PrimitiveType` and instances represent key types. Notes: * Two key types are considered equal iff their kind, min, count, and contiguous values are the same. * The static `IsValidDataKind` method returns true iff kind is `U1`, `U2`, `U4`, or `U8`. These are the only valid underlying data kinds for key types. * The inherited `KeyCount` property returns the same value as the `Count` property. ### `VectorType` Sealed Class The `VectorType` sealed class derives from `ColumnType` and instances represent vector types. The item type is specified as the first parameter to each constructor and the dimension information is inferred from the additional parameters. * The `DimCount` property indicates the number of dimensions and the `GetDim` method returns a particular dimension value. All dimension values are non- negative integers. A dimension value of zero indicates unknown (or variable) in that dimension. * The `VectorSize` property returns the product of the dimensions. * The `IsSubtypeOf(VectorType other)` method returns true if this is a subtype of `other`, in the sense that they have the same item type, and either have the same `VectorSize` or `other.VectorSize` is zero. * The inherited `Equals` method returns true if the two types have the same item type and the same dimension information. * The inherited `SameSizeAndItemType(ColumnType other)` method returns true if `other` is a vector type with the same item type and the same `VectorSize` value.