Type system

libqi does type erasure in similar way to Python. A type erased value is composed of a void* and a TypeInterface* (see AnyReferenceBase). The TypeInterface class is abstract and has a kind() method which returns an enum value.

To each kind corresponds a TypeInterface, like ListTypeInterface, ObjectTypeInterface, etc. A value can’t be of multiple kinds. The list of all kinds is in fwd.hpp and their mapping in signatures is in signature.hpp.

This system allows a value to hold anything, for example a PyObject representing an integer. This PyObject will not be converted to a C++ integer until it is needed. Note that it may be never needed if the value is passed back to Python code.

Type interfaces usually exist as unique instances within the program. This is not enforced though and it may happen that a type interface exist multiple times within a single program.

Type interfaces should be stateless, this is not enforced though and their functions are not marked const.

About kinds

A value of type Unknown cannot be used in a type-erased context. Its only use is when you use an unregistered type to do a type-erased call, if you give a type T and your function takes exactly a type T (the std::type_info must be equal), then the type system uses it as an Unknown value, but the call succeeds because the value is not transformed. You have the same behavior if you do qi::AnyValue::from(someT) and then yourvalue.as<T>().

A value of type Pointer is a typed pointer, it can be a raw pointer or a shared pointer. If it is a raw pointer, libqi does nothing to manage its lifetime, it’s up to the user to handle that.

A value of type Dynamic is an untyped pointer. There is no such thing in C++, but you can see it as boost::any. It is used to type-erase PyObject.

You can look at the interfaces of PointerTypeInterface and DynamicTypeInterface in typeinterface.hpp. PointerTypeInterface returns a static type that does not depends on the value (the pointedType function does not take the void* as argument), but the DynamicTypeInterface can return types different for each value through the get method.

A value of type Tuple, as the name suggests, is a list of values of different, fixed, types. Same as std::tuple in C++. C++ structures are represented as annotated tuples (the annotations correspond to the name of the structure and the name of its fields).

A value of type Raw is a raw buffer, conceptually like a std::vector<char> or a std::pair<char*, size_t>. qi::Buffer is implemented as Raw. This type is different from std::vector<char> which is a List of Int8. A List is something that gets introspected and that can be heterogeneous, a Raw is just a buffer of untyped data.

A value of type VarArgs is a variadic argument. Such object has single type, and any number of objects of that type. You can see it as Java’s variadic function which look like void f(String... strArray). If you need different types, you can use a VarArgs of Dynamic.

Relation between types and signatures

A type has a signature which is based on its kind and on some static information (functions you can call on the TypeInterface without a void* pointer). For example, a value that contains a string would have a kind String and a signature "s".

This is not based uniquely on the kind. For example, the signatures "i" and "I" both represent a kind Integer but one is signed and the other is unsigned. There exist also different signatures for different integer sizes.

Tuples are represented by parenthesis, for example "(sis)" is a tuple of String, Integer and String.

Lists are represented by square brackets, for example "[f]" is a list of Float. It is invalid to put more than one type inside the brackets.

Value’s states

A value can be invalid (for example when it is not initialized). An invalid value must never be used. Even though it looks like a Void value, it is not, and lot of functions will throw (or crash?) with an invalid value.

A value can be of Void kind. This corresponds to “no-value”. It has no equivalent in C++ but it works like Python’s None. You can instantiate such a value in C++ though:

qi::AnyValue value(qi::typeOf<void>());

Implementing type interfaces

Type interfaces must implement functions that receive a void* and must do the action on it. For example IntTypeInterface::get has the following interface and possible implementation:

int64_t Int64TypeInterface::get(void* storage)
{
  return *(int64_t*)ptrFromStorage(&storage);
}

These basic types for c++ are registered in the file registration.cpp. Note that the function may not be that trivial, for example for Python:

int64_t PyLongTypeInterface::get(void* storage)
{
  return PyLong_GetLong((PyObject*)ptrFromStorage(&storage));
}

This last function is implemented in pyobjectconvert.cpp.

Consider that ptrFromStorage(&storage) returns storage. For more information about this function, read the Pointer as pointer and pointer as value section below.

What functions a type interface must implement

A basic type interface must implement some basic functions. You can see the pure virtual functions in TypeInterface in detail/typeinterface.hpp.

These functions include default construction, copying, destruction and a less-than comparison operator. If default construction or copy is not possible, these functions are allowed to throw. In general, any function is allowed to throw if the operation is impossible. For example, lots of set throw in our Python type interfaces because Python values are usually immutable.

The less function is used to compare objects when they are stored in a container that need such a function to sort the items.

These functions are usually implemented through default implementations. Most (all?) type interfaces declare an alias to a class with default implementation and use a macro to bounce functions to it. For example:

using Methods = qi::DefaultTypeImplMethods<YourType>;

_QI_BOUNCE_TYPE_METHODS(Methods);

There are macros that bounce only some methods so that you can implement the others yourself.

The DefaultTypeImplMethods class lies in typeimpl.hxx.

Pointer as pointer and pointer as value

It is possible to apply an optimization so that when the type is small enough, instead of storing a pointer to it in the void* storage, you store the value itself. This is commonly called small buffer optimization. For example, you would have a value with an IntTypeInterface and a void* that would not be a int* pointing to a value of 42, but a void* that holds the value 42 itself (and must not be dereferenced of course).

This optimization is actually not enabled for integers and is usually not a good idea. This means that the following code would not have the expected behavior:

int i = 42;
auto ref = qi::AnyReference::from(i);
ref.setInt(10);
assert(i == 10);

You can see the code used to implement this optimization commented out in type.hxx, in the class TypeImplMethodsBySize. This class is actually only used for the C++ IntTypeInterface in inttypeinterface.hxx.

It is enabled though on all pointer types. It seems safe enough and this optimization relied upon in function type erasure, implemented in anyfunctionfactory.hxx.

Anyway, the behavior of your type interface depends on the behavior of the ptrFromStorage function. This function must return a void*. The only two possible implementation (that make sense) are the following:

void* ptrFromStorage(void** storage)
{
  return *storage; // use value as value
  // OR
  return storage; // use pointer as value
}

As said before, any function accessing the void* storage or void** storage must call this function. This is also why setter functions receive a void** instead of a void*, it allows the setter to change the pointer itself.

You can have a default implementation for this method too through the DefaultTypeImplMethods class. You configure its behavior through the second template argument you give it, which defaults to TypeByPointer<T> and is the expected behavior. If you want to use the other behavior, use TypeByValue<T>.

Construction, destruction, cloning

It is possible to change the behavior of the ptrFromStorage method, with the default construction method, cloning method, etc, by changing the second argument you give to TypeByPointer, for types that are non-copyable or non-default constructible. You can specify that for TypeByValue, the type will have default construction and default copy.

The second argument to TypeByPointer is a TypeManager which has an implementation for these functions. This TypeManager can be specialized and has a default implementation which allows the user to specify what is possible to do with their types thanks to some macros.

By default, the TypeManager makes default-constructible and copyable types for PODs, and non-default-constructible and non-copyable types for the rest, as can be seen in typeimpl.hxx.

It is possible to use the macros declared in typeinterface.hpp to specialize TypeManager to declare that a type (that may or may not be known to the type system, see Unknown kinds above) is or is not default-constructible or copyable.

If an attempt is made to default-construct a type that is non-default-constructible or to copy a type that is non-copyable, the corresponding function will just throw.

Finally, there is the TypeByPointerPOD access which forces default-constructibility and copyability.

How types are retrieved

You will reach a moment when the type system will receive a C++ object and will need to know its TypeInterface. This happens for example:

• when a struct is registered, the type system needs to know the type interfaces of the underlying fields,
• when an object’s method is registered, libqi needs the type interface of the parameters and return value
• when you use qi::typeOf<T>()
• when you use qi::AnyValue::from, which uses qi::TypeOf<T>

It will first search through a static map to find if the type has been registered at runtime. This part is done in typeinterface.cpp in the function getType.

If the type is unknown to the type system yet, it will instantiate TypeImpl<T> and use that. This part happens in type.hxx, in the function typeOfBackend.

How types are registered

This section is tightly tied to the previous one, be sure to read it first.

To register something in the global map, you usually use the function qi::registerType in typeinterface.cpp.

The other way to “register” a type is to specialize the qi::TypeImpl class. This may be useful to register full classes of types with C++ partial specialization. For example:

template <typename T>
class StdVectorTypeInterface : public ListTypeInterface
{
  // ...
};

template <typename T>
class TypeImpl<std::vector<T>> : public StdVectorTypeInterface
{};

The type interface of std::vector is implemented in listtypeinterface.hxx.

If you need to register a template class, this is the only way to make it work for any type T.

The default implementation of TypeImpl<T> is a type interface of Unknown kind which uses DefaultTypeImplMethods with default values to implement its methods. This means that it is configurable through the macros described in Construction, destruction, cloning.

About type_info comparison

To know if two types are equal, the type system uses simple type_info comparison. This feature is very weakly defined by C++, especially when shared libraries are used.

It is very important that the symbols of the type interfaces are exported in shared libraries so that the type_info themselves are exported and the runtime library can identify that two same types from different binaries are in fact the same.

Mac’s compiler has a very strict policy about that, so there is a hack where type_info are compared by mangled string representation instead of by value under Apple systems. This hack is found in typeinterface.cpp.

If someday, libqi stops using plain std::type_info and uses Boost.TypeIndex, this by-string-comparison behavior will be enforced on all platforms.

AnyValue and AnyReference

AnyValue has value semantic and AnyReference has pointer semantic. AnyReference never frees its value (unless destroy() is explicitly called). AnyValue usually owns its value and frees it automatically at destruction. It doesn’t own it sometimes, depending on the arguments given at its construction.

Users should always use AnyValue, AnyReference is usually for internal use.

Be careful when using AnyReference, do not create reference to temporaries. Your compiler won’t be able to detect it, you will not get a warning.

int i;
auto ref = qi::AnyReference::from(i); // good

ref = qi::AnyReference::from(boost::python::object(i)); // BAD
auto value = ref.toInt(); // undefined behavior