Improving secure enclaves interoperability

With the Secretarium SDK v3, we have introduced a radical improvement in enclaves communication, allowing for more rapid development cycles, proper type safety, version control, and communication across different languages.

A brief primer

Confidential computing is all about getting business logic into Trusted Execution Environments (TEEs). These isolated environments within the CPU are known as secure enclaves and allow sensitive computations to be performed securely.

Intra-app and intra-enclave communication

We needed a means to easily connect applications within or between enclaves transparently, and call methods in each apps as if they were a local function. We wanted the complicated details of local and remote attestation to be made transparent to the application developer. The idea being that apps could call themselves, call other apps within the same enclave, call an app in a separate enclave or even an app outside of the Secretarium domain.

Introducing the Secretarium IDL

A Secretarium app is a set of APIs. Developers need to provide an interface specification that describes their APIs, and then Secretarium registers the app and makes these APIs accessible by web clients and other apps.

To make it easier, we have introduced an Interface Definition File (IDL), which is a concept that has been around at least as far back as the 1980s. An IDL is a type and function calling declaration language used for generating code for Remote Procedure Calls (RPC). Intel too uses this idea in their EDL files to define a list of functions that is callable by either side of the secure enclaves.

The most well-known implementation of this technology is Microsoft's COM/DCOM and the Object Management Group's CORBA. They both went out of favour when the internet took off and web developers wanted to use RESTful type API's. COM and CORBA failed on this as their respective transports were too opaque and/or inconsistent with http communication and security protocols of that era.

Our IDL is a separate, but almost identical file format from normal C++ headers. It needs additional attributes such as whether a function parameter is an inbound value or an outbound value. Custom attributes are also useful such as tagging a function to indicate it should run as a transactional function in a dedicated thread (Secretarium has total ordering of transactions when used over a cluster of machines).

Our IDL implementation supports namespaces, and we strongly recommend that when designing components, namespaces with versioning is used. Once an interface is in production, it cannot change. If a change is required, a new interface should be declared, and this way we can guarantee ABI compliance between different systems. This is also true inside Secretarium Core, as our regression tests guarantee that we do not break backwards compatibility. When an interface is designed, the Secretarium RPC (sRPC) code generator assigns the interface a unique fingerprint for that interface and related types.

With our CMake build system, the IDL code is transformed into a set of headers and other function marshalling code.

How it works in practice

We have introduced a helper-based template class called service::v3::route_reflection. With this variadic class, it is possible to enumerate all the functions that an app is supporting and their calling convention.

To communicate with other apps, we have introduced the get_service function. It supports IntelliSense and offers a great developer experience. The returned interface can then be called such as this:

auto err = sdk::v3::get_service<io::v3::i_io>("Secretarium.IO_SERVICE")->https_query(request, response);

Although one would normally write code defensively, for brevity this one liner does several things. First, it tries to connect to a service called "IO_SERVICE", and then interrogates it to see if it supports the interface io::v3::i_io. If successful, a callable shared pointer is returned.

The function https_query is then called on that interface function. If the service is running in a different location to the application, the sRPC library will serialise the output parameters into a byte array and send it to the destination service where it is deserialised and called. Once complete, the output parameters are then serialised and deserialised in the opposite direction back to the original caller.

The returned pointer from get_service is a rpc::shared_ptr, semantically identical to a std::shared_ptr but with additional features necessary for communicating with remote objects. For instance, there is an rpc::dynamic_pointer_cast function that will interrogate a remote object to see if it supports a particular interface, so identical in design intention to the STL.

Calling an enclave from another enclave can be messy in that it has to be routed via the host before being sent to the second enclave. If the call stack is too big, the caller will need to spawn a new thread to continue the call. If the enclave is talking to another enclave for the first time, then a local or remote attestation is used to verify the measurement, and a Diffie Helman exchange secures the communication. The sRPC library manages all these things to provide a seamless experience for the application developer.

Future plans

We already have SPSC queues successfully working between enclaves and their host for fire and forget messaging for telemetry. We are extending that to allow communication between enclaves which can share the same memory for more efficient communication.

We can then bypass the host, which will allow us to explore the use of executors and coroutines to increase aggregate throughput of non-transactional communication, scaling our platform considerably. Currently, we are running one function call per thread, when we could have thousands.

Finally, we plan to work on the integration of sRPC with Google RPC (gRPC) and further increase the interoperability of the confidential computing services we host.