Getting Started

For an easy start simply visit the Quickstart or Quickstart from Hub page.

Installation / Integration into CMake projects

Integrating blace.ai into your project is super simple. You need to:

1. Download the blace.ai package from github.com/blace-ai/blace-ai/releases and place it in your project.
2. Include blace.ai cmake configuration in your CMakeLists.txt:
   include("../cmake/FindBlace.cmake")
3. Link your target to blace.ai:
   target_link_libraries(<your_target> PRIVATE 3rdparty::BlaceAI)
4. Add a post-build phase to copy all blace.ai libraries next to your executable:

   foreach(DLL_FILE ${BLACE_AI_COPY_LIBS})
       add_custom_command(TARGET <your_target> POST_BUILD COMMAND ${CMAKE_COMMAND} -E 
           copy "${DLL_FILE}" $<TARGET_FILE_DIR:your_target>
       )
   endforeach()

Important: This setup is the same on all operating systems so you can start developing e.g. on Windows and later deploy from Ubuntu by simply downloading a different (the Ubuntu) version of the package there. The folder structure is always the same. The same goes for coding: The same code works across all operating systems.

Usage

Add

#include "blace_ai.h"

at the top of your source file. This will make all headers available.

blace.ai makes use of a so-call computation graph to execute all commands. This allows for implicit caching of reusable results (like model inferences with exact same arguments) across runs. Therefore usage is split into two phases, graph construction and graph execution.

Model registering

All models that are coming from the model hub or our converter tool (to be released soon) consist of two artifacts: The .h model header and the .bin payload.

In order to register the model you need to

1. Include the header:

   #include "gemma_v1_default_v1_ALL_export_version_v10.h"

2. Register the model with the std::vector<char> contained in the header:

   blace::util::registerModel(gemma_v1_default_v1_ALL_export_version_v10, blace::util::getPathToExe());

   The second argument is the path where the .bin model payload is searched during execution. In this case we expect it next to the executable.

Now you can use the identifier gemma_v1_default_v1_ALL_export_version_v10_IDENT to refer to the declared model.

Graph building

First, you construct the computation graph (a DAG). Refer to public_ops.h to see an overview of all available operators (we have a limited set during beta but will roll out the rest of the operators soon).

Such a construction could look like (taken from the Gemma demo project):

auto text_t = CONSTRUCT_OP_GET(blace::ops::FromTextOp(str));
 
auto output_len = CONSTRUCT_OP_GET(blace::ops::FromIntOp(200));
auto temperature = CONSTRUCT_OP_GET(blace::ops::FromFloatOp(0.));
auto top_p = CONSTRUCT_OP_GET(blace::ops::FromFloatOp(0.9));
auto top_k = CONSTRUCT_OP_GET(blace::ops::FromIntOp(50));
 
blace::ml_core::InferenceArgsCollection infer_args;
// get available accelerator (cuda or metal device)
infer_args.inference_args.device = blace::util::get_accelerator().value();
 
auto infer_op = CONSTRUCT_OP_GET(blace::ops::InferenceOp(gemma_v1_default_v1_ALL_export_version_v10,
            {text_t, output_len, temperature, top_p, top_k}, infer_args, 0));

Note how five input nodes are constructed and fed into the infer_op construction. All relevant model inference arguments are hold by a blace::ml_core::InferenceArgsCollection object. Important: At this point, no model is loaded or executed. We simply define the execution structure.

Graph execution

Now that we have constructed the graph we can execute it. We do so by constructing a blace::computation_graph::GraphEvaluator from the last node (whichs result we want to get) and running the evaluation:

blace::computation_graph::GraphEvaluator evaluator(infer_op);
auto answer = evaluator.evaluateToString().value();
std::cout << "Answer: " << answer << std::endl;

If the evaluation fails answer will hold a std::nullopt.

Error handling

Our library will never throw exceptions at you. Instead, all calls to methods through the api wrap the result in a std::optional which will hold a std::nullopt in case of failure. The console will print the error message in this case.

Using models from the hub

Our model hub contains a growing list of compatible models which you can integrate in your application with a few lines of code. Check Quickstart from Hub to learn how to run the provided demo projects.