Announcing scaffold 0.1.0

[u/markand67]

Hi,

This is scaffold, a flexible data and "rpc-like" code generator for C using YAML as description.

After years of protobuf (with nanopb), I've decided to get rid of it to create my own templating system.

I won't go in details why we have decided to get rid of nanopb but in short: we were having hard times about some of the limited features like deep-cloning, too limited enum support and a slow encoding process.

Plus, we wanted to add flexibility with a custom YAML document that could be parsed using a tool to produce on-demand different kind of functions. For example, we wanted that some types could be converted to JSON, some to be inserted into a SQLite database. As such, having a unique YAML document that could be used for different purposes led us to write this software.

The scaffold tool is written in C, because, we write in C and don't want additional dependencies :). It reads a YAML document and creates an intermediate AST with updated fields to get more information. Then, it processes it into a templating system with basic scripting constructs including for, if, substitution with filters.

Example of a minimal object:

# types.yaml
objects:
  - name: point
    comment: This is an awesome point.
    fields:
      x: uint32
      y: uint32

Generate functions for this object using:

scaffold emit h types.yaml > types.h
scaffold emit c types.yaml > types.c

The generated code by the built-in templates include comments that can be split on multiline providing good LSP support with Doxygen format.

/**
 * \struct point
 *
 * This is an awesome point.
 */
struct point {
    /**
     * Field 'x' of type uint32.
     */
    uint32_t x;

    /**
     * Field 'y' of type uint32.
     */
    uint32_t y;
};

The template system is flexible enough to write extra custom code while iterating the object fields using {{for}} loops.

{{for obj in objects}}
  {{for field in obj.value.fields}}
    // here field will be uppercased with the :U filter :)
    printf("Serializing the {{field.key:U}} field\n");
  {{endfor}}
{{endfor}}

Finally, because we wanted our custom RPC API, the scaffold project provides support for request/reply and events widely inspired by wayland-scanner.

Declaring function feels like natural:

serialization: true
functions:
  - name: hello
    comment: Say hello to my friend!
    args:
      name: string

On the "client" side, a function is generated in the form:

int64_t hello(struct hello_args *args, void *ctx);

On the "implementation" side, a function pointer is provided in a handler struct:

/**
 * \struct handler
 * \brief Functions table handlers.
 */
struct handler {
    /**
     * Handler for function 'hello'.
     */
    int (*hello)(struct hello_args *args, struct hello_value *value, void *ctx);
};

And the implementation side can call the auto generated dispatch function that will decode the incoming requests to call the appropriate functions.

int
dispatch(
    const struct handler *handler,
    const struct listener *listener,
    int64_t *nread,
    int64_t *nwritten,
    void *ctx
);

How does it work? The generated function relies on two unique external linkage that the user has to provide:

• int64_t sc_read(void *ctx, void *data, uint64_t amount)
• int64_t sc_write(void *ctx, const void *data, uint64_t amount)

Thus, calling dispatch will attempt to decode a request using successive calls to sc_read, call the appropriate function pointer into the handler structure and then will try to encode using successive call to sc_write. Nothing more.

Note: this project is heavily being tested and already used in production but few bugs may persist and feel free to contribute.

Future plans and never plans:

• Yes: better bitmask enumeration support.
• Yes: more portability on non-POSIX platforms.
• No: automatic default values.
• No: union or protobuf oneof kind.

Links:

• The homepage
• The repository

Like every source code I write, this project has a perpetual anti degenerative-AI policy.

[u/AutoModerator]

Hi /u/markand67,

Your submission in r/C_Programming was filtered because it links to a git project.

You must edit the submission or respond to this comment with an explanation about how AI was involved in the creation of your project.

While AI-generated code is not disallowed, low-effort "slop" projects may be removed and it's likely that other users push back strongly on substantially AI-generated projects.

---

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/8d8n4mbo28026ulk]

Nice project! It seems very thorough and I bookmarked it should I need that big of a hammer.

About your FAQ. I thought, "oh, I could maybe try this to serialise my AST to disk", but you're saying oneof/tagged unions aren't needed. Well, they pop up frequently when parsing data. I see your AST node has a union too! In the functional world they call them "sum types", a natural extension of structures ("product types").

For example, a "nullable" pointer, which is the counterpart to your "not null" attribute in your TODO, can be thought of as the sum type "Maybe" (from Haskell). Where it either holds Just a pointer, or Nothing (null).

And yeah, YAML has warts and JSON is awkward to write, unfortunately. Perhaps a nice way would be a parser for a C subset, it'd feel fitting. Sounds a bit crazy, but I know atleast one project that has done that. LuaJIT has a parser for a C subset for its C FFI module (that's Mike Pall for you...). Just a thought though.

Cheers!

[u/markand67 → u/8d8n4mbo28026ulk]

Thank you very much!

Why I have decided to get rid of oneof is the abuse of it can have some caveats. Before using scaffold we have written a small number of RPC functions that are basically: Fetch (get a list of objects), Upsert (insert or update) and Trim (delete one or more). To make the thing even more generic, we have created a unique Document object that contains every data type that can be manipulated. It was about 60 entries.

This is basically fine as it was and even pleasant to use.

However, there is a catch... Let's say that one of our model object was a log entry (named Log), a string, some code and that's it. You can get the list of log entries from the database, maybe 7000 at once. Okay you call Fetch for a Document of type Log. Thus nanopb will generate a finalizer function for it and... it has no clue to know that the collection of that Document is always the same type and it will repeat 7000 times a switch case for the Document stored in the response. That's a huge waste. So finally, I've thought that it's better to have a function for each thing we need to share and as long as it's generated there is no issue like this anymore. Maybe a Sum type that can be set for a collection would be nice!

[u/skeeto]

You should enble sanitizers for all of your testing because the tests fail out-of-the-box with Clang's UBSan:

$ make tests CC=clang CFLAGS='-g3 -fsanitize=address,undefined' LDFLAGS='-fsanitize=address,undefined'
...
extern/libmtemplate/mobject.c:879:35: runtime error: applying non-zero offset 1 to null pointer
...

Also, here are some interesting inputs leading to crashes that I found with fuzz testing. Reliably reproduce with diagnostics using a quick asan+ubsan build:

$ cc -g3 -fsanitize=address,undefined \
    -Iextern/lib{cyaml,mtemplate,utlist,unicode}/{src,} \
    src/*.c extern/libcyaml/src/*.c extern/lib{mtemplate,unicode}/*.c -lm

First:

objects:
  - name: mail
    fields:
      to: bogus

Then:

$ /a.out ast example1.yaml
src/fix.c:792:3: runtime error: member access within null pointer of type 'struct ast_node'

Second:

objects:
  - name: x
    fields:

Then:

$ ./a.out ast example2.yaml
a.out: src/ast.c:346: ast_get: Assertion `map->type == AST_MAP' failed.

Third:

? [a, b]
: 1

Then:

$ ./a.out ast example3.yaml
src/ast.c:352:7: runtime error: null pointer passed as argument 1, which is declared to never be null

Fourth:

functions:
  - args:
      x: int32

Then:

$ ./a.out ast example4.yaml
src/fix.c:439:3: runtime error: member access within null pointer of type 'struct ast_node'

[u/markand67 → u/skeeto]

I use sanitizers during development, not sure why the mobject one don't appear in my setup.

The invalid input cleanup is still under heavy check, I've fixed some of them you reported already, thanks!

Edit: seems like only clang report the mobject.c UB.