Ask HN: A retrofitted C dialect?
Hi I'm Anqur, a senior software engineer with different backgrounds where development in C was often an important part of my work. E.g.
1) Game: A Chinese/Vietnam game with C/C++ for making server/client, Lua for scripting [1]. 2) Embedded systems: Switch/router with network stack all written in C [2]. 3) (Networked) file system: Ceph FS client, which is a kernel module. [3]
(I left some unnecessary details in links, but are true projects I used to work on.)
Recently, there's a hot topic about Rust and C in kernel and a message [4] just draws my attention, where it talks about the "Rust" experiment in kernel development:
> I'd like to understand what the goal of this Rust "experiment" is: If we want to fix existing issues with memory safety we need to do that for existing code and find ways to retrofit it.
So for many years, I keep thinking about having a new C dialect for retrofitting the problems, but of C itself.
Sometimes big systems and software (e.g. OS, browsers, databases) could be made entirely in different languages like C++, Rust, D, Zig, etc. But typically, like I slightly mentioned above, making a good filesystem client requires one to write kernel modules (i.e. to provide a VFS implementation. I do know FUSE, but I believe it's better if one could use VFS directly), it's not always feasible to switch languages.
And I still love C, for its unique "bare-bone" experience:
1) Just talk to the platform, almost all the platforms speak C. Nothing like Rust's PAL (platform-agnostic layer) is needed. 2) Just talk to other languages, C is the lingua franca (except Go needs no libc by default). Not to mention if I want WebAssembly to talk to Rust, `extern "C"` is need in Rust code. 3) Just a libc, widely available, write my own data structures carefully. Since usually one is writing some critical components of a bigger system in C, it's just okay there are not many choices of existing libraries to use. 4) I don't need an over-generalized generics functionality, use of generics is quite limited.
So unlike a few `unsafe` in a safe Rust, I want something like a few "safe" in an ambient "unsafe" C dialect. But I'm not saying "unsafe" is good or bad, I'm saying that "don't talk about unsafe vs safe", it's C itself, you wouldn't say anything is "safe" or "unsafe" in C.
Actually I'm also an expert on implementing advanced type systems, some of my works include:
1) A row-polymorphic JavaScript dialect [5]. 2) A tiny theorem prover with Lean 4 syntax in less than 1K LOC [6]. 3) A Rust dialect with reuse analysis [7].
Language features like generics, compile-time eval, trait/typeclass, bidirectional typechecking are trivial for me, I successfully implemented them above.
For the retrofitted C, these features initially come to my mind:
1) Code generation directly to C, no LLVM IR, no machine code. 2) Module, like C++20 module, to eliminate use of headers. 3) Compile-time eval, type-level computation, like `malloc(int)` is actually a thing. 4) Tactics-like metaprogramming to generate definitions, acting like type-safe macros. 5) Quantitative types [8] to track the use of resources (pointers, FDs). The typechecker tells the user how to insert `free` in all possible positions, don't do anything like RAII. 6) Limited lifetime checking, but some people tells me lifetime is not needed in such a language.
Any further insights? Shall I kickstart such project? Please I need your ideas very much.
[1]: https://vi.wikipedia.org/wiki/V%C3%B5_L%C3%A2m_Truy%E1%BB%81n_K%E1%BB%B3
[2]: https://e.huawei.com/en/products/optical-access/ma5800
[3]: https://docs.ceph.com/en/reef/cephfs/
[4]: https://lore.kernel.org/rust-for-linux/Z7SwcnUzjZYfuJ4-@infradead.org/
[5]: https://github.com/rowscript/rowscript
[6]: https://github.com/anqurvanillapy/TinyLean
[7]: https://github.com/SchrodingerZhu/reussir-lang
[8]: https://bentnib.org/quantitative-type-theory.html
There are approaches with at least partly the same goals as you mentioned, e.g. Zig. Personally I have been working on my own C replacement for some time which meets many of your points (see https://github.com/micron-language/specification); but the syntax is derived from my Oberon+ language, not from C (even if I use C and C++ for decades, I don't think it's a good syntax); it has compile-time execution, inlines and generic modules (no need for macros or a preprocessor); the current version is minimal, but extensions like inheritance, type-bound procedures, Go-like interfaces or the finally clause (for a simple RAII or "deferred" replacement) are already prepared.
> There are approaches e.g. Zig.
Yes! Zig has done a great job on many C-related stuff, e.g. they've already made it possible to cross-compile C/C++ projects with Zig toolchain years ago. But I'm still quite stupidly obsessed with source-level compatibility with C, don't know if it's good, but things like "Zig uses `0xAA` on debugging undefined memory, not C's traditional `0xCC` byte" make me feel Zig is not "bare-bone" enough to the C world.
> Micron and Oberon+ programming language.
They look absolutely cool to me! The syntax looks inspired from Lua (`end` marker) and OCaml (`of` keyword), CMIIW. The features are pretty nice too. I would look into the design of generic modules and inheritance more, since I'm not sure what a good extendability feature would look like for the C users.
Well BTW, I found there's only one following in your GitHub profile and it's Haoran Xu. Any story in here lol? He's just such a genius making a better LuaJIT, a baseline Python JIT and a better Python interepreter all happen in real life.
> The syntax looks inspired from Lua (`end` marker) and OCaml (`of` keyword), CMIIW
Oberon+ and Micron are mostly derived from Wirth's Oberon and Pascal lineage. Lua inherited many syntax features from Modula-2 (yet another Wirth language), and also OCaml (accidentally?) shares some keywords with Pascal. If you are interested in even more Lua similarities, have a look at https://github.com/rochus-keller/Luon, which I published recently, but which compiles to LuaJIT and thus serves different use-cases than C.
> I would look into the design of generic modules
I found generic modules to be a good compromise with simplicity in mind; here is an article about some of the motivations and findings: https://oberon-lang.github.io/2021/07/17/considering-generic...
> Haoran Xu, making a better LuaJIT
You mean this project: https://github.com/luajit-remake/luajit-remake? This is a very interesting project and as it seems development continues after a break for a year.
We seem to have the same desire for a “cleaned up C.” Could you say more about how metaprogramming would work? I doubt you want to put lifetimes into the type system to any degree. The reason C compiles so much quicker than C++ is the lack of features. Every feature must be crucial. Modules are crucial to preserving C.
> We seem to have the same desire for a “cleaned up C.”
That's so great! But sad that no enough ideas and argument came up here. :'(
> How metaprogramming would work?
When it comes to "tactics" in Coq and Lean 4 (i.e. DSL to control the typechecker, e.g. declare a new variable), there are almost equivalent features like "elaborator reflection" in Idris 1/2 [1] (e.g. create some AST nodes and let typechecker check if it's okay), and most importantly, in Scala 3 [2], you could use `summonXXX` APIs to generate new definitions to the compiler (e.g. automatically create an instance for the JSON encoding trait, if all fields of a record type is given).
So the idea is like: Expose some typechecker APIs to the user, with which one could create well-typed or ready-to-type AST nodes during compile time.
[1]: https://docs.idris-lang.org/en/latest/elaboratorReflection/e...
[2]: https://docs.scala-lang.org/scala3/reference/contextual/deri...
> Lifetime and compilation speed.
Yes exactly, I was considering features from Featherweight Rust [3], some subset of it might be partially applied. But yes it should be super careful on bringing new features in in case of compilation speed.
It's also worth to mention that C compiler itself would do some partial "compile-time eval" like constant folding, during optimization. I know some techniques [4] to achieve this during typechecking, not in another isolated pass, and things like incremental compilation and related caching could bring benefits here.
[3]: https://dl.acm.org/doi/10.1145/3443420
[4]: https://en.wikipedia.org/wiki/Normalisation_by_evaluation
> Every feature must be crucial.
I want to hear more of your ideas on designing such language too, and what's your related context and background for it BTW, for my curiosity?
I think you don't need any rants but here it goes anyway.
Ditching headers does not solve anything at least if your language targets include performance or my beloved example Gamedev =) . You will have to consume headers until operating systems will not stop using them. It is a people problem not language problem.
Big elephants in the room I do not see in your list:
1) "threading" was bolted onto languages like C and C++ without much groundwork. Rust kinda has an idea there but its really alien to everything I saw in my entire 20+ career with C++. I am not going to try to explain it here to not get downvoted into oblivion. Just want you to think that threading has to be natural in any language targeting multicore hardware.
2) "optimization" is not optional. Languages also will have to deal with strict aliasing and UB catastrophes. Compilers became real AGI of the industry. There are no smart developers outsmarting optimizing compilers anymore. You either with the big compilers on optimization or your language performance is not relevant. Providing even some ways to control optimization is something sorely missed every time everything goes boom with a minor compiler update.
3) "hardware". If you need performance you need to go back to hardware not hide from it further behind abstract machines. C and C++ lack real control of anything hardware did since 1985. Performant code really needs to be able to have memory pages and cache lines and physical layout controls of machine code. Counter arguments that these hardware things are per platform and therefore outside of language are not really helping. Because they need to be per platform and available in the language.
4) "libc" is a problem. Most of it being used in newly written code has to be escalated straight to bug reporting tool. I used to think that C++ stl was going to age better but not anymore. Assumptions baked into old APIs are just not there anymore.
I guess it does not sound helpful or positive for any new language to deal with those things. I am pretty sure we can kick all those cans down the road if our goal is to keep writing software compatible with PDP that somehow limps in web browser (sorry bad attempt at joking).
Exactly the kind of thoughts and insights I need from more of the users. Thank you for pointing out many concerns.
> Headers.
C++20 modules are left unstable and unused in major compilers there, but it’s a standard. And C is ironically perfect for FFI, as I said, almost every programming language speaks C: Rust WebAssembly API is extern C, JNI in Java, every scripting language, even Go itself talks to OS solely using syscall ABI, foreign-function calls are only possible with Cgo. C was not just an application/systems language for some sad decades.
> Big elephants.
Since I was in the zoo watching tigers:
Mostly three groups of people are served under a language: Application writers, library writers, compiler writers (language itself).
I narrowed down and started “small” to see if people writing programs crossing kernel and user space would have more thoughts about C since it’s the only choice. That’s also my job, I made distributed block device (AWS EBS replacement) using SPDK, distributed filesystem (Ceph FS replacement) using FUSE, packet introspection module in router using DPDK. I know how it feels.
Then for the elephants you mentioned, I see them more fitted into a more general library and application development, so here we go:
> Threading.
Async Rust is painful, Send + Sync + Pin, long signatures of trait bounds, no async runtimes are available in standard libraries, endless battles in 3rd party runtimes.
I would prefer Go on such problems. Not saying goroutines and channels are perfect (stackful is officially the only choice, when goroutine stacks somehow become memory intensive, going stackless is only possible with 3rd party event loops), but builtin deadlock and race detection win much here. So it just crashes on violation, loops on unknown deadlocks, I would probably go to this direction.
> Optimization, hardware.
Quite don’t understand why these concerns are “concerns” here.
It’s the mindset of having more known safer parts in C, like a disallow list, rather than under a strong set of rules, like in Rust, an allowlist (mark `unsafe` to be nasty). Not making everything reasonable, safe and generally smart, which is surreal.
C is still, ironically again, the best language to win against assembly upon an optimizing performance, if you know these stories:
- They increased 30% speed on CPython interpreter recently on v3.14.
- The technique was known 3 years ago to be applied in LuaJIT-Remake, they remade a Lua interpreter to win against the original handwritten assembly version, without inline caching.
- Sub-techniques of it exist more than a decade even it’s in Haskell LLVM target, and they theoretically exist before C was born.
It is essentially just an approach to matching how the real abstract machine looks like underneath.
> libc.
Like I said, C is more than a language. Ones need to switch a new allocator algorithm upon malloc/free, Rust quits using jemalloc by default and uses just malloc instead. Libc is somewhat a weird de facto interface.
I guess I need to illustrate my points a bit because I never needed to poke kernels and my concerns are mostly from large games. I am trying to imagine writing large games in your language so please bear with me for a moment.
>Modules
Nobody plans to provide other interfaces to oses/middlewares/large established libraries. Economy is just not there.
>Threading
I was not talking about I/O at all. All of that you mention will be miles better in any high level language because waiting can be done in any language. Using threads for computation intensive things is a niche for low level languages. I would go further say that copying stuff around and mutexes also will be fine in high level languages.
>Optimization/Hardware
Is very important to me. I don't know how it was not relevant to your plan of fixing low level language. Here goes couple of examples to try to shake things up.
The strlen implementation in glibc is not written in C. UB just do not allow to implement the same algorithm. Because reading up until memory page end is outside of abstract machine. Also note how sanitizers are implemented to avoid checking strlen implementation.
Pointer provenance that is both present in each major compiler and impossible to define atm. You need to decide if your language goes with abstract machine or gcc or clang or linux. None of them agree on it. A good attempt to add into C standard a logical model of pointer provenance did not produced any results. If you want to read up on that there was HN thread about it recently.
>libc
I am pretty sure I can't move you on that. Just consider platforms that need to use new APIs for everything and have horrendous 'never to be used' shims to be posix 'compatible'. Like you can compile legacy things but running it does not make sense. Games tend to run there just fine because games used to write relevant low level code per platform anyway.
> Imagine writing large games in your language.
You don’t. Read the features I listed. One ends up with a C alternative frontend (Cfront, if you love bad jokes) including type system like Zig without any standard library. No hash tables, no vectors. You tended to write large games with this.
Like I said the main 3 groups of users, if you’re concerned about application writing, ask it. Rest of the comments talked about possible directions of langdev.
> Modules.
You write C++ and don’t know what a standard is. Motivating examples, real world problems (full and incremental compilation, better compilation cache instead of precompiled headers), decades spent on discussions. Economy would come for projects with modern C++ features.
> Threading.
If you know Rust and Go, talk about them more. Go creates tasks and uses futexes, with bare-bone syscall ABI. Higher level primitives are easy to use. Tools and runtime are friendly to debugging.
I wrote Go components with channels running faster than atomics with waits, in a distributed filesystem metadata server.
On CPU intensiveness, I would talk about things like automatic vectorization, smarter boxing/unboxing, smarter memory layout (aka levity, e.g. AoS vs SoA). Not threading niche.
> Strlen implementation and plan of low level programming.
Because I keep talking about designing a general purpose language. One can also use LLVM IR to implement such algorithms.
The design space here is to write these if necessary. Go source code is full of assembly.
> Pointer provenance.
Search for Andras Kovacs implementation of 2ltt in ICFP 2024 (actually he finished it in 2022), and his dtt-rtcg, you would realize how trivial these features could be implemented “for a new language”. I design new languages.
> libc.
Like I said, your happy new APIs invoke malloc.
C3 to C compiler could be a proposal.
Ah that should be good for source-level compatibility. But I'm thinking about extending existing codebase that crosses between the kernel and user space, e.g. DPDK, SPDK, FUSE, kernel module, etc. Curious that how C3 would be adopted in such projects.
Start small.
And then? https://github.com/anqurvanillapy/TinyLean
Very very interesting for me. I always wanted to do something similar for Maude in Golang (Python is not a bad choice).
Currently my focus is on data engineering, but I can use it as an inspiration.
I talked about C3 to C translator, this is what I said start small.