Search for a command to run...
Is programming art, engineering or both? This is what I try to figure out, while striving towards user friendliness, simplicity and synergy. Philosophical when writing, precise when coding. DCI (Data, Context, Interaction) has been my major area of research for the last 10 years. Creator of Superforms for SvelteKit.
No comments yet. Be the first to comment.
Welcome back to the DCI series! Here are the previous parts if you're new. Error handling in Contexts As promised, the topic for today is error handling with the least amount of surprise. This could lead us to ask what is surprising when coding and d...
What is this, the bold statement of the year? Only two rules needed in UX, an area filled with so many "must-read" books, expensive design schools and huge project checklists? Call me bold, but I'd say yes. You have common sense within you. You have ...
How easy it is to create a multi-language site nowadays!
Ready for more DCI? If you're new, check out the other parts first. We talked a lot about Roles in the previous part, concluding that Roles are a quite natural way of expressing ourselves in code. This has led to many attempts at adding Roles to lang...
Superforms is a popular SvelteKit library that simplifies numerous aspects of form handling: Validation, error handling, nested data, loading spinners, browser constraints, and much more. About a month after the announcement that Superforms won Svelte Hack 2023, version 1.0 was released (in June 2023), and it has been a quite stable journey since then.
Thanks to a comprehensive test suite and browser automation testing, over 170 issues have been closed on Github with almost no regressions, and the new features added from 1.0 to the current 1.11 are always non-intrusive and backward-compatible.
This is much thanks to the simple API, which works on the deconstruct principle that you only expose what you need. So when you call superForm (basically the only API you need on the client), the initial call is very simple:
// Only deconstruct what you need
const { form } = superForm(data.form);
And as you move along and introduce additional features like client-side validation, error handling, loading spinners, you just keep deconstructing what you need, together with configuration as an optional parameter:
const { form, errors, enhance, delayed } = superForm(data.form, {
resetForm: true
});
This encapsulates the whole form behavior within a single function call, making things simple to both understand and expand upon.
So in general, all is well with version 1, but there's one thing that comes up now and then, touching on a problem that, when you enter the rabbit hole, goes quite deep.
This text is written in two parts, the first one is the background to the problem, and the second is how to solve it.
Looking at the numerous validation libraries out there, it's obvious that validation is quite a popular problem to tackle, having led to a friendly competition where the focus seems to be mostly on performance and bundle size. That is certainly a too simple view in most cases, so I apologize, but I say it because the problem itself - validation - has already been solved.
Most validation libraries don't add any brand-new features. Instead, they bring higher speed, smaller size and a different syntax, where at least the first two can be useful, unless it's the evil premature optimization.
More could be said about this, but even if we accept that people fill the ecosystem with similar libraries and consider that a good thing for everyone, there's a deeper issue at hand.
A frequent question for Superforms is "Does it support validation library X?" If it did, this writing wouldn't exist, so naturally the answer is no, it only supports Zod. This can be disappointing, given that there are smaller and faster libraries out there. So am I a Zod fanboy, made a big oversight, or is there another reason for not supporting any other library?
I do think Zod is great, but I did plenty of research before settling on it because there's a big difference between a schema validation library, which just handles validating data, and a form validation library like Superforms, which must handle numerous other aspects of the validation process. What others? Let's take a look at a rather overlooked one, type safety.
The idea with Superforms is that the validation schema encapsulates a single form, so the schema can be a single source of truth for the form, including its data, which type can be inferred from the schema. This means that if you have a schema like:
const schema = z.object({
name: z.string().min(2)
email: z.string().email()
})
The inferred type of the data is:
type Schema = z.infer<typeof schema>;
// Resulting type:
{
name: string;
email: string;
}
Note that the types here are string, not string | undefined, since the fields aren't optional in the schema. So when you're using this data, the fields must be set to a string. You can't just use an empty object, any operation on the supposed string data will then fail.
Some validation libraries solve this with default values, but adding that to every field is rather tedious:
const schema = z.object({
name: z.string().min(2).default('')
email: z.string().email().default('')
score: z.number().default(0)
// ...and 10+ more fields
})
Others don't care, as they just focus on the data validation. In either case, it puts an additional burden on the library consumer - supply default values, or your data won't be type-safe.
Having an extra data structure with default values for every schema is not only tedious but redundant since the information is already there. Looking at the schema, you know that name and email are strings. But how to access this information programmatically? This leads us to the problem at hand: You must be able to introspect the schema, to avoid burdening the user.
Introspection is the ability to examine the type or properties of an object at runtime. A related concept is reflection, which goes a step further and also allows manipulation of these properties.
This is essential for form validation because if it would be possible to introspect on the above schema, we could loop through the properties and see that name is a string, and then set a default value, an empty string in this case, that makes the data type-safe. No burden on the user, no additional data structure.
You may have figured out the problem already - which of the popular validation libraries support introspection?
Assuming that it must be a documented and quite stable feature, the answer is, as of November 2023, not many except Zod do.
Some libraries like Ajv are using JSON Schema, which is a formidable attempt at a validation standard, self-reflecting since it's just JSON. Compared to the fluent syntax of a validation library with IDE autocompletion, it can appear quite verbose though. Compare this Zod schema:
const schema = z.object({
name: z.string().min(2)
email: z.string().email()
})
To the equivalent JSON Schema:
{
"type": "object",
"properties": {
"name": { "type": "string", "minLength": 2 },
"email": { "type": "string", "format": "email" }
},
"required": [ "name", "email" ],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}
Because of that, libraries like Typebox have been created to simplify JSON Schema generation. But wouldn't it be nice to use only your favorite library, instead of having to jump through the hoops of
Typebox → Ajv → Validate data → Get default values from JSON Schema → Type-safe form data
Zod makes this quite easy since you can access the schema as objects:
function defaultValue(zodType: ZodTypeAny) {
switch(zodType._def.typeName) {
case 'ZodString': return '';
case 'ZodNumber': return 0;
case 'ZodObject': {
const zodObject = zodType as AnyZodObject
for (const [key, value] of Object.entries(zodObject.shape)) {
// Call defaultValue recursively to assign each property
}
}
// ... and so on
}
}
(typeName is used instead of instanceof to avoid prototype mismatch, which can happen if packages use different versions of Zod. Kudos to the author Colin McDonnell for acknowledging this.)
Now, for smaller systems, a few default values or extra data structures for the schemas could be manageable. But it gets more problematic when we look at another aspect, one where introspection turns into a dire need.
HTML5 introduced constraint validation, attributes on the form elements that improve validation by giving feedback in the browser, even if Javascript is disabled.
They can be easily added to any input field:
<input name="name" required="true" minlength="2" />
However, given that we have a schema with the purpose of defining validation constraints, applying these manually on every field is also quite redundant. Mapping the schema to browser validation constraints would be quite preferable, since there are plenty of attributes to deal with. Min/max values for numbers, the essential required attribute, pattern matching, etc.
To do this, you need to be able to extract constraints based on every schema field. The required attribute is an especially good example since it requires knowledge about whether the field is optional or nullable.
With Superforms introspecting the Zod schema, you'll get this for free and can use the constraints just by spreading its variable on an input field:
<script lang="ts">
export const data;
const { form, constraints } = superForm(data.form);
</script>
Name: <input name="name" bind:value={$form.name} {...$constraints.name} />
(Thanks Svelte for this cool feature!)
This is very convenient and useful, but puts some demands on the introspection capabilities of the validation library. Implementing that requires some planning beforehand, or a larger rewrite otherwise, so it may not be a high priority. But by not supporting introspection, the library is doing its users a considerable disservice.
Every validation library has its DSL (domain-specific language) for the domain-specific task of validating data. This DSL is also known as the schema syntax. The syntax itself is a part of the programming language and is not easily accessible at runtime, so we must rely on the library to expose the schema metadata through its API. If we cannot access and transform it into something else, suddenly your library of choice may force you to manually handle default values and constraints separately, even though you see everything you need to automate it, every time you look at the schema.
This is essentially the same as being committed and locked into a specific vendor, that refuses to open up a way of accessing its data or integrating its features.
That's why I'm strongly advocating that the popular validation libraries, and all future ones, must add introspection capabilities, even if it's more enjoyable to work on shaving microseconds off the benchmark comparison instead.
Of course, due to the nature of Open Source Software, it's not possible to demand this of the library maintainers. Most of us do it in our spare time, because it's enjoyable and interesting, like a hobby. But here it is in writing, at least, and if the point of publishing a library is about doing something good for others (otherwise, please don't publish it), I hope that adding a decidedly important feature is desirable as well as tinkering with the fun parts.
To summarize part 1, the problem is as follows:
We want to use a favorite validation library together with Superforms.
We want the data to be inferred from its schema, for type-safety and DX.
We don't want to deal with default values and constraints by hand - they should be derived from the schema at runtime.
And of course, without feeling a bit demanding on the open-source contributors that work for free, we want all this even if our favorite validation library doesn't have any introspection capabilities! In which case we're prepared to make things a bit more redundant, to enjoy the speed, size and/or syntax of the chosen library.
This may look daunting, but here's the upside of OSS - there may be a lot of libraries out there, some similar to each other, but thanks to the sheer amount, there will also be libraries that solve a specific problem for almost anything.
As mentioned, the "simple problem" of pure validation has been solved already. We want to take some data as input and get validation errors as output. As this is what validation is about at its core, it should be possible to unify most libraries to solve our first problem.
And it comes as no surprise that this has been made already, by a library called TypeSchema. It has adapters currently for 14 validation libraries, unifying the validation process in a common API. If Superforms was only about error reporting, we'd be done by now. We're not, but this is still a tremendous help.
What about problem 2, type inference for the form data? This is even simpler. TypeSchema handles it very well, as can be seen by its coverage table. The rest, like transforming the inferred data type to an error type, is handled by Typescript and its mapped types.
But we also need introspection. Is there a unified solution even for that?
Here's where JSON Schema comes in as a true savior. If we can generate a JSON Schema for each library and its schemas, we have a standard to follow, which can be converted to default values and constraints with ease. This is how standards should be used, behind the scenes, so their verbose style won't get in the way of users. Many thanks to the JSON Schema creators for taking the time to painstakingly define - and thereby in detail solve - the problem of data validation.
And guess what, converting schemas back and forth has also been solved already! At least for a few libraries. Zod schemas can be converted to JSON Schema with a library not surprisingly called zod-to-json-schema. Other libraries like Ajv are already using JSON Schema, so the schema can just be lifted from there. If needed, we could use another nice piece of software like json-schema-to-ts to infer the types directly from the JSON Schema.
For the others, the best would be to raise the issue at the respective library repo, so a JSON Schema exporter can be built in, or at least introspection capabilities so a converter can be made. Until that, coding the JSON Schema by hand is a tedious option, so as a middle way, a data structure with default values could be converted to a JSON Schema with to-json-schema. There won't be any constraints with such a solution, but that may not always be a requirement anyway.
With all the problems solved, we can now formulate an API for Superforms version 2 based on the capabilities of each validation library. The function used to validate data in Superforms is called superValidate. It takes 1-3 parameters:
schema: The validation schema.
[Optional] data: Data that should be validated, can be from a DB or FormData in an HTTP request. If missing, use default values.
[Optional] options: Extra options.
const form = await superValidate(schema, data)
This is the best case, where everything is handled conveniently behind the scenes by mapping the schema metadata to JSON Schema.
const form = await superValidate(schema, data, { jsonSchema })
If a JSON Schema can be created by some other means, passing it as an extra option will ensure that default values and constraints are generated properly.
const form = await superValidate(schema, data, { defaults })
If a JSON Schema isn't available and there is no introspection, passing an object of default values will at least provide type safety. No constraints will be available, but everything else will work.
It can be difficult to detect which of the validation libraries the schema belongs to at runtime. Therefore, it may be useful to handle the introspection in an adapter function, which also would help with tree shaking, quite important when we're about to support 14+ validation libraries. We only want to include the runtime for the library being used. Here's what it could look like:
import { zod } from 'sveltekit-superforms/adapters'
const form = await superValidate(zod(schema), data)
These ideas are not final, and help is much appreciated. Please open an issue at the repo on Github, or join the Discord server so we can have a chat in the #v2 channel. There are also a few 2.0 milestone issues that can be worth checking out.
There we have the plan for making Superforms support almost any validation library! A few tests have been made already and it looks like there should be no problems in making it happen.
I estimate that the validation part, superValidate, will be finished rather quickly, as the other libraries will do most of the heavy lifting there. The client part, superForm, is more intricate but a lot will be reused from version 1. Some parts are still dependent on Zod or have to be rewritten though, like the client-side validation, because they are frankly a bit messy. The test suite and browser automation tests must also be ported.
Unfortunately, there's not enough time and money to do all this rapidly. Maintaining and giving support for version 1 takes up a bit of time already, but with donations, I can at least distribute the time and prioritize the development.
So if you want to see Superforms v2 happen, there are a few ways of donating on the Superforms website. Any amount helps, and a monthly donation of $10 or more will be listed on the Sponsors page. A huge thanks to the people that are sponsoring, or have donated in the past.
Whether you donate or not, you are welcome to the Discord server to have a chat, give a word of encouragement, and of course get help for the current version of Superforms. See you there!