Skip to content

Add filter shader hooks #7582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 2, 2025
Merged

Add filter shader hooks #7582

merged 3 commits into from
Mar 2, 2025

Conversation

davepagurek
Copy link
Contributor

This is in service of #7188: the JS shader API will not generate full shaders, just shader hooks, so to be able to make filter shaders this way, we need filter shader hooks.

Changes

  • Creates a new baseFilterShader() method
  • Updated the filter runner in 2D mode to use WebGL 2 if available

Unexpected things during implementation

Passing a texture to a hook

In other hooks, we just pass in a struct with all the inputs. A sampler2D can't work in a struct, though, since it isn't a basic data type. In the past we've avoided this by first sampling the texture, then passing the resulting color into the hook. Filters might need to sample the texture multiple times, though, so this is unavoidable.

Instead, the hook format for filter shaders takes two parameters: vec4 getColor(FilterInputs inputs, in sampler2D content). The other weirdness is that it has to be an in sampler2D, not just a sampler2D, so we have to be aware that there may be qualifiers in hook parameters when parsing these into JS nodes for #7188.

Pre-multiplied alpha

One of the hooks changes I've made in 2.0 is to make shader hooks deal exclusively in unmultiplied alpha. We need to use premultiplied alpha in general for rendering to appear correct. If you want to output 50% opacity white, you would have to output vec4(0.5, 0.5, 0.5, 0.5), which is likely unintuitive since colors in the rest of p5 don't work this way. So, for other hooks, I divide out the alpha, pass that to the hook, and then assume the result is also unmultiplied, and multiply the alpha back in.

In filter shaders, since we're passing the texture in directly for users to sample themselves, we can't unmultiply the alpha for them. To deal with this, I've created a getTexture(in sampler2D tex, vec2 coord) that users should use instead. This does the same thing as the native texture(tex, coord), but it returns unmultiplied alpha. When we are creating the JS shader API, we should likely map all texture calls to this new function for more expected results.

@davepagurek
Copy link
Contributor Author

To make it easier to generate the JS types, I'm going to add some more to this PR to add an API to get the parameter types of hooks (hopefully tonight!) I'm thinking that you could call shader.hookTypes(hookName) and get back something like this:

{
  parameters: [
    {
      name: 'inputs',
      type: {
        typeName: 'FilterInputs',
        properties: {
          texCoord: {
            typeName: 'vec2'
          },
          canvasSize: {
            typeName: 'vec2'
          },
          texelSize: {
            typeName: 'vec2'
          },
        }
      }
    },
    {
      name: 'content',
      type: {
        qualifiers: ['in'],
        typeName: 'sampler2D'
      }
    },
  ],
  returns: {
    typeName: 'vec4'
  }
}

In TypeScript terms, that'd be:

type TypeInfo = {
  typeName: string
  qualifiers?: Array<string>
  properties: { [key: string]: TypeInfo }
}
type HookTypes = {
  parameters: Array<{ name: string; type: TypeInfo }>
  returns: TypeInfo
}

@lukeplowden
Copy link
Collaborator

lukeplowden commented Feb 28, 2025

To make it easier to generate the JS types, I'm going to add some more to this PR to add an API to get the parameter types of hooks (hopefully tonight!) I'm thinking that you could call shader.hookTypes(hookName) and get back something like this:

Thank you for the work on this. That will be really useful for generating the hook functions in the JS API. Once it's done I will pull it into my branch.

I am happy with the under-the-hood call to getTexture() to keep consistency, nothing to add there. Passing in sampler2D content makes sense too. What do you think about the more verbose sketchContent, canvasContent, or sketchTexture for the parameter name? As this API will be useful for beginners to shaders, the extra explicitness could help contextualise.

@davepagurek
Copy link
Contributor Author

Makes sense! I renamed it to canvasContent.

Also I added an implementation of hookTypes now!

Copy link
Collaborator

@lukeplowden lukeplowden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, I think it's ready to pull

@davepagurek davepagurek merged commit 8ce0269 into dev-2.0 Mar 2, 2025
2 checks passed
@davepagurek davepagurek deleted the filter-shader-hooks branch March 2, 2025 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants