Serving layer for sync/async HTTP requests on Numaflow

[vigith]

Summary

We need to build a serving layer (supporting HTTP 1 and 2) on top of the Numaflow engine to support both sync/async endpoints. The serving layer can also optionally run independently without Numaflow pipeline for low-latency endpoints, provided the processing logic is written using Numaflow Map SDK.

Example Use Cases

• Model serving using inference graph (powered by Numaflow DAG) for complex models
• Model execution for proxying traffic to simple models (without Numaflow) but models are interfaced using Numaflow Map SDK.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Tasks

Preview Give feedback

1. feat: serving and tracking endpoint for Numaflow #1765
   +0
2. controller to scaffold serving endpoint colocated with JetStream source #1813
   serving +1
3. Fault tolerant pod termination #1981
   serving +1
   
4. update servesink to tonic 0.12
5. enable HTTPS for Serving #1980
   serving +1
   
6. E2E Test for Serving #1979
   serving +1
7. TTL for expiring tracking and store entries
8. add metrics for monitoring (requests, failures,) #1982
   serving +1
   
9. publish serve image #1857
   serving +1
   
10. use published image in the container creation
11. avoid creating serving stream if serving source is not enabled #1843
    +0
    
12. UI to track messages for Serving
13. Support multiple serving sources in the pipeline #1842
    +0
14. rename serving dir to ext #1876
    rust +1
    
15. integrate with Redis ISB for serving source #2407
    area/controller serving +2
    
16. Serving as a builtin souce #2244
    9 of 12
    area/controller area/forward area/monovertex serving +4
    
17. endpoint to fetch result from the store using the given id
18. support flatmap operation in serving #2325
    +0
19. async serve endpoint
20. serving should work for monovertex
21. merge http source and serving
22. making tracking optional for http source
23. callback failures should fallback to direct DB insertion
24. integrate with external stores like DBD
25. Make serving sink as a builtin
26. Expose response serve functionality in SDKs for send the response back to the serving source #2408
    serving +1
    
27. Autoscaling support
28. document response_serve
29. we should only wait until the draining is complete, not a blind sleep