Show HN: PDF2MD – Rust+Redis+ClickHouse+VLLM conversion pipeline for PDFs https://bit.ly/4fCXhaj

Show HN: PDF2MD – Rust+Redis+ClickHouse+VLLM conversion pipeline for PDFs If you just want to use it, try here - https://bit.ly/4fBCowo . I think the LLM's are astoundingly good at converting complex powerpoint style infographics. I wouldn't normally think folks on HN would find this interesting as the general concept has been posted about already in the past few months. We were heavily inspired by Zerox[1]. However, the stack we went with was fun and over-engineered which is more likely to create interesting discussion. We use all the same tools at Trieve (our main product), but wanted to see if they would be a good fit for something that needed to get built in a tighter timeline and we think they were! Took us 2 weeks to get this setup end-to-end and it's by no means complete (see roadmap in linked README). However, it's cool that a relatively cookie cutter web service like this can be created with pure open-source dependencies and non-standard Rust tooling so quickly. Rust won't kill your startup! - Minijinja templates for the UI[2] - PDFObject for doc display in-browser[3] - actix/actix-web HTTP server framework[4] - Redis queue macro for worker async processing[5] - Clickhouse for task storage[6] - chm CLI to handle Clickhouse migrations[7] - MinIO S3 for object storage[8] [1]: https://bit.ly/3zRSvWH [2]: https://bit.ly/3CDcytc [3]: https://bit.ly/4fQaCMs [4]: https://bit.ly/4hSRpLE [5]: https://bit.ly/3Z45InU... [6]: https://bit.ly/4fEzabd [7]: https://bit.ly/4i33Lko [8]: https://bit.ly/4hTnD9P https://bit.ly/4hYYAC8 November 21, 2024 at 10:05PM