Design and build up the realtime advertisement order attribution mechanism based on Redis to support more than 10k place order QPS.
Spearheaded the design and implementation of the next-generation advertisement reporting system, decouple the data processing module and data storage module to double the throughput and support 1 million realtime data processing QPS, employed the lambda architecture to streamline both online and offline data processing flows catering to diverse downstream users.
Utilized the OLAP database Druid to build up the new reporting system with PB level data. Compared with the old system based on KV database, not only save more than 50% storage size and obviously reducing the cost, but also reduce the development difficulty and improve the system robustness.
Senior Software Engineer
Shopee
01.2023 - Current
Implemented a Spark-based offline reporting processing service, introducing the offline backfilling data ability and enhancing flexibility for downstream integration into the data warehouse.
Implement the new ads reporting querier system, flexible to read multiple data layer with business use-cases and reduce 80% code.
Introduce the CPM data processing flow in our ads tracking and ads reporting system, design and implement the live stream ads data flow.
Engineered a robust cost per mile deduction mechanism for our advertisement deduction service, streamlining the cost calculation process and improving overall efficiency.