Adopting Parallel Processing for Rapid Generation of Transcripts in Multimedia-rich Online Information Environment
Mert Can Çakmak, Obianuju Okeke, Billy Spann, Nitin Agarwal
Abstract
The growth of online media platforms, particularly multimedia-rich social networks such as YouTube, has resulted in a demand for efficient data collection and analysis techniques. One of the critical data elements for multimedia-rich social platforms is the video transcript, which is not readily available through social platforms. Traditional methods for transcript generation are time-consuming and are challenged by the vast amount of data. This study proposes a methodology that leverages parallel computing and the Python multiprocessing library to improve the speed of transcript collection from YouTube. The methodology utilizes YouTube’s Transcript API to extract YouTube-generated transcripts and OpenAI’s Whisper model to generate transcriptions on videos without native YouTube transcriptions. Additionally, the Googletrans Translation API was used to translate transcriptions from non-English videos. The results showed a significant improvement in processing time and performance, enabling researchers to conduct various studies on a larger scope of YouTube data with ease. With parallel processing, the YouTube Transcript API showed a 2100.88% performance increase, the Whisper model showed a 29.45% improvement, and the Googletrans API showed a 738.46% increase compared to the sequential processing baseline using the same process. The total time consumption was reduced by 25.54% from 105.64 hours to 78.66 hours. The methodology developed in this study is not limited to YouTube and can be applied to other social media platforms, making it a versatile solution for data collection and analysis.