Evaluating the Performance of the Hugging Chat Assistant as a Grammar Checker for EFL Learners: A Comparison with Grammarly View Digital Media

Poster Session
Hsin Yueh Yang  

With the rise of artificial intelligence technologies, large language models (LLMs) and natural language processing (NLP) systems have given rise to a variety of AI assistants. In particular, AI assistants tailored to learners’ needs have great potential as helpful grammar checkers for improving learners’ writing ability and reducing teachers’ workload for checking grammar errors. However, the validation of AI-based grammar checkers requires further research. This study evaluates the effectiveness of the Hugging Chat Assistant, powered by Meta Llama 3, as a grammar checker for English as a Foreign Language (EFL) learners. The researcher developed custom prompts for the Hugging Chat Assistant to detect grammar errors based on the principles from existing studies and compared its performance to Grammarly. A dataset of 100 grammatically incorrect sentences, targeting five common grammar errors faced by intermediate learners, was used to assess the tools’ precision and recall rates. The findings suggest that the Hugging Chat Assistant outperforms Grammarly in terms of accuracy and sensitivity in detecting grammar errors. Additionally, its ability to provide contextualized and personalized explanations was evaluated positively by English major college students. However, a questionnaire revealed that EFL learners hold mixed opinions regarding the user interface and feedback from both checkers on their writing tasks. While the Hugging Face Assistant demonstrates significant potential, further research is needed to validate the findings with a larger dataset and through long-term studies to track user experiences and the potential impact of these tools on EFL learners’ writing development.