Hostname: page-component-77c89778f8-gvh9x Total loading time: 0 Render date: 2024-07-20T07:39:12.011Z Has data issue: false hasContentIssue false
  • English
  • Français

Towards improving coherence and diversity of slogan generation

Published online by Cambridge University Press:  04 February 2022

Yiping Jin Open the ORCID record for Yiping Jin [Opens in a new window] ,
Akshay Bhatia ,
Dittaya Wanvarie  and
Phu T. V. Le
Yiping Jin
Affiliation:
Department of Mathematics and Computer Science, Faculty of Science, Chulalongkorn University, Bangkok, Thailand 10300
Akshay Bhatia
Affiliation:
Knorex, 140 Robinson Road, #14-16 Crown @ Robinson, Singapore 068907
Dittaya Wanvarie*
Affiliation:
Department of Mathematics and Computer Science, Faculty of Science, Chulalongkorn University, Bangkok, Thailand 10300
Phu T. V. Le
Affiliation:
Knorex, 140 Robinson Road, #14-16 Crown @ Robinson, Singapore 068907
*
*Corresponding author. E-mail: Dittaya.W@chula.ac.th
Get access Rights & Permissions [Opens in a new window]

Abstract

Previouswork in slogan generation focused on utilising slogan skeletons mined from existing slogans. While some generated slogans can be catchy, they are often not coherent with the company’s focus or style across their marketing communications because the skeletons are mined from other companies’ slogans. We propose a sequence-to-sequence (seq2seq) Transformer model to generate slogans from a brief company description. A naïve seq2seq model fine-tuned for slogan generation is prone to introducing false information. We use company name delexicalisation and entity masking to alleviate this problem and improve the generated slogans’ quality and truthfulness. Furthermore, we apply conditional training based on the first words’ part-of-speech tag to generate syntactically diverse slogans. Our best model achieved a ROUGE-1/-2/-L $\mathrm{F}_1$ score of 35.58/18.47/33.32. Besides, automatic and human evaluations indicate that our method generates significantly more factual, diverse and catchy slogans than strong long short-term memory and Transformer seq2seq baselines.

Keywords

Natural language generationSequence-to-sequence modelSlogan generation
Type
Article
Information
Natural Language Engineering , Volume 29 , Issue 2 , March 2023 , pp. 254 - 286
Check for updates
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abrams, Z. and Vee, E. (2007). Personalized ad delivery when ads fatigue: An approximation algorithm. In Proceedings of the International Workshop on Web and Internet Economics, Bangalore, India. Springer, pp. 535–540.CrossRefGoogle Scholar
Ackley, D.H., Hinton, G.E. and Sejnowski, T. J. (1985). A learning algorithm for boltzmann machines. Cognitive Science 9(1), 147–169.CrossRefGoogle Scholar
Alnajjar, K. and Toivonen, H. (2021). Computational generation of slogans. Natural Language Engineering 27(5), 575–607.CrossRefGoogle Scholar
Angeli, G., Premkumar, M.J.J. and Manning, C.D. (2015). Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China. Association for Computational Linguistics, pp. 344–354.CrossRefGoogle Scholar
Bahdanau, D., Cho, K. and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA.Google Scholar
Boigne, J. (2020). Building a slogan generator with gpt-2. Available at https://jonathanbgn.com/gpt2/2020/01/20/slogan-gen-erator.html (accessed 14 January 2020).Google Scholar
Bruce, N.I., Murthi, B. and Rao, R.C. (2017). A dynamic model for digital advertising: The effects of creative format, message content, and targeting on engagement. Journal of Marketing Research 54(2), 202–218.CrossRefGoogle Scholar
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J. and Charlin, L. (2019). Language gans falling short. In Proceedings of the International Conference on Learning Representations, New Orleans, Louisiana.Google Scholar
Cao, M., Dong, Y., Wu, J. and Cheung, J.C.K. (2020). Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 6251–6258.CrossRefGoogle Scholar
Cao, Z., Wei, F., Li, W. and Li, S. (2018). Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, New Orleans, Louisiana.CrossRefGoogle Scholar
Chen, S., Zhang, F., Sone, K. and Roth, D. (2021). Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online. Association for Computational Linguistics,pp. 5935–5941.CrossRefGoogle Scholar
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. and Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar. Association for Computational Linguistics,pp. 1724–1734.CrossRefGoogle Scholar
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46.CrossRefGoogle Scholar
Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 4171–4186.Google Scholar
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J.C.K. and Liu, J. (2020). Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 9320–9331.CrossRefGoogle Scholar
Durmus, E., He, H. and Diab, M. (2020). Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 5055–5070.CrossRefGoogle Scholar
Eyal, M., Baumel, T. and Elhadad, M. (2019). Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 3938–3948.CrossRefGoogle Scholar
Falke, T., Ribeiro, L.F., Utama, P.A., Dagan, I. and Gurevych, I. (2019). Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics, pp. 2214–2220.CrossRefGoogle Scholar
Fan, A., Lewis, M. and Dauphin, Y. (2018). Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics, pp. 889–898.CrossRefGoogle Scholar
Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y. and Gao, J. (2021). GO FIGURE: A meta evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online. Association for Computational Linguistics, pp. 478–487.CrossRefGoogle Scholar
Gao, X., Lee, S., Zhang, Y., Brockett, C., Galley, M., Gao, J. and Dolan, W.B. (2019). Jointly optimizing diversity and relevance in neural response generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, USA. Association for Computational Linguistics, pp. 1229–1238.CrossRefGoogle Scholar
Gatti, L., Özbal, G., Guerini, M., Stock, O. and Strapparava, C. (2015). Slogans are not forever: Adapting linguistic expressions to the news. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, pp. 2452–2458.Google Scholar
Gatti, L., Özbal, G., Stock, O. and Strapparava, C. (2017). To sing like a mockingbird. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain. Association for Computational Linguistics, pp. 298–304.CrossRefGoogle Scholar
Goodrich, B., Rao, V., Liu, P.J. and Saleh, M. (2019). Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, Alaska. Association for Computing Machinery, pp. 166–175.CrossRefGoogle Scholar
He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J. and Li, M. (2019). Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. Institute of Electrical and Electronics Engineers, pp. 558–567.CrossRefGoogle Scholar
Hermann, K.M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M. and Blunsom, P. (2015). Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, Montreal, Canada, pp. 1693–1701.Google Scholar
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation 9(8), 1735–1780.CrossRefGoogle ScholarPubMed
Holtzman, A., Buys, J., Du, L., Forbes, M. and Choi, Y. (2019). The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations, New Orleans, Louisiana.Google Scholar
Howard, J. and Gugger, S. (2020). Fastai: A layered API for deep learning. Information 11(2), 108.CrossRefGoogle Scholar
Hua, X., Sreevatsa, A. and Wang, L. (2021). DYPLOC: Dynamic planning of content using mixed language models for text generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics, pp. 6408–6423.CrossRefGoogle Scholar
Hughes, J.W., Chang, K.-h. and Zhang, R. (2019). Generating better search engine text advertisements with deep reinforcement learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, Alaska. Association for Computing Machinery, pp. 2269–2277.CrossRefGoogle Scholar
Iwama, K. and Kano, Y. (2018). Japanese advertising slogan generator using case frame and word vector. In Proceedings of the 11th International Conference on Natural Language Generation, Tilburg, The Netherlands. Association for Computational Linguistics, pp. 197–198.CrossRefGoogle Scholar
Jin, D., Jin, Z., Zhou, J.T., Orii, L. and Szolovits, P. (2020). Hooks in the headline: Learning to generate headlines with controlled styles. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 5082–5093.CrossRefGoogle Scholar
Kanungo, Y.S., Negi, S. and Rajan, A. (2021). Ad headline generation using self-critical masked language model. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers. Association for Computational Linguistics, pp. 263–271.CrossRefGoogle Scholar
Katragadda, R., Pingali, P. and Varma, V. (2009). Sentence position revisited: A robust light-weight update summarization ‘baseline’ algorithm. In Proceedings of the Third International Workshop on Cross Lingual Information Access: Addressing the Information Need of Multilingual Societies (CLIAWS3), Boulder, Colorado. Association for Computational Linguistics, pp. 46–52.CrossRefGoogle Scholar
Keskar, N.S., McCann, B., Varshney, L., Xiong, C. and Socher, R. (2019). CTRL - A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.Google Scholar
Kryscinski, W., McCann, B., Xiong, C. and Socher, R. (2020). Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 9332–9346.CrossRefGoogle Scholar
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V. and Zettlemoyer, L. (2020). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics, pp. 7871–7880.CrossRefGoogle Scholar
Li, J., Monroe, W. and Jurafsky, D. (2016). A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562.Google Scholar
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L. and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.Google Scholar
Lucas, D.B. (1934). The optimum length of advertising headline. Journal of Applied Psychology 18(5), 665.CrossRefGoogle Scholar
Luong, M.-T., Pham, H. and Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal. Association for Computational Linguistics, pp. 1412–1421.CrossRefGoogle Scholar
Matsumaru, K., Takase, S. and Okazaki, N. (2020). Improving truthfulness of headline generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics,pp. 1335–1346.CrossRefGoogle Scholar
Maynez, J., Narayan, S., Bohnet, B. and McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 1906–1919.CrossRefGoogle Scholar
Mieder, B. and Mieder, W. (1977). Tradition and innovation: Proverbs in advertising. Journal of Popular Culture 11(2), 308.CrossRefGoogle Scholar
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, volume 26, Lake Tahoe, Nevada, USA,pp. 3111–3119.Google Scholar
Misawa, S., Miura, Y., Taniguchi, T. and Ohkuma, T. (2020). Distinctive slogan generation with reconstruction. In Proceedings of Workshop on Natural Language Processing in E-Commerce, Barcelona, Spain. Association for Computational Linguistics, pp. 87–97.Google Scholar
Mishra, S., Verma, M., Zhou, Y., Thadani, K. and Wang, W. (2020). Learning to create better ads: Generation and ranking approaches for ad creative refinement. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. Association for Computing Machinery, pp. 2653–2660.CrossRefGoogle Scholar
Munigala, V., Mishra, A., Tamilselvam, S.G., Khare, S., Dasgupta, R. and Sankaran, A. (2018). Persuaide! an adaptive persuasive text generation system for fashion domain. In Companion Proceedings of the The Web Conference 2018, Lyon, France. Association for Computing Machinery, pp. 335–342.CrossRefGoogle Scholar
Nan, F., Nallapati, R., Wang, Z., Nogueira dos Santos, C., Zhu, H., Zhang, D., K., McKeown and Xiang, B. (2021). Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, pp. 2727–2733.CrossRefGoogle Scholar
Niu, X., Xu, W. and Carpuat, M. (2019). Bi-directional differentiable input reconstruction for low-resource neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, USA. Association for Computational Linguistics, pp. 442–448.CrossRefGoogle Scholar
Özbal, G., Pighin, D. and Strapparava, C. (2013). Brainsup: Brainstorming support for creative sentence generation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria. Association for Computational Linguistics, pp. 1446–1455.Google Scholar
Pagnoni, A., Balachandran, V. and Tsvetkov, Y. (2021). Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp. 4812–4829.CrossRefGoogle Scholar
Phillips, B.J. and McQuarrie, E.F. (2009). Impact of advertising metaphor on consumer belief: Delineating the contribution of comparison versus deviation factors. Journal of Advertising 38(1), 49–62.CrossRefGoogle Scholar
Qi, P., Zhang, Y., Zhang, Y., Bolton, J. and Manning, C.D. (2020). Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online. Association for Computational Linguistics, pp. 101–108.CrossRefGoogle Scholar
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog 1(8), 9.Google Scholar
Reddy, R. (1977). Speech understanding systems: A summary of results of the five-year research effort. Carnegie Mellon University.Google Scholar
Rogers, A., Kovaleva, O. and Rumshisky, A. (2020). A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics 8, 842–866.CrossRefGoogle Scholar
Scialom, T., Lamprier, S., Piwowarski, B. and Staiano, J. (2019). Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics, pp. 3237–3247.CrossRefGoogle Scholar
See, A., Liu, P.J. and Manning, C.D. (2017). Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada. Association for Computational Linguistics, pp. 1073–1083.CrossRefGoogle Scholar
Sun, J., Ma, X. and Peng, N. (2021). AESOP: Paraphrase generation with adaptive syntactic control. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics, pp. 5176–5189.CrossRefGoogle Scholar
Sutskever, I., Vinyals, O. and Le, Q.V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27, Montreal, Quebec, Canada, pp. 3104–3112.Google Scholar
Tenney, I., Das, D. and Pavlick, E. (2019). Bert rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics, pp.4593–4601.CrossRefGoogle Scholar
Tomašic, P., Znidaršic, M. and Papa, G. (2014). Implementation of a slogan generator. In Proceedings of 5th International Conference on Computational Creativity, volume 301, Ljubljana, Slovenia, pp. 340–343.Google Scholar
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, Long Beach, CA, USA.Google Scholar
Vempati, S., Malayil, K.T., Sruthi, V. and Sandeep, R. (2020). Enabling hyper-personalisation: Automated ad creative generation and ranking for fashion e-commerce. In Fashion Recommender Systems. Springer, pp. 25–48.CrossRefGoogle Scholar
Wang, A., Cho, K. and Lewis, M. (2020). Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 5008–5020.CrossRefGoogle Scholar
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K. and Weston, J. (2019). Neural text generation with unlikelihood training. In Proceedings of the International Conference on Learning Representations, New Orleans, Louisiana.Google Scholar
White, G.E. (1972). Creativity: The X factor in advertising theory. Journal of Advertising 1(1), 28–32.CrossRefGoogle Scholar
Williams, A., Nangia, N. and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.Google Scholar
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q. and Rush, A. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online. Association for Computational Linguistics, pp. 38–45.CrossRefGoogle Scholar
Zhang, H., Duckworth, D., Ippolito, D. and Neelakantan, A. (2021). Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), Online. Association for Computational Linguistics, pp. 25–33.Google Scholar
Zhang, J., Zhao, Y., Saleh, M. and Liu, P. (2020a). Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the International Conference on Machine Learning. PMLR, pp. 11328–11339.Google Scholar
Zhang, Y., Merck, D., Tsai, E., Manning, C.D. and Langlotz, C. (2020b). Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 5108–5120.CrossRefGoogle Scholar
Zhu, C., Hinthorn, W., Xu, R., Zeng, Q., Zeng, M., Huang, X. and Jiang, M. (2021). Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online. Association for Computational Linguistics, pp. 718–733.CrossRefGoogle Scholar