Deep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design System

Deep Learning in a Computational Model f or Conceptual Shifts in a Co-Cr eativ e Design System Pegah Karimi 1 , Mary Lou Maher 1 , Nicholas Da vis 1 , Kazjon Grace 2 1 UNC Charlotte, 2 The Univ ersity of Sydney 1 USA, 2 Australia pkarimi@uncc.edu, m.maher@uncc.edu, ndavis64@uncc.edu, kazjon.grace@sydne y .edu.au Abstract This paper presents a computational model for concep- tual shifts, based on a no velty metric applied to a v ector representation generated through deep learning. This model is integrated into a co-creativ e design system, which enables a partnership between an AI agent and a human designer interacting through a sketching can- vas. The AI agent responds to the human designer’ s sketch with a new sketch that is a conceptual shift: in- tentionally varying the visual and conceptual similarity with increasingly more novelty . The paper presents the results of a user study showing that increasing novelty in the AI contribution is associated with higher creati ve outcomes, whereas low nov elty leads to less creativ e outcomes. Introduction Creativ e systems are computational systems that either model human creati vity in some manner or are designed to support and inspire creati vity . Over the last few years, three main approaches to these systems hav e emerged: fully autonomous creative systems, creativity support tools, and co-creativ e systems. Fully autonomous creati ve systems, part of the field of computational creativity , are designed to generate creativ e artifacts or exhibit creati ve behaviors (Colton et al. 2015; Das and Gamb ¨ ack 2014). Creati v- ity support tools, on the other hand, are technologies that can support human creativity by accelerating or augment- ing some facets of the creativ e process (Shneiderman 2007; V oigt, Nieha ves, and Becker 2012). Finally , co-creati ve sys- tems incorporate concepts from both fully autonomous sys- tems and creativity support tools: they enable human users and computer systems to work together on a shared creativ e task (Da vis et al. 2015a; Y annakakis, Liapis, and Alexopou- los 2014). In this paper , we introduce the algorithms for a co- creativ e sketching tool called the Creati ve Sketching Part- ner (CSP), which in v olves collaboration between a designer and an AI agent on a shared design task. Figure 1 illus- trates the CSP tool, in which the design task is described at the top and the three sketches belo w represent the re- sponses to this task. The two sketches at the top represent the user’ s initial sketch on the left and the AI agent’ s re- sponding sketch and label for the sketch on the right. The Figure 1: The Creati ve Sketching Partner interf ace. sketch at the bottom of the can v as is the user’ s ne w sk etch, with the shaded re gion showing the user’ s additions inspired by the AI agent’ s sketch. The system utilizes a compu- tational model of conceptual shifts (Karimi et al. 2018b; Karimi et al. 2018a) to guide users tow ard different aspects of the design space based on the amount of visual and con- ceptual similarity to the user’ s sk etch input. V isual similarity entails identifying a sketch that shares some structural char- acteristics, whereas conceptual similarity identifies a con- cept that has some semantic relationship. W e present users with stimuli that ha ve either both high visual and conceptual similarity (like a pen and a pencil) or low visual and concep- tual similarity (like a dolphin and a chair). Karimi et al. (2018c) introduced a framework of ways to ev aluate creativity in co-creativ e systems. It was found that current co-creative systems research tends to focus on measuring the usability of the system, rather than on op- erationalising creativity . This demonstrates an opportunity for adopting metrics from computational creati ve systems in order to empower co-creativ e systems with the capacity to measure the creativity of their contrib utions to the output. For our conceptual shift model, we adopt one of the most commonly measured components of creati vity from compu- tational creativ e systems: nov elty (Grace et al. 2015). Nov- elty is associated with measuring how different an artifact is compared to another set of artifacts (Grace et al. 2015). The novelty can be based on a comparison with a univ er- sal set of artifacts, which we will call a universal measur e , or on a set of artifacts that the user has previously experi- enced, which we will call a personal measur e . In this paper , we use a uni versal measure based on a lar ge dataset of la- belled sketches and deep learning that enables two kinds of representation: one that enables a measure of visual similar - ity and one that enables a measure of conceptual similarity . From these metrics, we ha ve constructed a univ ersal com- posite measurement of nov elty that is a combination of the distance between feature vectors in the visual space and the conceptual space. W e hypothesize that, when a system provides stimulus in the form of design concept responses that are highly nov el to the user’ s design, it leads to more transformative creati ve outcomes. In these cases, the designer is able to draw upon distant visual and semantic features to inspire their creati ve process, such as adding features from another design do- main. In contrast, when the system displays stimulus design concepts that are less novel to the user’ s design, it corre- sponds to less creativ e outcomes. The features of similar designs do not pro vide highly novel input to the process, leading to design iterations that share many attrib utes with the designer’ s original sketch. T o explore this hypothesis we performed a user study utilizing a W izard of Oz system to see how altering the nov elty of the AI agent’ s response affected the creati vity of the user’ s response. Participants experienced three conditions: low , intermediate, and high nov elty in the system’ s response. After the sketching expe- rience, participants were interviewed and surveyed to deter- mine how the AI agent’ s responses affected their creati vity . W e found that, based both on our quantitative and qualitati ve results, the high nov elty conceptual shifts stimulated more creativ e thinking than the low nov elty ones. Related Research Over the last fe w decades, digital tools ha ve been introduced as a way to support design creativity (Johnson et al. 2009). These tools offer a variety of functions that allow designers to share their digital sketches and suggest ne w ideas to facil- itate creati vity . More recently , intelligent systems ha ve been dev eloped that enable collaboration with designers in real time. These systems, also referred to as computational co- creativ e systems, work alongside human users to encourage their creati vity , support inspiration, and stimulate the user to continue creating. V ie wPoints AI (Jacob et al. 2013) is an example of an artistic co-creati ve system that has applica- tions in dance and theater . It uses a compositional technique that perceiv es and analyzes human movements and gestures to facilitate an AI response in real time. Morai Maker (Guz- dial et al. 2019) is an example of a co-creativ e game le vel design tool that assists users in authoring game le vel content. Co-creativ e sketching systems are an acti ve area of re- search in the computational creativity community . One such example is the Drawing Apprentice, which is a co-creative drawing partner that collaborates with users in real time (Davis et al. 2015b). The system uses sketch recognition to identify objects dra wn by the user and selects a comple- mentary object to display on the screen. Complementarity is defined by the semantic distance between the user’ s sk etched object and the target object. DuetDraw (Oh et al. 2018) is another example of a co-creative sk etching tool that works alongside the user by recognizing what the user dra ws and drawing related content to complete a shared scene. In our work, we use visual and conceptual similarity to select an object from a distinct category to be drawn on the screen in order to support the design process. Instead of selecting a sketch from the same conceptual category , such as Drawing Apprentice, the CSP uses a computational model of concep- tual shifts (Karimi et al. 2018b) to determine an appropriate target sk etch from a dataset. Conceptual shifts in design can occur when a sketch of one concept is recognized as being similar to a sketch of an- other concept (Karimi et al. 2018b). Identifying and capital- izing on conceptual shifts is an important component of the design process, as it allows designers to perceiv e their de- sign ideas from dif ferent perspecti ves. There are two modes of perception that hav e been defined in design: seeing-that and seeing-as (Suwa and Tv ersky 1997). Seeing-that refers to the concrete properties of a sketch and their function in the ov erall design, whereas seeing-as refers to interpretation, in which sketch elements can be considered through multiple perspectiv es. Conceptual shifts hav e the potential to inspire designers to adopt the seeing-as mode of perception, explor - ing ho w their emer ging design could be connected to a vari- ety of distinct concepts presented as stimuli. Identifying conceptual shifts could also help users over - come design fixation (Purcell and Gero 1996). Designers often have a hard time disengaging from the ideas the y de- veloped and learned o ver time. This effect, called fixation, may be reduced by presenting designers with a sketch of another object that shares some visual and conceptual in- formation. W e presume that, when presenting a conceptual shift successfully triggers seeing-as perception, a designer could be distracted from fixation, and potentially develop nov el contributions to their design. This could lead to the discov ery of innov ativ e solutions for a design task. The study of creativ e design has lead to a characteriza- tion of dif ferent types of creati vity . Gero (2000) has intro- duced six forms of design creativity that can form the basis for computational aids: combination, exploration, transfor- mation, analogy , emergence, and first principles. Combina- tion happens when tw o distinct design concepts are added. Exploration relates to changing some variable v alues asso- ciated with a design concept. T ransformation in volves alter - ing one or more variables of a design concept through ex- ternal processes. Analogy is characterized by mapping be- tween structural elements of two dissimilar objects. Emer- gence occurs when extensional properties of a design con- cept are identified beyond the intentional ones. First princi- ples use computational knowledge to relate function to be- haviour and beha viour to structure. The CSP introduced in this paper can be considered a computational aid to design that can support the first four of these forms of creati vity in a co-creativ e design context: combination, exploration, trans- formation, and analogy . Quantifying Conceptual Shifts Quantifying conceptual shifts is challenging because con- cepts are not typically represented or ev aluated numerically . Our premise is that the larger the shift, the more creati ve the resulting design. In order to quantify the scale of a concep- tual shift between two sketches (in our case the user’ s sketch and the system’ s proposed response), we need a representa- tion space in which we can measure similarity or nov elty . The more similar the second sketch is to the first, the less nov el the second item is and (we hypothesize) the less likely that it will trigger a conceptual shift. When the two items are less similar , the more novel the stimulus and (again, we hypothesize) the more likely it will result in a conceptual shift. W e focus on nov elty in generating conceptual shifts be- cause it has been shown to be a ke y component in predicting creativity (Grace et al. 2015). The assumption in measuring nov elty is the existence of a representation that allo ws ob- jectiv e measurement of dif ference. In (Grace et al. 2015), the corpus of designs in the design space were represented as a set of features that formed the basis for correlation and regression analysis. The feature set was e xtracted from a database in which the information about the designs was manually entered as a set of features with categorical and numerical values. This representation enabled various ways to measure nov elty , b ut not a single nov elty score. In the CSP , we measure novelty by comparing two sketches: an initial sketch presented by the user and a second sketch selected from a large dataset of sketches. Novelty is a combination of two components: the visual similarity based on the visual data and the conceptual similarity based on the label for the sketch. W e use deep learning models to extract a vector representation in two design spaces: a visual space using a large dataset of sketches, and a semantic space us- ing a word embedding model. W e consider the nov elty to be a combination of the classification of visual novelty in the visual space and conceptual novelty in the word embedding space. W e classify novelty into three categories: lo w , intermedi- ate, and high. Low novelty occurs when two sketches share a large amount of visual and conceptual information, inter - mediate no velty is when two sketches share some visual and conceptual information, and high nov elty occurs when two sketches share little visual and conceptual information. W e presume that low no velty lies within the expectation of the user , and that the system’ s response might be most likely to help the designer add more details to their initial design. In- termediate novelty could instead inspire the designer to ex- plore possible new design ideas associated with their initial design. High novelty has the potential to widen the user’ s thinking process, making it more likely to help them in- corporate ne w design features from a completely different design space. Based on this presumption, we hypothesize that increasing the novelty of the CSP stimulus will corre- late with more creativ e outputs. Conceptual Shift Algorithm In this section we describe an AI model of conceptual shifts. The model selects an object from a database of sketches to be displayed on the canv as as a stimulus during a co-creati ve session. Our model has two components: visual similarity and conceptual similarity . V isual similarity recognizes pairs of sketches from distinct categories that share some underly- ing visual information. Conceptual similarity identifies the semantic similarity between the labels of the sketches. Figure 2 shows the computational model the AI agent uses to select a sketch of the desired level of novelty in response to the user’ s input. The visual similarity module computes the distances between the cluster centroids of distinct cate- gories and maps the user’ s input to the most similar sketches from categories to which it does not belong. The concep- tual similarity module takes the pairs of selected category names from the previous step and computes their semantic similarity . In this section, we describe ho w CSP generates a numerical value for visual and conceptual similarity and determines the conceptual shift candidates based on high, intermediate, and low no velty . V isual Similarity Module The visual similarity module uses a lar ge public dataset of human-drawn sketches, called QuickDraw! (QD) (Jongejan et al. 2016), with more than 50 million labeled sketches grouped into 345 categories. In preparation for calculat- ing visual similarity , we have 2 steps: a learning step and a clustering step. In the learning step, the sketches are used to b uild a vector representation of the sketch’ s features. In the clustering step, we use the resulting feature vectors for sketches in each cate gory to create clusters of visually simi- lar sketches. This process provides a feature vector represen- tation for calculating the novelty between the user’ s initial sketch and sketches in the QD dataset using visual similar- ity . Deep Learning Model of Sketches for V isual Similarity As in the case of natural images, sketches can also be pro- cessed as a grid of pixels, ( h, w, d ) , in which h is the height, w is the width, and d is the number of channels. Howe ver , in this case, d will be 1 because the sketches are monochrome. T o de velop a representation for visual similarity we em- ployed a con volutional neural network (CNN) model due to their success in providing high level visual information and discriminating visual appearances, such as shapes and orientations (LeCun, Bengio, and Hinton 2015). W e started with a pre-trained model, V GG16 (Simonyan and Zisserman 2014), with 13 con v olutional layers, two fully connected layers, and a softmax output layer . The model is primar - ily trained on the ImageNet dataset (Deng et al. 2009) that contains more than 20 million labeled natural images. W e then fine-tune this model on the QD dataset with the objec- tiv e of classifying a sketch into one of the 345 categories. W e use 30,000 training samples and 10,000 validation sam- ples per category , and trained for 1.5 million training steps. Observation shows that the accuracy reaches 52.1% after 1 million steps and remains the same afterw ards. W e e xtract a neural representation of each sketch by taking the output of the first fully connected layer , for 4096 v alues per sk etch. Howe ver , this model has low accuracy and a high compu- tational cost because of the large number of parameters in the V GG16 architecture and processing sketches as a grid of pixels. Figure 2: Computational steps for identifying conceptual shifts. T op: Identifying visually similar categories to the user’ s input. Bottom: Balancing visual similarity with conceptual similarity and identifying conceptual shifts with high, intermediate, and low no velty . In order to solve this problem, we tried another represen- tation of sketches: a sequence of pen strokes, inspired by the work done by Ha and Eck on Recurrent Neural Network drawing (Ha and Eck 2017). In this case, each stroke is a list of points with 3 elements: ( ∆ x, ∆ y , p ). ∆ x and ∆ y are the coordinates with respect to the previous point, and p is a binary number that determines whether the stroke is drawn or not (i.e. just moves the pen). Here we use a deep learning model called Conv olutional Neural Network-Long Short T erm Memory (CNN-LSTM) (Carb une 2017). The model has three one-dimensional con volutional layers and three LSTM layers. W e train the model from scratch on the QD dataset with the same objectiv e, training, and validation samples as the CNN-only model. Results show that, after 1 million training steps, accuracy reaches 73.4% and remains the same afterw ards. Each sketch is represented by the last LSTM layer , for 256 values per sketch. T able 1 summarizes the results for accurac y and the a verage-per cate gory infer- ence time for both models. Accuracy measures a true posi- tiv e rate, while inference time represents the total amount of time it takes to extract features from all sk etches of a cate- gory . The CNN-LSTM model is clearly both faster and more accurate, and we use it hereafter . Clustering visually similar sketches in each category The sketches in a category exhibit a large variability visu- ally . For our visual similarity measure to be meaningful, we group the sketches in each category into clusters and use the feature vector of the cluster centroid as the representa- tiv e sketch. This process is a form of denoising, where the intra-cluster variability is suppressed. W e perform cluster- ing using a K-means algorithm and determine the optimal number of clusters via the elbow method. By analyzing the variance versus the number of clusters, we observed that for most categories the optimal number of clusters is between 7 and 12—we set the number of clusters to 10 across all cat- egories. The distances between the cluster centroids from distinct categories are computed and stored in a matrix of size 3450 × 3450 : 10 clusters of sketches for each of 345 categories. Giv en the source sketch and label from the user, L S , we first extract visual features using the pre-trained CNN- LSTM model that produces 256 values. W e then locate the representativ e cluster within its category (according to the label of the user’ s sketch) by selecting the closest centroid based on the L2 (i.e. Euclidean) distance. Using the distance V GG-16 CNN-LSTM Accuracy 52.1% 73.4% Inference time 18,000S 960S T able 1: Classification accuracy and the inference time using two dif ferent deep learning models. matrix, we then select the top 20 most visually similar target clusters from other categories, L T , as the ones with mini- mum distance from the representati ve cluster . The similarity is computed as 1 − d v , where d v is the Euclidean distance normalized across the most visually similar candidates. As the similarity values for the selected tar get sk etches change smoothly , we classify those that fall in the top 33rd per- centile of the distribution as low novelty , between 33rd and 66th percentile as intermediate nov elty , and above 66th per - centile as high nov elty . Conceptual Similarity Module The conceptual similarity module uses a word embedding model (Mikolo v 2016) trained on the Google Ne ws corpus with 3 million distinct words. The visual similarity module provides a set of candidate sketches to the conceptual sim- ilarity module based on the categories of low , intermediate, and high novelty . W e extract the word2vec word embedding features (Mikolov 2016) from these cate gory names. The similarity between the category of the source sk etch and the selected target sketch is computed as 1 − d c , where d c is the cosine distance between the feature vectors of category names. The lar ger number indicates that the two sketch cate- gories are more likely to appear in the same context, whereas a smaller number indicates that the two are less associated with each other . In order to determine the conceptual shift categories, we select those where the visual and conceptual similarity are both high, medium, or low . This is done by selecting candidates for which the dif ference between vi- sual and conceptual similarity values are below 0.05 and the ov erall similarity component is computed as the average of visual and conceptual values. User Study W e conducted a user study to ev aluate the effecti veness of our conceptual shift model in a co-creative design ses- sion. W e inv estigated how the novelty of the system’ s re- sponse could inspire user creativity and correspond to differ- ent types of design behaviors. Our hypothesis is that increas- ing the novelty of the system’ s response can help designers add new features and/or functions from another design space to their initial drawing, thus leading to more creativ e out- comes. By contrast, when the system is in the lo w novelty condition, the designer is presented with the similar features to the initial drawing, which leads to less creativ e outcomes. In this study , we used a within-subjects design, such that each participant experienced three conditions with a two- minute break between them. In the first condition the de- sign task is a chair , and the system produces a result that is highly nov el with respect to the participant’ s sketch. In the second condition the design task is a streetlight, and the system produces a result associated with intermediate nov- elty . In the third condition the design task is a bridge, and the system produces a result that is classified as lo w nov elty . Participants were not aware whether they were in a high, in- termediate, or low nov elty condition. A context is provided to help guide each design task, such as “draw a streetlight for safety at night on a city street of a small to wn. ” When the system’ s output object is presented to the user , it is accom- panied by a label indicating what the object is. Each design task takes approximately 7 minutes. The order of the three conditions for each participant was randomized to account for any ordering ef fects. W e used an online sketching tool, called SketchT ogether (Bonazza 2019), that enables multiple users to contribute to a shared can vas in real time. This application allo wed us to run a W izard-of-Oz interaction for the user study in which we used the results of the deep learning model for determin- ing high, intermediate, and low novelty sketches, but a per- son performed the interaction of placing the selected sketch on the shared can v as. Participants underwent a 5-minute training session that included an explanation about the in- terface tool and the design tasks. After training, participants are asked to start the first design task. The instruction given to the participants were to draw an object according to the design task and iterate on that drawing based on inspiration from the system’ s response to their sk etch. Follo wing each experimental condition, we asked participants Likert scale surve y questions associated with that design session. The questions we asked after each task were: 1. Did the system’ s sk etch response inspire you to come up with creativ e ideas for your design objects? 2. Did the system’ s sketch response lead you to come up with a different type of design object? The answers to the surve y questions were recorded for quantitativ e analysis. After the last design session, we ask ed participants the following questions in an intervie w: 1. How did the sketches presented by the system affect your creativ e process? 2. W as it more helpful when the sketches presented by the system were more or less similar to your input? 3. In which of the three design tasks did the system’ s sketch inspire you most? 4. Do you have any comments for participating in this study? The answers to the interview questions were used for qualitativ e analysis. The entire session for each participant took almost 30 minutes. Results The user study included 24 participants recruited from the College of Architecture at a public univ ersity in North America. Gender distrib ution was 15 males and 9 females. The criterion for participating was whether students perform sketching frequently for their design practice. W e recorded surve y and interview responses for all participants. In this Figure 3: The total percentage of high, intermediate, and lo w survey responses for (a) inspired creative ideas, and (b) led to different design. section, we describe our analysis based on the participants’ responses in order to in vestigate our hypothesis. Quantitative Analysis W e compared the results from the user’ s feedback on the three design tasks associated with high, intermediate, and low novelty conditions. W e grouped the responses into high, neutral, and lo w ratings: 4 and 5 are considered high, 3 is neutral, and 1 and 2 are low . For each condition we count the number of ratings based on this grouping. Analysis of creative ideas Participants were asked to rate the responses provided by the system after each design session. W ith this question, we aimed to understand whether increasing the novelty of the system’ s response inspired their creati ve thoughts. W e found that 91.66% of the participants thought that the sys- tem’ s response inspired creativity when the system was in the high novelty condition (HNC) compared to 29.16% in the lo w novelty condition (LNC). These results indicate that when the system’ s response is more nov el with respect to the user’ s sketch (HNC), it is associated with more creativ e outcomes, which may encourage the user to come up with new design ideas for their initial drawing. When the system was in intermediate novelty condition (INC), 54.16% of the participants were highly inspired by the system’ s response. Figure 3a sho ws the distribution of the ratings for the three conditions. Analysis of design object inspiration T ransformational creativity happens when a designer changes one or more structural variables of the current de- sign object to produce new variables (Gero 2000). This im- plies that the system’ s response has the potential to inspire the user to transform some features of a design concept by adding new features from another design space related to the system’ s response. W e e xplored whether increasing the no v- elty of the system’ s response can lead to transformational creativity in which the participant’ s designed object signif- icantly de viates from their initial sketch. All participants rated high in response to changing their design when the system was in HNC. This indicates that when the system’ s response was less similar to the participant’ s input (HNC), they were able to transform their initial sketch. By contrast, when the system was in LNC, none of the participants re- ported that the system helped them come up with a dif ferent type of design object. when the system was in INC, 41.66% of the participants rated high in response to changing their design and 58.33% rated low or neutral (see Figure 3b). Qualitative Analysis T o understand how the nov elty of the system’ s response can help designers come up with creativ e ideas for their initial task we analyzed the participants responses to the interview questions conducted after the design tasks were complete. W e aimed to explore the relationship between stimulus nov- elty and design thinking. Thematic Analysis W e performed a thematic analysis of the responses the par- ticipants ga ve to the interview questions. Ov erall, three main themes were found from the interview answers. • The tool helps with the design process • High novelty helps changing the design • Low nov elty helps completing the design In the follo wing section, we elaborate on each of these themes. Supporting the design process Most participants found the tool useful, as it can help with the design-thinking process as well as iterating and gener- ating new design ideas. P11 ex emplifies ho w the sketching tool helped their design process, “ The sketches presented af- ter I did my initial sketch, change the cr eative pr ocess, mak- ing me think of differ ent object and using that design phi- losophy and then the second object to affect the first. ” This participant described ho w the system’ s output sketch helped them think of different design ideas and iterate on their ini- tial design sketch. This demonstrates that the tool gener- ally supports the iterati ve nature of the early design process. Additionally , P14 comments: “ it sort of help[ed] me to see how I think about design, like the y teach us just to design, I never r eally thought about how I go about that pr ocess of designing and so having this sort of precedent to work with is more useful to me. ” This participant sho ws the role such a tool could play in design education. It helps to provide precedents that can inform the design process and inspire additional thinking on the topic. P4 described how helpful the system is when they say , “I think the system’s response is very helpful, because it gives me a lever age on adding to my initial design or just give me some clue or hint to chang e my design to make it better . ” Here, the participant comments about how the tool helps them iterate on their design by adding or changing different elements of the initial sketch based on the ‘clues’ or ‘hints’ provided by the system’ s output. P5 agrees with this senti- ment when they said, “the way that we communicate is gr eat because you add something and I am going to r edesign it and so it’ s gr eat. ” This participant focused on the communi- cation channel established between the user and system, and described ho w this channel helped in the redesign process. In a similar vein, P25 describes ho w “it kind of guided me thr ough some con ventional ways of impr oving my design” which shows how the tool serves to shepherd users through the design process by pro viding new a venues to explore and inspiration to change the user’ s initial design. High novelty inspires changing the design W e found that high novelty conceptual shifts inspire partic- ipants to change the ov erall shape of their design by adding new features from another design space related to the tar - get sk etch. In this condition, 21/24 reported that it is more inspiring when the system’ s response is less similar to their initial design. P11 commented: “I think to create an inter - esting r esult it was mor e helpful to have a dissimilar object as opposed to a similar , because it allows you to change the form and differ ent ideas instead of just kind of a similar shape affecting it. ” This participant indicates that when the system’ s response is less similar to their initial design (high nov elty condition), it helps to change the structure, such that it is possible to incorporate different ideas from the target sketch. Similarly , P10 commented: “It was easier to make changes when it was mor e differ ent. I think when something is alr eady similar sometimes my brain already has a same set of ideas, but when I am pr esented with something dif- fer ent the contrast helps me to generate a new idea. ” This participant was able to come with a new idea when he/she was presented with a sketch that was less similar to the ini- tial drawing. When P16 was presented with a sketch of an aircraft- carrier after designing a chair , they described ho w the sys- tem’ s sketch opened up new possibilities for them, “The air craft-carrier may have chairs b ut it doesn’ t elicit specific form especially giving the pr ompt that is going to be at the kitchen table. Thinking about ne w possibilities that can hap- pen definitely opens the new design criteria. ” This example shows that the chairs of the aircraft-carrier introduced ne w design criteria that inspired the participant to sketch a ne w kitchen chair with the features of aircraft-carrier seats, such as more comfort. Additionally , when P21 was presented with a sketch of a speedboat after designing a chair , they also found new possibilities in the design space, “The r e- lationship between the two, even though they ar e used both in the same task or same function because of the dif fer ence that one is on water , one needs to be outdoor , the dif fer ent needs and purposes between the two was influencing me bet- ter to cr eate something new between them. ” Similarly , P22 used the features of the system’ s response to reason about their initial sketch, “The air craft, because of its curves and the materiality , so thinking about the skin of the material, maybe thinking about its curves so that led me to think about the curves which maybe helped me to think of armr est. ” In this example both the structure and the concept of the tar- get sketch inspired the participant to change the shape to be curvy as well as adding ne w functionalities such as armrests. Low novelty helps complete the design Overall, 3/24 participants commented that it is more help- ful when the sketch that is presented to them is more sim- ilar to their initial dra wing (low novelty condition). P4 ex- plains why the sketch of fence that was highly similar to their initial drawing of bridge was more helpful, “because ther e were clear features and structures that could help by adding, mainly the similar featur es. ” In this case, the par - ticipant preferred to finalize the original drawing by adding more details and structures rather than changing the exist- ing features. Similarly , P9 commented: “I lik e the pr oduct of end results when stuff [is] mor e similar . Because I could pull fr om the pr ofile of fence and add to the bridge...So, you take something fr om it and add it to your design. ” From both P4 and P9, we can conclude that when the system is in low nov elty mode the designer mainly adds more details to the initial drawing rather than transforming the shape or adding new features to the dra wing. Most participants found the lo w nov elty condition less helpful. F or instance, P12 de- scribed how they liked less similar designs, “I would say it was more helpful when it was less similar because then you ar e not just copying the instances fr om the other design. ” P8 agreed with this sentiment when they said: “high similarity is kind of within my expectation. ” In both cases of P8 and P12, the low novelty conceptual shift designs do not help to significantly change the original drawing. Instead, the y are used to combine some elements of the two sketches. P13 echoes this general viewpoint when they said: “I think if you ar e presenting something that is al- most exactly the same, you ar e going to intr oduce the same idea again. ” Similar to P8, this participant also emphasizes that lo w novelty conceptual shifts are within their expecta- tion. P22 also commented: “I feel that similar designs didn’ t give me as muc h cr eative fr eedom. ” These examples demon- strate that lo w novelty conceptual shifts may help to com- bine the elements of the tw o sketches, rather than encourag- ing the user’ s creativ e thoughts. Both likely hav e a role in co-creativ e design systems, serving different purposes. Conclusion This paper presents a computational model of conceptual shifts for a co-creativ e design system called the Creative Sketching P artner . The tool is meant to inspire design cre- ativity by presenting a sketch of a distinct category that shares some visual and conceptual information with the user’ s input sketch. W e describe the role of deep learning in creating a representation space for measuring distance be- tween the visual and conceptual features of a sketch. W e hav e detailed the process for classifying potential response sketches as lo w , intermediate, or high novelty with respect to the designer’ s sketch. A user study is presented in which the participants are giv en a design task and then experience three different versions of the tool: low , intermediate, and high no velty responses. Both quantitati ve and qualitati ve re- sults from the user study demonstrate that the high novelty conceptual shift designs inspire creative thinking more than the low no velty condition. Acknowledgements The research reported in this article is funded by NSF IIS1618810 CompCog: RI: Small: Pique: A cognitive model of curiosity for personalizing sequences of learning resources. References [Bonazza 2019] Bonazza, M. 2019. SketchT ogether . https://sketchtogether.com . Accessed: 2019- 02-27. [Carbune 2017] Carbune, V . 2017. Recur- rent neural networks for drawing classification. https://www.tensorflow.org/tutorials/ sequences/recurrent_quickdraw . [Colton et al. 2015] Colton, S.; Halskov , J.; V entura, D.; Gouldstone, I.; Cook, M.; and Ferrer , B. P . 2015. The paint- ing fool sees! new projects with the automated painter . In ICCC , 189–196. [Das and Gamb ¨ ack 2014] Das, A., and Gamb ¨ ack, B. 2014. Poetic machine: Computational creativity for automatic po- etry generation in bengali. In ICCC , 230–238. [Davis et al. 2015a] Davis, N.; Hsiao, C.-P .; Popova, Y .; and Magerko, B. 2015a. An enactive model of creativity for computational collaboration and co-creation. In Creativity in the Digital Age . Springer . 109–133. [Davis et al. 2015b] Davis, N. M.; Hsiao, C.-P .; Singh, K. Y .; Li, L.; Moningi, S.; and Magerko, B. 2015b . Drawing ap- prentice: An enacti ve co-creative agent for artistic collabo- ration. In Cr eativity & Cognition , 185–186. [Deng et al. 2009] Deng, J.; Dong, W .; Socher, R.; Li, L.- J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A lar ge-scale hierarchical image database. [Gero 2000] Gero, J. S. 2000. Computational models of in- nov ative and creative design processes. T echnological fore- casting and social change 64(2-3):183–196. [Grace et al. 2015] Grace, K.; Maher , M. L.; Fisher , D.; and Brady , K. 2015. Modeling expectation for ev aluating sur - prise in design creativity . In Design Computing and Cogni- tion’14 . Springer . 189–206. [Guzdial et al. 2019] Guzdial, M.; Liao, N.; Chen, J.; Chen, S.-Y .; Shah, S.; Shah, V .; Reno, J.; Smith, G.; and Riedl, M. 2019. Friend, collaborator , student, manager: How de- sign of an ai-driven game lev el editor af fects creators. arXiv pr eprint arXiv:1901.06417 . [Ha and Eck 2017] Ha, D., and Eck, D. 2017. A neu- ral representation of sketch drawings. arXiv pr eprint arXiv:1704.03477 . [Jacob et al. 2013] Jacob, M.; Coisne, G.; Gupta, A.; Sysoev , I.; V erma, G. G.; and Magerko, B. 2013. V iewpoints ai. In Ninth Artificial Intelligence and Interactive Digital Enter - tainment Confer ence . [Johnson et al. 2009] Johnson, G.; Gross, M. D.; Hong, J.; Do, E. Y .-L.; et al. 2009. Computational support for sketching in design: a revie w . F oundations and T r ends R  in Human–Computer Interaction 2(1):1–93. [Jongejan et al. 2016] Jongejan, J.; Rowley , H.; Kawashima, T .; Kim, J.; and Fox-Gie g, N. 2016. The quick, draw!-ai experiment. Mount V iew , CA, accessed F eb 17:2018. [Karimi et al. 2018a] Karimi, P .; Davis, N.; Grace, K.; and Maher , M. L. 2018a. Deep learning for identifying poten- tial conceptual shifts for co-creati ve drawing. arXiv pr eprint arXiv:1801.00723 . [Karimi et al. 2018b] Karimi, P .; Grace, K.; Davis, N.; and Maher , M. L. 2018b . Creati ve sketching apprentice: Sup- porting conceptual shifts in sketch ideation. In International Confer ence on-Design Computing and Cognition , 721–738. Springer . [Karimi et al. 2018c] Karimi, P .; Grace, K.; Maher , M. L.; and Davis, N. 2018c. Evaluating creativity in computational co-creativ e systems. In ICCC , 104–111. [LeCun, Bengio, and Hinton 2015] LeCun, Y .; Bengio, Y .; and Hinton, G. 2015. Deep learning. natur e 521(7553):436. [Mikolov 2016] Mikolo v , T . 2016. word2vec: T ool for computing continuous distributed representations of words. https://code.google.com/p/word2vec . [Oh et al. 2018] Oh, C.; Song, J.; Choi, J.; Kim, S.; Lee, S.; and Suh, B. 2018. I lead, you help but only with enough de- tails: Understanding user experience of co-creation with ar- tificial intelligence. In Pr oceedings of the 2018 CHI Confer - ence on Human F actor s in Computing Systems , 649. A CM. [Purcell and Gero 1996] Purcell, A. T ., and Gero, J. S. 1996. Design and other types of fixation. Design studies 17(4):363–383. [Shneiderman 2007] Shneiderman, B. 2007. Creativity sup- port tools: Accelerating discov ery and innovation. Commu- nications of the A CM 50(12):20–32. [Simonyan and Zisserman 2014] Simonyan, K., and Zisser - man, A. 2014. V ery deep con volutional networks for large- scale image recognition. arXiv pr eprint arXiv:1409.1556 . [Suwa and Tversk y 1997] Suwa, M., and Tversk y , B. 1997. What do architects and students percei ve in their design sketches? a protocol analysis. Design studies 18(4):385– 403. [V oigt, Niehav es, and Becker 2012] V oigt, M.; Nieha ves, B.; and Becker , J. 2012. T o wards a unified design theory for creativity support systems. In International Conference on Design Science Resear ch in Information Systems , 152–173. Springer . [Y annakakis, Liapis, and Alexopoulos 2014] Y annakakis, G. N.; Liapis, A.; and Alexopoulos, C. 2014. Mixed- initiativ e co-creativity . In FDG .